Debian Codegraph @

doc/build/MIGRATING.txt

raw
1 New features
2 ************
3
4 This document describes the major changes occurring between versions
5 of Modules. It provides an overview of the new features and changed
6 behaviors that will be encountered when upgrading.
7
8
9 v5.7
10 ====
11
12 This new version is backward-compatible with previous version 5
13 release. It fixes bugs but also introduces new functionalities that
14 are described in this section. See the 5.7 release notes for a
15 complete list of the changes between Modules v5.6 and v5.7.
16
17
18 Improving evaluation performances
19 ---------------------------------
20
21 We observed that Modules could feel slow when evaluating more than one
22 hundred modulefiles within a single command. To address this, several
23 improvements have been implemented:
24
25 * In-memory caching of expensive operations: the results of costly
26 operations, such as module specification comparisons, are now cached
27 in memory.
28
29 * Reworked environment synchronization between Tcl interpreters:
30 environment synchronization during Tcl modulefile evaluation has
31 been redesigned to rely on Tcl’s internal mechanisms rather than a
32 custom reimplementation. As part of this change, calls to the Tcl
33 "append" command have been replaced with "set", which reliably
34 triggers synchronization when environment variables are modified.
35
36 * More efficient dependency checks: when verifying dependencies
37 against loaded modules, comparisons are now limited to the subset of
38 loaded modules that match the module root name specification,
39 instead of comparing against all loaded modules.
40
41 * Lighter-weight resolution of an exact "name/version" specification:
42 sites with a modulepath directory holding hundreds or thousands of
43 sibling modulefiles reported slow "module load" commands even when
44 the requested version already exists. Correctly computing the
45 implicit "default" and "latest" symbolic versions of such a
46 directory used to require validating every sibling modulefile in it.
47 The lookup now walks candidate versions from the dictionary-highest
48 to the dictionary-lowest and stops validating as soon as this can be
49 answered: either a higher valid version is found (the requested one
50 is then known not to be "default" or "latest"), or the requested
51 version itself is reached, in which case nothing higher exists. This
52 bypasses the exhaustive scan without changing the resolved result.
53
54 For this work, the **mb** utility was extended with new benchmark
55 tests that loads 137 modulefiles ("load2"), lists the loaded modules
56 ("list2"), and then purges them. Running this benchmark highlights the
57 performance improvements, as shown in the table below:
58
59 +-----------+---------------+----------------------+
60 | | v5.6.1 | v5.7.0 |
61 |===========|===============|======================|
62 | load2 | 1038 ms | 384 ms (-63%) |
63 +-----------+---------------+----------------------+
64 | list2 | 222 ms | 29 ms (-87%) |
65 +-----------+---------------+----------------------+
66 | purge | 568 ms | 303 ms (-46%) |
67 +-----------+---------------+----------------------+
68
69 A "load3" benchmark test was also added, loading a low, non-default
70 version of a module out of 500 sibling versions in the same directory.
71 It specifically exercises the lighter-weight resolution described
72 above:
73
74 +-----------+---------------+----------------------+
75 | | v5.6.1 | v5.7.0 |
76 |===========|===============|======================|
77 | load3 | 30 ms | 19 ms (-36%) |
78 +-----------+---------------+----------------------+
79
80 These measurements were obtained on a system where all modulefiles are
81 stored on local flash storage. If your modulefiles reside on a shared
82 network filesystem, using a Module cache is strongly recommended.
83
84 A module cache file aggregates the content of all the modulercs and
85 modulefiles found in a modulepath and this whole file is evaluated
86 when a module search occurs, even if the search targets specific
87 modules. On setups with a very large number of modulefiles, resulting
88 in cache files of several megabytes, evaluating the whole cache file
89 may take longer than walking through the modulepath directory to find
90 the few requested modules. The new "ignore_cache_subcmds"
91 configuration option defines a list of module sub-commands on which
92 module cache files are ignored. For instance to keep cache benefit on
93 listing sub-commands but avoid its penalty on targeted ones:
94
95 $ module config ignore_cache_subcmds load:try-load:load-any
96
97 When the configuration option is changed from its default value using
98 the "config" sub-command, the "MODULES_IGNORE_CACHE_SUBCMDS"
99 environment variable is defined accordingly. See
100 "MODULES_IGNORE_CACHE_SUBCMDS" for details.
101
102
103 Reorder existing entries in path-like variables
104 -----------------------------------------------
105
106 A new "path_entry_reorder" configuration option is introduced to
107 control how path-like environment variables are modified when
108 "prepend-path", "append-path" or "use" target a path entry that is
109 already defined in the environment variable.
110
111 By default, this configuration option is set to "0", meaning that when
112 a path entry already exists, the environment variable is left
113 unchanged. When set to "1", an existing entry is reordered to the
114 beginning or the end of the variable, depending on the command used,
115 unless duplicates are explicitly allowed.
116
117 $ module append-path PATHVAR /foo
118 $ module append-path PATHVAR /bar
119 $ module append-path PATHVAR /foo
120 $ echo $PATHVAR
121 /foo:/bar
122 $ module config path_entry_reorder 1
123 $ module append-path PATHVAR /foo
124 $ echo $PATHVAR
125 /bar:/foo
126 $ module append-path --duplicates PATHVAR /bar
127 $ echo $PATHVAR
128 /bar:/foo:/bar
129
130 The default value of "path_entry_reorder" can be defined at
131 installation time with the "--enable-path-entry-reorder" option. When
132 the configuration option is changed from its default value using the
133 "config" sub-command, the "MODULES_PATH_ENTRY_REORDER" environment
134 variable is defined accordingly. See "MODULES_PATH_ENTRY_REORDER" for
135 details.
136
137
138 Linked environment variables
139 ----------------------------
140
141 A new configuration option, "linked_envvars", can be used to
142 automatically mirror environment variable changes onto other
143 environment variables.
144
145 When a modulefile modifies an environment variable using commands such
146 as "setenv", "unsetenv", "prepend-path", "append-path", "remove-path"
147 or "pushenv", the same operation is also applied to all variables
148 linked through the "linked_envvars" configuration option.
149
150 For example, with the following configuration:
151
152 $ module config linked_envvars "CPATH&INCLUDE_PATH"
153
154 loading a module that prepends a directory to "CPATH":
155
156 prepend-path CPATH /opt/pkg/include
157
158 will also prepend the same directory to "INCLUDE_PATH".
159
160 Links are directional. In the previous example, changes made to
161 "CPATH" are also applied to "INCLUDE_PATH", but changes made directly
162 to "INCLUDE_PATH" are not applied back to "CPATH" unless an explicit
163 reverse link is configured.
164
165 When enabling this feature, note that existing values are not
166 synchronized when the link is established. Only subsequent
167 modifications performed through modulefile environment variable
168 commands are propagated to linked variables.
169
170 The output of "module display" also includes the environment variable
171 changes applied to linked environment variables, making these
172 propagated updates visible before a module is loaded.
173
174
175 Initialize environment variables on first modification
176 ------------------------------------------------------
177
178 A new configuration option, "init_envvars", has been added to define
179 initial values for environment variables that are modified through
180 path-like modulefile commands.
181
182 This feature is particularly useful for variables such as "MANPATH",
183 which require a leading or trailing colon character to preserve access
184 to the system default search paths. Previously, modifying an undefined
185 "MANPATH" could prevent system manual pages from being found. See the
186 Handle man pages search path cookbook for details.
187
188 The "init_envvars" option defines a colon-separated list of
189 "VARIABLE_NAME=initial_value" entries. When a path-like operation is
190 applied to an environment variable that is not currently set, the
191 configured initial value is assigned before the modification is
192 performed.
193
194 For example, with the following configuration:
195
196 $ module config init_envvars MANPATH=
197
198 loading a module that prepends a directory to "MANPATH":
199
200 prepend-path MANPATH /opt/pkg/man
201
202 will keep a colon character at the end of "MANPATH"
203
204 $ echo $MANPATH
205 /opt/pkg/man:
206
207 This mechanism applies to all path-like environment variable
208 management commands, including "append-path", "prepend-path" and
209 "remove-path".
210
211 When values are later removed from an environment variable, it is
212 automatically unset if its resulting value matches the configured
213 initial value and no explicit reference counter is associated with it.
214
215
216 Hook API
217 --------
218
219 Site code that needs to run right before or after a modulefile or
220 modulerc evaluation could so far only be attached through the "trace"
221 Tcl command or by renaming an internal "modulecmd.tcl" procedure, both
222 of which bind to implementation details that may change from one
223 Modules version to the next.
224
225 A new "add-hook" siteconfig command is introduced to register a
226 procedure on one of four stable events without relying on such
227 internal details: "before-modulefile-eval", "after-modulefile-eval",
228 "before-modulerc-eval" and "after-modulerc-eval". Several procedures
229 can be registered on the same event; they are then called in their
230 registration order.
231
232 proc auditRequestedLoad {modfile modname modnamevr modspec mode requested} {
233 if {$requested && $mode eq {load}} {
234 set fd [open /var/log/modules-audit.log a]
235 puts $fd "[clock format [clock seconds]] $modnamevr"
236 close $fd
237 }
238 }
239 add-hook before-modulefile-eval auditRequestedLoad
240
241 An error raised by a hook procedure is reported but does not abort the
242 running "module" command nor prevent other procedures registered on
243 the same event from running, since these hooks fire on nearly every
244 modulefile or modulerc evaluation.
245
246 See the *Hooks* section of *module(1)* man page for the full argument
247 contract of each event. The "trace"/rename techniques remain available
248 for anything not (yet) covered by a hook event.
249
250 These four events are only the first ones introduced through "add-
251 hook"; additional hook events will be added in the future as real site
252 needs for them come up. See Add new hook event for the guide
253 contributors can follow to propose one.
254
255
256 Ignore files in modulepath
257 --------------------------
258
259 Modulepath directories sometimes hold files that are not modulefiles,
260 like helper scripts sourced by modulefiles or software installation
261 files when modulefiles are mixed with the software trees they
262 describe. The new "modulepath-ignore" modulefile command lets
263 modulepath maintainers describe from rc files such content to ignore
264 when searching for modulefiles. Ignored files and directories are
265 handled as if they do not exist: they are not checked, not walked
266 down, not reported and not recorded in modulepath cache file. Cache
267 build time and size are thus kept proportional to the number of actual
268 modulefiles, whatever the amount of other files stored in modulepath.
269
270 Patterns follow the same syntax and behavior than git with
271 ".gitignore" files. For instance to ignore every file with the ".tcl"
272 extension and the whole content of the "soft" directory, add to the
273 ".modulerc" file at the root of the modulepath:
274
275 modulepath-ignore *.tcl soft/
276
277 Negated patterns switch to an *ignore all, accept some* logic. For
278 instance to only take files with the ".mod" extension into account:
279
280 modulepath-ignore * !*/ !*.mod
281
282 When defined in a ".modulerc" file located within a modulepath,
283 patterns are relative to the directory of this rc file. When defined
284 in a global or user rc file, patterns apply to every modulepath. As
285 ignored elements are excluded from cache files when they are built,
286 pattern definitions should not be conditional.
287
288 The "modulepath_ignore" configuration option, enabled by default,
289 controls whether pattern definitions are applied. Patterns can be
290 skipped for one command execution with the "--no-modulepath-ignore"
291 command line switch, which helps finding out whether a module is
292 missing because it is ignored.
293
294
295 v5.6
296 ====
297
298 This new version is backward-compatible with previous version 5
299 release. It fixes bugs but also introduces new functionalities that
300 are described in this section. See the 5.6 release notes for a
301 complete list of the changes between Modules v5.5 and v5.6.
302
303
304 spider sub-command
305 ------------------
306
307 Support has been added for the "spider" sub-command introduced by the
308 Lmod project. This sub-command lists available modules found in
309 enabled modulepaths and recursively found in modulepaths enabled by
310 available modules.
311
312 $ module avail
313 --------------------- /path/to/modulefiles ---------------------
314 foo/1 foo/2
315 $ module show foo/1
316 -------------------------------------------------------------------
317 /path/to/modulefiles/foo/1:
318
319 conflict foo
320 module use /path/to/modulefiles.2
321 -------------------------------------------------------------------
322 $ module show foo/2
323 -------------------------------------------------------------------
324 /path/to/modulefiles/foo/2:
325
326 conflict foo
327 append-path MODULEPATH /path/to/modulefiles.3
328 -------------------------------------------------------------------
329 $ module spider
330 --------------------- /path/to/modulefiles ---------------------
331 foo/1 foo/2
332
333 -------------- /path/to/modulefiles.2 (via foo/1) --------------
334 bar/1 bar/2 bar/3
335
336 -------------- /path/to/modulefiles.3 (via foo/2) --------------
337 bar/3 bar/4
338
339 -------------- /path/to/modulefiles.3 (via bar/1) --------------
340 qux/1 qux/2
341
342 The "spider" sub-command relies on the Extra match search mechanism to
343 scan modulefiles and identify the modulepaths they enable. Since all
344 modulefiles are evaluated during this scan, it is recommended to build
345 and use a Module cache to improve search performance.
346
347 The output of "spider" is similar to that of the "avail" sub-command
348 and supports the same set of options and queries.
349
350 $ module spider -t bar@2:
351 /path/to/modulefiles.2:
352 bar/2
353 bar/3
354
355 /path/to/modulefiles.3:
356 bar/3
357 bar/4
358
359 The "spider" sub-command has its own configuration options to control
360 the content of its output: "spider_indepth", "spider_output", and
361 "spider_terse_output".
362
363 These settings make it possible to define output for "spider" that
364 differs from the "avail" sub-command.
365
366 $ module config spider_output -modulepath
367 $ module config spider_indepth 0
368 $ module spider
369 bar/ foo/ qux/
370
371 When a modulepath is enabled by a specific module, that modulepath is
372 considered available *via* that module. When the module is loaded,
373 this *via* information is stored in the "__MODULES_LMUSE" environment
374 variable to track which modules enable which modulepaths.
375
376 By default, the *via* information is included in the standard output
377 of the "spider" sub-command. Additionally, the JSON output for both
378 the "avail" and "spider" sub-commands has been updated to include
379 *via* details for each modulefile.
380
381 $ module spider -j bar@2:
382 {"/path/to/modulefiles.2": {
383 "bar/2": { "name": "bar/2", "type": "modulefile", "symbols": [], "tags": [], "pathname": "/path/to/modulefiles.2/bar/2", "via": "foo/1"}
384 "bar/2": { "name": "bar/3", "type": "modulefile", "symbols": [], "tags": [], "pathname": "/path/to/modulefiles.2/bar/3", "via": "foo/1"}
385 },
386 "/path/to/modulefiles.3": {
387 "bar/3": { "name": "bar/3", "type": "modulefile", "symbols": [], "tags": [], "pathname": "/path/to/modulefiles.3/bar/3", "via": "foo/2"},
388 "bar/4": { "name": "bar/4", "type": "modulefile", "symbols": [], "tags": [], "pathname": "/path/to/modulefiles.3/bar/4", "via": "foo/2"}
389 }}
390
391
392 Requiring *via* module
393 ----------------------
394
395 A loaded module that enables a modulepath is considered the *via*
396 module of other loaded modules whose modulefiles are stored in this
397 modulepath. The "require_via" configuration option is introduced to
398 consider a *via* module a requirement for the loaded module stored in
399 the modulepath it enables.
400
401 A *module hierarchy* mechanism, as introduced by the Lmod project, is
402 now supported with this feature. It allows organizing modulefiles
403 through a primary modulepath, where loading certain modules enables
404 additional modulepaths. "require_via" maintains a link between
405 modulefiles in these additional paths and the *via* module that
406 activated them.
407
408 The "require_via" setting is disabled by default to maintain
409 compatibility with previous Modules 5 releases. However, users are
410 encouraged to enable this feature if they want to use the *module
411 hierarchy* mechanism.
412
413 $ module config require_via 1
414
415 When unloading a *via* module, all the modules stored in its enabled
416 modulepath are automatically unloaded thanks to the *Dependent Unload*
417 mechanism of the Automated module handling mode.
418
419 $ module load foo/1 bar/2
420 $ module avail
421 -------------- /path/to/modulefiles.2 (via foo/1) --------------
422 bar/1 bar/2 bar/3
423
424 --------------------- /path/to/modulefiles ---------------------
425 foo/1 foo/2
426
427 Key:
428 loaded modulepath
429 $ module unload foo
430 Unloading foo/1
431 Unloading dependent: bar/2
432
433 When a *via* module is replaced by another, any modulepaths it enabled
434 are updated accordingly. If the newly enabled modulepath provides
435 alternatives to modules previously loaded from the original path,
436 those modules are reloaded from the new location. If no alternatives
437 are found, the dependent modules are unloaded.
438
439 $ module config conflict_unload 1
440 $ module load foo/1 bar/3
441 $ module load foo/2
442 Loading foo/2
443 Unloading dependent: bar/3
444 Unloading conflict: foo/1
445 Reloading dependent: bar/3
446 $ module avail
447 -------------- /path/to/modulefiles.3 (via foo/2) --------------
448 bar/3 bar/4
449
450 --------------------- /path/to/modulefiles ---------------------
451 foo/1 foo/2
452
453 Key:
454 loaded modulepath
455
456
457 Always see hidden modules
458 -------------------------
459
460 The "hidden" element is a new allowed value for the "avail_output",
461 "avail_terse_output", "list_output", "list_terse_output",
462 "spider_output" and "spider_terse_output" configuration options. When
463 set into the value list of these options hidden modules are shown on
464 the output of corresponding commands ("avail", "list" and "spider").
465
466 --------------- /path/to/modulefiles ---------------
467 bar/1.0
468 $ module config avail_output +hidden
469 $ ml av
470 --------------- /path/to/modulefiles ---------------
471 bar/1.0 foo/1.0 <H>
472
473 When set, this is equivalent to always adding the "--all" command-line
474 option.
475
476
477 Issue warning upon module evaluation
478 ------------------------------------
479
480 The mfcmd:*module-warn* modulefile Tcl command is introduced to define
481 warning message to print when loading specified modulefiles.
482
483 $ cat /path/to/modulefiles/foo/1.0
484 #%Module
485 module-warn --message {Please consider using "bar" module instead} foo
486 $ module load foo
487 Loading foo/1.0
488 WARNING: Please consider using "bar" module instead
489
490 The warning message is printed when targeted modulefile is evaluated
491 in *load*, *display*, *help* or *test* mode.
492
493 "module-warn" supports the same kind of options than "module-tag" or
494 "module-hide": it is possible to specify a list of affected or
495 unaffected users or groups. Warning may also be only effective before
496 or after a particular date.
497
498 The "warning" tag is set on modules targeted by a mfcmd:*module-warn*
499 command.
500
501
502 Simplified way to define module help message
503 --------------------------------------------
504
505 Until now module help message was defined through a specific Tcl
506 procedure that outputs text to *stderr* channel:
507
508 #%Module
509 proc ModulesHelp {} {
510 puts stderr {Module Name: foo, Version: 1.0}
511 puts stderr {This modulefile does this and that}
512 }
513
514 A new modulefile Tcl command is introduced, "module-help", to simplify
515 a bit the definition of help message:
516
517 #%Module
518 module-help {Module Name: foo, Version: 1.0}
519 module-help {This modulefile does this and that}
520
521 With this modulefile command there is no need to define the
522 "ModulesHelp" procedure anymore. As "module-help" is reported on
523 module *display* evaluation, help message will also be visible there.
524
525
526 Support for newline character in environment variable value
527 -----------------------------------------------------------
528
529 It is now possible on *sh*, *bash*, *ksh*, *zsh* and *fish* shells to
530 set value for environment variable that contains newline character:
531
532 $ cat /path/to/modulefiles/bar/1.0
533 #%Module
534 setenv BAR "multinlinenvalue"
535 $ module load bar
536 $ echo "$BAR"
537 multi
538 line
539 value
540
541 Generated shell code on these shells has been changed to enclose value
542 within single quotes rather escaping each special characters:
543
544 $ module mod-to-sh bash bar/1.0
545 BAR='multi
546 line
547 value'; export BAR;
548
549
550 Declare additional elements provided by a module
551 ------------------------------------------------
552
553 New "provide" modulefile command is added to declare additional
554 elements included in a modulefile. When the modulefile is loaded, each
555 argument specified in the "provide" command is registered as an alias
556 for the module.
557
558 $ module show foo
559 -------------------------------------------------------------------
560 /path/to/modulefiles/foo/1:
561
562 provide liba/1.2 libb/2.1
563 conflict foo
564 conflict liba libb
565 -------------------------------------------------------------------
566 $ module load foo
567
568 It is advised to define "conflict" over the name of these additional
569 elements to ensure only one provider is found loaded in user
570 environment.
571
572 $ module show bar/1
573 -------------------------------------------------------------------
574 /path/to/modulefiles/bar/1:
575
576 provide liba/1.2 libb/2.1
577 conflict bar
578 conflict liba libb
579 -------------------------------------------------------------------
580 $ module load bar
581 Loading bar/1
582 ERROR: Module cannot be loaded due to a conflict.
583 HINT: Might try "module unload foo/1" first.
584
585 Once a module with a provided alias is loaded, it fulfills any
586 requirements declared by other modules for that name and version, just
587 like any other type of module alias.
588
589 $ module show qux
590 -------------------------------------------------------------------
591 /path/to/modulefiles/qux/1:
592
593 prereq liba/1.2
594 -------------------------------------------------------------------
595 $ module load qux
596 $ module list
597 Currently Loaded Modulefiles:
598 1) foo/1 2) qux/1
599
600 Module aliases defined via the "provide" command are not available
601 during module resolution, as modulefiles are not evaluated in that
602 context. Therefore, it is recommended to define a global module alias
603 in a "modulerc" file using the "module-alias" command to specify a
604 default target for the alias.
605
606 $ cat /path/to/modulefiles/.modulerc
607 #%Module
608 module-alias liba/1.2 bar/1
609 $ module purge
610 $ module load qux
611 Loading qux/1
612 Loading requirement: bar/1
613
614 The "provide" modulefile command also serves as an alias for the
615 "extensions" command, ensuring compatibility with modulefiles written
616 for Lmod.
617
618 By default an "avail" command includes module aliases defined in
619 "modulerc" files. To also get aliases defined within modulefiles with
620 "provide" or "family", output element "provided-alias" has to be
621 included. It is supported on "avail_output", "avail_terse_output",
622 "spider_output" and "spider_terse_output" configuration options.
623
624 Having "provided-alias" included in output triggers Extra match search
625 mechanism that scans all modulefiles to find these aliases. It is thus
626 advised to have Module cache built and used to improve search speed.
627
628 $ module avail
629 --------------------- /path/to/modulefiles ---------------------
630 bar/1 foo/1 liba/1.2 qux/1
631
632 Key:
633 modulepath module-alias
634 $ module avail -o +provided-alias
635 --------------------- /path/to/modulefiles ---------------------
636 bar/1 foo/1 liba/1.2 libb/2.1 qux/1
637
638 Key:
639 modulepath module-alias
640
641 The "provided-alias" Extra specifier is also introduced to find
642 modules that define such alias with "provide" or "family" modulefile
643 commands.
644
645 $ module avail provided-alias:liba/1.2
646 --------------------- /path/to/modulefiles ---------------------
647 bar/1 foo/1
648
649 Key:
650 modulepath
651
652
653 v5.5
654 ====
655
656 This new version is backward-compatible with previous version 5
657 release. It fixes bugs but also introduces new functionalities that
658 are described in this section. See the 5.5 release notes for a
659 complete list of the changes between Modules v5.4 and v5.5.
660
661
662 Logging activity
663 ----------------
664
665 The ability to log module command activity is now available out of the
666 box. It could be enabled on previous versions by using a specific
667 "siteconfig.tcl" configuration script as described in the Log module
668 command cookbook recipe.
669
670 Integrated logging feature relies on two configuration options:
671
672 * "logger", the command run to transmit messages to the log system
673
674 * "logged_events", list of module event to log
675
676 "logger" option relies on the **logger** command by default, which is
677 usually available and already installed on most systems.
678
679 The "logged_events" option is empty by default, so no content is sent
680 to logs by default. It recognizes the following events:
681
682 * "requested_cmd": record module commands typed by users
683
684 * "requested_eval": record modulefile evaluations requested by users
685
686 * "auto_eval": record modulefile evaluations automatically triggered
687
688 $ module config logged_events +requested_cmd:requested_eval
689 $ ml av
690 --------------- /path/to/modulefiles ---------------
691 bar/1.0 foo/1.0 qux/1.0
692 $ module load bar
693 Loading bar/1.0
694 Loading requirement: qux/1.0
695 $ module purge
696
697 In the above example the module command and modulefile evaluations
698 directly requested by users are sent to the system log:
699
700 $ journalctl -q -t modules
701 Apr 29 07:47:42 hostname modules[3777797]: user="username" command="avail" arguments=""
702 Apr 29 07:48:10 hostname modules[3777876]: user="username" command="load" arguments="bar"
703 Apr 29 07:48:10 hostname modules[3777876]: user="username" mode="load" module="bar/1.0" specified="bar" modulefile="/path/to/modulefiles/bar/1.0" requested="1"
704 Apr 29 07:48:17 hostname modules[3777914]: user="username" command="purge" arguments=""
705 Apr 29 07:48:17 hostname modules[3777914]: user="username" mode="unload" module="bar/1.0" specified="bar/1.0" modulefile="/path/to/modulefiles/bar/1.0" requested="1"
706
707 Some messages can also be sent during modulefile evaluation by using
708 the "log" channel of the "puts" command:
709
710 $ cat /path/to/modulefiles/foo/1.0
711 #%Module
712 puts log {some message sent to log}
713 $ ml foo/1.0
714 $ journalctl -q -t modules
715 Apr 29 07:47:51 hostname modules[3777835]: user="username" command="ml" arguments="foo/1.0"
716 Apr 29 07:47:51 hostname modules[3777835]: some message sent to log
717 Apr 29 07:47:51 hostname modules[3777835]: user="username" mode="load" module="foo/1.0" specified="foo/1.0" modulefile="/path/to/modulefiles/foo/1.0" requested="1"
718
719
720 Negating extra match search criteria
721 ------------------------------------
722
723 The "not:" prefix string is introduced for extra specifier and variant
724 criteria. When this prefix is used, the search criteria where it is
725 applied is negated.
726
727 $ module config avail_output +variant
728 $ module config variant_shortcut toolchain=%
729 $ module avail
730 --------------------- /path/to/modulefiles ---------------------
731 bar/1.0{%x86_64} foo/2.0{%x86_64,arm64}
732 bar/2.0{%x86_64,arm64} qux/1.0{%x86_64}
733 foo/1.0{%x86_64} qux/2.0{%x86_64,arm64}
734 $ module avail not:envvar:FOO,BAR
735 --------------------- /path/to/modulefiles ---------------------
736 qux/1.0{%x86_64} qux/2.0{%x86_64,arm64}
737
738 The above search query returns all modules not defining "FOO" and
739 "BAR" environment variables.
740
741 $ module avail not:envvar:BAR not:%arm64
742 --------------------- /path/to/modulefiles ---------------------
743 foo/1.0{%x86_64} qux/1.0{%x86_64}
744
745 The above example returns all modules not defining "BAR" environment
746 variable and not defining "toolchain" variant or defining this variant
747 without "arm64" among the possible values.
748
749 This prefix is recognized on module search context (i.e., "avail",
750 "whatis" and "paths" sub-commands).
751
752
753 PowerShell support
754 ------------------
755
756 Support for PowerShell ("pwsh") is added. This shell may be used with
757 Modules on either Unix or Windows platforms. All main kind of
758 environment changes are supported.
759
760 A "envmodule" shell function is configured on PowerShell, as "module"
761 is a reserved keyword on this shell.
762
763 The Windows-specific distribution zipball of Modules has been updated
764 to also contain the required files to setup PowerShell support.
765
766
767 Conflict unload
768 ---------------
769
770 Automated module handling is enhanced with the addition of a new
771 mechanism: *Conflict Unload*. When loading a module, this new
772 mechanism automatically unloads conflicting modules and their
773 dependent.
774
775 $ module load foo/1
776 Loading foo/1{-debug}
777 Loading requirement: qux/1 bar/1
778 $ module load foo/2
779 Loading foo/2
780 ERROR: Module cannot be loaded due to a conflict.
781 HINT: Might try "module unload foo/1" first.
782 $ module config conflict_unload 1
783 $ module load foo/2
784 Loading foo/2{-debug}
785 Unloading conflict: foo/1{-debug}
786
787 As seen in the above example, *Conflict Unload* is controlled by the
788 "conflict_unload" configuration option. It is disabled by default as
789 it changes behaviors of the "automated module handling mode". But
790 everyone is encouraged to enable this new option to benefit from an
791 highly automated experience. This option can be changed at
792 installation time with "--enable-conflict-unload" configure script
793 option.
794
795 *Conflict Unload* handles all kind of conflicts: conflicts defined by
796 already loaded modules, conflict declared by loading module through
797 "conflict", "family" or "module unload" commands. It also supports the
798 unload of a module that is loaded again but with different variant
799 values.
800
801 $ module load foo/2 +debug
802 Loading foo/2{+debug}
803 Unloading conflict: foo/2{-debug}
804
805 Along with *Conflict Unload*, the *Useless Requirement Unload*
806 mechanism is added to load evaluation. With it, auto loaded
807 requirements of modules unloaded by *Conflict Unload* mechanism are
808 automatically unloaded.
809
810 $ module load foo/3
811 Loading foo/3{-debug}
812 Unloading conflict: foo/2{+debug} bar/1
813 Loading requirement: bar/2
814 Unloading useless requirement: qux/1
815
816 Dependent modules of *Conflict Unload* modules are considered
817 *Dependent Reload* modules. They are unloaded prior unloading the
818 *Conflict Unload* module that depend on them. Reload occurs after
819 loading main module and only if the loaded environment satisfies the
820 dependencies of the module to reload.
821
822 $ module load bar/3
823 Loading bar/3
824 Loading requirement: qux/2
825 $ module load qux/1
826 Loading qux/1
827 Unloading dependent: bar/3
828 Unloading conflict: qux/2
829 Reloading dependent: bar/3
830
831
832 Improved automated module handling mechanisms
833 ---------------------------------------------
834
835 Along with the introduction of *Conflict Unload* mechanism, some of
836 the existing automated module handling mechanisms have been adapted
837 and enhanced.
838
839 *Useless Requirement Unload* (UReqUn) mechanism is moved during a
840 "switch" sub-command from the end of the unload phase to the end of
841 the load phase. Both *UReqUn* modules coming from unload and load
842 phases are this way treated jointly. If *UReqUn* modules resulting
843 from the switch unload phase are detected as conflict to the switched-
844 on module, it is advised to enable the "conflict_unload" mechanism to
845 automatically handle them.
846
847 $ module switch foo/3 foo/4
848 Loading foo/4
849 ERROR: Module cannot be loaded due to a conflict.
850 HINT: Might try "module unload bar/2" first.
851
852 Switching from foo/3{-debug} to foo/4
853 ERROR: Load of switched-on foo/4 failed
854 $ module config conflict_unload 1
855 $ module switch foo/3 foo/4
856 Switching from foo/3{-debug} to foo/4
857 Unloading conflict: bar/2
858
859 *Dependent Reload* (DepRe) modules that also are *UReqUn* modules are
860 now unloaded during the *DepRe* unload phase instead of during the
861 *UReqUn* process. As a consequence these modules are now unloaded
862 prior main module action, as they are a dependent of this main module.
863 This kind of *UReqUn* modules may not be a dependency of a module
864 unloaded in the current processing.
865
866 Modules from *Dependent Unload* (DepUn) and *Dependent Reload* (DepRe)
867 mechanisms are now mixed together to proceed to their unload. They are
868 this way unloaded in their reverse loading order. So if a module is
869 part of the *DepUn* process and one of its requirement is part of the
870 *DepRe* process, the requirement is now unloaded after its dependent.
871
872 Reload of *Dependent Reload* (DepRe) module is now skipped if this
873 module is not found loadable in environment. Either because its
874 requirements are not loaded or a conflict is spotted. Such module is
875 considered *Dependent Unload* rather *Dependent Reload*. If *DepRe*
876 module is sticky, reload is attempted anyway.
877
878 Reload of all *Dependent Reload* (DepRe) modules now occurs after main
879 module evaluation rather after each sub module evaluation that
880 triggered the *Dependent Reload* unload phase.
881
882 When a loading module requires a *Dependent Unload* (DepUn) module,
883 the unload of this module was forced when it was triggered by a
884 Conflict Unload or a switched-off module. The unload of such *DepUn*
885 module is not forced anymore by default and an error is raised if a
886 loading module requiring the *DepUn* module is detected.
887
888 When unloading *Dependent Reload* modules, do not force by default
889 this unload unless asked with "--force" option. As a result an error
890 is obtained when trying to reload Dependent Reload modules that
891 conflict with other loaded modules. This error may be by-passed with "
892 --force" command-line option.
893
894
895 Specific modulepath for requirements
896 ------------------------------------
897
898 A "--modulepath" option is introduced on the "always-load", "depends-
899 on", "prereq", "prereq-all" and "prereq-any" modulefile commands. This
900 option indicates that requirement should be specifically searched in
901 the provided list of modulepaths. Such mechanism helps to ensure
902 loaded requirements are those expected and they are not coming from an
903 unexpected module tree.
904
905 In the following example, "foo" module requires "bar". It expects the
906 "bar" module from the same modulepath than "foo". User's environment
907 has enabled another modulepath that also provides a "bar" module. Use
908 of "--modulepath" option guides the requirement load mechanism to the
909 expected module.
910
911 $ module avail
912 --------------------- /path/to/othermfiles ---------------------
913 bar/1
914
915 --------------------- /path/to/modulefiles ---------------------
916 bar/1 foo/1 foo/2
917 $ module show foo/1
918 -------------------------------------------------------------------
919 /path/to/modulefiles/foo/1:
920
921 prereq --modulepath .. bar/1
922 -------------------------------------------------------------------
923 $ module load foo/1
924 Loading foo/1
925 Loading requirement: bar/1
926 $ echo $_LMFILES_
927 /path/to/modulefiles/bar/1:/path/to/modulefiles/foo/1
928
929 Specific modulepath set may guide to a directory not necessarily
930 enabled (i.e., not defined in "MODULEPATH" environment variable):
931
932 $ module show foo/2
933 -------------------------------------------------------------------
934 /path/to/modulefiles/foo/2:
935
936 prereq --modulepath /path/to/alternatefiles bar/2
937 -------------------------------------------------------------------
938 $ module load foo/2
939 Loading foo/2
940 Loading requirement: bar/2
941 $ module list
942 Currently Loaded Modulefiles:
943 1) bar/2 2) foo/2
944 $ echo $_LMFILES_
945 /path/to/alternatefiles/bar/2:/path/to/modulefiles/foo/2
946
947 If a required module is already loaded from a modulepath that does not
948 correspond to the requirement definition, an error is obtained.
949 However if the *Conflict Unload* mechanism described above is enabled,
950 the conflicting module will automatically be unloaded.
951
952 $ module purge
953 $ module load bar/1
954 $ echo $_LMFILES_
955 /path/to/othermfiles/bar/1
956 $ module load foo/1
957 Loading bar/1
958 ERROR: Module already loaded from a different modulepath
959
960 Loading foo/1
961 ERROR: Load of requirement bar/1 (specific path) failed
962 $ module config conflict_unload 1
963 $ module load foo/1
964 Loading foo/1
965 Unloading conflict: bar/1
966 Loading requirement: bar/1
967 $ echo $_LMFILES_
968 /path/to/modulefiles/bar/1:/path/to/modulefiles/foo/1
969
970
971 v5.4
972 ====
973
974 This new version is backward-compatible with previous version 5
975 release. It fixes bugs but also introduces new functionalities that
976 are described in this section. See the 5.4 release notes for a
977 complete list of the changes between Modules v5.3 and v5.4.
978
979
980 Enhancing extra specifiers
981 --------------------------
982
983 Extra specifier "tag" is introduced to search modules based on tags
984 applied. It applies to "avail", "whatis" and "paths" sub-commands. Tag
985 abbreviation may also be used as extra specifier name.
986
987 $ module avail tag:sticky
988 --------------------- /path/to/modulefiles ---------------------
989 bar/1
990
991 Multiple names can now be specified on one extra specifier criterion
992 to select modules matching any of these names.
993
994 $ module avail tag:S,F
995 --------------------- /path/to/modulefiles ---------------------
996 bar/1 qux/1
997
998 Multiple values can now be specified on one variant criterion to
999 select modules providing any of these variant values.
1000
1001 $ module config variant_shortcut toolchain=%
1002 $ module avail %a,c
1003 --------------------- /path/to/modulefiles ---------------------
1004 baz/1{%c} foo/1{%a,b,c}
1005 $ module avail %a,c %b
1006 --------------------- /path/to/modulefiles ---------------------
1007 foo/1{%a,b,c}
1008
1009
1010 Purging sticky modules
1011 ----------------------
1012
1013 The "sticky_purge" configuration option is added to define the
1014 behavior of "purge" sub-command when unloading a sticky or super-
1015 sticky module.
1016
1017 By default an "error" is raised. "sticky_purge" can be changed to emit
1018 a "warning" message instead or to be "silent".
1019
1020 $ module purge
1021 Unloading foo/1.0
1022 ERROR: Unload of sticky module skipped
1023 $ module config sticky_purge warning
1024 $ module purge
1025 Unloading foo/1.0
1026 WARNING: Unload of sticky module skipped
1027 $ module config sticky_purge silent
1028 $ module purge
1029 $ module list
1030 Currently Loaded Modulefiles:
1031 1) foo/1.0
1032
1033 Key:
1034 sticky
1035
1036
1037 Specific modulepath labels
1038 --------------------------
1039
1040 "modulepath-label" command is introduced to define a label to use to
1041 designate modulepath in module "avail" output. This new command should
1042 be used in global or modulepath-specific rc files.
1043
1044 $ cat /path/to/modulefiles/.modulerc
1045 #%Module
1046 modulepath-label . Tools
1047 $ module avail foo
1048 ---------------------- Tools ----------------------
1049 foo/1.0 foo/2.0
1050
1051
1052 Unique module name loaded
1053 -------------------------
1054
1055 The configuration option "unique_name_loaded" is introduced to allow
1056 only one module loaded per module name. It is similar to the *One name
1057 rule* feature introduced by the Lmod project.
1058
1059 When enabled, "unique_name_loaded" produces a conflict definition at
1060 the start of modulefile evaluation for each actual and alternative
1061 module root name. For instance when loading a module named
1062 "foo/sub/1.0" with a "qux/1.0" alias, it produces a conflict against
1063 "foo" and "qux".
1064
1065 When loading a module that shares a name with an already loaded
1066 module, an error is raised due to the conflict definition. This error
1067 aborts the module load evaluation.
1068
1069 $ cat /path/to/modulefiles/bar/1.0
1070 #%Module
1071 $ cat /path/to/modulefiles/bar/2.0
1072 #%Module
1073 $ module config unique_name_loaded 1
1074 $ module load bar/1.0
1075 $ module load bar/2.0
1076 Loading bar/2.0
1077 ERROR: Module cannot be loaded due to a conflict.
1078 HINT: Might try "module unload bar" first.
1079
1080 "unique_name_loaded" is disabled by default. It can be changed with
1081 module "config" sub-command or at installation time with "--enable-
1082 unique-name-loaded" configure script option.
1083
1084
1085 Cache sourced files
1086 -------------------
1087
1088 You may have gathered common Tcl code files used by all your
1089 modulefiles in Tcl files that are sourced with *source(n)* Tcl
1090 command. When loading multiple modulefiles at once, these sourced
1091 files may be read for every loaded modules, which may induce some I/O
1092 load.
1093
1094 The configuration option "source_cache" is introduced to cache files
1095 evaluated through "source" Tcl command in order to only read them once
1096 during a "modulecmd.tcl" execution. When this option is enabled,
1097 sourced file read also benefits from optimized I/O calls of Modules
1098 Tcl extension library.
1099
1100 In the following example, 20 modules are loaded, each of them sources
1101 the same Tcl file:
1102
1103 $ syscall_list=close,fcntl,ioctl,newfstatat,openat,read,readlink
1104 $ strace -f -e $syscall_list -c $MODULES_CMD bash load foo/20
1105 % time seconds usecs/call calls errors syscall
1106 ------ ----------- ----------- --------- --------- ----------------
1107 31.48 0.000781 1 466 read
1108 28.58 0.000709 5 131 16 openat
1109 14.19 0.000352 2 142 3 newfstatat
1110 9.71 0.000241 1 124 close
1111 8.02 0.000199 0 212 211 readlink
1112 4.55 0.000113 1 67 47 ioctl
1113 3.47 0.000086 1 60 fcntl
1114 ------ ----------- ----------- --------- --------- ----------------
1115 100.00 0.002481 2 1202 277 total
1116
1117 When sourced file is cached, it is only read once which saves a
1118 substantial amount of I/O operations:
1119
1120 $ module config source_cache 1
1121 $ strace -f -e $syscall_list -c $MODULES_CMD bash load foo/20
1122 % time seconds usecs/call calls errors syscall
1123 ------ ----------- ----------- --------- --------- ----------------
1124 34.41 0.000523 4 112 16 openat
1125 27.37 0.000416 0 428 read
1126 11.84 0.000180 1 122 3 newfstatat
1127 11.51 0.000175 1 105 close
1128 7.11 0.000108 2 47 27 ioctl
1129 5.99 0.000091 0 92 91 readlink
1130 1.78 0.000027 0 40 fcntl
1131 ------ ----------- ----------- --------- --------- ----------------
1132 100.00 0.001520 1 946 137 total
1133
1134
1135 Abort on error
1136 --------------
1137
1138 When an error occurs during the evaluation of a modulefile, several
1139 module sub-commands, like "load" or "unload", continue their
1140 processing. It means these sub-commands do not stop if they have
1141 multiple modulefiles to evaluate. This is named the *continue on
1142 error* behavior.
1143
1144 The "abort_on_error" configuration option is introduced to abort, for
1145 the sub-commands listed in its value, the evaluation of multiple
1146 modulefiles if one fails to evaluate.
1147
1148 The *abort on error* behavior is applied by default on **ml** command,
1149 "reload" sub-command and the unload phase of "switch" sub-command.
1150 This behavior may also be turned on for "load", "mod-to-sh", "purge",
1151 "switch", "try-load" and "unload" sub-commands. If a command is
1152 removed from "abort_on_error"'s value, it applies the *continue on
1153 error* behavior.
1154
1155 In the following example, *abort on error* behavior is applied to
1156 "load" sub-command to stop evaluation instead of loading the remaining
1157 modulefile in the sequence if first modulefile fails to evaluate.
1158
1159 $ module load foo bar
1160 Loading foo/1
1161 ERROR: Module evaluation aborted
1162 $ module list
1163 Currently Loaded Modulefiles:
1164 1) bar/1
1165 $ module purge
1166 $ module config abort_on_error +load
1167 $ module load foo bar
1168 Loading foo/1
1169 ERROR: Module evaluation aborted
1170 $ module list
1171 No Modulefiles Currently Loaded.
1172
1173 When enabled for "switch" sub-command, whole action aborts if
1174 switched-on modulefile fails to load. By default, "switch" only aborts
1175 if switched-off modulefile fails to unload.
1176
1177 $ module load bar
1178 $ module switch bar foo
1179 Loading foo/1
1180 ERROR: Module evaluation aborted
1181
1182 Switching from bar/1 to foo/1
1183 WARNING: Load of switched-on foo/1 failed
1184 $ module list
1185 No Modulefiles Currently Loaded.
1186 $ module load bar
1187 $ module config abort_on_error +switch
1188 $ module switch bar foo
1189 Loading foo/1
1190 ERROR: Module evaluation aborted
1191
1192 Switching from bar/1 to foo/1
1193 WARNING: Load of switched-on foo/1 failed
1194 $ module list
1195 Currently Loaded Modulefiles:
1196 1) bar/1
1197
1198 "abort_on_error" configure option has no impact on "module" commands
1199 defined in modulefiles. When "--force" option is set, sub-commands
1200 apply the *continue on error* behavior.
1201
1202 When "--enable-new-features" installation option is set, the *abort on
1203 error* behavior is also enabled on "load" and "switch" sub-commands.
1204
1205
1206 Improve error reporting
1207 -----------------------
1208
1209 Module evaluation error reports have been tweaked to appear where
1210 these errors occur rather reporting them on the message block of the
1211 main action. Such change helps to better understand the module
1212 evaluation flow.
1213
1214 $ module load baz
1215 Loading baz/1
1216 Loading requirement: foo/1
1217 $ module switch foo bar
1218 Unloading baz/1
1219 Module ERROR: invalid command name "bad_command"
1220 while executing
1221 "bad_command"
1222 (file "/path/to/modulefiles/baz/1" line 4)
1223 Please contact <root@localhost>
1224
1225 Unloading foo/1
1226 ERROR: Unload of dependent baz/1 failed
1227
1228 Switching from foo/1 to bar
1229 ERROR: Unload of switched-off foo/1 failed
1230
1231 "switch" sub-command specific error messages are not reported when
1232 "module switch" command is run from a modulefile (like *unload of
1233 switched-off module failed*, *load of switched-on module failed*
1234 messages or *Switching* block message). On "switch" sub-command, load
1235 failure of switched-on module is now reported as an error.
1236
1237 $ module load foo
1238 $ module switch foo qux
1239 Loading qux/1
1240 Module ERROR: invalid command name "bad_command"
1241 while executing
1242 "bad_command"
1243 (file "/path/to/modulefiles/qux/1" line 2)
1244 Please contact <root@localhost>
1245
1246 Switching from foo/1 to qux/1
1247 ERROR: Load of switched-on qux/1 failed
1248
1249 Conflict error messages now describe the loaded module name and
1250 version that conflicts rather the generic conflict module
1251 specification.
1252
1253 $ cat /path/to/modulefiles/bar/1
1254 #%Module
1255 conflict foo
1256 $ module load foo
1257 $ module load bar
1258 Loading bar/1
1259 ERROR: Module cannot be loaded due to a conflict.
1260 HINT: Might try "module unload foo/1" first.
1261
1262
1263 New options for source-sh modulefile command
1264 --------------------------------------------
1265
1266 Support for new *shell* mode named "bash-eval" is added to "source-sh"
1267 modulefile command. With this new mode, the generated output of the
1268 bash shell script is evaluated to get the environment changes instead
1269 of sourcing this script.
1270
1271 $ cat /path/to/modulefiles/foo/foo-output.sh
1272 #!/bin/bash
1273 cat <<EOF
1274 export FOO=value;
1275 alias foo_alias='echo foo_alias;'
1276 foo_function() {
1277 echo foo_function;
1278 }
1279 EOF
1280 $ cat /path/to/modulefiles/foo/1
1281 #%Module
1282 set foo_script [file dirname $ModulesCurrentModulefile]/foo-output.sh
1283 source-sh bash-eval $foo_script
1284
1285 $ module show foo/1
1286 -------------------------------------------------------------------
1287 /path/to/modulefiles/foo/1:
1288
1289 set-alias foo_alias {echo foo_alias;}
1290 set-function foo_function {
1291 echo foo_function}
1292 setenv FOO value
1293 -------------------------------------------------------------------
1294 $ module load foo/1
1295 $ foo_alias
1296 foo_alias
1297
1298 "bash-eval" *shell* support is also available on "sh-to-mod" sub-
1299 command.
1300
1301 A new option is added to "source-sh" modulefile command: "--ignore".
1302 It filters shell elements changed by script, not to get these changes
1303 applied when loading modulefile. This option accepts a list, separated
1304 by colon character, of shell elements: "envvar", "alias", "function",
1305 "complete" or "chdir".
1306
1307 $ module purge
1308 $ cat /path/to/modulefiles/foo/2
1309 #%Module
1310 set foo_script [file dirname $ModulesCurrentModulefile]/foo-output.sh
1311 source-sh --ignore alias:function bash-eval $foo_script
1312
1313 $ module show foo/2
1314 -------------------------------------------------------------------
1315 /path/to/modulefiles/foo/2:
1316
1317 setenv FOO value
1318 -------------------------------------------------------------------
1319 $ module load foo/2
1320 $ foo_alias
1321 bash: foo_alias: command not found
1322
1323 This "--ignore" option can be used with any *shell* supported by
1324 "source-sh".
1325
1326
1327 v5.3
1328 ====
1329
1330 This new version is backward-compatible with previous version 5
1331 release. It fixes bugs but also introduces new functionalities that
1332 are described in this section. See the 5.3 release notes for a
1333 complete list of the changes between Modules v5.2 and v5.3.
1334
1335
1336 Module cache
1337 ------------
1338
1339 A module cache file can be created under each modulepath directory
1340 with new "cachebuild" sub-command. Cache file is named ".modulecache"
1341 and contains in one file all modulefiles and modulercs found in
1342 modulepath directory.
1343
1344 When cache file is available, module search mechanism uses this file
1345 rather walking through the content of modulepath directory. I/O
1346 operations are saved this way which reduces search processing time.
1347
1348 When searching for available modules without cache, each file
1349 contained in enabled modulepaths is opened to check if it is a
1350 modulefile or not. Such checks lead to a large number of I/O
1351 operations on large module setup like in the below example where a
1352 total of 1051 modulefiles are available:
1353
1354 $ module -o "" avail -t | wc -l
1355 1051
1356 $ syscall_list=access,close,getdents64,newfstatat,openat,read
1357 $ strace -f -e $syscall_list -c $MODULES_CMD bash avail
1358 % time seconds usecs/call calls errors syscall
1359 ------ ----------- ----------- --------- --------- ----------------
1360 31.09 0.003776 2 1424 9 openat
1361 28.56 0.003469 2 1649 3 newfstatat
1362 14.08 0.001710 1 1421 close
1363 11.85 0.001439 3 460 getdents64
1364 10.88 0.001321 0 1505 read
1365 3.54 0.000430 4 107 5 access
1366 ------ ----------- ----------- --------- --------- ----------------
1367 100.00 0.012145 1 6566 17 total
1368
1369 After building cache file for every enabled modulepaths in this
1370 example setup, a lot of I/O operations are saved when searching for
1371 available modules:
1372
1373 $ module cachebuild
1374 Creating /path/to/modulefiles/.modulecache
1375 Creating /path/to/modulefiles.2/.modulecache
1376 Creating /path/to/modulefiles.3/.modulecache
1377 $ module config cache_buffer_bytes 1000000
1378 $ strace -f -e $syscall_list -c $MODULES_CMD bash avail
1379 % time seconds usecs/call calls errors syscall
1380 ------ ----------- ----------- --------- --------- ----------------
1381 70.19 0.000544 2 255 read
1382 13.16 0.000102 2 38 9 openat
1383 8.90 0.000069 1 35 close
1384 6.06 0.000047 1 31 2 newfstatat
1385 1.68 0.000013 1 11 2 access
1386 ------ ----------- ----------- --------- --------- ----------------
1387 100.00 0.000775 2 370 13 total
1388
1389 A significant execution time drop may be noticed, especially if
1390 modulepath directories are stored on heavily loaded network
1391 filesystem.
1392
1393 To further optimize I/O operation count, the "cache_buffer_bytes"
1394 configuration option can be set like in the above example to use fewer
1395 number of "read" operation to load cache file content.
1396
1397 To build cache file, user should be granted write access on modulepath
1398 directory. Modulefiles or directories that are not accessible for
1399 everyone are not recorded in cache. An indication is saved instead to
1400 test these limited access elements when cache is loaded to determine
1401 if they are available to currently running user.
1402
1403 Cache file can be ignored with "--ignore-cache" command line switch or
1404 more permanently with "ignore_cache" configuration option.
1405
1406 Cache file is valid indefinitely by default but "cache_expiry_secs"
1407 configuration option can be used to define the number of seconds a
1408 cache file is considered valid after being generated. Expired cache
1409 file is ignored.
1410
1411 Cache file of enabled modulepaths can be deleted all at once with
1412 "cacheclear" sub-command.
1413
1414
1415 Querying available module variants
1416 ----------------------------------
1417
1418 A new mechanism named Extra match search is introduced to evaluate
1419 modulefiles during a module search to find those matching an extra
1420 query on a variant value, a dependency or an environment variable
1421 definition.
1422
1423 During this specific evaluation, modulefiles are interpreted in *scan*
1424 mode to collect the different Tcl modulefile commands they use.
1425 Special care should be given when writing modulefiles to ensure they
1426 cope with such evaluation mode.
1427
1428 Extra match search mechanism is available on "avail", "whatis" and
1429 "paths" sub-commands.
1430
1431 With this new mechanism, it is possible to list all available variant
1432 defined in modulefiles with their associated values:
1433
1434 $ module config avail_output modulepath:alias:dirwsym:sym:tag:key:variant
1435 $ module config variant_shortcut toolchain=%
1436 $ module avail
1437 --------------------- /path/to/modulefiles ---------------------
1438 bar/1.0{%a,b} foo/1.0{%a} qux/1.0{%a,b}
1439 bar/2.0{%b} foo/2.0{%a,b,c} qux/2.0{%b,c}
1440
1441 Key:
1442 modulepath {%value}={toolchain=value}
1443 default-version {variant=value}
1444
1445 You can also search for modules defining a specific variant value:
1446
1447 $ module avail %a
1448 --------------------- /path/to/modulefiles ---------------------
1449 bar/1.0{%a,b} foo/1.0{%a} foo/2.0{%a,b,c} qux/1.0{%a,b}
1450
1451 Key:
1452 modulepath {%value}={toolchain=value}
1453 default-version {variant=value}
1454
1455 Note:
1456
1457 As extra match search implies additional modulefile evaluations, it
1458 is advised to build and use Module cache to improve search speed.
1459
1460 Two new elements, "variant" and "variantifspec", are added to the
1461 allowed value list of "avail_output" and "avail_terse_output"
1462 configuration options. The latter is set in the default value list of
1463 both options. When "variant" is set, variants and their possible
1464 values are reported along module they are associated to. When
1465 "variantifspec" is set, available variants are reported only if a
1466 variant is specified in search query.
1467
1468 $ module config --reset avail_output
1469 $ module config avail_output
1470 Modules Release 5.3.0 (2023-05-14)
1471
1472 - Config. name ---------.- Value (set by if default overridden) ---------------
1473 avail_output modulepath:alias:dirwsym:sym:tag:variantifspec:key
1474 $ module avail foo
1475 --------------------- /path/to/modulefiles ---------------------
1476 foo/1.0 foo/2.0
1477
1478 Key:
1479 modulepath
1480 $ module avail foo %a
1481 --------------------- /path/to/modulefiles ---------------------
1482 foo/1.0{%a} foo/2.0{%a,b,c}
1483
1484 Key:
1485 modulepath {%value}={toolchain=value}
1486 default-version {variant=value}
1487
1488
1489 Extra specifiers
1490 ----------------
1491
1492 Extra specifiers are introduced to query content of modulefiles. They
1493 can be specified with *element:name* syntax as part of module
1494 specification on module search commands ("avail", "paths" and and
1495 "whatis").
1496
1497 Extra specifiers trigger Extra match search mechanism when found in
1498 module specification. Available modulefiles are evaluated in *scan*
1499 mode to collect the different Tcl modulefile commands they use.
1500
1501 $ module avail variant:toolchain
1502 --------------------- /path/to/modulefiles ---------------------
1503 bar/1.0 bar/2.0 foo/1.0 foo/2.0 qux/1.0 qux/2.0
1504
1505 In the above example, all modulefiles defining a "toolchain" variant
1506 are returned.
1507
1508 Most Tcl modulefile commands can be queried with extra specifiers:
1509 "variant", "setenv", "unsetenv", "append-path", "prepend-path",
1510 "remove-path", "pushenv", "complete", "uncomplete", "set-alias",
1511 "unset-alias", "set-function", "unset-function", "chdir", "family",
1512 "prereq", "prereq-any", "prereq-all", "depends-on", "always-load",
1513 "load", "load-any", "try-load", "switch", "switch-on", "switch-off",
1514 "conflict" and "unload". Commands that handle environment variables
1515 may be aliased "envvar". Commands that define a module requirement may
1516 be aliased "require" and those that define a module incompatibility
1517 may be aliased "incompat".
1518
1519 When several extra specifiers are set in query, modules returned are
1520 those matching both conditions. In the following example, all
1521 modulefiles interacting with "PATH" environment variable and requiring
1522 "foo" module are returned.
1523
1524 $ module avail envvar:PATH require:foo
1525 --------------------- /path/to/modulefiles ---------------------
1526 bar/1.0 bar/2.0
1527
1528 Extra specifiers related to module requirement or incompatibility may
1529 leverage the Advanced module version specifiers syntax. On following
1530 example, modulefiles returned are those defining a requirement on
1531 "foo" module with version higher or equal to "1.2" and variant
1532 "toolchain=a" selected.
1533
1534 $ module avail "require:[email protected]: toolchain=a"
1535 --------------------- /path/to/modulefiles ---------------------
1536 bar/1.0
1537
1538 Note:
1539
1540 Module aliases or symbolic versions used either in modulefile
1541 definitions or as extra specifier values are not resolved.
1542
1543
1544 Append or subtract elements to current option value
1545 ---------------------------------------------------
1546
1547 For command-line switches and configuration options whose value is a
1548 colon-separated list, it is now possible to append or subtract
1549 elements to the current value. With a "+" prefix elements are appended
1550 and with a "-" prefix elements are subtracted.
1551
1552 This new feature applies to "--output"/"-o" command-line switches and
1553 "avail_output", "avail_terse_output", "list_output",
1554 "list_terse_output", "colors", "protected_envvars",
1555 "shells_with_ksh_fpath", "tag_abbrev", "tag_color_name",
1556 "variant_shortcut" configuration options.
1557
1558 For instance, to output available *foo* modules without modulepath and
1559 tag information:
1560
1561 $ module -o -modulepath:tag avail foo
1562 foo/1.0 foo/2.0
1563
1564 On following example, terse output mode of *list* sub-command is
1565 permanently updated to report variant information:
1566
1567 $ module config list_terse_output
1568 Modules Release 5.3.0 (2023-05-14)
1569
1570 - Config. name ---------.- Value (set by if default overridden) ---------------
1571 list_terse_output header
1572 $ module config list_terse_output +variant
1573 $ module config list_terse_output
1574 Modules Release 5.3.0 (2023-05-14)
1575
1576 - Config. name ---------.- Value (set by if default overridden) ---------------
1577 list_terse_output header:variant (env-var)
1578
1579
1580 v5.2
1581 ====
1582
1583 This new version is backward-compatible with previous version 5
1584 release. It fixes bugs but also introduces new functionalities that
1585 are described in this section. See the 5.2 release notes for a
1586 complete list of the changes between Modules v5.1 and v5.2.
1587
1588
1589 Optional requirements
1590 ---------------------
1591
1592 The "--optional" option has been added to the "prereq", "prereq-all",
1593 "depends-on" and "always-load" modulefile commands to indicate that
1594 specified requirement is optional. An automatic load attempt is also
1595 performed for optional requirements. If requirement is not found or
1596 cannot be loaded, the dependency is yet considered satisfied as it is
1597 optional.
1598
1599 $ module show foo
1600 -------------------------------------------------------------------
1601 /path/to/modulefiles/foo:
1602
1603 prereq --optional bar
1604 -------------------------------------------------------------------
1605 $ module load foo
1606 Loading foo
1607 Loading requirement: bar
1608
1609 If the optional requirement is unloaded or loaded afterward, the
1610 dependent module is automatically reloaded thanks to the
1611 "auto_handling" mechanism.
1612
1613 $ module unload bar
1614 Unloading bar
1615 Unloading dependent: foo
1616 Reloading dependent: foo
1617 $ module list
1618 Currently Loaded Modulefiles:
1619 1) foo
1620 $ module load bar
1621 Loading bar
1622 Unloading dependent: foo
1623 Reloading dependent: foo
1624
1625 Modules loaded by other modules with the "module try-load" command are
1626 now considered optional requirements. Dependent module can be loaded
1627 without the *try-load* modules, but now it gets automatically reloaded
1628 if *try-load* module is loaded afterward, to take it into account.
1629
1630
1631 Linting modulefiles
1632 -------------------
1633
1634 Static analysis of modulefile, modulerc and global/user rc is now
1635 possible with "lint" sub-command. It relies on an external program
1636 defined with "tcl_linter" configuration option. Modules or files
1637 specified on the command-line are resolved then passed to the Tcl
1638 linter program.
1639
1640 $ cat /path/to/modulefiles/foo/1.0
1641 #%Module
1642 if {"str" eq} {
1643 else {
1644 }
1645 $ module lint foo/1.0
1646 Linting /path/to/modulefiles/foo/1.0
1647 ERROR line 2: Could not complete statement.
1648 One close brace would complete the first line
1649 One close brace would complete at end of line 4.
1650 One close brace would complete the script body at line 5.
1651 Assuming completeness for further processing.
1652 ERROR line 2: Bad expression: missing operand at _@_
1653 in expression ""str" eq_@_"
1654 WARNING line 3: Unknown command "else"
1655 NOTICE line 4: Close brace not aligned with line 3 (1 0)
1656
1657 Nagelfar is the Tcl linter recommended for Modules and set by default.
1658 This default can be changed at installation time with "--with-tcl-
1659 linter" and "--with-tcl-linter-opts" options. It can also be
1660 configured later on through "tcl_linter" config option.
1661
1662 Specific syntax databases and plugins for Nagelfar are added by
1663 Modules to precisely lint modulefile commands syntax in addition to
1664 regular Tcl syntax. The installation of these specific files is
1665 controlled with "--enable-nagelfar-addons" option (enabled by
1666 default). Their location is controlled by the "--nagelfardatadir"
1667 option.
1668
1669 $ module lint bar@:1 /path/to/modulefiles/.modulerc
1670 Linting /path/to/modulefiles/.modulerc
1671 ERROR line 35: Wrong number of arguments (3) to "module-alias"
1672 ERROR line 41: Wrong number of arguments (3) to "module-virtual"
1673
1674 Linting /path/to/modulefiles/bar/1.2
1675 WARNING line 19: Unknown command "unk"
1676 $ module lint ~/.modulerc
1677 Linting /home/user/.modulerc
1678 WARNING line 2: Command "setenv" should not be be used in global rc file
1679
1680 "lint" sub-command outputs messages returned by the Tcl linter
1681 program. Nagelfar produces NOTICE, WARNING and ERROR messages. If
1682 linter does not report a thing, "lint" sub-command will be silent,
1683 unless if the "--verbose"/"-v" is set.
1684
1685 $ module lint /path/to/modulefiles/bar/.version bar/1.4
1686 $ module lint -v /path/to/modulefiles/bar/.version bar/1.4
1687 Linting /path/to/modulefiles/bar/.version
1688 Linting /path/to/modulefiles/bar/1.4
1689
1690 When no file is specified to "lint" sub-command, all the global/user
1691 rc files and all the modulefiles and modulercs from enabled
1692 modulepaths are analyzed. If the "--all"/"-a" option is set, all
1693 hidden modulefiles are also linted.
1694
1695 $ module lint
1696 Linting /home/user/.modulerc
1697 WARNING line 2: Command "setenv" should not be be used in global rc file
1698
1699 Linting /path/to/modulefiles/.modulerc
1700 ERROR line 35: Wrong number of arguments (3) to "module-alias"
1701 ERROR line 41: Wrong number of arguments (3) to "module-virtual"
1702
1703 Linting /path/to/modulefiles/bar/1.2
1704 WARNING line 19: Unknown command "unk"
1705 ...
1706
1707 To use Nagelfar as Tcl linter for Modules, this open source tool has
1708 to be installed on your system. When installing from tarball
1709 distribution, make sure **nagelfar.tcl** command is found through a
1710 "PATH" lookup or that "tcl_linter" option is set to its full path
1711 location. Nagelfar is also made available as a RPM package in EPEL and
1712 Fedora repositories.
1713
1714
1715 mod-to-sh sub-command
1716 ---------------------
1717
1718 New sub-command is added to translate modulefile into shell code:
1719 "mod-to-sh". It evaluates modulefiles passed as argument and produces
1720 code for specified shell.
1721
1722 $ cat /path/to/modulefiles/foo
1723 #%Module
1724 setenv FOO value
1725 set-function foo {echo foo}
1726 $ module mod-to-sh bash foo
1727 FOO=value; export FOO;
1728 foo () { echo foo; }; export -f foo;
1729
1730 Designated modulefiles are evaluated as if they were loading. But
1731 instead of producing shell code that is evaluated in current shell
1732 session, **module** command outputs this shell code.
1733
1734 $ cat /path/to/modulefiles/bar
1735 #%Module
1736 setenv BAR othervalue
1737 set-alias bar {echo bar}
1738 $ module mod-to-sh fish foo bar
1739 set -xg FOO value;
1740 set -xg BAR othervalue;
1741 alias bar echobar;
1742 function foo; echo foo; end;
1743 $ module list
1744 No Modulefiles Currently Loaded.
1745
1746 All shells supported by "modulecmd.tcl" script are supported by "mod-
1747 to-sh".
1748
1749 $ module mod-to-sh python foo bar
1750 import os
1751 os.environ['FOO'] = 'value'
1752 os.environ['BAR'] = 'othervalue'
1753
1754
1755 Initial environment
1756 -------------------
1757
1758 When Modules initializes, it evaluates the "initrc" and "modulespath"
1759 configuration files to enable default modulepaths and load default
1760 modules. Initial environment corresponds to the environment state
1761 after this initialization.
1762
1763 Initial environment is now saved in an environment variable
1764 ("__MODULES_LMINIT") in current shell session to remember what are the
1765 initial modulepaths and initial modules with their tags and variants
1766 if any.
1767
1768 "reset" sub-command is introduced, in a similar fashion than on Lmod,
1769 to restore the initial environment. Here, "reset" relies on the
1770 collection mechanism based and the content of "__MODULES_LMINIT".
1771 Currently enabled modulepaths and loaded modules are respectively
1772 unused and unloaded to use the modulepaths and load the modules with
1773 tags and variants as described by initial environment.
1774
1775 $ module list
1776 Currently Loaded Modulefiles:
1777 1) foo/1.0 2) bar/1.0
1778 $ module switch bar/1.0 qux/1.0
1779 $ module reset
1780 Unloading qux/1.0
1781 Loading bar/1.0
1782
1783 "restore" sub-command has been adapted to reinitialize the environment
1784 to its initial state when no collection name is provided and no
1785 *default* collection exists or if "__init__" virtual collection name
1786 is provided.
1787
1788 It is possible to view the content of the initial environment with
1789 "saveshow" sub-command. It is displayed when no argument is provided
1790 and no collection exists or if "__init__" name is provided.
1791
1792 $ module saveshow __init__
1793 -------------------------------------------------------------------
1794 initial environment:
1795
1796 module use --append /path/to/modulefiles
1797 module load foo/1.0
1798 module load bar/1.0
1799
1800 -------------------------------------------------------------------
1801
1802 Users have the ability to define what is their initial environment
1803 state and thus adapt the behavior of "reset" sub-command with
1804 "reset_target_state" configuration option. Default value is "__init__"
1805 and it corresponds to the behavior described above. When set to
1806 "__purge__", a "purge" command is performed when resetting. Any other
1807 value corresponds to the name of a collection to restore.
1808
1809 $ module config reset_target_state __purge__
1810 $ module reset
1811 Unloading bar/1.0
1812 Unloading foo/1.0
1813
1814
1815 Stashing environment
1816 --------------------
1817
1818 The ability to stash current environment is added with the
1819 introduction of the "stash" sub-command. When called current
1820 environment is saved in a *stash* collection then initial environment
1821 is restored.
1822
1823 $ module list
1824 Currently Loaded Modulefiles:
1825 1) bar/2.0 2) foo/2.0
1826 $ module stash
1827 Unloading foo/2.0
1828 Unloading bar/2.0
1829 $ module list
1830 No Modulefiles Currently Loaded.
1831
1832 Sub-commands are added to specifically handle stash collections. Their
1833 names are prefixed with *stash*, like "stashlist" to list existing
1834 stash collections or "stashshow" to display their content.
1835
1836 $ module stashlist
1837 Stash collection list:
1838 0) stash-1665377597432 1) stash-1664946764252
1839 $ module stashshow
1840 -------------------------------------------------------------------
1841 /home/user/.module/stash-1665377597432:
1842
1843 module use --append /path/to/modulefiles
1844 module load bar
1845 module load foo
1846
1847 -------------------------------------------------------------------
1848
1849 Stash collections can be designated on sub-commands by their
1850 collection name or stash index. Most recent stash collection has index
1851 "0", the one after is designated with index "1", and so on. When no
1852 stash collection is specified, most recent one is assumed.
1853
1854 $ module stashshow 1
1855 -------------------------------------------------------------------
1856 /home/user/.module/stash-1664946764252:
1857
1858 module use --append /path/to/modulefiles
1859 module load foobar
1860
1861 -------------------------------------------------------------------
1862
1863 Stash collections are restored with the "stashpop" sub-command. It
1864 changes the user environment to match the stash definition, then it
1865 deletes the stash collection file.
1866
1867 $ module stashpop
1868 Loading bar/2.0
1869 Loading foo/2.0
1870 $ module stashlist
1871 Stash collection list:
1872 0) stash-1664946764252
1873
1874 Stash collections can be deleted one by one with "stashrm" sub-command
1875 or all together with "stashclear".
1876
1877 $ module stashrm
1878 $ module stashlist
1879 No stash collection.
1880
1881
1882 Siteconfig hook variables
1883 -------------------------
1884
1885 Several Tcl variables are introduced for Site-specific configuration
1886 script to define specific commands and variables in the evaluation
1887 context of modulefiles and modulercs. These commands and variables
1888 setup in "siteconfig.tcl" can be used in modulefile or modulerc. Sites
1889 can easily extend modulefile and modulerc syntax with specific
1890 elements.
1891
1892 "modulefile_extra_cmds" variable defines a list of commands to expose
1893 in the modulefile evaluation context and the associated procedure to
1894 run when this command is called. This variable has to be defined in
1895 "siteconfig.tcl" located for instance at "/etc/environment-
1896 modules/siteconfig.tcl".
1897
1898 In the following example "modulefile_extra_cmds" is used to define the
1899 "sys" command and bound it to the "sys" procedure that is also defined
1900 in "siteconfig.tcl".
1901
1902 proc sys {mode} {
1903 switch -- $mode {
1904 name { return myhost-$::tcl_platform(machine) }
1905 default { error "Unknown mode '$mode'" }
1906 }
1907 }
1908 set modulefile_extra_cmds {sys sys}
1909
1910 Once "siteconfig.tcl" is setup, the "sys" command can be called by
1911 modulefiles. In the following example it is used to determine the
1912 application path.
1913
1914 $ cat /path/to/modulefiles/foo/1.2
1915 #%Module
1916 append-path PATH /path/to/apps/foo-1.2/[sys name]/bin
1917
1918 $ module show foo/1.2
1919 -------------------------------------------------------------------
1920 /path/to/modulefiles/foo/1.2:
1921
1922 append-path PATH /path/to/apps/foo-1.2/myhost-x86_64/bin
1923 -------------------------------------------------------------------
1924
1925 "modulerc_extra_cmds" follows the same approach than
1926 "modulefile_extra_cmds" and makes specific commands available during
1927 modulerc evaluation.
1928
1929 "modulefile_extra_vars" variable defines a list of variables to expose
1930 in the modulefile evaluation context and their associated value. This
1931 variable has to be defined in "siteconfig.tcl".
1932
1933 In the following example "modulefile_extra_vars" is used to define the
1934 "APP_ROOT" variable with "/path/to/apps" as value.
1935
1936 set modulefile_extra_vars {APP_ROOT /path/to/apps}
1937
1938 Once "siteconfig.tcl" is setup, the "APP_ROOT" variable can be used in
1939 modulefiles.
1940
1941 $ cat /path/to/modulefiles/bar/2.1
1942 #%Module
1943 append-path PATH $APP_ROOT/bar-2.1/[sys name]/bin
1944
1945 $ module show bar/2.1
1946 -------------------------------------------------------------------
1947 /path/to/modulefiles/bar/2.1:
1948
1949 append-path PATH /path/to/apps/bar-2.1/myhost-x86_64/bin
1950 -------------------------------------------------------------------
1951
1952 "modulerc_extra_vars" follows the same approach than
1953 "modulefile_extra_vars" and makes specific variables available during
1954 modulerc evaluation.
1955
1956
1957 v5.1
1958 ====
1959
1960 This new version is backward-compatible with previous version 5
1961 release. It fixes bugs but also introduces new functionalities that
1962 are described in this section. See the 5.1 release notes for a
1963 complete list of the changes between Modules v5.0 and v5.1.
1964
1965
1966 Control output redirection
1967 --------------------------
1968
1969 Since version 4.0, the **module** function is initialized differently
1970 on *sh*, *bash*, *ksh*, *zsh* and *fish* shells when their session is
1971 found interactive. In such situation **module** redirects its output
1972 from *stderr* to *stdout*. Once initialized the redirection behavior
1973 is inherited in sub-sessions.
1974
1975 The "redirect_output" configuration option is introduced in version
1976 5.1, to supersede the default behavior set at initialization time.
1977
1978 $ module load unknown >/dev/null
1979 $ module config redirect_output 0
1980 $ module load unknown >/dev/null
1981 ERROR: Unable to locate a modulefile for 'unknown'
1982
1983 The "--redirect" and "--no-redirect" command-line switches are also
1984 added to change the output redirection behavior for a single command:
1985
1986 $ module load unknown --redirect >/dev/null
1987 $ module load unknown --no-redirect >/dev/null
1988 ERROR: Unable to locate a modulefile for 'unknown'
1989
1990
1991 Change modulefile command behavior
1992 ----------------------------------
1993
1994 Depending on the evaluation mode of the modulefile (e.g. *load*,
1995 *unload*, *display*, etc) commands have different behavior. Most
1996 common example is the "setenv" command that sets an environment
1997 variable when modulefile is loaded and unsets it when it is unloaded.
1998 A different behavior may be wished sometimes for commands. This is why
1999 options are introduced for some modulefile commands to control what
2000 happens on particular evaluation mode.
2001
2002 The "--return-value" option is added to the "getenv" and "getvariant"
2003 modulefile commands to ensure that the value of the designated
2004 environment variable or variant is returned even if modulefile is
2005 evaluated in *display* mode:
2006
2007 $ cat /path/to/modulefiles/foo/1.0
2008 #%Module5.1
2009 if {[getenv --return-value VAR] eq {}} {
2010 setenv VAR value
2011 }
2012 $ module display foo/1.0
2013 -------------------------------------------------------------------
2014 /path/to/modulefiles/foo/1.0:
2015
2016 setenv VAR value
2017 -------------------------------------------------------------------
2018
2019 "--remove-on-unload", "--append-on-unload", "--prepend-on-unload" and
2020 "--noop-on-unload" options are added to the "remove-path" and "module
2021 unuse" modulefile commands to control the behavior applied when
2022 modulefile is unloaded. With these options it is possible for instance
2023 to restore the paths unset at load time or to set other paths:
2024
2025 $ module display bar/1.0
2026 -------------------------------------------------------------------
2027 /path/to/modulefiles/bar/1.0:
2028
2029 module unuse --prepend-on-unload /path/to/dir1
2030 module use /path/to/dir2
2031 -------------------------------------------------------------------
2032 $ module use
2033 Search path for module files (in search order):
2034 /path/to/dir1
2035 /path/to/modulefiles
2036 $ module bar/1.0
2037 $ module use
2038 Search path for module files (in search order):
2039 /path/to/dir2
2040 /path/to/modulefiles
2041 $ module unload bar/1.0
2042 $ module use
2043 Search path for module files (in search order):
2044 /path/to/dir1
2045 /path/to/modulefiles
2046
2047 Following the same trend, the "--unset-on-unload" and "--noop-on-
2048 unload" options are added to the "unsetenv" modulefile command to be
2049 able to choose between unsetting variable, setting a value or
2050 performing no operation when modulefile is unloaded.
2051
2052
2053 Reducing number of I/O operations
2054 ---------------------------------
2055
2056 A new configuration option named "mcookie_check" is introduced to
2057 control the verification made to files to determine if they are
2058 modulefiles. By default this configuration option is set to "always"
2059 and when searching for modulefiles within enabled modulepaths each
2060 file below these directories is opened to check if it starts with the
2061 Modules magic cookie (i.e., "#%Module" file signature).
2062
2063 These historical checks lead to a large number of I/O operations on
2064 large module setup like in the below example where a total of 1098
2065 modulefiles are available:
2066
2067 $ module -o "" avail -t | wc -l
2068 1098
2069 $ module config mcookie_check always
2070 $ strace -f -e open,openat,read,close -c $MODULES_CMD bash avail
2071 ...
2072 % time seconds usecs/call calls errors syscall
2073 ------ ----------- ----------- --------- --------- ----------------
2074 44.29 0.044603 25 1734 166 open
2075 34.15 0.034389 16 2056 close
2076 11.87 0.011949 24 483 5 openat
2077 9.69 0.009761 4 2146 read
2078 ------ ----------- ----------- --------- --------- ----------------
2079 100.00 0.100702 6419 171 total
2080
2081 For each file, 3 I/O operations ("open", "read" and "close") are
2082 achieved to determine if it is a modulefile and include it in search
2083 results. When modulefiles are located in a shared filesystem
2084 concurrently accessed by hundreds of users, a "module avail" command
2085 may take some time to finish.
2086
2087 When setting the "mcookie_check" configuration option to the "eval"
2088 value, files are not checked anymore when searching for modulefiles,
2089 only when evaluating them. All files under modulepaths are considered
2090 modulefiles, so the content of these directories must be carefully
2091 checked to use this "mcookie_check" mode which lead to a significant
2092 reduction of I/O operations:
2093
2094 $ module config mcookie_check eval
2095 $ strace -f -e open,openat,read,close -c $MODULES_CMD bash avail
2096 ...
2097 % time seconds usecs/call calls errors syscall
2098 ------ ----------- ----------- --------- --------- ----------------
2099 30.56 0.013717 14 944 close
2100 28.76 0.012911 21 612 156 open
2101 26.41 0.011857 24 483 5 openat
2102 14.26 0.006403 6 1034 read
2103 ------ ----------- ----------- --------- --------- ----------------
2104 100.00 0.044888 3073 161 total
2105
2106 A substantial reduction of execution time may be noticed depending on
2107 the storage setup used to host the modulepath directories. A special
2108 care should be given to the content of these directories to ensure
2109 they only contain modulefiles (see "MODULES_MCOOKIE_CHECK").
2110
2111 See the Reduce I/O load cookbook recipe to learn additional features
2112 of Modules that could be leveraged to lower the number of I/O
2113 operations.
2114
2115
2116 Shell command completion
2117 ------------------------
2118
2119 New modulefile commands "complete" and "uncomplete" are added to get
2120 the ability to respectively define and unset command completion.
2121 *bash*, *tcsh* and *fish* shells are supported.
2122
2123 > module display foo
2124 -------------------------------------------------------------------
2125 /path/to/modulefiles/foo/1.0:
2126
2127 append-path PATH /path/to/foo-1.0/bin
2128 complete fish foo {-s V -l version --description 'Command version'}
2129 complete fish foo {-s h -l help --description 'Command help'}
2130 -------------------------------------------------------------------
2131 > module load foo
2132 > foo -<TAB>
2133 -h --help (Command help) -V --version (Command version)
2134
2135 "sh-to-mod" sub-command and "source-sh" modulefile command have also
2136 been updated to track shell completion changes.
2137
2138 $ module sh-to-mod bash /path/to/foo-1.0/share/setup-env.sh
2139 #%Module
2140 complete bash foo {-o default -F _foo}
2141 append-path PATH /path/to/foo-1.0/bin
2142 set-function _foo {
2143 ...bash completion code...}
2144
2145
2146 Lmod Tcl modulefile support
2147 ---------------------------
2148
2149 With this new version, Modules now supports Tcl modulefiles written
2150 for Lmod, the alternative **module** implementation developed in Lua.
2151 Such modulefiles can be safely evaluated by Modules without raising
2152 error.
2153
2154 Support has been added for the following modulefile commands
2155 introduced by Lmod: "depends-on", "prereq-any", "always-load", "module
2156 load-any", "pushenv", "require-fullname" and "family".
2157
2158 The Compatibility with Lmod Tcl modulefile section in the modulefile
2159 man page describes the differences existing between the two
2160 implementations.
2161
2162 Note that when processing a "family" command, the "LMOD_FAMILY_<NAME>"
2163 environment variable is defined by Modules to be compatible with
2164 existing modulefiles or scripts relying on such variable.
2165
2166
2167 More tagging capabilities
2168 -------------------------
2169
2170 The new "--tag" option helps to define extra tags onto a loading
2171 module. These tags comes in addition to those inherited from the
2172 module state or those associated with the "module-tag" modulefile
2173 command.
2174
2175 The "--tag" option is available on "load", "load-any", "switch" and
2176 "try-load" sub-commands and on "always-load", "depends-on", "module",
2177 "prereq", "prereq-all" and "prereq-any" modulefile commands.
2178
2179 Informational messages of module evaluation have been updated to
2180 mention in module denomination the tags applying to it, as it is done
2181 in "list" sub-command output:
2182
2183 $ module load -v --tag=sticky:bar foo/1.0
2184 Loading foo/1.0 <bar>
2185
2186 In case the designated module is already loaded, the additional tags
2187 are added to the list of tags already applied to this module.
2188
2189 $ module list
2190 Currently Loaded Modulefiles:
2191 1) foo/1.0
2192 $ module load -v --tag=sticky:bar foo/1.0
2193 Tagging foo/1.0 <bar>
2194
2195 The "keep-loaded" tag is introduced in this version. It avoids an
2196 auto-loaded module to get automatically unloaded when its dependent
2197 modules are unloaded. This new tag can be set with the "module-tag"
2198 modulefile command or when module is loaded with the "always-load"
2199 modulefile command. Default "tag_abbrev" configuration option has been
2200 updated to add the "kL" abbreviation for "keep-loaded" tag. Default
2201 dark and light color palettes have been updated too.
2202
2203 $ module show bar/1.0
2204 -------------------------------------------------------------------
2205 /path/to/modulefiles/bar/1.0:
2206
2207 always-load foo/1.0
2208 -------------------------------------------------------------------
2209 $ module load bar/1.0
2210 Loading bar/1.0
2211 Loading requirement: foo/1.0
2212 $ module unload bar/1.0
2213 $ module list
2214 Currently Loaded Modulefiles:
2215 1) foo/1.0
2216
2217 Key:
2218 auto-loaded keep-loaded
2219
2220 When saving a collection, the tags defined with "--tag" option are
2221 recorded to set them again when collection is restored. Tags resulting
2222 from module load state, like "auto-loaded" and "keep-loaded", are also
2223 recorded.
2224
2225 $ module load --tag=sticky bar/1.0
2226 Loading bar/1.0
2227 Loading requirement: foo/1.0
2228 $ module save
2229 $ module saveshow
2230 -------------------------------------------------------------------
2231 /home/user/.module/default:
2232
2233 module use --append /path/to/modulefiles
2234 module load --tag=auto-loaded:keep-loaded foo
2235 module load --tag=sticky bar
2236
2237 -------------------------------------------------------------------
2238
2239 The "collection_pin_tag" configuration option is added to record in
2240 collection all tags set on loaded modules. This configuration option
2241 is disabled by default.
2242
2243 Note:
2244
2245 Collection saved now starts with a "#%Module5.1" file signature if "
2246 --tag" option is recorded in it. Such collection could only be
2247 handled by Modules version 5.1 and above.
2248
2249
2250 v5.0
2251 ====
2252
2253 With this new major version the **module** experience has been updated
2254 to benefit by default from all the advanced behaviors and features
2255 developed over the Modules 4 versions. Modules 5.0 also introduces
2256 some breaking changes to improve the consistency of the whole
2257 solution. See the 5.0 release notes for a complete list of the changes
2258 between Modules v4.8 and v5.0. The changes document gives an in-depth
2259 view of the modified behaviors.
2260
2261
2262 Upgraded default configuration
2263 ------------------------------
2264
2265 Release after release, new advanced features were added on Modules 4.
2266 They were set off by default to avoid breaking change during the
2267 version 4 cycle. With the move to a new major release, these features
2268 are set on to improve by default usages for everybody. These features
2269 enabled by default are:
2270
2271 * Automated module handling mode which provides the automatic
2272 dependency resolution when loading and unloading modules
2273
2274 $ module load foo/1.0
2275 Loading foo/1.0
2276 Loading requirement: bar/1.0
2277
2278 * Extended default that enables to select a module when the first
2279 number in its version is specified
2280
2281 $ module load -v foo/1
2282 Loading foo/1.2.3
2283
2284 * Advanced module version specifiers, an improved syntax to designate
2285 module version in range or list and module variant
2286
2287 $ module load foo@:2.2 %gcc11
2288 Loading foo/2.1{%gcc11}
2289 Loading requirement: bar/1.2{-debug:%gcc11}
2290
2291 * Colored output to graphically enhance parts of the produced output
2292 to improve readability
2293
2294 $ ml av
2295 ------------------ /path/to/modulefiles ------------------
2296 bar/1.0 bar/2.0 foo/1.0 foo/2.0 foo/2.2
2297
2298 Key:
2299 modulepath module-alias sticky
2300 default-version forbidden
2301
2302 * Specify modules in a case insensitive manner search that matches
2303 modules using a different character case than the one expressed in
2304 search query
2305
2306 $ ml av liba
2307 ------------------ /path/to/modulefiles ------------------
2308 LibA/1.0 LibA/2.0
2309
2310 Some other features that were enabled by default during the Modules 4
2311 cycle have been turned off as they may not be useful for basic usages.
2312 Among other things setting off the following features makes the
2313 definition of the **module** function simpler. Even if off by default,
2314 these features can now be enabled once Modules is installed through
2315 the "initrc" configuration file.
2316
2317 * *Set shell startup* files to ensure the **module** command is
2318 defined once shell has been initialized. See the "set_shell_startup"
2319 configuration option to activate.
2320
2321 * *Quarantine mechanism* that protects the **module** command run-time
2322 environment from side effect coming from the current environment
2323 definition. See the "quarantine_support" configuration option to
2324 activate.
2325
2326 * *Silent shell debug* which disables the debugging property set on
2327 current shell session for the duration of the **module** command.
2328 See the "silent_shell_debug" configuration option to activate.
2329
2330
2331 Removing compatibility version
2332 ------------------------------
2333
2334 The ability to co-install version 3.2 of Modules along newer version
2335 is discontinued. The installation option "--enable-compat-version",
2336 **switchml** shell function and "MODULES_USE_COMPAT_VERSION"
2337 environment variables are thus removed.
2338
2339 The interesting features of Modules 3.2 that were missing in the
2340 initial Modules 4 release in 2017 have been reintroduced progressively
2341 (like "clear" sub-command or "--icase" search). With Modules 5.0, the
2342 "refresh" sub-command is even changed to the behavior it had on
2343 Modules 3.2. So it is a good time for the big jump.
2344
2345 If you are still using Modules 3.2, please refer to the Changes
2346 between versions document that describes the differences of this
2347 version compared to the newer releases.
2348
2349
2350 Improving Modules initialization
2351 --------------------------------
2352
2353 Modules initialization files are now installed by default in the *etc*
2354 directory designated by the "--etcdir" installation option. The
2355 initialization configuration file is named "initrc" in this directory,
2356 and the modulepath-specific configuration file is named "modulespath".
2357 When both files exist, now they are both evaluated instead of just the
2358 "modulespath" file.
2359
2360 Modules magic cookie (i.e., "#%Module" file signature) is now required
2361 at the start of "initrc". An error is produced if the magic cookie is
2362 missing or if the optional version number placed after the cookie
2363 string is higher than the version of the "modulecmd.tcl" script in
2364 use.
2365
2366 Note that "initrc" configuration file can host more than "module use"
2367 and "module load" commands. **module** configuration can also be
2368 achieved with this file through the use of "module config" commands.
2369
2370 Modules initialization has been enhanced for situations where a module
2371 environment is found already defined. In this case the loaded modules
2372 are automatically refreshed which is useful to re-apply the non-
2373 persistent environment configuration (i.e., shell alias and function
2374 that are not exported to the sub-shell). For instance when starting a
2375 sub-shell session it ensures that the loaded environment is fully
2376 inherited from parent shell:
2377
2378 $ ml show foo/1.0
2379 -------------------------------------------------------------------
2380 /path/to/modulefiles/foo/1.0:
2381
2382 set-alias foo {echo foo}
2383 -------------------------------------------------------------------
2384 $ ml foo/1.0
2385 $ alias foo
2386 alias foo='echo foo'
2387 $ bash
2388 $ ml
2389 Currently Loaded Modulefiles:
2390 1) foo/1.0
2391 $ alias foo
2392 alias foo='echo foo'
2393
2394
2395 Simplifying path-like variable reference counting
2396 -------------------------------------------------
2397
2398 The reference counting mechanism used for path-like environment
2399 variable enables to determine if a path entry has been added several
2400 times (by several loaded modules for instance) to know whether or not
2401 this path entry should be unset when unloading a module. Entry is not
2402 removed if multiple loaded modules rely on it.
2403
2404 The mechanism is not applied anymore to the Modules-specific path
2405 variables (like "LOADEDMODULES") as an element entry in these
2406 variables cannot be added multiple times without duplication. For
2407 instance, a given module name and version cannot be added twice in
2408 "LOADEDMODULES" as this module is only loaded once. With this change a
2409 thinner environment is produced by **module**. An exception is made
2410 for "MODULEPATH" environment variable where the mechanism still
2411 applies.
2412
2413 $ $MODULES_CMD bash load foo/2.0
2414 _LMFILES_=/path/to/modulefiles/foo/2.0; export _LMFILES_;
2415 LOADEDMODULES=foo/2.0; export LOADEDMODULES;
2416 __MODULES_LMTAG=foo/2.0&mytag; export __MODULES_LMTAG;
2417 test 0;
2418
2419 Reference counting mechanism has also been simplified for entries in
2420 path-like variable that are only referred once. For such entries no
2421 entry is set in the reference counting variable (which are now called
2422 "__MODULES_SHARE_<VAR>"). A reference count entry is set only if the
2423 entry in the path-like variable is referred more than one time.
2424
2425 $ ml foo/3.0
2426 $ echo $PATHVAR
2427 /path/to/dir1
2428 $ echo $__MODULES_SHARE_PATHVAR
2429
2430 $ ml bar/1.0
2431 $ echo $PATHVAR
2432 /path/to/dir1
2433 $ echo $__MODULES_SHARE_PATHVAR
2434 /path/to/dir1:2
2435 $ ml -foo/3.0
2436 $ echo $PATHVAR
2437 /path/to/dir1
2438 $ echo $__MODULES_SHARE_PATHVAR
2439
2440 $
2441
2442 When the "use" and "unuse" module sub-commands are not called during a
2443 modulefile evaluation, the reference counter associated with each
2444 entry in "MODULEPATH" environment variable is ignored. In such
2445 context, a "module use" will not increase the reference counter of a
2446 path entry already defined and a "module unuse" will remove specified
2447 path whatever its reference counter value. Same change applies for
2448 "append-path", "prepend-path" and "remove-path" module sub-commands
2449 when called from the command-line.
2450
2451 $ echo $MODULEPATH
2452 /path/to/modulefiles
2453 $ echo $__MODULES_SHARE_MODULEPATH
2454 /path/to/modulefiles:2
2455 $ ml use /path/to/modulefiles
2456 $ echo $__MODULES_SHARE_MODULEPATH
2457 /path/to/modulefiles:2
2458 $ ml unuse /path/to/modulefiles
2459 $ echo $MODULEPATH
2460
2461 $ echo $__MODULES_SHARE_MODULEPATH
2462
2463 $
2464
2465
2466 v4.8
2467 ====
2468
2469 This new version is backward-compatible with previous version 4
2470 releases. It fixes bugs but also introduces new functionalities that
2471 are described in this section. See the 4.8 release notes for a
2472 complete list of the changes between Modules v4.7 and v4.8.
2473
2474
2475 Editing modulefiles
2476 -------------------
2477
2478 "edit" sub-command is introduced to give the ability to open
2479 modulefiles in a text editor. Modulefiles can be specified like with
2480 any other sub-command: using regular, symbolic or aliased names or
2481 using advanced version specifiers.
2482
2483 $ ml edit foo
2484
2485 "edit" sub-command resolves the path toward the designated modulefile
2486 then call configured text editor to open this modulefile with it.
2487 Below, the modulefile is opened with the "vi" command:
2488
2489 #%Module
2490 module-whatis [module-info name]
2491 setenv PATH /path/to/foo-1.0/bin
2492 ~
2493 ~
2494 ~
2495 "/path/to/modulefiles/foo/1.0" 3L, 42B 1,1 All
2496
2497 The "editor" configuration option controls the editor command to use.
2498 This option can be configured at installation time with the "--with-
2499 editor" installation option. If not set, "editor" configuration option
2500 is set by default to "vi".
2501
2502 "editor" configuration option can be changed with the "config" sub-
2503 command. Which sets the "MODULES_EDITOR" environment variable.
2504
2505 The "VISUAL" or the "EDITOR" environment variables override the
2506 default value of "editor" configuration option but are overridden by
2507 the "MODULES_EDITOR" environment variable.
2508
2509
2510 Using version range in version list
2511 -----------------------------------
2512
2513 The Advanced module version specifiers mechanism has been improved to
2514 allow the use of version range (*@:version*, *@vers1:vers2* or
2515 *@version:*) within version list (*@version1,version2,...*).
2516
2517 It is now possible to write for instance "mod@:1.2,1.4:1.6,1.8:" to
2518 designate all versions of module *mod*, except versions *1.3* and
2519 *1.7*.
2520
2521 This improvement is available where the advanced version specifier
2522 syntax is supported. Thus it can be either used from the command-line
2523 or when writing modulefiles, for instance to hide or tag modules or to
2524 declare requirements.
2525
2526
2527 Try module load with no complain if not found
2528 ---------------------------------------------
2529
2530 Add the "try-load" sub-command that tries to load the modulefile
2531 passed as argument, like the "load" sub-command, but does not raise an
2532 error if this modulefile cannot be found.
2533
2534 $ module load unknown
2535 ERROR: Unable to locate a modulefile for 'unknown'
2536 $ echo $?
2537 1
2538 $ module try-load unknown
2539 $ echo $?
2540 0
2541 $ module list
2542 No Modulefiles Currently Loaded.
2543
2544 This sub-command first introduced by the Lmod project is added to
2545 Modules to improve the compatibility between the two "module"
2546 implementations.
2547
2548 "try-load" is also available within modulefile context to continue the
2549 evaluation of a modulefile in case no module is found in its attempt
2550 to load another modulefile
2551
2552 $ module display foo/1.0
2553 -------------------------------------------------------------------
2554 /path/to/modulefiles/foo/1.0:
2555
2556 module try-load unknown/1.0
2557 -------------------------------------------------------------------
2558 $ module load foo/1.0
2559 $ module list
2560 Currently Loaded Modulefiles:
2561 1) foo/1.0
2562
2563
2564 Module variants
2565 ---------------
2566
2567 Module variants is a new mechanism that allows to pass arguments to
2568 evaluated modulefiles in order to achieve different environment
2569 variable or module requirement setup with a single modulefile.
2570
2571 Variant specification relies on the Advanced module version specifiers
2572 mechanism, which leverages the variant syntax of the Spack package
2573 manager:
2574
2575 $ module config advanced_version_spec 1
2576 $ module load -v bar/1.2 toolchain=a -debug
2577 Loading bar/1.2{-debug:toolchain=a}
2578
2579 Variants are defined in modulefile with the "variant" command, which
2580 defines the variant type and its accepted values:
2581
2582 #%Module4.8
2583 variant toolchain a b c
2584 variant --boolean --default off debug
2585
2586 # select software build depending on variant values
2587 set suffix -[getvariant toolchain]
2588 if {$ModuleVariant(debug)} {
2589 append suffix -dbg
2590 }
2591
2592 prepend-path PATH /path/to/bar-1.2$suffix/bin
2593 prepend-path LD_LIBRARY_PATH /path/to/bar-1.2$suffix/lib
2594
2595 The *bar/1.2* modulefile defines a "toolchain" variant, which accepts
2596 the "a", "b" and "c" values, and a "debug" Boolean variant, which is
2597 set "off" by default. Once these two variants are declared, their
2598 value specified on module designation are instantiated in the
2599 "ModuleVariant" array variable which could also be queried with the
2600 "getvariant" modulefile command. Selected variant values enable to
2601 define a specific installation build path for the *bar/1.2* software.
2602
2603 If a variant is not specified when designating module and if this
2604 variant is not declared with a default value, an error is obtained:
2605
2606 $ module purge
2607 $ module load [email protected]
2608 Loading bar/1.2
2609 ERROR: No value specified for variant 'toolchain'
2610 Allowed values are: a b c
2611
2612 Once module is loaded, selected variants are reported on the "list"
2613 sub-command output:
2614
2615 $ module load [email protected] toolchain=b
2616 $ module list
2617 Currently Loaded Modulefiles:
2618 1) bar/1.2{-debug:toolchain=b}
2619
2620 Key:
2621 {-variant}={variant=off} {variant=value}
2622
2623 Note:
2624
2625 The default value of the "--with-list-output" installation option
2626 has been updated to include variant information.
2627
2628 Variant specification could be used where the Advanced module version
2629 specifiers is supported. For instance a module may express a
2630 dependency over a specific module variant:
2631
2632 $ module show foo/2.1 toolchain=c
2633 -------------------------------------------------------------------
2634 /path/to/modulefiles/foo/2.1:
2635
2636 variant toolchain a b c
2637 prereq [email protected] toolchain={toolchain}
2638 prepend-path PATH /path/to/foo-2.1-{toolchain}/bin
2639 prepend-path LD_LIBRARY_PATH /path/to/foo-2.1-{toolchain}/lib
2640 -------------------------------------------------------------------
2641
2642 In this example, *foo/2.1* module depends on *bar/1.2* and the same
2643 toolchain variant should be selected for both modules in order to load
2644 two software builds that are compatible between each other.
2645
2646 $ module purge
2647 $ module config auto_handling 1
2648 $ module load foo/2.1 toolchain=a
2649 Loading foo/2.1{toolchain=a}
2650 Loading requirement: bar/1.2{-debug:toolchain=a}
2651
2652
2653 Variant shortcuts
2654 -----------------
2655
2656 The "variant_shortcut" configuration option is added to define
2657 shortcut characters for easily specifying variants. Instead of writing
2658 the variant name to specify it in module designation (e.g.,
2659 *name=value*), the shortcut associated to this variant could be used
2660 (i.e., *<shortcut>value*):
2661
2662 $ module purge
2663 $ module config variant_shortcut toolchain=%
2664 $ module load foo/2.1 %a
2665 Loading foo/2.1{%a}
2666 Loading requirement: bar/1.2{-debug:%a}
2667
2668 Configured shortcuts are also used to report the loaded variant on
2669 "list" sub-command output (shortcuts are explained in key section):
2670
2671 $ module list
2672 Currently Loaded Modulefiles:
2673 1) bar/1.2{-debug:%a} 2) foo/2.1{%a}
2674
2675 Key:
2676 auto-loaded {-variant}={variant=off} {%value}={toolchain=value} {variant=value}
2677
2678
2679 v4.7
2680 ====
2681
2682 This new version is backward-compatible with previous version 4
2683 releases. It fixes bugs but also introduces new functionalities that
2684 are described in this section. See the 4.7 release notes for a
2685 complete list of the changes between Modules v4.6 and v4.7.
2686
2687
2688 Determining module implementation and version
2689 ---------------------------------------------
2690
2691 New Modules variables are introduced to determine during the
2692 evaluation of a modulefile or a modulerc what *module* implementation
2693 is currently in use. The "ModuleTool" variable corresponds to the name
2694 of the *module* implementation and is set to "Modules" for this
2695 project. The "ModuleToolVersion" variable corresponds to the version
2696 number of the implementation (e.g. "4.7.0").
2697
2698 With these new variables it is possible to precisely know what
2699 *module* command is in use then adapt modulefile code to handle a
2700 specific behavior or leverage a new feature.
2701
2702 The modulefile command "versioncmp" is also introduced to provide a
2703 simple way to compare two version strings and return if first version
2704 string is less than, equal to or greater than second one.
2705
2706 if {[info exists ModuleTool] && $ModuleTool eq {Modules}
2707 && [versioncmp $ModuleToolVersion 4.7] >= 0} {
2708 # here some code specific for Modules 4.7 and later versions
2709 }
2710
2711 The "ModuleTool" and "ModuleToolVersion" variables and the
2712 "versioncmp" modulefile command are supported by the Lmod project
2713 starting version "8.4.8".
2714
2715
2716 Symbolic version to designate module loaded version
2717 ---------------------------------------------------
2718
2719 When the Advanced module version specifiers is enabled, the "loaded"
2720 symbolic version may be used to designate the currently loaded version
2721 of specified module.
2722
2723 $ ml display foo@loaded
2724 -------------------------------------------------------------------
2725 /path/to/modulefiles/foo/1.0:
2726
2727 module-whatis foo/1.0
2728 -------------------------------------------------------------------
2729
2730 If no version of specified module can be found loaded, an error is
2731 returned.
2732
2733 $ ml display foo@loaded
2734 ERROR: No loaded version found for 'foo' module
2735
2736
2737 Module tags
2738 -----------
2739
2740 Module tags are piece of information that can be associated to
2741 individual modulefiles. Tags could be purely informational or may lead
2742 to specific behaviors.
2743
2744 Module tags may be inherited from the module state set by a modulefile
2745 command or consequence of a module action. Tags may also be associated
2746 to modules by using the new "module-tag" modulefile command.
2747
2748 Module tags are reported along the module they are associated to on
2749 "avail" and "list" sub-command results. Tags could be reported either:
2750
2751 * along the module name, all tags set within angle brackets, each tag
2752 separated from the others with a colon character (e.g., "foo/1.2
2753 <tag1:tag2>").
2754
2755 $ cat /path/to/modulefiles/foo/.modulerc
2756 #%Module
2757 module-tag mytag foo
2758 module-tag othertag foo/1.0
2759 $ ml av
2760 --------------- /path/to/modulefiles ---------------
2761 foo/1.0 <mytag:othertag> foo/2.0 <mytag>
2762 $ ml foo/1.0
2763 $ ml
2764 Currently Loaded Modulefiles:
2765 1) foo/1.0 <mytag:othertag>
2766
2767 * graphically rendered over the module name for each tag associated to
2768 a Select Graphic Rendition (SGR) code in the color palette (see
2769 "MODULES_COLORS")
2770
2771 $ # set SGR code to report 'mytag' with blue background color
2772 $ ml config colors "hi=1:di=94:L=90;47:mytag=102"
2773 $ ml av
2774 --------------- /path/to/modulefiles ---------------
2775 foo/1.0 <othertag> foo/2.0
2776 $ ml
2777 Currently Loaded Modulefiles:
2778 1) foo/1.0 <othertag>
2779
2780 The "tag_abbrev" configuration option is available to define
2781 abbreviated strings for module tags and then use these abbreviations
2782 instead of tag names when reporting tags on "avail" and "list" command
2783 results.
2784
2785 $ # add abbreviation for 'othertag' tag
2786 $ ml config tag_abbrev loaded=L:othertag=oT
2787 $ ml av
2788 --------------- /path/to/modulefiles ---------------
2789 foo/1.0 <oT> foo/2.0
2790 $ ml
2791 Currently Loaded Modulefiles:
2792 1) foo/1.0 <oT>
2793
2794 When a SGR code is set for a tag in the color palette, this graphical
2795 rendition is applied by default over the module name and the tag name
2796 or its abbreviation is not displayed. If tag name or abbreviation is
2797 added to the "tag_color_name" configuration option, graphical
2798 rendering is applied to the tag name or abbreviation rather than over
2799 the module name they are attached to.
2800
2801 $ # add SGR code for 'oT' tag and set rendition over tag name
2802 $ ml config colors "hi=1:di=94:L=90;47:mytag=44:oT=41"
2803 $ ml config tag_color_name oT
2804 $ ml av
2805 --------------- /path/to/modulefiles ---------------
2806 foo/1.0 <oT> foo/2.0
2807 $ ml
2808 Currently Loaded Modulefiles:
2809 1) foo/1.0 <oT>
2810
2811 Tags inherited from module state, consequence of a module action or
2812 set by using "module-tag" but that have a special meaning currently
2813 are:
2814
2815 +--------------------+---------------------------+--------------------------+---------+
2816 | Tag | Description | Set with | Abbr. |
2817 |====================|===========================|==========================|=========|
2818 | auto-loaded | Module has been loaded | Inherited | aL |
2819 | | automatically | | |
2820 +--------------------+---------------------------+--------------------------+---------+
2821 | forbidden | Module cannot be loaded | Inherited from "module- | F |
2822 | | | forbid" | |
2823 +--------------------+---------------------------+--------------------------+---------+
2824 | hidden | Module is not visible on | Inherited from "module- | H |
2825 | | "avail" | hide" | |
2826 +--------------------+---------------------------+--------------------------+---------+
2827 | hidden-loaded | See Hiding loaded modules | Inherited from "module- | H |
2828 | | | hide" | |
2829 +--------------------+---------------------------+--------------------------+---------+
2830 | loaded | Module is currently | Inherited | L |
2831 | | loaded | | |
2832 +--------------------+---------------------------+--------------------------+---------+
2833 | nearly-forbidden | Module will soon not be | Inherited from "module- | nL |
2834 | | able to load anymore | forbid" | |
2835 +--------------------+---------------------------+--------------------------+---------+
2836 | sticky | See Sticky modules | "module-tag" | S |
2837 +--------------------+---------------------------+--------------------------+---------+
2838 | super-sticky | See Sticky modules | "module-tag" | sS |
2839 +--------------------+---------------------------+--------------------------+---------+
2840
2841
2842 Hiding loaded modules
2843 ---------------------
2844
2845 The "--hidden-loaded" option has been added to the "module-hide"
2846 modulefile command and it indicates that designated hidden modules
2847 remain hidden after being loaded.
2848
2849 $ cat /path/to/modulefiles/foo/1.0
2850 #%Module
2851 module load bar
2852 $ cat /path/to/modulefiles/bar/.modulerc
2853 #%Module4.7
2854 module-hide --soft --hidden-loaded bar
2855
2856 In this example, *foo* depends on *bar* which is set soft hidden and
2857 hidden once loaded. As a consequence, automated load of *bar* module
2858 will not be reported and *bar/1.0* will not appear in loaded module
2859 list by default:
2860
2861 $ ml foo
2862 $ ml
2863 Currently Loaded Modulefiles:
2864 1) foo/1.0
2865
2866 However *bar/1.0* is loaded. Hidden loaded modules can be unveiled
2867 with the "--all"/"-a" option set on the "list" sub-command. "hidden-
2868 loaded" tag (abbreviated by default to "H" when colored output is
2869 disabled) applies to such modules.
2870
2871 $ ml -a
2872 Currently Loaded Modulefiles:
2873 1) bar/1.0 2) foo/1.0
2874
2875 To also get the informational messages about hidden loaded module
2876 automated load or unload, the new verbosity level "verbose2" can be
2877 used (with "-vv" option for instance):
2878
2879 $ ml purge
2880 $ ml -vv foo
2881 Loading bar/1.0
2882
2883 Loading foo/1.0
2884 Loading requirement: bar/1.0
2885
2886
2887 Sticky modules
2888 --------------
2889
2890 Module stickiness is introduced, in a similar fashion than on the Lmod
2891 project, to allow to glue modules to the loaded environment. A sticky
2892 module cannot be unloaded, unless if the unload action is forced or if
2893 the module reloads after being unloaded.
2894
2895 A modulefile is declared *sticky* by applying it the "sticky" tag with
2896 the "module-tag" modulefile command.
2897
2898 $ cat /path/to/modulefiles/foo/.modulerc
2899 #%Module4.7
2900 module-tag sticky foo/1.0
2901 $ ml
2902 Currently Loaded Modulefiles:
2903 1) foo/1.0
2904 $ ml -foo
2905 Unloading foo/1.0
2906 ERROR: Unload of sticky module 'foo/1.0' skipped
2907 $ ml
2908 Currently Loaded Modulefiles:
2909 1) foo/1.0
2910 $ ml --force -foo
2911 Unloading foo/1.0
2912 WARNING: Unload of sticky module 'foo/1.0' forced
2913 $ ml
2914 No Modulefiles Currently Loaded.
2915
2916 Modulefile can also be defined "super-sticky" by applying the
2917 corresponding module tag. *Super-sticky* module cannot be unloaded
2918 even if the unload action is forced. It can only be unloaded if the
2919 module reloads afterward.
2920
2921 $ cat /path/to/modulefiles/bar/.modulerc
2922 #%Module4.7
2923 module-tag super-sticky bar/1.0
2924 $ ml
2925 Currently Loaded Modulefiles:
2926 1) bar/1.0
2927 $ ml purge
2928 Unloading bar/1.0
2929 ERROR: Unload of super-sticky module 'bar/1.0' skipped
2930 $ ml purge -f
2931 Unloading bar/1.0
2932 ERROR: Unload of super-sticky module 'bar/1.0' skipped
2933 $ ml
2934 Currently Loaded Modulefiles:
2935 1) bar/1.0
2936
2937 Modulefiles targeted by a "sticky" or a "super-sticky" tag are colored
2938 on "avail" and "list" sub-command outputs to indicate such tag
2939 applies. If colored output is disabled a tag abbreviation is reported
2940 along module designation (respectively "S" and "sS").
2941
2942 In case the stickiness applies to the generic module name (and does
2943 not target a specific module version or version-set), one version of
2944 the sticky or super-sticky module can be swapped by another version of
2945 this same module:
2946
2947 $ cat /path/to/modulefiles/baz/.modulerc
2948 #%Module4.7
2949 module-tag sticky baz
2950 $ ml
2951 Currently Loaded Modulefiles:
2952 1) baz/2.0
2953 $ ml switch baz/1.0
2954 $ ml
2955 Currently Loaded Modulefiles:
2956 1) baz/1.0
2957
2958
2959 Explaining avail/list output
2960 ----------------------------
2961
2962 A *Key* section is added at the end of the "avail" and "list" sub-
2963 commands output to give hints on the meaning of the graphical
2964 rendition applied to elements or what the elements set in parentheses
2965 or chevrons along module name stand for.
2966
2967 $ ml av
2968 ------------------ /path/to/modulefiles ------------------
2969 foo/1.0 <oT> foo/2.0 foo/3.0
2970
2971 Key:
2972 loaded default-version sticky <oT>=othertag
2973 modulepath module-alias <module-tag>
2974
2975
2976 Configuring avail/list output
2977 -----------------------------
2978
2979 New configuration options are introduced to control what content to
2980 output in addition to modules names on the regular and terse output
2981 modes of the "avail" and "list" sub-commands.
2982
2983 These new configuration options named "avail_output",
2984 "avail_terse_output", "list_output" and "list_terse_output" can be
2985 updated using the "config" sub-command or set at installation time
2986 respectively with the "--with-avail-output", "--with-avail-terse-
2987 output", "--with-list-output" and "--with-list-terse-output" configure
2988 options.
2989
2990 The four options accept a colon separated list of elements as value.
2991 Accepted elements for the "avail"-related options are: "modulepath",
2992 "alias", "dirwsym", "sym", "tag" and "key". Accepted elements for the
2993 "list"-related options are: "header", "idx", "sym", "tag" and "key".
2994
2995 In the following example, default output configuration for the "avail"
2996 sub-command is checked then module tags and key section are removed to
2997 get a simpler output:
2998
2999 $ ml config avail_output
3000 Modules Release 4.7.0 (2021-02-19)
3001
3002 - Config. name ---------.- Value (set by if default overridden) ---------------
3003 avail_output modulepath:alias:dirwsym:sym:tag:key
3004 $ ml av
3005 ------------------ /path/to/modulefiles ------------------
3006 bar/1.0 bar/2.0 foo/1.0 foo/2.0 foo/2.2
3007
3008 Key:
3009 modulepath module-alias sticky
3010 default-version forbidden
3011 $ ml config avail_output modulepath:alias:dirwsym:sym
3012 $ ml av
3013 ------------------ /path/to/modulefiles ------------------
3014 bar/1.0 bar/2.0 foo/1.0 foo/2.0 foo/2.2
3015
3016 The "--output"/"-o" switches are added to define a specific output
3017 configuration for the duration of the associated command line. The
3018 following example shows how to limit the content reported on a module
3019 "list" to the loaded index and the symbolic versions in addition to
3020 the module names:
3021
3022 $ ml
3023 Currently Loaded Modulefiles:
3024 1) bar/1.0 2) foo/2.0
3025
3026 Key:
3027 default-version sticky
3028 $ ml -o idx:sym
3029 1) bar/1.0 2) foo/2.0
3030
3031 When the new configuration options or command line switches are set to
3032 an empty value, the module names are the sole information reported:
3033
3034 $ ml -t -o ""
3035 bar/1.0
3036 foo/2.0
3037
3038 In case the "modulepath" element is withdrawn from the "avail" sub-
3039 command output configuration, the available modules from all enabled
3040 modulepaths are reported as a single list:
3041
3042 $ ml av
3043 --------------- /path/to/other/modulefiles ---------------
3044 baz/1.0 baz/2.0
3045
3046 ------------------ /path/to/modulefiles ------------------
3047 bar/1.0 bar/2.0 foo/1.0 foo/2.0 foo/2.2
3048
3049 Key:
3050 modulepath module-alias sticky
3051 default-version forbidden
3052 $ ml av --output=alias:tag
3053 bar/1.0 baz/1.0 foo/1.0 foo/2.2
3054 bar/2.0 baz/2.0 foo/2.0
3055
3056 Note:
3057
3058 The "avail_report_dir_sym" and "avail_report_mfile_sym" locked
3059 configuration options have been removed. Their behaviors can now be
3060 obtained by respectively adding the "dirwsym" and "sym" elements to
3061 the "avail_output" or "avail_terse_output" configuration options.
3062
3063
3064 v4.6
3065 ====
3066
3067 This new version is backward-compatible with previous version 4
3068 releases. It fixes bugs but also introduces new functionalities that
3069 are described in this section. See the 4.6 release notes for a
3070 complete list of the changes between Modules v4.5 and v4.6.
3071
3072
3073 sh-to-mod sub-command
3074 ---------------------
3075
3076 The "sh-to-mod" sub-command is added to output as a modulefile content
3077 the environment changes done by the evaluation of a shell script
3078 passed as argument. "sh-to-mod" is especially useful for software
3079 providing a shell script for their enablement in shell session: it can
3080 convert these scripts into modulefiles.
3081
3082 Say for instance, a *foo* software has been installed and it provides
3083 a "foo-setup.sh" script to activate *foo* software in user
3084 environment:
3085
3086 $ cat /path/to/foo-1.2/foo-setup.sh
3087 #!/bin/sh
3088 export FOOENV="$1"
3089 export PATH=/path/to/foo-1.2/bin:$PATH
3090 alias foo='foobin -q -l'
3091
3092 Calling "module sh-to-mod" on this shell script outputs the
3093 environment changes it performs as a modulefile content:
3094
3095 $ module sh-to-mod sh /path/to/foo-1.2/foo-setup.sh arg1
3096 #%Module
3097 prepend-path PATH /path/to/foo-1.2/bin
3098 set-alias foo {foobin -q -l}
3099 setenv FOOENV arg1
3100
3101 Changes on environment variables, shell aliases, shell functions and
3102 current working directory are tracked. The following shells are
3103 supported: *sh*, *dash*, *csh*, *tcsh*, *bash*, *ksh*, *ksh93*, *zsh*
3104 and *fish*.
3105
3106 "sh-to-mod" acts as a full replacement for the standalone
3107 **createmodule.sh** and **createmodule.py** scripts. However those two
3108 scripts are currently still provided for compatibility purpose.
3109
3110
3111 source-sh modulefile command
3112 ----------------------------
3113
3114 The "source-sh" modulefile command is introduced to source environment
3115 changes done by the evaluation of a shell script passed as argument.
3116 With newly introduced "sh-to-mod" sub-command resulting environment
3117 changes done by script are output as modulefile commands. "source-sh"
3118 applies those modulefile commands as if they were directly written in
3119 loading modulefile.
3120
3121 "source-sh" is useful for software providing a shell script for their
3122 enablement. If you want to enable such software with **module** yet
3123 using shell script provided by software for this task, just write a
3124 modulefile using "source-sh" command to call the shell script.
3125
3126 Keeping the same example used to describe "sh-to-mod" sub-command:
3127 *foo* software provides a "foo-setup.sh" script for its activation.
3128 Create a modulefile "foo/1.2" that calls this script:
3129
3130 $ cat /path/to/modulefiles/foo/1.2
3131 #%Module4.6
3132 source-sh sh /path/to/foo-1.2/foo-setup.sh arg1
3133
3134 Displaying this modulefile indicates the environment changes done by
3135 script:
3136
3137 $ module display foo/1.2
3138 -------------------------------------------------------------------
3139 /path/to/modulefiles/foo/1.2:
3140
3141 prepend-path PATH /path/to/foo-1.2/bin
3142 set-alias foo {foobin -q -l}
3143 setenv FOOENV arg1
3144 -------------------------------------------------------------------
3145
3146 Loading the modulefile applies the environment changes seen above:
3147
3148 $ module load -v foo/1.2
3149 Loading foo/1.2
3150 $ echo $FOOENV
3151 arg1
3152 $ alias foo
3153 alias foo='foobin -q -l'
3154
3155 Track of these changes is kept in user environment to be able to undo
3156 them when modulefile is unloaded:
3157
3158 $ module unload -v foo/1.2
3159 Unloading foo/1.2
3160 $ echo $FOOENV
3161
3162 $ alias foo
3163 bash: alias: foo: not found
3164
3165 Changes on environment variables, shell aliases, shell functions and
3166 current working directory are tracked. The following shells are
3167 supported: *sh*, *dash*, *csh*, *tcsh*, *bash*, *ksh*, *ksh93*, *zsh*
3168 and *fish*.
3169
3170
3171 Querying user's name and groups membership
3172 ------------------------------------------
3173
3174 Two new sub-commands are introduced for the "module-info" modulefile
3175 command: "username" and "usergroups". They respectively fetch the name
3176 of the user currently running "modulecmd.tcl" or the name of all the
3177 groups this user is member of.
3178
3179 These two new modulefile commands can help to adapt code to specific
3180 users or groups. Like for instance to instantiate a modulefile for
3181 each group the user is member of:
3182
3183 $ cat /path/to/modulefiles/foo/.modulerc
3184 #%Module4.6
3185 foreach grp [module-info usergroups] {
3186 module-virtual foo/$grp .common
3187 }
3188 $ id -G -n
3189 grp1 grp2 grp3
3190 $ module avail
3191 --------------- /path/to/modulefiles ---------------
3192 foo/grp1 foo/grp2 foo/grp3
3193
3194 "username" and "usergroups" sub-commands of "module-info" modulefile
3195 command are only supported on Unix platform.
3196
3197
3198 Hiding modules
3199 --------------
3200
3201 The newly introduced "module-hide" modulefile command enables to
3202 dynamically hide modulefiles, module aliases or symbolic versions
3203 specified to it:
3204
3205 $ cat /path/to/modulefiles/bar/.modulerc
3206 #%Module4.6
3207 module-version bar/1.0 old
3208 # hide 'old' symbolic version
3209 module-hide bar/old
3210 # hide all version 2 and above
3211 module-hide bar@2:
3212 $ cat /path/to/modulefiles/.modulerc
3213 #%Module4.6
3214 # hide all versions of foo module
3215 module-hide foo
3216
3217 "module-hide" commands should be placed in module rc files and can
3218 leverage the Advanced module version specifiers syntax as shown in the
3219 above example.
3220
3221 Hidden modules are excluded from available module search or module
3222 selection unless query refers to hidden module by its exact name:
3223
3224 $ ml av
3225 --------------- /path/to/modulefiles ---------------
3226 bar/1.0 bar/2.0
3227 $ module load -v foo
3228 ERROR: Unable to locate a modulefile for 'foo'
3229 $ module load -v foo/1.0
3230 Loading foo/1.0
3231 $ module avail bar/old
3232 --------------- /path/to/modulefiles ---------------
3233 bar/1.0(old)
3234
3235 "module-hide" command accepts a "--soft" option to apply a lighter of
3236 hiding to modules:
3237
3238 $ cat /path/to/modulefiles/qux/.modulerc
3239 #%Module4.6
3240 # softly hide all qux modules
3241 module-hide --soft qux
3242
3243 The soft hiding mode enables to hide modules from full availability
3244 listing yet keeping the ability to select such module for load without
3245 having to use module exact name:
3246
3247 $ ml av
3248 --------------- /path/to/modulefiles ---------------
3249 bar/1.0 bar/2.0
3250 $ ml av qux
3251 --------------- /path/to/modulefiles ---------------
3252 qux/1.0 qux/2.0
3253 $ module load -v qux
3254 Loading qux/2.0
3255
3256 Alternatively, a "--hard" option can be set on "module-hide" command
3257 to ensure designated modules do not unveil even if referred by their
3258 exact name:
3259
3260 $ cat /path/to/modulefiles/qux/.modulerc
3261 #%Module4.6
3262 # softly hide all qux modules
3263 module-hide --soft qux
3264 # set highest version of qux hard hidden
3265 module-hide --hard qux/3.0
3266 $ ml av qux/3.0
3267 $ ml qux/3.0
3268 ERROR: Unable to locate a modulefile for 'qux/3.0'
3269
3270 Some users or groups can be set unaffected by hiding mechanism with
3271 the "--not-user" or "--not-group" options:
3272
3273 $ cat /path/to/modulefiles/quuz/.modulerc
3274 #%Module4.6
3275 # hiding does not apply to grp1 and grp2 groups
3276 module-hide --not-group {grp1 grp2} quuz
3277
3278 $ id --groups --name
3279 grp1 grp7
3280 $ ml av quuz
3281 --------------- /path/to/modulefiles ---------------
3282 quuz/1.0 quuz/2.0
3283 $ ml -v quuz
3284 Loading quuz/2.0
3285
3286 Hiding mechanism can also be set effective only before or after a
3287 given date time with the "--before" and "--after" options. Accepted
3288 date time format is "YYYY-MM-DD[THH:MM]".
3289
3290 $ cat /path/to/modulefiles/fum/.modulerc
3291 #%Module4.6
3292 # hide only before a given date
3293 module-hide --hard --before 2020-09-01T12:00 fum/1.0
3294 # hide only after a given date
3295 module-hide --hard --after 2020-09-01 fum/2.0
3296
3297 $ date
3298 Fri 04 Sep 2020 06:21:48 AM CEST
3299 $ ml av fum
3300 --------------- /path/to/modulefiles ---------------
3301 fum/1.0
3302
3303 Hidden modules can be included in available module searches if option
3304 "--all"/"-a" is set on "avail", "aliases", "whatis" or "search" sub-
3305 commands. Hard hidden modules are unaffected by this option and stay
3306 hidden.
3307
3308 $ ml av -a
3309 --------------- /path/to/modulefiles ---------------
3310 bar/1.0(old) foo/1.0 fum/1.0 quuz/2.0 qux/2.0
3311 bar/2.0 foo/2.0 quuz/1.0 qux/1.0
3312
3313
3314 Forbidding use of modules
3315 -------------------------
3316
3317 The "module-forbid" modulefile command is added to dynamically forbid
3318 the evaluation of modulefiles it specifies. When forbidden, a module
3319 cannot be loaded and an access error is returned when an attempt is
3320 made to evaluate it.
3321
3322 $ cat /path/to/modulefiles/foo/.modulerc
3323 #%Module4.6
3324 module-forbid foo@1:
3325 $ ml foo/1.0
3326 ERROR: Access to module 'foo/1.0' is denied
3327 $ ml
3328 No Modulefiles Currently Loaded.
3329
3330 "module-forbid" statements can be coupled with "module-hide"
3331 statements to hide modules in addition to forbid their use. "module-
3332 forbid" supports the "--not-user", "--not-group", "--before" and "--
3333 after" options to still allow some users or forbid modules before or
3334 after a given date time.
3335
3336 An additional error message can be defined with the "--message" option
3337 to guide for instance users when they try to evaluate a forbidden
3338 module:
3339
3340 $ cat /path/to/modulefiles/bar/.modulerc
3341 #%Module4.6
3342 module-forbid --message {Software bar/1.0 is decommissioned, please now use\
3343 bar/2.0} --after 2020-09-01 bar/1.0
3344 $ ml bar/1.0
3345 ERROR: Access to module 'bar/1.0' is denied
3346 Software bar/1.0 is decommissioned, please now use bar/2.0
3347
3348 When an evaluated module will soon be forbidden, a message is returned
3349 to the user to warn them of the near limit. An additional warning
3350 message can also be defined here with the "--nearly-message" option to
3351 guide users.
3352
3353 $ cat /path/to/modulefiles/qux/.modulerc
3354 #%Module4.6
3355 module-forbid --nearly-message {Version 1.0 will soon expire, please now use\
3356 version 2.0} --after 2020-09-15 qux/1.0
3357 $ date
3358 Tue 08 Sep 2020 06:49:43 AM CEST
3359 $ ml qux/1.0
3360 Loading qux/1.0
3361 WARNING: Access to module will be denied starting '2020-09-15'
3362 Version 1.0 will soon expire, please now use version 2.0
3363
3364 The range of time the *nearly forbidden* warning appears can be
3365 controlled with the "nearly_forbidden_days" configuration option,
3366 whose value equals to the number of days prior the module starts to be
3367 forbidden. This configuration is set to "14" (days) by default and
3368 this value can be controlled at "configure" time with "--with-nearly-
3369 forbidden-days" option. When the "nearly_forbidden_days" configuration
3370 is set through the "config" sub-command, the
3371 "MODULES_NEARLY_FORBIDDEN_DAYS" environment variable is set.
3372
3373
3374 Tracing module execution
3375 ------------------------
3376
3377 The "trace" verbosity is introduced between the "verbose" and "debug"
3378 levels to report details on module searches, resolutions, selections
3379 and evaluations. Trace mode can be enabled by setting the "verbosity"
3380 config to the "trace" value or by using the "-T"/"--trace" command-
3381 line switches.
3382
3383 To specifically render trace messages, the "tr" key is added to the
3384 color palette with a default value of "2" (decreased intensity).
3385
3386 $ ml -T foo
3387 Evaluate modulerc: '/path/to/modulefiles/.modulerc'
3388 Get modules: {foo} matching 'foo' in '/path/to/modulefiles'
3389 Resolve: 'foo' into 'bar'
3390 Get modules: {bar bar/1.0} matching 'bar' in '/path/to/modulefiles'
3391 Select module: 'bar/1.0' (/path/to/modulefiles/bar/1.0) matching 'bar/1.0'
3392
3393 Loading bar/1.0
3394 Evaluate modulefile: '/path/to/modulefiles/bar/1.0' as 'bar/1.0'
3395
3396
3397 v4.5
3398 ====
3399
3400 This new version is backward-compatible with previous version 4
3401 releases. It fixes bugs but also introduces new functionalities that
3402 are described in this section. See the 4.5 release notes for a
3403 complete list of the changes between Modules v4.4 and v4.5.
3404
3405
3406 ml command
3407 ----------
3408
3409 The "ml" command is added to Modules. "ml" is a frontend to the
3410 "module" command that reduces the number of characters to type to
3411 trigger module actions.
3412
3413 With no argument provided "ml" is equivalent to "module list", "ml
3414 foo" corresponds to "module load foo" and "ml -foo" means "module
3415 unload foo":
3416
3417 $ ml foo
3418 $ ml
3419 Currently Loaded Modulefiles:
3420 1) foo/2
3421 $ ml -foo
3422 $ ml
3423 No Modulefiles Currently Loaded.
3424
3425 Multiple modules to either load or unload can be combined on a single
3426 command. The unloads are first processed then the loads.
3427
3428 "ml" accepts all command-line switches and sub-commands accepted by
3429 "module" command:
3430
3431 $ ml avail -t foo
3432 foo/1
3433 foo/2
3434
3435 This handy interface has been originally developed by the Lmod
3436 project. Having this command line interface also supported on Modules
3437 helps to provide a similar user experience whatever the module
3438 implementation used.
3439
3440
3441 JSON format output
3442 ------------------
3443
3444 The "-j" and "--json" command line switches are added for the "avail",
3445 "list", "savelist", "whatis" and "search" module sub-commands. When
3446 set, the output result of these sub-commands is rendered in JSON
3447 format:
3448
3449 $ module avail --json bar | python -mjson.tool
3450 {
3451 "/path/to/modulefiles": {
3452 "bar/2.3": {
3453 "name": "bar/2.3",
3454 "pathname": "/path/to/modulefiles/bar/2.3",
3455 "symbols": [
3456 "default"
3457 ],
3458 "type": "modulefile"
3459 },
3460 "bar/3.4": {
3461 "name": "bar/3.4",
3462 "pathname": "/path/to/modulefiles/bar/3.4",
3463 "symbols": [],
3464 "type": "modulefile"
3465 }
3466 }
3467 }
3468 $ ml whatis -j foo/1.2.3 | python -mjson.tool
3469 {
3470 "/path/to/modulefiles": {
3471 "foo/1.2.3": {
3472 "name": "foo/1.2.3",
3473 "whatis": [
3474 "The foo/1.2.3 modulefile"
3475 ]
3476 }
3477 }
3478 }
3479
3480
3481 Improved Windows support
3482 ------------------------
3483
3484 A new option to the "./configure" script named "--enable-windows-
3485 support" is introduced to install additional files relative to the
3486 enablement of Modules on the Windows platform. When set, this option
3487 installs "module.cmd", "ml.cmd" and "envml.cmd" scripts in "bindir"
3488 and initialization script "cmd.cmd" in "initdir". With these four
3489 files the Modules installation may be used from either a Unix or a
3490 Windows platform.
3491
3492 "module.cmd", "ml.cmd" and "envml.cmd" scripts respectively provide
3493 the "module", "ml" and "envml" commands for Windows "cmd" terminal
3494 shell, relying on "modulecmd.tcl" script which was already able to
3495 produce shell code for this Windows shell. Initialization script
3496 "cmd.cmd" adds the directory of "module.cmd", "ml.cmd" and "envml.cmd"
3497 to "PATH".
3498
3499 These Windows-specific files are relocatable: "module.cmd", "ml.cmd"
3500 and "envml.cmd" scripts expect to find initialization script "cmd.cmd"
3501 in the "init" directory next to them (to setup Modules-specific
3502 variables in current environment) and "cmd.cmd" expects
3503 "modulecmd.tcl" to be found in "libexec" directory and the 3 commands
3504 in "bin" directory next to it.
3505
3506 Starting from this "4.5" release a distribution zipball is published
3507 to install Modules on Windows. This zip archive ships an install and
3508 an uninstall scripts ("INSTALL.bat" and "UNINSTALL.bat"). The zipball
3509 can be built locally from Modules sources by running "make dist-win".
3510
3511 The Installing Modules on Windows document describes how to install
3512 Modules on Windows from the distribution zipball.
3513
3514
3515 Error stack trace
3516 -----------------
3517
3518 Error messages will now embed a stack trace for unknown errors to help
3519 localize the root cause of issues. This change applies to modulefile
3520 evaluation:
3521
3522 Loading foo/1.2
3523 Module ERROR: add-path cannot handle path equals to separator string
3524 while executing
3525 "append-path PATH :"
3526 (file "/path/to/modulefiles/foo/1.2" line 24)
3527 Please contact <root@localhost>
3528
3529 A stack trace is also returned when an unknown error occurs in
3530 "modulecmd.tcl" script, which facilitates issue report and analysis:
3531
3532 $ module load bar
3533 ERROR: invalid command name "badcommand"
3534 while executing
3535 "badcommand"
3536 (procedure "module" line 14)
3537 invoked from within
3538 "module load bar"
3539 ("eval" body line 1)
3540 invoked from within
3541 "eval $execcmdlist"
3542 Please report this issue at https://github.com/envmodules/modules/issues
3543
3544
3545 Automatic default and latest symbolic versions
3546 ----------------------------------------------
3547
3548 When the implicit default mechanism and the Advanced module version
3549 specifiers are both enabled, a "default" and a "latest" symbolic
3550 versions are automatically defined for each module name.
3551
3552 This new feature gives the ability to select the highest version
3553 available for a module, without knowing beforehand this version name:
3554
3555 $ module load -v foo@latest
3556 Loading foo/1.10
3557
3558 The symbolic versions are automatically defined unless a symbolic
3559 version, an alias or a regular module version already exists for these
3560 "default" or "latest" version names.
3561
3562
3563 v4.4
3564 ====
3565
3566 This new version is backward-compatible with previous version 4
3567 releases. It fixes bugs but also introduces new functionalities that
3568 are described in this section. See the 4.4 release notes for a
3569 complete list of the changes between Modules v4.3 and v4.4.
3570
3571 Warning:
3572
3573 Modules configuration option handling has been reworked internally
3574 to provide a unified way for all options to get initialized,
3575 retrieved or set. Existing site-specific configuration script should
3576 be reviewed to make use of the new "getConf", "setConf", "unsetConf"
3577 and "lappendConf" procedures to manipulate configuration options.
3578
3579
3580 Specify modules in a case insensitive manner
3581 --------------------------------------------
3582
3583 The ability to match module name in a case insensitive manner has been
3584 added. This feature can be enabled at different level with the
3585 following values set to the "icase" configuration option:
3586
3587 * "never": a case sensitive match is applied in any cases
3588
3589 * "search": a case insensitive match is applied to the "avail",
3590 "whatis" and "paths" sub-commands
3591
3592 * "always": a case insensitive match is applied to search contexts and
3593 also to the other module sub-commands and modulefile Tcl commands
3594 for the module specification they receive as argument.
3595
3596 It can help for instance to load a module without knowing the case
3597 used to name its relative modulefile:
3598
3599 $ module config icase always
3600 $ module load -v mysoftware
3601 Loading MySoftware/1.0
3602
3603 Insensitive case match activation can be controlled at configure time
3604 with the "--with-icase" option, which could be passed any of the above
3605 activation levels. This option could be superseded with the
3606 "MODULES_ICASE" environment variable, which could be set through the
3607 **config** sub-command with the "icase" option. Command-line switch
3608 **--icase** supersedes in turns any other icase configurations. When
3609 this command-line switch is passed, "icase" mode equals "always".
3610
3611
3612 Extended default
3613 ----------------
3614
3615 The extended default mechanism has been introduced to help selecting a
3616 module when only the first numbers in its version are specified.
3617 Starting portion of the version, part separated from the rest of the
3618 version string by a "." character, could be used to refer to a more
3619 precise version number.
3620
3621 This mechanism is activated through the new configuration option
3622 "extended_default". It enables to refer to a module named "foo/1.2.3"
3623 as "foo/1.2" or "foo/1":
3624
3625 $ module config extended_default 1
3626 $ module load -v foo/1
3627 Loading foo/1.2.3
3628
3629 When multiple versions match partial version specified and only one
3630 module should be selected, the default version (whether implicitly or
3631 explicitly defined) among matches is returned. The following example
3632 shows that "foo/1.1.1", the *foo* module default version, is selected
3633 when it matches query. Elsewhere the highest version (also called the
3634 latest version or the implicit default) among matching modules is
3635 returned:
3636
3637 $ module av foo
3638 --------------- /path/to/modulefiles ---------------
3639 foo/1.1.1(default) foo/1.2.1 foo/1.10
3640 foo/1.1.10 foo/1.2.3
3641 $ module load -v foo/1.1
3642 Loading foo/1.1.1
3643 $ module purge
3644 $ module load -v foo/1.2
3645 Loading foo/1.2.3
3646 $ module purge
3647 $ module load -v foo/1
3648 Loading foo/1.1.1
3649
3650 In case "implicit_default" option is disabled and no explicit default
3651 is found among matches, an error is returned:
3652
3653 $ module config implicit_default 0
3654 $ module load -v foo/1.2
3655 ERROR: No default version defined for 'foo/1.2'
3656
3657 When it is enabled, extended default applies everywhere a module could
3658 be specified, which means it could be used with any module sub-command
3659 or any modulefile Tcl command receiving a module specification as
3660 argument. It may help for instance to declare dependencies between
3661 modules:
3662
3663 $ module show bar/3
3664 ----------------------------------------------------------
3665 /path/to/modulefiles/bar/3.4:
3666
3667 prereq foo/1.2
3668 ----------------------------------------------------------
3669 $ module load --auto bar/3
3670 Loading bar/3.4
3671 Loading requirement: foo/1.2.3
3672
3673 Extended default activation can be controlled at configure time with
3674 the "--enable-extended-default" option. This option could be
3675 superseded with the "MODULES_EXTENDED_DEFAULT" environment variable,
3676 which could be set through the **config** sub-command with the
3677 "extended_default" option.
3678
3679
3680 Advanced module version specifiers
3681 ----------------------------------
3682
3683 The ability to specify finer constraints on module version has been
3684 added to Modules. It enables to filter the module selection to a given
3685 version list or range by specifying after the module name a version
3686 constraint prefixed by the "@" character.
3687
3688 This new feature leverages the version specifier syntax of the Spack
3689 package manager as this syntax covers all the needs for a fine-grained
3690 selection of module versions. It copes very well with command-line
3691 typing, by avoiding characters having a special meaning on shells.
3692 Moreover the users of Spack that also are users of Modules may already
3693 be familiar with this syntax.
3694
3695 The mechanism introduced here is called *advanced module version
3696 specifier* and it can be activated through the new configuration
3697 option "advanced_version_spec". Constraints can be expressed to refine
3698 the selection of module version to:
3699
3700 * a single version with the "@version" syntax, for instance
3701 "[email protected]" syntax will select module "foo/1.2.3"
3702
3703 * a list of versions with the "@version1,version2,..." syntax, for
3704 instance "[email protected],1.10" will match modules "foo/1.2.3" and
3705 "foo/1.10"
3706
3707 * a range of versions with the "@version1:", "@:version2" and
3708 "@version1:version2" syntaxes, for instance "[email protected]:" will select
3709 all versions of module "foo" greater than or equal to "1.2",
3710 "foo@:1.3" will select all versions less than or equal to "1.3" and
3711 "[email protected]:1.3" matches all versions between "1.2" and "1.3" including
3712 "1.2" and "1.3" versions
3713
3714 This new feature enables for instance to list available versions of
3715 module "foo" higher or equal to "1.2":
3716
3717 $ module config advanced_version_spec 1
3718 $ module av foo
3719 --------------- /path/to/modulefiles ---------------
3720 foo/1.1.1(default) foo/1.2.1 foo/1.10
3721 foo/1.1.10 foo/1.2.3
3722 $ module av [email protected]:
3723 --------------- /path/to/modulefiles ---------------
3724 foo/1.2.1 foo/1.2.3 foo/1.10
3725
3726 Then choose to load for instance a version higher than or equal to
3727 "1.2" and less than or equal to "1.3". Default version is selected if
3728 it corresponds to a version included in the range, elsewhere the
3729 highest version (also called latest version or implicit default) is
3730 selected:
3731
3732 $ module load -v [email protected]:1.3
3733 Loading foo/1.2.3
3734
3735 In case "implicit_default" option is disabled and no explicit default
3736 is found among version specifier matches, an error is returned:
3737
3738 $ module config implicit_default 0
3739 $ module load -v [email protected]:1.3
3740 ERROR: No default version defined for '[email protected]:1.3'
3741
3742 When advanced module version specifier is enabled, it applies
3743 everywhere a module could be specified, which means it could be used
3744 with any module sub-command or any modulefile Tcl command receiving a
3745 module specification as argument. It may help for instance to declare
3746 smoother dependencies between modules:
3747
3748 $ module show bar@:2
3749 ----------------------------------------------------------
3750 /path/to/modulefiles/bar/2.3:
3751
3752 prereq [email protected],1.2.1
3753 ----------------------------------------------------------
3754 $ module load --auto bar@:2
3755 Loading bar/2.3
3756 Loading requirement: foo/1.2.1
3757
3758 Advanced specification of single version or list of versions may
3759 benefit from the activation of the Extended default mechanism (range
3760 of versions natively handles abbreviated versions):
3761
3762 $ module config extended_default 1
3763 $ module load -v [email protected]
3764 Loading foo/1.2.3
3765 $ module unload -v foo @1.2,1.5
3766 Unloading foo/1.2.3
3767
3768 Advanced module version specifier activation can be controlled at
3769 configure time with the "--enable-advanced-version-spec" option. This
3770 option could be superseded with the "MODULES_ADVANCED_VERSION_SPEC"
3771 environment variable, which could be set through the **config** sub-
3772 command with the "advanced_version_spec" option.
3773
3774
3775 v4.3
3776 ====
3777
3778 This new version is backward-compatible with previous version 4
3779 releases. It fixes bugs but also introduces new functionalities that
3780 are described in this section. See the 4.3 release notes for a
3781 complete list of the changes between Modules v4.2 and v4.3.
3782
3783
3784 Modulepath rc file
3785 ------------------
3786
3787 A ".modulerc" file found at the root of an enabled modulepath
3788 directory is now evaluated when modulepath is walked through to locate
3789 modulefiles. This modulepath rc file gives for instance the ability to
3790 define module alias whose name does not correspond to any module
3791 directory in this modulepath. Thus this kind of module alias would not
3792 be found unless if it is defined at the modulepath global scope.
3793
3794
3795 Further I/O operations optimization
3796 -----------------------------------
3797
3798 Additional work has been performed to save a significant number of
3799 filesystem I/O operations made to search and evaluate modulefiles.
3800
3801 When fully read, the content of a modulefile is now cached in memory
3802 to avoid new I/O operations in case this modulefile should be read one
3803 more time during the same module command evaluation.
3804
3805 Except for "path", "paths", "list", "avail" and "aliases" module
3806 commands always fully read a modulefile whether its full content is
3807 needed or just its header to verify its validity. This way modulefiles
3808 are only read once on commands that first check modulefile validity
3809 then read again valid files to get their full content.
3810
3811 Last but not least, Modules Tcl extension library is introduced to
3812 extend the Tcl language in order to provide more optimized I/O
3813 commands to read a file or a directory content than native Tcl
3814 commands do. This library is built and enabled in "modulecmd.tcl"
3815 script with "--enable-libtclenvmodules" configure argument (it is
3816 enabled by default). As this library is written in C, it must be
3817 compiled and "--with-tcl" or "--with-tclinclude" configure arguments
3818 may be used to indicate where to find Tcl development files.
3819
3820 Modules Tcl extension library greatly reduces the number of filesystem
3821 I/O operations by removing unneeded "ioctl", "fcntl" and "lstat"
3822 system calls done (by Tcl "open" command) to read each modulefile.
3823 Directory content read is also improved by fetching hidden and regular
3824 files in one pass. Moreover ".modulerc" and ".version" read access is
3825 tested only if these files are found in the directory.
3826
3827
3828 Colored output
3829 --------------
3830
3831 The ability to graphically enhance some part of the produced output
3832 has been added to improve readability. Among others, error, warning
3833 and info message prefixes can be colored as well as modulepath, module
3834 alias and symbolic version.
3835
3836 Color mode can be set to "never", "auto" or "always". When color mode
3837 is set to "auto", output is colored only if the standard error output
3838 channel is attached to a terminal.
3839
3840 Default color mode could be controlled at configure time with the "--
3841 enable-color" and the "--disable-color" option, which respectively
3842 correspond to the "auto" and "never" color mode. This default mode
3843 could be superseded with the "CLICOLOR", "CLICOLOR_FORCE" and
3844 "MODULES_COLOR" environment variables and the "--color" command-line
3845 switch.
3846
3847 Color to apply to each element can be controlled with the
3848 "MODULES_COLORS" environment variable or the "--with-dark-background-
3849 colors" and "--with-light-background-colors" configure options. These
3850 variable and options take as value a colon-separated list in the same
3851 fashion "LS_COLORS" does. In this list, output item that should be
3852 highlighted is designated by a key which is associated to a Select
3853 Graphic Rendition (SGR) code.
3854
3855 The "MODULES_TERM_BACKGROUND" environment variable and the "--with-
3856 terminal-background" configure option help Modules to determine if the
3857 color set for dark background or the color set for light background
3858 should be used to color output in case no specific color set is
3859 defined with the "MODULES_COLORS" variable.
3860
3861 Output items able to be colorized and their relative key are:
3862 highlighted element ("hi"), debug information ("db"), tag separator
3863 ("se"); Error ("er"), warning ("wa"), module error ("me") and info
3864 ("in") message prefixes; Modulepath ("mp"), directory ("di"), module
3865 alias ("al"), module symbolic version ("sy"), module "default" version
3866 ("de") and modulefile command ("cm").
3867
3868 For instance the default color set for a terminal with dark background
3869 is defined to:
3870
3871 hi=1:db=2:se=2:er=91:wa=93:me=95:in=94:mp=1;94:di=94:al=96:sy=95:de=4:cm=92
3872
3873 When colored output is enabled and a specific graphical rendition is
3874 defined for module *default* version, the "default" symbol is omitted
3875 and instead the defined graphical rendition is applied to the relative
3876 modulefile. When colored output is enabled and a specific graphical
3877 rendition is defined for module alias, the "@" symbol is omitted.
3878
3879 "CLICOLOR" and "CLICOLOR_FORCE" environment variables are also honored
3880 to define color mode. The "never" mode is set if "CLICOLOR" equals to
3881 "0". If "CLICOLOR" is set to another value, it corresponds to the
3882 "auto" mode. The "always" mode is set if "CLICOLOR_FORCE" is set to a
3883 value different than "0". Color mode set with these two variables is
3884 superseded by mode set with "MODULES_COLOR" environment variable.
3885
3886
3887 Configure modulecmd with config sub-command
3888 -------------------------------------------
3889
3890 The **config** sub-command has been added to "module" to help getting
3891 or setting the **modulecmd.tcl** options. With no additional command-
3892 line argument, this sub-command reports the current value of all
3893 existing options with a mention to indicate if this value has been
3894 overridden from a command-line switch or from an environment variable.
3895
3896 See the description of this sub-command in the module man page for a
3897 complete reference on existing configuration options.
3898
3899 Most of the options can be altered by passing the option name and a
3900 value to the sub-command. Setting an option by this mean overrides its
3901 default value, set at installation time in **modulecmd.tcl** script,
3902 by defining the environment variable which supersedes this default.:
3903
3904 $ module config auto_handling 1
3905 $ module config auto_handling
3906 Modules Release 4.3.0 (2019-07-26)
3907
3908 - Config. name ---------.- Value (set by if default overridden) ---------------
3909 auto_handling 1 (env-var)
3910
3911 Setting options with "module config" could be done in the Modules
3912 initialization RC file to change default value of options when
3913 "module" command is initialized.
3914
3915 When command-line switch "--reset" and an option name is passed to the
3916 **config** sub-command, it restores default value for configuration
3917 option by unsetting related environment variable.
3918
3919 With command-line switch "--dump-state", the **config** sub-command
3920 reports, in addition to currently set options, the current state of
3921 **modulecmd.tcl** script and Modules-related environment variables.
3922 Providing the output of the "module config --dump-state" command when
3923 submitting an issue to the Modules project will help to analyze the
3924 situation.
3925
3926
3927 Control module command verbosity
3928 --------------------------------
3929
3930 The ability to control message verbosity has been added so "module"
3931 command can be configured whether it should display more or less
3932 information. Available verbosity levels from the least to the most
3933 verbose are:
3934
3935 * "silent": turn off error, warning and informational messages but
3936 does not affect module command output result.
3937
3938 * "concise": enable error and warning messages but disable
3939 informational messages.
3940
3941 * "normal": turn on informational messages, like a report of the
3942 additional module evaluations triggered by loading or unloading
3943 modules, aborted evaluation issues or a report of each module
3944 evaluation occurring during a **restore** or **source** sub-
3945 commands.
3946
3947 * "verbose": add additional informational messages, like a systematic
3948 report of the loading or unloading module evaluations.
3949
3950 * "debug": print debugging messages about module command execution.
3951
3952 Default verbosity level can be controlled at configure time with the "
3953 --with-verbosity" option, which could be passed any of the above level
3954 names. This default verbosity level could be superseded with the
3955 "MODULES_VERBOSITY" environment variable, which could be set through
3956 the **config** sub-command with the "verbosity" option. Command-line
3957 switches **--silent**, **--verbose** and **--debug** supersede in
3958 turns any other verbosity configuration to respectively set module
3959 command silent, verbose or in debug mode.
3960
3961
3962 Other new sub-commands, command-line switches and environment variables
3963 -----------------------------------------------------------------------
3964
3965 * The **avail** sub-command gets two new command-line switches:
3966 **--indepth** and **--no-indepth**. These options control whether
3967 search results should recursively include or not modulefiles from
3968 directories matching search query. Shell completion scripts have
3969 been updated to complete available modulefiles in the no in depth
3970 mode.
3971
3972 * The **MODULES_AVAIL_INDEPTH** environment variable defines if the
3973 **avail** sub-command should include or exclude by default the
3974 modulefiles from directories matching search query. Its value is
3975 superseded by the use of the **--indepth** and **--no-indepth**
3976 command-line switches.
3977
3978 * The **clear** sub-command, which was available on Modules version
3979 3.2, has been reintroduced. This sub-command resets the Modules
3980 runtime information but does not apply further changes to the
3981 environment at all. This sub-command now leverages the **--force**
3982 command-line switch to skip its confirmation dialog.
3983
3984 * The **MODULES_SITECONFIG** environment variable defines an
3985 additional siteconfig script which is loaded if it exists after the
3986 siteconfig script configured at build time in "modulecmd.tcl". This
3987 ability is enabled by default and could be disabled with configure
3988 option "--with-locked-configs=extra_siteconfig".
3989
3990 * The **MODULES_UNLOAD_MATCH_ORDER** environment variable sets whether
3991 the firstly or the lastly loaded module should be selected for
3992 unload when multiple loaded modules match unload request. Configure
3993 option "--with-unload-match-order" defines this setting which can be
3994 superseded by the environment variable. By default, lastly loaded
3995 module is selected and it is recommended to keep this behavior when
3996 used modulefiles express dependencies between each other.
3997
3998 * The **MODULES_IMPLICIT_DEFAULT** environment variable sets whether
3999 an implicit default version should be defined for modules with no
4000 default version explicitly defined. When enabled, which is the
4001 default behavior, a module version is automatically selected (latest
4002 one) when the generic name of the module is passed. When implicit
4003 default is disabled and no default version is explicitly defined for
4004 a module, the name of this module to evaluate should be fully
4005 qualified elsewhere an error is returned. Configure option "--
4006 enable-implicit-default" defines this setting which can be
4007 superseded by the environment variable. This superseding mechanism
4008 can be disabled with configure option "--with-locked-
4009 configs=implicit_default".
4010
4011 * The **MODULES_SEARCH_MATCH** environment variable defines the
4012 matching style to perform when searching for available modules. With
4013 **starts_with** value, modules whose name begins by search query
4014 string are returned. When search match style is set to **contains**,
4015 modules returned are those whose fully qualified name contains
4016 search query string. Configure option "--with-search-match" defines
4017 this setting which can be superseded by the environment variable,
4018 which in turns can be superseded by the **--starts-with** and
4019 **--contains** command-line switches of **avail** module sub-
4020 command.
4021
4022 * The **MODULES_SET_SHELL_STARTUP** environment variable controls
4023 whether or not shell startup file should be set to ensure "module"
4024 command is defined once shell has been initialized. When enabled,
4025 the "ENV" and "BASH_ENV" environment variables are set, when
4026 "module" function is defined, to the Modules bourne shell
4027 initialization script. Configure options "--enable-set-shell-
4028 startup" and "--disable-set-shell-startup" define this setting which
4029 can be superseded by the environment variable.
4030
4031 * When initializing the "module" command in a shell session,
4032 initialization configuration files stored in the defined
4033 configuration directory are taken into account if present instead of
4034 the configuration files stored in the initialization script
4035 directory. When they are stored in the configuration directory,
4036 these configuration files are named "initrc" and "modulespath"
4037 instead of respectively "modulerc" and ".modulespath". The location
4038 of the installation of those files can be controlled with configure
4039 option "--with-initconf-in", which accepts "etcdir" and "initdir"
4040 values.
4041
4042 * The **MODULES_WA_277** environment variable helps to define an
4043 alternative "module" alias on Tcsh shell when set to *1*. It
4044 workarounds an issue on Tcsh history mechanism occurring with
4045 default "module" command alias: erroneous history entries are
4046 recorded each time the "module" command is called. However the
4047 alternative definition of the module alias weakens shell evaluation
4048 of the code produced by modulefiles. Characters with special meaning
4049 for Tcsh shell (like *{* and *}*) may not be used anymore in shell
4050 alias definition elsewhere the evaluation of the code produced by
4051 modulefiles will return a syntax error.
4052
4053
4054 v4.2
4055 ====
4056
4057 This new version is backward-compatible with previous version 4
4058 releases. It fixes bugs but also introduces new functionalities that
4059 are described in this section. See the 4.2 release notes for a
4060 complete list of the changes between Modules v4.1 and v4.2.
4061
4062
4063 Modulefile conflict constraints consistency
4064 -------------------------------------------
4065
4066 With the **conflict** modulefile command, a given modulefile can list
4067 the other modulefiles it conflicts with. To load this modulefile, the
4068 modulefiles it conflicts with cannot be loaded.
4069
4070 This constraint was until now satisfied when loading the modulefile
4071 declaring the **conflict** but it vanished as soon as this modulefile
4072 was loaded. In the following example "a" modulefile declares a
4073 conflict with "b":
4074
4075 $ module load b a
4076 WARNING: a cannot be loaded due to a conflict.
4077 HINT: Might try "module unload b" first.
4078 $ module list
4079 Currently Loaded Modulefiles:
4080 1) b
4081 $ module purge
4082 $ module load a b
4083 $ module list
4084 Currently Loaded Modulefiles:
4085 1) a 2) b
4086
4087 Consistency of the declared **conflict** is now ensured to satisfy
4088 this constraint even after the load of the modulefile declaring it.
4089 This is achieved by keeping track of the conflict constraints of the
4090 loaded modulefiles in an environment variable called
4091 "MODULES_LMCONFLICT":
4092
4093 $ module load a b
4094 ERROR: WARNING: b cannot be loaded due to a conflict.
4095 HINT: Might try "module unload a" first.
4096 $ module list
4097 Currently Loaded Modulefiles:
4098 1) a
4099
4100 An environment variable is used to keep track of this conflict
4101 information to proceed the same way than used to keep track of the
4102 loaded modulefiles with the "LOADEDMODULES" environment variable.
4103
4104 In case a conflict constraint toward a modulefile is set by an already
4105 loaded modulefile, loading the conflicting modulefile will lead to a
4106 load evaluation attempt in order for this modulefile to get the chance
4107 to solve the constraint violation. If at the end of the load
4108 evaluation, the conflict has not been solved, modulefile load will be
4109 discarded.
4110
4111 Warning:
4112
4113 On versions "4.2.0" and "4.2.1", a conflict constraint set by an
4114 already loaded modulefile forbade the load of the conflicting
4115 modulefile. This has been changed starting version "4.2.2" to better
4116 cope with behaviors of previous Modules version: an evaluation
4117 attempt of the conflicting modulefile is made to give it the
4118 opportunity to solve this conflict by using **module unload**
4119 modulefile command.
4120
4121
4122 Modulefile prereq constraints consistency
4123 -----------------------------------------
4124
4125 With the **prereq** modulefile command, a given modulefile can list
4126 the other modulefiles it pre-requires. To load this modulefile, the
4127 modulefiles it pre-requires must be loaded prior its own load.
4128
4129 This constraint was until now satisfied when loading the modulefile
4130 declaring the **prereq** but, as for the declared **conflict**, it
4131 vanished as soon as this modulefile was loaded. In the following
4132 example "c" modulefile declares a prereq on "a":
4133
4134 $ module load c
4135 WARNING: c cannot be loaded due to missing prereq.
4136 HINT: the following module must be loaded first: a
4137 $ module list
4138 No Modulefiles Currently Loaded.
4139 $ module load a c
4140 $ module list
4141 Currently Loaded Modulefiles:
4142 1) a 2) c
4143 $ module unload a
4144 $ module list
4145 Currently Loaded Modulefiles:
4146 1) c
4147
4148 Consistency of the declared **prereq** is now ensured to satisfy this
4149 constraint even after the load of the modulefile declaring it. This is
4150 achieved, like for the conflict consistency, by keeping track of the
4151 prereq constraints of the loaded modulefiles in an environment
4152 variable called "MODULES_LMPREREQ":
4153
4154 $ module load a c
4155 $ module list
4156 Currently Loaded Modulefiles:
4157 1) a 2) c
4158 $ module unload a
4159 ERROR: WARNING: a cannot be unloaded due to a prereq.
4160 HINT: Might try "module unload c" first.
4161 $ module list
4162 Currently Loaded Modulefiles:
4163 1) a 2) c
4164
4165
4166 By-passing module defined constraints
4167 -------------------------------------
4168
4169 The ability to by-pass a **conflict** or a **prereq** constraint
4170 defined by modulefiles is introduced with the "--force" command line
4171 switch ("-f" for short notation) for the **load**, **unload** and
4172 **switch** sub-commands.
4173
4174 With this new command line switch, a given modulefile is loaded even
4175 if it conflicts with other loaded modulefiles or even if the
4176 modulefiles it pre-requires are not loaded. Some example reusing the
4177 same modulefiles "a", "b" and "c" than above:
4178
4179 $ module load b
4180 $ module load --force a
4181 WARNING: a conflicts with b
4182 $ module list
4183 Currently Loaded Modulefiles:
4184 1) b 2) a
4185 $ module purge
4186 $ module load --force c
4187 WARNING: c requires a loaded
4188 $ module list
4189 Currently Loaded Modulefiles:
4190 1) c
4191
4192 "--force" also enables to unload a modulefile required by another
4193 loaded modulefiles:
4194
4195 $ module load a c
4196 $ module list
4197 Currently Loaded Modulefiles:
4198 1) a 2) c
4199 $ module unload --force a
4200 WARNING: a is required by c
4201 $ module list
4202 Currently Loaded Modulefiles:
4203 1) c
4204
4205 In a situation where some of the loaded modulefiles have unsatisfied
4206 constraints corresponding to the **prereq** and **conflict** they
4207 declare, the **save** and **reload** sub-commands do not perform and
4208 return an error.
4209
4210
4211 Automated module handling mode
4212 ------------------------------
4213
4214 An automatic management of the dependencies between modulefiles has
4215 been added and it is called *automated module handling mode*. This new
4216 mode consists in additional actions triggered when loading or
4217 unloading a modulefile to satisfy the constraints it declares.
4218
4219 When loading a modulefile, following actions are triggered:
4220
4221 * Requirement Load (ReqLo): load of the modulefiles declared as a
4222 **prereq** of the loading modulefile.
4223
4224 * Dependent Reload (DepRe): reload of the modulefiles declaring a
4225 **prereq** onto loaded modulefile or declaring a **prereq** onto a
4226 modulefile part of this reloading batch.
4227
4228 When unloading a modulefile, following actions are triggered:
4229
4230 * Dependent Unload (DepUn): unload of the modulefiles declaring a non-
4231 optional **prereq** onto unloaded modulefile or declaring a non-
4232 optional **prereq** onto a modulefile part of this unloading batch.
4233 A **prereq** modulefile is considered optional if the **prereq**
4234 definition order is made of multiple modulefiles and at least one
4235 alternative modulefile is loaded.
4236
4237 * Useless Requirement Unload (UReqUn): unload of the **prereq**
4238 modulefiles that have been automatically loaded for either the
4239 unloaded modulefile, an unloaded dependent modulefile or a
4240 modulefile part of this useless requirement unloading batch.
4241 Modulefiles are added to this unloading batch only if they are not
4242 required by any other loaded modulefiles. "MODULES_LMNOTUASKED"
4243 environment variable helps to keep track of these automatically
4244 loaded modulefiles and to distinguish them from modulefiles asked by
4245 user.
4246
4247 * Dependent Reload (DepRe): reload of the modulefiles declaring a
4248 **conflict** or an optional **prereq** onto either the unloaded
4249 modulefile, an unloaded dependent or an unloaded useless requirement
4250 or declaring a **prereq** onto a modulefile part of this reloading
4251 batch.
4252
4253 In case a loaded modulefile has some of its declared constraints
4254 unsatisfied (pre-required modulefile not loaded or conflicting
4255 modulefile loaded for instance), this loaded modulefile is excluded
4256 from the automatic reload actions described above.
4257
4258 For the specific case of the **switch** sub-command, where a
4259 modulefile is unloaded to then load another modulefile. Dependent
4260 modulefiles to Unload are merged into the Dependent modulefiles to
4261 Reload that are reloaded after the load of the switched-to modulefile.
4262
4263 This automated module handling mode integrates concepts (like the
4264 Dependent Reload mechanism) of the Flavours extension, which was
4265 designed for Modules compatibility version. As a whole, automated
4266 module handling mode can be seen as a generalization and as an
4267 expansion of the Flavours concepts.
4268
4269 This new feature can be controlled at build time with the "--enable-
4270 auto-handling" configure option. This default configuration can be
4271 superseded at run-time with the "MODULES_AUTO_HANDLING" environment
4272 variable or the command line switches "--auto" and "--no-auto".
4273
4274 By default, automated module handling mode is disabled and will stay
4275 so until the next major release version (5.0) where it will be enabled
4276 by default. This new feature is currently considered experimental and
4277 the set of triggered actions will be refined over the next feature
4278 releases.
4279
4280
4281 Consistency of module load/unload commands in modulefile
4282 --------------------------------------------------------
4283
4284 With the **module load** modulefile command, a given modulefile can
4285 automatically load a modulefile it pre-requires. Similarly with the
4286 **module unload** modulefile command, a given modulefile can
4287 automatically unload a modulefile it conflicts with.
4288
4289 Both commands imply additional actions on the loaded environment
4290 (loading or unloading extra modulefiles) that should cope with the
4291 constraints defined by the loaded environment.
4292
4293 Additionally **module load** and **module unload** modulefile commands
4294 express themselves constraints on loaded environment that should stay
4295 satisfied to ensure consistency.
4296
4297 To ensure the consistency of **module load** modulefile command once
4298 the modulefile defining it has been loaded, this command is
4299 assimilated to a **prereq** command. Thus the defined constraint is
4300 recorded in the "MODULES_LMPREREQ" environment variable. Same approach
4301 is used for **module unload** modulefile command which is assimilated
4302 to a **conflict** command. Thus the defined constraint is recorded in
4303 the "MODULES_LMCONFLICT" environment variable.
4304
4305 To ensure the consistency of the loaded environment, the additional
4306 actions of the **module load** and **module unload** modulefile
4307 commands have been adapted in particular situations:
4308
4309 * When unloading modulefile, **module load** command will unload the
4310 modulefile it targets only if no other loaded modulefile requires it
4311 and if this target has not been explicitly loaded by user.
4312
4313 * When unloading modulefile, **module unload** command does nothing as
4314 the relative conflict registered at load time ensure environment
4315 consistency and will forbid conflicting modulefile load.
4316
4317 Please note that loading and unloading results may differ than from
4318 previous Modules version now that consistency is checked:
4319
4320 * Modulefile targeted by a **module load** modulefile command may not
4321 be able to load due to a registered conflict in the currently loaded
4322 environment. Which in turn will break the load of the modulefile
4323 declaring the **module load** command.
4324
4325 * Modulefile targeted by a **module unload** modulefile command may
4326 not be able to unload due to a registered prereq in the loaded
4327 environment. Which in turn will break the load of the modulefile
4328 declaring the **module unload** command.
4329
4330 * If automated module handling mode is enabled, **module load**
4331 modulefile command is interpreted when unloading modulefile as part
4332 of the Useless Requirement Unload (UReqUn) mechanism not through
4333 modulefile evaluation. As a consequence, an error occurring when
4334 unloading the modulefile targeted by the **module load** command
4335 does not break the unload of the modulefile declaring this command.
4336 Moreover unload of the **module load** targets is done in the
4337 reverse loaded order, not in the **module load** command definition
4338 order.
4339
4340
4341 Modulefile alias and symbolic modulefile name consistency
4342 ---------------------------------------------------------
4343
4344 With the **module-alias** and **module-version** modulefile commands,
4345 alternative names can be given to a modulefile. When these names are
4346 used to load for instance a modulefile, they are resolved to the
4347 modulefile they target which is then processed for the load action.
4348
4349 Until now, the alias and symbolic version names were correctly
4350 resolved for the **load** and **unload** actions and also for the
4351 querying sub-commands (like **avail** or **whatis**). However this
4352 alternative name information vanishes once the modulefile it resolves
4353 to is loaded. As a consequence there was no consistency over these
4354 alternative designations. In the following example "f" modulefile
4355 declares a conflict on "e" alias which resolves to "d" modulefile:
4356
4357 $ module load e
4358 $ module list
4359 Currently Loaded Modulefiles:
4360 1) d
4361 $ module info-loaded e
4362 $ module load f
4363 $ module list
4364 Currently Loaded Modulefiles:
4365 1) d 2) f
4366
4367 Consistency of the alternative names set on a modulefile with
4368 **module-alias** and **module-version** commands is now ensured to
4369 enable modulefile commands **prereq**, **conflict**, **is-loaded** and
4370 **module-info loaded** using these alternative designations as
4371 argument. This consistency is achieved, like for the conflict and
4372 prereq consistencies, by keeping track of the alternative names of the
4373 loaded modulefiles in an environment variable called
4374 "MODULES_LMALTNAME":
4375
4376 $ module load e
4377 $ module list
4378 Currently Loaded Modulefiles:
4379 1) d
4380 $ module info-loaded e
4381 d
4382 $ module load f
4383 WARNING: f cannot be loaded due to a conflict.
4384 HINT: Might try "module unload e" first.
4385 $ module list
4386 Currently Loaded Modulefiles:
4387 1) d
4388
4389
4390 Environment variable change through modulefile evaluation context
4391 -----------------------------------------------------------------
4392
4393 All environment variable edition commands ("setenv", "unsetenv",
4394 "append-path", "prepend-path" and "remove-path") have been updated to:
4395
4396 * Reflect environment variable value change on the environment of the
4397 current modulefile Tcl interpreter. So using "$env(VAR)" will return
4398 the currently defined value for environment variable "VAR", not the
4399 one found prior modulefile evaluation.
4400
4401 * Clear environment variable content instead of unsetting it on the
4402 environment of the current modulefile Tcl interpreter to avoid
4403 raising error about accessing an undefined element in "$env()". Code
4404 is still produced to purely unset environment variable in shell
4405 environment.
4406
4407 Exception is made for the "whatis" evaluation mode: environment
4408 variables targeted by variable edition commands are not set to the
4409 defined value in the evaluation context during this "whatis"
4410 evaluation. These variables are only initialized to an empty value if
4411 undefined. This exception is made to save performances on this global
4412 evaluation mode.
4413
4414
4415 Express Modules compatibility of modulefile with versioned magic cookie
4416 -----------------------------------------------------------------------
4417
4418 Any modulefile should start with the "#%Module" file signature (also
4419 called the Modules magic cookie) and sometimes a version number may be
4420 placed right after this string. Until now this version number
4421 corresponded to a modulefile format version but it was never checked.
4422
4423 Starting with this new Modules release, this version number reflects
4424 the minimum version of Modules required to interpret the modulefile.
4425 If the version number is set along the magic cookie string it is now
4426 checked and the modulefile is interpreted only if Modules version is
4427 greater or equal to this version number. For instance, if a modulefile
4428 begins with the "#%Module4.3" string, it can only be evaluated by
4429 Modules version 4.3 and above. Elsewhere the modulefile is ignored
4430 like files without the "#%Module" magic cookie set.
4431
4432
4433 Improved module message report
4434 ------------------------------
4435
4436 Module sub-commands like "load", "unload" or "switch", may perform
4437 multiple load or unload modulefile evaluations in a row. Also these
4438 kind of evaluation modes may sometimes trigger additional load or
4439 unload evaluations, when for instance a modulefile contains a "module
4440 load" command.
4441
4442 To improve the readability of the module messages produced relatively
4443 to a load or an unload evaluation, these messages are now stacked
4444 under a *Loading* or an *Unloading* message block that gathers all the
4445 messages produced for a given modulefile evaluation:
4446
4447 $ module load --no-auto foo
4448 Loading foo/1.2
4449 ERROR: foo/1.2 cannot be loaded due to missing prereq.
4450 HINT: the following module must be loaded first: bar/4.5
4451
4452 In addition, foreground "load", "unload", "switch" and "restore"
4453 actions (i.e., asked on the command-line) now report a summary of the
4454 additional load and unload evaluations that were eventually triggered
4455 in the process:
4456
4457 $ module load --auto foo
4458 Loading foo/1.2
4459 Loading requirement: bar/4.5
4460
4461
4462 New modulefile commands
4463 -----------------------
4464
4465 2 new modulefile Tcl commands have been introduced:
4466
4467 * **set-function**: define a shell function on sh-kind and fish
4468 shells.
4469
4470 * **unset-function**: unset a shell function on sh-kind and fish
4471 shells.
4472
4473
4474 v4.1
4475 ====
4476
4477 This new version is backward-compatible with previous version 4
4478 releases. It fixes bugs but also introduces new functionalities that
4479 are described in this section. See the 4.1 release notes for a
4480 complete list of the changes between Modules v4.0 and v4.1.
4481
4482
4483 Virtual modules
4484 ---------------
4485
4486 A virtual module stands for a module name associated to a modulefile.
4487 The modulefile is the script interpreted when loading or unloading the
4488 virtual module which appears or can be found with its virtual name.
4489
4490 The **module-virtual** modulefile command is introduced to give the
4491 ability to define these virtual modules. This new command takes a
4492 module name as first argument and a modulefile location as second
4493 argument:
4494
4495 module-virtual app/1.2.3 /path/to/virtualmod/app
4496
4497 With this feature it is now possible to dynamically define modulefiles
4498 depending on the context.
4499
4500
4501 Extend module command with site-specific Tcl code
4502 -------------------------------------------------
4503
4504 "module" command can now be extended with site-specific Tcl code.
4505 "modulecmd.tcl" now looks at a **siteconfig.tcl** file in an "etcdir"
4506 defined at configure time (by default "$prefix/etc"). If it finds this
4507 Tcl script file, it is sourced within "modulecmd.tcl" at the beginning
4508 of the main procedure code.
4509
4510 "siteconfig.tcl" enables to supersede any global variable or procedure
4511 definitions made in "modulecmd.tcl" with site-specific code. A module
4512 sub-command can for instance be redefined to make it fit local needs
4513 without having to touch the main "modulecmd.tcl".
4514
4515
4516 Quarantine mechanism to protect module execution
4517 ------------------------------------------------
4518
4519 To protect the module command run-time environment from side effect
4520 coming from the current environment definition a quarantine mechanism
4521 is introduced. This mechanism, sets within module function definition
4522 and shell initialization script, modifies the "modulecmd.tcl" run-time
4523 environment to sanitize it.
4524
4525 The mechanism is piloted by environment variables. First of all
4526 "MODULES_RUN_QUARANTINE", a space-separated list of environment
4527 variable names. Every variable found in "MODULES_RUN_QUARANTINE" will
4528 be set in quarantine during the "modulecmd.tcl" run-time. Their value
4529 will be set empty or set to the value of the corresponding
4530 "MODULES_RUNENV_<VAR>" environment variable if defined. Once
4531 "modulecmd.tcl" is started it restores quarantine variables to their
4532 original values.
4533
4534 "MODULES_RUN_QUARANTINE" and "MODULES_RUNENV_<VAR>" environment
4535 variables can be defined at build time by using the following
4536 configure option:
4537
4538 --with-quarantine-vars='VARNAME[=VALUE] ...'
4539
4540 Quarantine mechanism is available for all supported shells except
4541 "csh" and "tcsh".
4542
4543
4544 Pager support
4545 -------------
4546
4547 The informational messages Modules sends on the *stderr* channel may
4548 sometimes be quite long. This is especially the case for the avail
4549 sub-command when hundreds of modulefiles are handled. To improve the
4550 readability of those messages, *stderr* output can now be piped into a
4551 paging command.
4552
4553 This new feature can be controlled at build time with the "--with-
4554 pager" and "--with-pager-opts" configure options. Default pager
4555 command is set to "less" and its relative options are by default
4556 "-eFKRX". Default configuration can be supersedes at run-time with
4557 "MODULES_PAGER" environment variables or command-line switches ("--no-
4558 pager", "--paginate").
4559
4560 Warning:
4561
4562 On version "4.1.0", the "PAGER" environment variable was taken in
4563 consideration to supersede pager configuration at run-time. Since
4564 version "4.1.1", "PAGER" environment variable is ignored to avoid
4565 side effects coming from the system general pager configuration.
4566
4567
4568 Module function to return value in scripting languages
4569 ------------------------------------------------------
4570
4571 On Tcl, Perl, Python, Ruby, CMake and R scripting shells, module
4572 function was not returning value and until now an occurred error led
4573 to raising a fatal exception.
4574
4575 To make "module" function more friendly to use on these scripting
4576 shells it now returns a value. False in case of error, true if
4577 everything goes well.
4578
4579 As a consequence, returned value of a module sub-command can be
4580 checked. For instance in Python:
4581
4582 if module('load', 'foo'):
4583 # success
4584 else:
4585 # failure
4586
4587
4588 New modulefile commands
4589 -----------------------
4590
4591 4 new modulefile Tcl commands have been introduced:
4592
4593 * **is-saved**: returns true or false whether a collection,
4594 corresponding to currently set collection target, exists or not.
4595
4596 * **is-used**: returns true or false whether a given directory is
4597 currently enabled in "MODULEPATH".
4598
4599 * **is-avail**: returns true or false whether a given modulefile
4600 exists in currently enabled module paths.
4601
4602 * **module-info loaded**: returns the exact name of the modulefile
4603 currently loaded corresponding to the name argument.
4604
4605 Multiple collections, paths or modulefiles can be passed respectively
4606 to "is-saved", "is-used" and "is-avail" in which case true is returned
4607 if at least one argument matches condition (acts as a OR boolean
4608 operation). No argument may be passed to "is-loaded", "is-saved" and
4609 "is-used" commands to return if anything is respectively loaded, saved
4610 or used.
4611
4612 If no loaded modulefile matches the "module-info loaded" query, an
4613 empty string is returned.
4614
4615
4616 New module sub-commands
4617 -----------------------
4618
4619 Modulefile-specific commands are sometimes wished to be used outside
4620 of a modulefile context. Especially for the commands managing path
4621 variables or commands querying current environment context. So the
4622 following modulefile-specific commands have been made reachable as
4623 module sub-commands with same arguments and properties as if called
4624 from within a modulefile:
4625
4626 * **append-path**
4627
4628 * **prepend-path**
4629
4630 * **remove-path**
4631
4632 * **is-loaded**
4633
4634 * **info-loaded**
4635
4636 The "is-loaded" sub-command returns a boolean value. Small Python
4637 example:
4638
4639 if module('is-loaded', 'app'):
4640 print 'app is loaded'
4641 else:
4642 print 'app not loaded'
4643
4644 "info-loaded" returns a string value and is the sub-command
4645 counterpart of the "module-info loaded" modulefile command:
4646
4647 $ module load app/0.8
4648 $ module info-loaded app
4649 app/0.8
4650
4651
4652 v4.0
4653 ====
4654
4655 Major evolution occurs with this v4.0 release as the traditional
4656 *module* command implemented in C is replaced by the native Tcl
4657 version. This full Tcl rewrite of the Modules package was started in
4658 2002 and has now reached maturity to take over the binary version.
4659 This flavor change enables to refine and push forward the *module*
4660 concept.
4661
4662 This document provides an outlook of what is changing when migrating
4663 from v3.2 to v4.0 by first describing the introduced new features.
4664 Both v3.2 and v4.0 are quite similar and transition to the new major
4665 version should be smooth. Slights differences may be noticed in a few
4666 use-cases. So the second part of the document will help to learn about
4667 them by listing the features that have been discontinued in this new
4668 major release or the features where a behavior change can be noticed.
4669
4670
4671 New features
4672 ------------
4673
4674 On its overall this major release brings a lot more robustness to the
4675 *module* command with now more than 4000 non-regression tests crafted
4676 to ensure correct operations over the time. This version 4.0 also
4677 comes with fair amount of improved functionalities. The major new
4678 features are described in this section.
4679
4680
4681 Additional shells supported
4682 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
4683
4684 Modules v4 introduces support for **fish**, **lisp**, **tcl** and
4685 **R** code output.
4686
4687
4688 Non-zero exit code in case of error
4689 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4690
4691 All module sub-commands will now return a non-zero exit code in case
4692 of error whereas Modules v3.2 always returned zero exit code even if
4693 issue occurred.
4694
4695
4696 Output redirect
4697 ~~~~~~~~~~~~~~~
4698
4699 Traditionally the *module* command output text that should be seen by
4700 the user on *stderr* since shell commands are output to *stdout* to
4701 change shell's environment. Now on *sh*, *bash*, *ksh*, *zsh* and
4702 *fish* shells, output text is redirected to *stdout* after shell
4703 command evaluation if shell is in interactive mode.
4704
4705
4706 Filtering avail output
4707 ~~~~~~~~~~~~~~~~~~~~~~
4708
4709 Results obtained from the **avail** sub-command can now be filtered to
4710 only get the default version of each module name with use of the
4711 **--default** or **-d** command line switch. Default version is either
4712 the explicitly set default version or the highest numerically sorted
4713 modulefile or module alias if no default version set.
4714
4715 It is also possible to filter results to only get the highest
4716 numerically sorted version of each module name with use of the
4717 **--latest** or **-L** command line switch.
4718
4719
4720 Extended support for module alias and symbolic version
4721 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4722
4723 Module aliases are now included in the result of the **avail**,
4724 **whatis** and **apropos** sub-commands. They are displayed in the
4725 module path section where they are defined or in a *global/user
4726 modulerc* section for aliases set in user's or global "modulerc" file.
4727 A **@** symbol is added in parenthesis next to their name to
4728 distinguish them from modulefiles.
4729
4730 Search may be performed with an alias or a symbolic version-name
4731 passed as argument on **avail**, **whatis** and **apropos** sub-
4732 commands.
4733
4734 Modules v4 resolves module alias or symbolic version passed to
4735 **unload** command to then remove the loaded modulefile pointed by the
4736 mentioned alias or symbolic version.
4737
4738 A symbolic version sets on a module alias is now propagated toward the
4739 resolution path to also apply to the relative modulefile if it still
4740 correspond to the same module name.
4741
4742
4743 Hiding modulefiles
4744 ~~~~~~~~~~~~~~~~~~
4745
4746 Visibility of modulefiles can be adapted by use of file mode bits or
4747 file ownership. If a modulefile should only be used by a given subset
4748 of persons, its mode an ownership can be tailored to provide read
4749 rights to this group of people only. In this situation, module only
4750 reports the modulefile, during an **avail** command for instance, if
4751 this modulefile can be read by the current user.
4752
4753 These hidden modulefiles are simply ignored when walking through the
4754 modulepath content. Access issues (permission denied) occur only when
4755 trying to access directly a hidden modulefile or when accessing a
4756 symbol or an alias targeting a hidden modulefile.
4757
4758
4759 Improved modulefiles location
4760 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4761
4762 When looking for an implicit default in a modulefile directory,
4763 aliases are now taken into account in addition to modulefiles and
4764 directories to determine the highest numerically sorted element.
4765
4766 Modules v4 resolves module alias or symbolic version when it points to
4767 a modulefile located in another modulepath.
4768
4769 Access issues (permission denied) are now distinguished from find
4770 issues (cannot locate) when trying to access directly a directory or a
4771 modulefile as done on **load**, **display** or **whatis** commands. In
4772 addition, on this kind of access not readable ".modulerc" or
4773 ".version" files are ignored rather producing a missing magic cookie
4774 error.
4775
4776
4777 Module collection
4778 ~~~~~~~~~~~~~~~~~
4779
4780 Modules v4 introduces support for module *collections*. Collections
4781 describe a sequence of **module use** then **module load** commands
4782 that are interpreted by Modules to set the user environment as
4783 described by this sequence. When a collection is activated, with the
4784 **restore** sub-command, modulepaths and loaded modules are unused or
4785 unloaded if they are not part or if they are not ordered the same way
4786 as in the collection.
4787
4788 Collections are generated by the **save** sub-command that dumps the
4789 current user environment state in terms of modulepaths and loaded
4790 modules. By default collections are saved under the "$HOME/.module"
4791 directory. Collections can be listed with **savelist** sub-command,
4792 displayed with **saveshow** and removed with **saverm**.
4793
4794 Collections may be valid for a given target if they are suffixed. In
4795 this case these collections can only be restored if their suffix
4796 correspond to the current value of the "MODULES_COLLECTION_TARGET"
4797 environment variable. Saving collection registers the target footprint
4798 by suffixing the collection filename with
4799 ".$MODULES_COLLECTION_TARGET".
4800
4801
4802 Path variable element counter
4803 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4804
4805 Modules 4 provides path element counting feature which increases a
4806 reference counter each time a given path entry is added to a given
4807 path-like environment variable. As consequence a path entry element is
4808 removed from a path-like variable only if the related element counter
4809 is equal to 1. If this counter is greater than 1, path element is kept
4810 in variable and reference counter is decreased by 1.
4811
4812 This feature allows shared usage of particular path elements. For
4813 instance, modulefiles can append "/usr/local/bin" to "PATH", which is
4814 not unloaded until all the modulefiles that loaded it unload too.
4815
4816
4817 Optimized I/O operations
4818 ~~~~~~~~~~~~~~~~~~~~~~~~
4819
4820 Substantial work has been done to reduce the number of I/O operations
4821 done during global modulefile analysis commands like **avail** or
4822 **whatis**. "stat", "open", "read" and "close" I/O operations have
4823 been cut down to the minimum required when walking through the
4824 modulepath directories to check if files are modulefiles or to resolve
4825 module aliases.
4826
4827 Interpretation of modulefiles and modulerc are handled by the minimum
4828 required Tcl interpreters. Which means a configured Tcl interpreter is
4829 reused as much as possible between each modulefile interpretation or
4830 between each modulerc interpretation.
4831
4832
4833 Sourcing modulefiles
4834 ~~~~~~~~~~~~~~~~~~~~
4835
4836 Modules 4 introduces the possibility to **source** a modulefile rather
4837 loading it. When it is sourced, a modulefile is interpreted into the
4838 shell environment but then it is not marked loaded in shell
4839 environment which differ from **load** sub-command.
4840
4841 This functionality is used in shell initialization scripts once
4842 **module** function is defined. There the "etc/modulerc" modulefile is
4843 sourced to setup the initial state of the environment, composed of
4844 *module use* and *module load* commands.
4845
4846
4847 Removed features and substantial behavior changes
4848 -------------------------------------------------
4849
4850 Following sections provide list of Modules v3.2 features that are
4851 discontinued on Modules v4 or features with a substantial behavior
4852 change that should be taken in consideration when migrating to v4.
4853
4854
4855 Package initialization
4856 ~~~~~~~~~~~~~~~~~~~~~~
4857
4858 "MODULESBEGINENV" environment snapshot functionality is not supported
4859 anymore on Modules v4. Modules collection mechanism should be used
4860 instead to **save** and **restore** sets of enabled modulepaths and
4861 loaded modulefiles.
4862
4863
4864 Command line switches
4865 ~~~~~~~~~~~~~~~~~~~~~
4866
4867 Some command line switches are not supported anymore on v4.0. When
4868 still using them, a warning message is displayed and the command is
4869 ran with these unsupported switches ignored. Following command line
4870 switches are concerned:
4871
4872 * "--force", "-f"
4873
4874 * "--human"
4875
4876 * "--verbose", "-v"
4877
4878 * "--silent", "-s"
4879
4880 * "--create", "-c"
4881
4882 * "--icase", "-i"
4883
4884 * "--userlvl" lvl, "-u" lvl
4885
4886
4887 Module sub-commands
4888 ~~~~~~~~~~~~~~~~~~~
4889
4890 During an **help** sub-command, Modules v4 does not redirect output
4891 made on stdout in *ModulesHelp* Tcl procedure to stderr. Moreover when
4892 running **help**, version 4 interprets all the content of the
4893 modulefile, then call the *ModulesHelp* procedure if it exists,
4894 whereas Modules 3.2 only interprets the *ModulesHelp* procedure and
4895 not the rest of the modulefile content.
4896
4897 When **load** is asked on an already loaded modulefiles, Modules v4
4898 ignores this new load order whereas v3.2 refreshed shell alias
4899 definitions found in this modulefile.
4900
4901 When **switching** on version 4 an *old* modulefile by a *new* one, no
4902 error is raised if *old* modulefile is not currently loaded. In this
4903 situation v3.2 threw an error and abort switch action. Additionally on
4904 **switch** sub-command, *new* modulefile does not keep the position
4905 held by *old* modulefile in loaded modules list on Modules v4 as it
4906 was the case on v3.2. Same goes for path-like environment variables:
4907 replaced path component is appended to the end or prepended to the
4908 beginning of the relative path-like variable, not appended or
4909 prepended relatively to the position hold by the swapped path
4910 component.
4911
4912 During a **switch** command, version 4 interprets the swapped-out
4913 modulefile in *unload* mode, so the sub-modulefiles loaded, with
4914 "module load" order in the swapped-out modulefile are also unloaded
4915 during the switch.
4916
4917 Modules 4 provides path element counting feature which increases a
4918 reference counter each time a given path entry is added to a given
4919 environment variable. This feature also applies to the "MODULEPATH"
4920 environment variable. As consequence a modulepath entry element is
4921 removed from the modulepath enabled list only if the related element
4922 counter is equal to 1. When **unusing** a modulepath if its reference
4923 counter is greater than 1, modulepath is kept enabled and reference
4924 counter is decreased by 1.
4925
4926 On Modules 3.2 paths composing the "MODULEPATH" environment variable
4927 may contain reference to environment variable. These variable
4928 references are resolved dynamically when "MODULEPATH" is looked at
4929 during module sub-command action. This feature has been discontinued
4930 on Modules v4.
4931
4932 Following Modules sub-commands are not supported anymore on v4.0:
4933
4934 * "clear"
4935
4936 * "update"
4937
4938
4939 Modules specific Tcl commands
4940 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4941
4942 Modules v4 provides path element counting feature which increases a
4943 reference counter each time a given path entry is added to a given
4944 environment variable. As a consequence a path entry element is not
4945 always removed from a path-like variable when calling to "remove-path"
4946 or calling to "append-path" or "append-path" at unloading time. The
4947 path element is removed only if its related element counter is equal
4948 to 1. If this counter is greater than 1, path element is kept in
4949 variable and reference counter is decreased by 1.
4950
4951 On Modules v4, **module-info mode** returns during an **unload** sub-
4952 command the "unload" value instead of "remove" on Modules v3.2.
4953 However if *mode* is tested against "remove" value, true will be
4954 returned. During a **switch** sub-command on Modules v4, "unload" then
4955 "load" is returned instead of "switch1" then "switch2" then "switch3"
4956 on Modules v3.2. However if *mode* is tested against "switch" value,
4957 true will be returned.
4958
4959 When using **set-alias**, Modules v3.2 defines a shell function when
4960 variables are in use in alias value on Bourne shell derivatives,
4961 Modules 4 always defines a shell alias never a shell function.
4962
4963 Some Modules specific Tcl commands are not supported anymore on v4.0.
4964 When still using them, a warning message is displayed and these
4965 unsupported Tcl commands are ignored. Following Modules specific Tcl
4966 commands are concerned:
4967
4968 * "module-info flags"
4969
4970 * "module-info trace"
4971
4972 * "module-info tracepat"
4973
4974 * "module-info user"
4975
4976 * "module-log"
4977
4978 * "module-trace"
4979
4980 * "module-user"
4981
4982 * "module-verbosity"
4983
4984
4985 Further reading
4986 ---------------
4987
4988 To get a complete list of the differences between Modules v3.2 and v4,
4989 please read the Changes between versions document.
4990
4991 A significant number of issues reported for v3.2 have been closed on
4992 v4. List of these closed issues can be found at:
4993
4994 https://github.com/envmodules/modules/milestone/1?closed=1