Debian Codegraph @

doc/source/cookbook/compiler-etc-dependencies.rst

raw
1 .. _compiler-etc-dependencies:
2
3 Handling Compiler and other Package Dependencies
4 ================================================
5
6 When creating a collection of software (applications and libraries)
7 for users to use, there is the problem of ensuring that the
8 user is using the correct builds of everything. Generally,
9 if an user is attempting to compile code making use of the system
10 software collection, you want to ensure that the user is compiling
11 their code with the same compiler that was used to compile the library.
12 This tends to be particularly true of C++ and Fortran code using modules,
13 and parallel codes using MPI libraries.
14
15 As a result, in environments supporting multiple compilers,
16 software libraries often end up with multiple installs of libraries
17 and applications of the same version, depending on the compiler
18 and other libraries used to build them. Sometimes there are
19 even additional installs for variants with different threading
20 models, number formats, level of vectorization support, etc.
21 This cookbook describes various strategies for handling the modulefiles
22 to support all these different builds for each package.
23
24 For each strategy, we will provide an overview of how it works,
25 and then show how an user might interact with it, usually a similar
26 sequence for each case. We wish to explore how, and how well, each strategy
27 succeeds in handling the multiple builds of the same version of a package,
28 including
29
30 #. Basic dependency handling: seeing how well the strategy supports
31 the loading of the correct build of a package depending on the previously
32 loaded dependencies. And if no appropriate build is available, they should
33 error accordingly.
34 #. The ``module switch`` command and more advanced dependency handling: how
35 well the strategy supports more advanced cases. E.g. a case wherein several
36 modules are loaded and the user replaces a module upon which other modules
37 currently loaded depend. In general, how well the strategy prevents the
38 user's set of loaded modules from being incompatible.
39 #. Visibility into what packages are available. This includes being able
40 to readily see all of the packages installed, seeing what versions of packages
41 are available for a given compiler/MPI/etc combination, and seeing for
42 which compiler/MPI/etc combinations a specific version of a package is available.
43 #. How easily the user can navigate the modules for the builds. This
44 includes how well partial modulenames (e.g. omitting version, etc) are
45 handled by the different strategies.
46
47 We then try to summarize the strengths, weaknesses, and other attributes of
48 each strategy. We also try to discuss differences in using on older (3.x)
49 and newer (4.x) Environment Modules versions.
50
51 In addition to displaying examples for each strategy in this
52 document, we have set up a the test environment as a playground in
53 which you can explore.
54
55
56 Contents
57 --------
58
59 #. `Overview of Examples`_
60 #. :ref:`Flavours_strategy`
61 #. :ref:`Homebrewed_flavors_strategy`
62 #. :ref:`Modulerc-based_strategy`
63 #. :ref:`Modulepath-based_strategy`
64 #. `Comparison of Strategies`_
65
66
67 Overview of Examples
68 --------------------
69
70 The examples are a bit more elaborate than in some other cookbooks, so
71 the directory structure under ``doc/example/compiler-etc-dependencies``
72 is similarly more complicated.
73
74 Example Software Library
75 ^^^^^^^^^^^^^^^^^^^^^^^^
76
77 For the purpose of the examples and the playground, we have created
78 a fake example software library, rooted at the subdirectory
79 ``doc/example/compiler-etc-dependencies/dummy-sw-root`` beneath where
80 you placed the modules source files. This software tree is intended
81 to represent some of the features you might see in a real software
82 tree, which supports various compiler and MPI libraries, and that has
83 been added to over time, and not always in the most systematic way.
84
85 The example software library does not contain any real code; there are
86 dummy scripts for e.g. ``gcc``, ``mpirun``, etc. which just echo then
87 name of the code and what version, compiler, etc. it was supposed to be
88 built for, which is handy to show in the examples that the modulefiles
89 are working as expected. It also shows how such a directory tree might
90 be laid out --- the details of the layout will affect some of the code
91 in the modulefiles, etc. The directory structure can be altered to
92 fit your standards, but would require some minor modification to the
93 modulefiles, etc.
94 Note that there are also a bunch of subdirectories named ``1``
95 containing symlinks, these are for the strategy using the Flavours add-on and are
96 discussed in that section.
97
98 The software in the example software library consists of:
99
100 * GNU compiler versions ``8.2.0`` and ``9.1.0``
101 * Intel Parallel Studio suite versions ``2018`` and ``2019`` (includes compilers, MPI and MKL)
102 * PGI compiler suite versions ``18.4`` and ``19.4``
103 * OpenMPI version ``4.0``, built for:
104
105 * ``gcc/9.1.0``
106 * ``intel/2019``
107 * ``pgi/19.4``
108
109 * OpenMPI version ``3.1``, built for:
110
111 * ``gcc`` versions ``8.2.0`` and ``9.1.0``
112 * ``intel`` versions ``2018`` and ``2019``
113 * ``pgi`` versions ``18.4`` and ``19.4``
114
115 * mvapich version 2.3.1, built for:
116
117 * ``gcc/9.1.0``
118 * ``intel/2019``
119 * ``pgi/19.4``
120
121 * mvapich version ``2.1``, built for:
122
123 * ``gcc`` versions ``8.2.0`` and ``9.1.0``
124 * ``intel`` versions ``2018`` and ``2019``
125 * ``pgi`` versions ``18.4`` and ``19.4``
126
127 * foo version ``2.4``, built for:
128
129 * ``gcc/9.1.0`` and ``openmpi/4.0``
130 * ``gcc/9.1.0`` and ``mvapich/2.3.1``
131 * ``gcc/9.1.0`` and no MPI
132 * ``intel/2019`` and ``openmpi/4.0``
133 * ``intel/2019`` and ``mvapich/2.3.1``
134 * ``intel/2019`` and ``intelmpi``
135 * ``intel/2019`` and no MPI
136 * ``pgi/19.4`` and ``openmpi/3.1``
137 * ``pgi/19.4`` and no MPI
138
139 * foo version ``1.1``, built for:
140
141 * ``gcc/8.2.0`` and ``openmpi/3.1``
142 * ``gcc/8.2.0`` and ``mvapich/2.1``
143 * ``gcc/8.2.0`` and no MPI
144 * ``intel/2018`` and ``openmpi/3.1``
145 * ``intel/2018`` and ``mvapich/2.1``
146 * ``intel/2018`` and ``intelmpi``
147 * ``intel/2018`` and no MPI
148 * ``pgi/18.4`` and ``openmpi/3.1``
149 * ``pgi/18.4`` and ``mvapich/2.1``
150 * ``pgi/18.4`` and no MPI
151
152 * bar version ``5.4``, built with:
153
154 * ``gcc/9.1.0`` and supporting ``avx2``
155 * ``gcc/9.1.0`` and supporting ``avx``
156
157 * bar version ``4.7``, built for:
158
159 * ``gcc/8.2.0`` and supporting ``avx``
160 * ``gcc/8.2.0`` and supporting ``sse4.1``
161
162 I.e., we have 3 families of compiler suites with 2 different versions each. And
163 two MPI families (openmpi and mvapich) with two versions each, with the
164 most recent version only built with the latest compiler version of each family,
165 and the older version built with both versions of each compiler family.
166 In addition, it is assumed that the intel compiler suites include Intel's MPI
167 library built for that compiler. The application foo depends on the compiler and optionally on MPI libraries
168 and has two versions; the newer version *mostly* has builds for the latest compiler
169 and MPI (for pgi it only supports the latest compiler and older openmpi), and the
170 older version *mostly* has builds for the older compiler and MPI. The bar application
171 depends on compiler and has variants depending on size of integers used in the API.
172
173 We also assume that the ``gcc/8.2.0`` compiler is the system default; i.e. it is
174 the compiler provided by default by the Linux distro used by the system, and
175 therefore might potentially be available to users without loading any modules.
176
177 More directories under doc/example/compiler-etc-dependencies
178 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
179
180 The modulefiles for the different strategies do not play well with each
181 other, in part because we use the same names for many of the modules
182 between strategies. So in addition to the ``dummy-sw-root`` subdirectory,
183 each strategy has its own modulepath tree subdirectory
184 underneath ``doc/example/compiler-etc-dependencies``. There are some minor
185 differences between the modulefiles for the :ref:`Modulerc-based_strategy`
186 depending on whether Environment Modules 3.x or 4.x is being used,
187 so we actually have two trees for that case (``modulerc3`` and ``modulerc4``).
188
189 As there are a fair number of modulefiles, we make use of various tricks
190 in the cookbook :ref:`tips-for-code-reuse` to minimize the amount of repeated
191 code. In general, the actual modulefiles are small "stubfiles", setting one or a few
192 Tcl variables, and then sourcing a ``common`` tcl file which does all the
193 real work. Symlinks are used where possible to avoid duplicating files.
194 The :ref:`Modulerc-based_strategy` also uses some complicated ``.modulerc`` files; these
195 are fairly generic and to avoid redundancy are symlinked into the appropriate
196 places in modulepath tree from the ``modrc_common`` directory.
197
198 We also in some cases use Tcl procedures; for the sake of the
199 examples these our sourced in the files as needed, but if one were to
200 use the strategies needing such in production it would be better to follow
201 the suggestions in :ref:`expose-procs-to-modulefiles` and
202 place the required procedures in site config script.
203 The various tcl procedures are placed in the ``tcllib`` sub-directory, outside
204 of the modulepaths. These are actually broken up into multiple files
205 for the purpose of this cookbook (so that smaller chunks of code can be
206 looked at in this document).
207
208 The ``example-sessions`` subdirectory contains various shell scripts
209 used for the usage examples for each strategy (shell scripts are used
210 because there are some slight variations required between the strategies),
211 as well as the outputs of running such scripts. Subdirectories exist
212 for each strategy, and beneath them for each of the two Environment
213 Modules versions (``3.2.10`` and ``4.3.1``) used; for brevity not all of them
214 are shown in this document, especially as the 3.x and 4.x differences
215 are often small. In the example outputs, the Environment Modules
216 version and the strategy being employed is indicated in the shell prompt
217 (e.g. ``mod3-flavours`` or ``mod4 (modulerc)``).
218
219 Using the playground environments
220 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
221
222 Although we strive to provide a decent discussion in this cookbook,
223 you are encouraged to try things out in the playground in order to
224 get a better feel for things.
225
226 Because we use some modulefile names (e.g. gcc, intel, pgi, openmpi, etc)
227 that likely are present on your system as well, it is recommended
228 that if you wish to explore the playground environment that you
229 spawn a new shell, do a "module purge", and then set your ``MODULEPATH``
230 environment variable appropriately for the specific strategy.
231
232 The `Flavours_strategy`, as will be discussed, requires some modifications
233 to your Environment Modules installation. It is recommended that
234 you you make a copy or new installation (for Flavours, a 3.x install
235 works best), and then spawn a new shell and initialize the new Flavours
236 install in that first. Flavours code is not provided with this cookbook.
237
238 Some of the modulefiles, etc. require knowledge of where they were
239 installed. To avoid requiring you to update lines in numerous
240 files, we require you to set the environment variable
241 ``MOD_GIT_ROOTDIR`` to location where the modules git working directory
242 was cloned. E.g., if you issued the command
243 ``git clone https://github.com/envmodules/modules.git ~/modules.test``
244 you should set ``MOD_GIT_ROOTDIR`` to ``~/modules.test``. Please
245 ensure it is exported (use ``setenv`` in csh and related shells,
246 or ``export`` in Bourne derived shells like bash). This is just
247 a hack to make the examples work better; if you opt to use one of
248 these strategies in production, you will want to hard code some
249 relevant paths; the comments in the modulefiles will describe
250 what needs to be done.
251
252 Some more detail on setting up the playground is given at the
253 start of the Examples section for each strategy.
254
255 .. _Flavours_strategy:
256
257 Flavours Strategy
258 -----------------
259
260 The Flavours strategy uses the `Flavours extension`_ to Environment Modules.
261 Unlike the other strategies discussed, this requires the separate download and
262 installation of an extension to Environment Modules.
263
264 .. _Flavours extension: https://sourceforge.net/projects/flavours/
265
266 Installation and Implementation
267 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
268
269 More details can be found at the website for this extension, but to install this
270 you basically just need to:
271
272 #. Clone the git repo somewhere (``git clone https://git.code.sf.net/p/flavours/code flavours-code``)
273 #. Rename the standard Environment Modules ``modulecmd`` file (in the ``bin`` subdirectory
274 under the installation root) to ``modulecmd.wrapped``. (It is recommended that you do this
275 in a copy of your production installation, or better yet, in a new install of the 3.x Environment
276 Modules (as Flavours has been developed for Modules 3.x))
277 #. Copy the ``modulecmd.wrapper`` file from Flavours to the ``bin`` subdirectory above. Make sure
278 the ``modulecmd.wrapper`` file is executable.
279 #. Symlink ``modulecmd.wrapper`` to ``modulecmd``
280 #. Edit ``modulecmd.wrapper`` where indicated to give fully qualified path to ``modulecmd.wrapped``
281 #. Copy the ``flavours.tcl`` and ``pkgIndex.tcl`` files to some (possibly new) directory under
282 the modules installation root, and set ``TCLLIBPATH`` to that directory (you probably will want to
283 add that to the various modules init scripts)
284
285 The ``module`` command invokes ``modulecmd``, which in this case is results
286 in the Flavours wrapper bash script ``modulecmd.wrapper`` being invoked. This
287 calls the renamed standard ``modulecmd.wrapped`` command. This wrapper
288 command catches and processes certain output from the modulefile evaluation
289 intended for its consumption.
290
291 The modulefiles themselves make use of various commands in the Tcl module ``flavours``.
292 Many of these are just flavours variants of standard modulefile commands, e.g.
293 ``flavours prepend-path`` versus ``prepend-path``. Some important flavours
294 commands:
295
296 * ``package require flavours``: This loads the Tcl package flavours, and should occur
297 near the top of your modulefile
298 * ``flavours init``: This initializes the flavours package, and should be the first
299 of the flavours commands issued. Typically call right after the package load.
300 * ``flavours prereq``: Like the standard ``prereq`` command, this declares a prerequisite.
301 But it also does quite a bit more, as is discussed further below.
302 * ``flavours root``: This is used to set the root for where the package is actually
303 installed. This is used when generating the ``flavours path``
304 * ``flavours revision``: seem intended to allow for changes in the path format in future
305 versions of ``flavours``. It is used in constructing the final path to the package.
306 * ``flavours conflict``: This is similar to the standard conflict command, but enhanced
307 to recognize the flavours prereqs above.
308 * ``flavours commit``: This should be called after the ``root``, ``revision``, and ``prereq``
309 sub-commands of ``flavours`` are called, and before any of the ``path`` sub-commands.
310 It seems to be responsible for taking all those values to above and constructing the
311 path to the package.
312 * ``flavours path``: This returns a string with the path to the specific build of the package.
313 * ``flavours prepend-path``, ``flavour append-path``: These work much like the standard
314 ``prepend-path`` and ``append-path``, except that the value being prepended/appended
315 to the environment variable has the path (as returned by ``flavours path``) prepended
316 to it with the appropriate directory separator. E.g., to add to the ``PATH`` variable the bin subdirectory
317 of the root directory where the specific build was installed, use ``flavours prepend-path PATH bin``
318 * ``flavours cleanup``: This should be called after all ``flavours`` sub-commands are finished
319 and before exiting the script to ensure proper cleanup. Among other things, it ensures
320 that any packages that depend on this package will get reloaded if this package is switched out.
321
322 The ``flavours prereq`` command accepts the new ``-class`` parameter, allowing
323 it to require a class of packages; e.g. one could use ``-class compiler`` to indicate
324 that it has a prereq on a compiler (any of the modules ``gnu``, ``intel``, or ``pgi``).
325 The allowable classes, and the package basenames that are in each class, is defined in
326 ``flavours.tcl`` in the Tcl associative array ``_class``. The ones shipped by default are
327
328 * compiler: consisting of gnu, intel, and pgi
329 * mpi: consisting of openmpi, mvapich2, mvapich, intelmpi
330 * linalg: consisting of mkl, atlas, acml, netlib
331
332 You will likely want to adjust these if you go with flavours in production.
333
334 The ``flavours prereq`` command also accepts the parameter ``-optional``, which declares
335 optional prerequisites. Although it sounds a little oxymoronic, this comes into play
336 with the secondary purpose of the command in declaring the components of the path, as
337 discussed below. If a prereq is not optional, the modulefile will complain if nothing
338 satisfying the prereq has been module loaded previously. If the prereq is optional,
339 the modulefile will not complain if it was not loaded, but will use the prereq in
340 constructing the path to the build of the package if it was loaded.
341
342 The ``flavours prereq`` command also defines the components which will comprise the final
343 path to the directory containing the specific build of the package. The order of the
344 prereq commands controls the order of the components in the path.
345
346 The modulefile will check that all non-optional ``flavours prereq`` commands are satisfied,
347 and then construct a path to the installation root for this build of the package using
348 the packages satisfying the prereqs. The resultant path is composed of:
349
350 * the value from ``flavours root``
351 * directory separator (``/``)
352 * the value from ``flavours revision``
353 * directory separator (``/``)
354 * a ``prefix`` created by concatenating the package names satisfying the prereqs, in order.
355 The package name and version will be separated by a hyphen (-), as will the different
356 components.
357
358 So if ``flavours root`` was set to ``/local/software/foo/1.7``, ``revision`` to 1, and the
359 package had prereqs compiler and mpi, and ``gnu/9.1.0`` and ``openmpi/4.0`` were loaded, the
360 resulting path would be ``/local/software/foo/1.7/1/gnu-9.1.0-openmpi-4.0``.
361
362 The modulefile actually will test for the existence of that directory, and if not found will
363 return an error to the that the package was not built for that combination of prereqs.
364 You either need to install your packages using the above directory schema, or create
365 symlinks linking that scheme to where you actually install the packages.
366
367 Examples
368 ^^^^^^^^
369
370 We now look at the example modulefiles for flavours. To use the examples,
371 you must
372
373 #. Have `Flavours extension`_ installed. **NOTE** these examples will NOT work without
374 the Flavours installed.
375 #. Set (and export) ``MOD_GIT_ROOTDIR`` to where you git-cloned the modules source
376 #. Do a ``module purge``, and then set your ``MODULEPATH``
377 to ``$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/flavours``
378
379 We start with the ``module avail`` command:
380
381 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/flavours/modules4/modavail.out
382 :language: console
383
384 We note that we only see the package names and versions; e.g. ``foo/2.4``, without any mention
385 of the compilers and MPI libraries for which it is built. This terser type was an intentional
386 design goal of the authors. Also of note are the intelmpi and simd packages. The Flavours
387 approach relies on seeing what modules have been loaded previously in order to determine what
388 'flavor' of the requested package should be loaded. To support the different builds of ``bar``
389 which depend on the CPU vectorization commands supported, we need to add a "dummy" package ``simd``.
390 The module definition is quite trivial; a simple stub file like
391
392 .. literalinclude:: ../../example/compiler-etc-dependencies/flavours/simd/avx
393 :language: tcl
394
395 and the main content in the ``common`` file:
396
397 .. literalinclude:: ../../example/compiler-etc-dependencies/flavours/simd/common
398 :language: tcl
399
400 Basically it just declares a help procedure and whatis text. This way, an user can
401 load the appropriate simd module to control which variant of bar they will get. The
402 only interesting aspect is that near the beginning of the file we do a
403 ``package require flavours`` and ``flavours init``, and add a ``flavours cleanup``
404 near the bottom. The lines at the beginning instruct Tcl command to load the Flavours
405 package, and then initialize the package.
406 The ``flavours cleanup`` is required so that if the simd module is switched out,
407 any modulefiles that depend on it get reloaded.
408
409 In our example, we assumed that the Intel MPI libraries are automatically set up properly if one
410 were to load the ``intel`` module, and we assumed the Intel MPI libraries were not supported
411 for either the GNU or PGI compilers. However, we also wished to allow for ``foo`` to
412 be used without any MPI support. So we need a way to distinguish if someone wants to
413 use an Intel compiler build of ``foo`` without MPI or with the Intel MPI libraries. Our
414 choice for this example was to require one to explicitly module load ``intelmpi`` if one
415 wished to use the Intel MPI variant --- we do not bother with a real version number because
416 assuming the version is determined by the version of ``intel`` (the Intel Parallel Studio version).
417 So the intelmpi modulefile is similar to the simd modulefiles, a dummy modulefile. Again,
418 it includes the ``flavours init`` and ``flavours cleanup`` wrapping to ensure proper reloading
419 of dependent modules should it be switched out.
420
421 If you were to support Intel MPI for non-intel compilers, you could create your intelmpi
422 modulefiles as usual, and then add a ``default`` or ``intel`` "dummy" version to use the
423 version that is part of the ``intel`` Parallel Studio. Or you could separate the intelmpi
424 bits from the ``intel`` modulefile so both non-intel and intel compilers need to explicitly
425 module load intelmpi.
426
427 The modulefiles for the various compilers are all pretty much standard, except for the
428 same three ``flavours`` lines as the simd modulefile: ``package require flavours``,
429 ``flavours init``, and ``flavours cleanup``. These are required to
430 ensure dependent modulefiles get reloaded if the compiler is switched out.
431 We also note that the modulefile for the GNU Compiler Collection is referred to as ``gnu``, not ``gcc``
432 (this is due to how the ``compiler`` class is defined in ``flavours.tcl``, and we did not bother
433 to change that for the purposes of this cookbook).
434
435 With the openmpi and mvapich MPI libraries, things start to get interesting. These all
436 should setup the environment for a different build depending on the compiler loaded. The
437 real work is done in the ``common`` tcl file, as shown below:
438
439 .. literalinclude:: ../../example/compiler-etc-dependencies/flavours/openmpi/common
440 :language: tcl
441
442 Like the previous cases, the file starts with the Tcl command to load the
443 package, followed by the ``flavours init`` command.
444
445 The ``flavours prereq`` command states that this package requires a compiler to have been previously
446 loaded, and that the path to the specific build to use will depend on that. We note the use
447 of the ``-class`` parameter; the exact definition of the compiler class is in the ``compiler``
448 field of the Tcl associative hash ``_class`` defined in ``flavours.tcl``.
449
450 The ``flavours root`` sets the root directory of where the builds for this package is installed.
451 We use the ``MOD_GIT_ROOTDIR`` environment variable for convenience in this example, but in production
452 you would generally hardcode a path. The result of all the directives is that the build will be
453 found in a path named after the compiler (since in this case there is only one ``flavour prereq``);
454 e.g. for gcc version ``9.1.0``, we expect to find the build in ``$swroot/openmpi/4.0/1/gnu-9.1.0``.
455 If you do not use that naming convention for your installation directories, you can use symlinks
456 to fake it.
457
458 The ``flavours path`` command in the ``setenv MPI_DIR`` statement sets MPI_DIR to the aforementioned
459 build path. The ``flavours prepend-path`` commands prepend to the environment variable specified
460 by the first argument the result of prepending the ``flavours path`` to their second argument. E.g.,
461 the first such, assuming openmpi version ``4.0`` was requested and ``gnu/9.1.0`` loaded, would be basically
462 the same as a standard Modules command:
463
464 .. code-block:: tcl
465
466 prepend-path PATH $swroot/openmpi/4.0/1/gnu-9.1.0/bin
467
468 The following shows how this would appear to the user:
469
470 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/flavours/modules4/ompi-loads1.out
471 :language: console
472
473 Here we note that once a compiler is loaded, the ``PATH`` and the other environment
474 variables are set appropriately to point to the bin dir for the particular build of
475 ``openmpi/4.0``, as evidenced by the output of our dummy mpirun command. At the end, we attempt
476 to load ``openmpi/4.0`` for ``gnu/8.2.0``, and receive an error because our dummy SW library does not contain
477 a matching build. This is determined from the ``flavours path``; if the
478 path does not exist (in this example ``$swroot/openmpi/4.0/1/gnu-8.2.0``) it will
479 abort in this fashion.
480
481 In the above, we have explicitly unloaded openmpi, switched the compilers, and
482 then reloaded openmpi. A nice feature of the `Flavours extension`_ is that it can handle the
483 switching out of compilers or other modulefiles which other modulefiles depend on,
484 as:
485
486 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/flavours/modules3/ompi-switch.out
487 :language: console
488
489 Note that when we switched between the pgi and intel compilers above, Flavours
490 automatically "unloaded" and "reloaded" the openmpi module. This happens in
491 the ``flavours cleanup`` portion of the compiler modulefiles, and is due to
492 ``openmpi`` declaring a ``flavours prereq`` on the compiler class.
493
494 .. note:: The above behavior with ``switch`` was done with version ``3.2.10``
495 of Environment Modules; it does *not* appear to work with ``4.3.1``.
496
497 Note that when we further tried to replace version ``2019`` of the intel compiler with the ``2018`` version,
498 the module switch of the compilers failed because ``openmpi/4.0`` was not built with
499 ``intel/2018``. Since the user never explicitly requested version ``4.0`` of openmpi (it was
500 defaulted in the initial load as the latest version of openmpi available for ``pgi/19.4``),
501 it would have been nicer had the attempted reload of openmpi allowed it t:o default to the ``3.1``
502 version (as the latest version available for ``intel/2018``). Nevertheless, it behaved well
503 in this situation; the module switch failed with a reasonable error message and the resulting
504 set of modules was still consistent.
505
506 We also note that if we attempt to load openmpi without having previously loading a compiler,
507 we will get an error:
508
509 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/flavours/modules4/ompi-defaults.out
510 :language: console
511
512 In particular, there is no support for a "default" compiler; if e.g. you wished to make the
513 distribution supplied gcc the default compiler, you will need to have the initializations
514 scripts automatically do a module load of that compiler (possibly
515 a dummy modulefile like simd/intelmpi if the compiler is already in the user's path)
516 in your user's start up dot files or similar. We also note that there is no additional
517 intelligence in the version defaulting --- in the last example, we have ``gnu/8.2.0`` loaded and if
518 we try to load openmpi without specifying a version, it defaults to version ``4.0``
519 as that is the latest version of openmpi without regard for the fact that there
520 is no build of openmpi version ``4.0`` for ``gnu/8.2.0`` (but there is such for ``openmpi/3.1``).
521
522 The situation for ``foo`` is more complicated, as it depends both on the compiler and
523 optionally on the MPI library. But with Flavours, the modulefile is only slightly more
524 complicated, e.g. for the common file is:
525
526 .. literalinclude:: ../../example/compiler-etc-dependencies/flavours/foo/common
527 :language: tcl
528
529 Basically, the main difference is the addition of the
530 line ``flavours prereq -optional -class mpi``.
531 This instructs Flavours that there is an additional, optional prereq. The
532 order of the prereq lines matter, as that controls the resultant ``flavors path``.
533 With the current configuration, assuming ``gnu/9.1.0`` and ``openmpi/4.0`` were loaded,
534 the path would become ``$swroot/foo/2.4/1/gnu-9.1.0-openmpi-4.0``. If the order were
535 reversed, the ``openmpi-4.0`` would precede the ``gnu-9.1.0``. Because the MPI requirement
536 is optional, if ``gnu/9.1.0`` was loaded and no MPI library loaded, the path would
537 evaluate to ``$swroot/foo/2.4/1/gnu-9.1.0``.
538
539 We show how it works below:
540
541 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/flavours/modules4/foo-loads.out
542 :language: console
543
544 So basically, if the user loads a compiler, the
545 the environment variables (``PATH``, etc) are set up for the correct build of foo.
546 If no MPI library was loaded, a version of foo built without MPI will be loaded,
547 otherwise, a version of foo built with the loaded MPI library will be loaded.
548 This is shown by the output of the ``foo`` command. Note
549 also how we use the dummy ``intelmpi`` package to indicate a desire for the
550 intelmpi enabled version.
551
552 The 3.x version of Environment Modules supports using the switch command on
553 either the compiler or MPI library, and will result in reloading of foo and the MPI
554 library.
555
556 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/flavours/modules3/foo-switch.out
557 :language: console
558
559 In particular note the final case, wherein we load ``intel/2019`` then foo, and get the
560 version of foo built without MPI. When we subsequently load openmpi, foo is reloaded
561 to be the openmpi version (this is because the hooks to reload foo are in the ``flavours cleanup``
562 part of the openmpi modulefile, and foo declared its optional dependency on MPI).
563 Also, we don't bother showing it, but if you were to attempt to load foo without
564 at least a compiler loaded, it would display an error.
565
566 Our final example for flavours is the ``bar`` command. Here in addition to the
567 compiler dependency, we have versions for different SIMD vectorization supported.
568 Again, the difference in the modulefile is small, e.g.
569
570 .. literalinclude:: ../../example/compiler-etc-dependencies/flavours/bar/common
571 :language: tcl
572
573 Basically, the optional ``flavours prereq`` on the mpi class from the ``foo`` package
574 is replaced by a (mandatory) ``flavours prereq`` on the ``simd`` dummy package.
575 We note that Flavours package knows nothing about our ``simd`` dummy package until
576 we add it as a prereq for bar. (This is in contrast to the compiler and mpi classes).
577 Usage would be like:
578
579 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/flavours/modules4/bar-loads.out
580 :language: console
581
582 Here we note that as both the compiler and simd prereqs are non-optional, it complains
583 unless both have been previously loaded. When both have been loaded, the ``PATH`` and
584 other environment variables are set appropriately for the requested build; and if
585 it does not exist and error is produced.
586
587 Summary of Flavours
588 ^^^^^^^^^^^^^^^^^^^
589
590 * It is an external extension to Environment Modules, requiring additional installation steps.
591 * The git repository appears to have been last updated in 2013; although which means
592 that it has not been updated for Environment Modules 4.x, a simple experimentation
593 indicates that it still works, with the exception of automatic reloading of a module if any of
594 the modules it depends on are switched.
595 However, the `Flavours extension`_ does not appear to be actively supported.
596 * The Flavours package (using Environment Modules 3.x) fully supports the ``module switch`` syntax,
597 with the switching out of a dependency (e.g. a compiler) causing the reload of all modulefiles depending on it.
598 (however the test of this feature failed when using Environment Modules 4.x.)
599 * The syntax for modulefiles is elegant, and one can easily extend the basic compiler dependency
600 modulefile to add additional dependencies. Even for packages/dummy packages that the Flavours
601 extension knows nothing about (e.g. simd in the above example).
602 * The Flavours package will provide a shorter module avail output, only e.g. giving package name and
603 version and not listing a separate modulefile for each combination of package, version, compiler+version,
604 MPI library+version, etc.
605 * The Flavours package will fail with an error message if user tries to load a package which was
606 not built for the values of the dependency packages loaded.
607 * The Flavours package will fail to load with an error message if any dependent package is not
608 already loaded. In particular, it will not attempt to default these. So if such defaults are
609 desired, you will need to have initialization scripts automatically load the appropriate modules.
610 * The Flavours package does not include any mechanism for more intelligent defaulting. I.e., if an
611 user requests to load a package without specifying the version desired, the version will be defaulted
612 to the latest version (or whatever the ``.modulerc`` file specifies) without regard for which versions
613 support the versions of the compiler and other prereq-ed packages the user has loaded. While one
614 could write custom ``.modulerc`` files for such, Flavours does not provide any tools for simplifying
615 such.
616
617
618 .. _Homebrewed_flavors_strategy:
619
620 Homebrewed Flavors Strategy
621 ---------------------------
622
623 Although the `Flavours extension`_ described above has an elegance about it,
624 one can achieve much of the same functionality in modulefiles using standard
625 Environment Modules and Tcl commands. This can be facilitated by the definition
626 of some useful Tcl procedures. For lack of a better name, we will refer to
627 this strategy as *Homebrewed flavors*.
628
629 Implementation
630 ^^^^^^^^^^^^^^
631
632 This strategy just makes use of standard Environment Modules and Tcl procedures
633 to query what modules of a given type are loaded and to construct the path to the
634 software package accordingly. To avoid needless (and error prone) repetition of
635 code, we collect these into several Tcl procedures of our own. Ideally, these
636 should be placed in a site configuration Tcl file and exposed to modulefiles
637 as explained in the cookbook :ref:`expose-procs-to-modulefiles`.
638 However, to avoid the need for that in these examples, we instead have placed them
639 into a file and use the ``MOD_GIT_ROOTDIR`` to locate and source that file in the
640 relevant modulefiles. (Actually, we have a single tcl file that is sourced both
641 for this and some other strategies, and it sources several files so that we can
642 break up the discussion of the the Tcl procedures. All of that is just for the
643 purposes of this cookbook; normally you just put the procedures you need in the
644 one site config file).
645
646 We discuss the various Tcl procedures here, as they are what provide most of the
647 functionality. We start with the routines for generic loaded modules:
648
649 .. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/LoadedModules.tcl
650 :language: tcl
651 :lines: 4-
652
653 This defines the two Tcl procedures:
654
655 * **GetLoadedModules** : this returns the list of loaded modules, from the ``LOADEDMODULES``
656 * **GetTagOfModuleLoaded** : this takes as argument the base name of a package,
657 and returns the first full spec for the matching package, or an empty
658 string if no matching package found.
659
660 The Tcl procedure **GetTagOfModuleLoaded** can be used to find out what version
661 of a given package is loaded, and is enough for many packages. However, for
662 compilers, and similar, a bit more is needed. For compilers:
663
664 .. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/CompilerUtils.tcl
665 :language: tcl
666 :lines: 4-
667
668 We defined four procedures above:
669
670 * **GetDefaultCompiler** : this simply returns the name of our default compiler, which for
671 this example is ``gcc/8.2.0``
672 * **RequireCompiler** : this simply does a module load on the specified compiler.
673 It is kept as a separate procedure just in case you wish to intercept and prevent
674 the loading of the default compiler (e.g. because no modulefile exists for it).
675 In our example, there is a modulefile for it and so it is just a wrapper for ``module load``.
676 * **GetKnownCompilerFamilies** : this simply returns a Tcl list of known
677 compiler families.
678 * **GetLoadedCompiler**: this is the procedure that does the main work, and is
679 described in detail below.
680
681 The **GetLoadedCompiler** procedure basically checks if any packages matching
682 the names in **GetKnownCompilerFamilies** have been previously loaded. If so,
683 it returns the modulefile specification for the first one found, and returns.
684 If not, if ``pathDefault`` is set and there is a recognized compiler name and version
685 in the last two components of the module specification, it will return
686 that compiler. Otherwise, if the optional flag ``useDefault`` is set, it will return the
687 value from **GetDefaultCompiler**. If all else fails, returns the empty string.
688
689 If the optional parameter ``loadIt`` is set, if a compiler was defaulted (i.e.
690 not returned because it was already loaded), the procedure will call ``RequireCompiler``
691 to module load it.
692
693 If the optional parameter ``requireIt`` is set, we invoke ``prereq`` on the compiler
694 found before returning.
695
696 A similar set of procedures exist for the MPI libraries, namely:
697
698 .. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/MpiUtils.tcl
699 :language: tcl
700 :lines: 4-
701
702 The three procedures here are analogues of the compiler versions:
703
704 * **RequireMPI** : this basically does a module load of the specified MPI library. It
705 has some added logic so that it will not do a module load if the MPI library is ``nompi``.
706 Also, if the optional parameter ``noLoadIntel`` is set, if the MPI library is ``intelmpi``
707 (or a variant of that name) and the loaded compiler in ``intel``, we assume that no
708 additional module needs to be loaded. For this strategy, we want to load ``intelmpi``
709 modules, because, just like in the :ref:`Flavours_strategy`, we need to provide dummy
710 ``intelmpi`` modules to allow one to request the use of the Intel MPI library.
711 * **GetKnownMpiFamilies** : this returns a list of known MPI library family names.
712 Used in **GetLoadedMPI**
713 * **GetLoadedMPI** : This is the analogue of **GetLoadedCompiler**. If an MPI library is
714 loaded, it will return the name of that module. If the optional ``requireIt`` flag
715 is set, it will do a ``prereq`` on the MPI library before returning. The first optional
716 argument, ``useIntel``, indicates whether this module should return ``intelmpi`` if no
717 MPI library is loaded but an Intel compiler is loaded.
718
719 The modulefiles for the compilers are basically standard; unlike the
720 :ref:`Flavours_strategy` there is nothing special needed in these. Likewise
721 for the dummy ``simd`` and ``intelmpi`` modules (the latter is only this
722 basic because we assume intelmpi is only available if an Intel compiler
723 is loaded. If one allowed for intelmpi with other compilers, it would
724 more closely resemble the other MPI libraries).
725
726 We also define some Tcl procedures for generating warning and error messages, namely
727
728 .. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/WarningUtils.tcl
729 :language: tcl
730 :lines: 4-
731
732 These procedures:
733
734 * **PrintIfLoading**: will print supplied text to stderr only when in ``load`` mode
735 * **PrintLoadInfo**: will print supplied text as informational text, but only when trying to load a module
736 * **PrintLoadWarning**: will print supplied text as warning text, but only when trying to load a module
737 * **PrintLoadError**: will print supplied text as error text and abort, but only when trying to load a module
738
739 The gist of this is that we might wish to print errors if an user tries to
740 load an incompatible modulefile, but do not wish to print errors if they are merely
741 doing a help, display, or whatis command.
742
743 The interesting bit begins with the openmpi and mvapich modulefiles. These
744 both depend on the compiler, we show the main part of the openmpi modulefile
745 below:
746
747 .. literalinclude:: ../../example/compiler-etc-dependencies/homebrewed/openmpi/common
748 :language: tcl
749
750 We begin by sourcing the ``common_utilities`` file which defined the previously
751 described Tcl procedures. Normally it is recommended that you put those
752 procedures in a site config Tcl script and expose them to the modulefiles
753 using the techniques described in the cookbook :ref:`expose-procs-to-modulefiles`.
754 Even if you opt against that and decide to source a Tcl file, it is recommended
755 to hard code the path.
756
757 The next interesting bit comes when we set the local Tcl variable ``ctag``
758 by calling the ``GetLoadedCompiler`` procedure. We allow the procedure to
759 default the compiler, and because we have a default compiler defined we
760 should always get a value. (If no default compiler was defined, one would
761 have to handle the error if no compiler was loaded/defaulted.) We then use
762 the value of ``ctag`` to set the path to the build of the package. To ensure
763 that the package is built for this compiler, we do a quick check that the
764 package installation path exists.
765
766 The modulefile for ``foo`` is a bit more complex:
767
768 .. literalinclude:: ../../example/compiler-etc-dependencies/homebrewed/foo/common
769 :language: tcl
770
771 The main difference between this modulefile, depending on both compiler and
772 optionally MPI, and the openmpi modulefile above, is that in addition to
773 detecting which compiler is loaded, we call ``GetLoadedMPI`` to determine
774 the MPI library which was loaded, and use both of them in constructing the
775 prefix to the installed foo.
776
777 Examples
778 ^^^^^^^^
779
780 We now look at the example modulefiles for the *Homebrewed flavors* strategy.
781 To use the examples, you must
782 #. Set (and export) ``MOD_GIT_ROOTDIR`` to where you git-cloned the modules source
783 #. Do a ``module purge``, and then set your ``MODULEPATH`` to:
784
785 .. code-block:: shell
786
787 $MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/homebrewed
788
789 The *Homebrewed flavors* strategy behaves much like the :ref:`Flavours_strategy`
790 in practice. The module avail command,
791
792 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/modavail.out
793 :language: console
794
795 looks basically the same, showing the a concise listing of packages and
796 versions without information on the compilers and MPI libraries they were
797 built with.
798
799 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/ompi-loads1.out
800 :language: console
801
802 Again, once a compiler is loaded, loading openmpi will set the ``PATH``, etc. for
803 the correct build of openmpi, as evidenced by the output of the dummy
804 mpirun command. Notice that the ``module list`` command only shows
805 the version of openmpi loaded, and contains no information about what
806 compiler it was built with (you just have to assume it matches the loaded
807 compiler). And again we note that if one attempts to load a version
808 of openmpi (e.g. ``4.0``) that was not built for the specified compiler (e.g.
809 ``gcc/8.2.0``), an error is generated.
810
811 Unlike in :ref:`Flavours_strategy`, we did not put any code in the modulefiles to cause
812 dependent modulefiles to be reloaded if a module they depend on gets switched
813 out. However, starting with Environment Modules ``4.2``, a feature called
814 :ref:`Automated module handling <v42-automated-module-handling-mode>` was added. Without this feature, attempting to
815 switch out a module upon which other modules depended could be problematic,
816 as evidenced in this sequence below (using Environment Modules ``3.2.10`` and
817 so without automated module handling mode):
818
819 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules3/ompi-switch.out
820 :language: console
821
822 Here we note that we were able to switch out the pgi compiler for the intel compiler,
823 but the openmpi module was not reloaded and the environment is still set for openmpi
824 compiled with the pgi compiler, and that this inconsistency is not readily determined
825 from the module list command.
826
827 Environment Modules 4.x, even with automated module handling disabled, is better ---
828 in a command sequence as above the module switch from pgi to intel would fail due to the
829 prereq module. However, with the :ref:`Automated module handling <v42-automated-module-handling-mode>`
830 enabled (this feature is disabled by default but could be enabled on a per-command basis with the ``--auto``
831 flag), things work much better, as evidenced below:
832
833 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/ompi-switch.out
834 :language: console
835
836 When we switch out the pgi compiler for the ``intel/2019`` compiler, the openmpi
837 module is automatically unloaded before the compiler switch and reload afterwards,
838 so we end up with the correct build of openmpi for the newly loaded ``intel/2019``
839 compiler. If one then switches out ``intel/2019`` replacing it with ``intel/2018``,
840 the openmpi module is first unloaded, the compilers are switched, and as there
841 is no ``openmpi/4.0`` build for ``intel/2018``, a warning is given and the openmpi module
842 is left unloaded. Since the user never specifically requested version ``4.0`` of openmpi
843 (it was defaulted in the original module load of openmpi as that was the latest version
844 available for ``pgi/19.4``), it would have been nicer if on the switch of intel compiler
845 versions the reload only attempted a ``module load openmpi`` instead of ``module load openmpi/4.0``,
846 but nevertheless this well behaved. The openmpi module is dropped with a warning and the
847 user has a consistent set of modules loaded.
848
849 We note that the modulefile is able to
850 default the compiler, so when we attempt to load openmpi without having
851 previously loaded a compiler, as in
852
853 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/ompi-defaults.out
854 :language: console
855
856 it will default to the default compiler, ``gcc/8.2.0``. Note however, that if
857 one does not specify version ``3.1`` of openmpi, it will still default to ``4.0``
858 and fail to load as there is no build of ``openmpi/4.0`` for ``gcc/8.2.0``.
859
860 The situation is similar for foo:
861
862 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/foo-loads.out
863 :language: console
864
865 Again, one can load a compiler without an MPI library to get the non-MPI version
866 of foo, or a compiler and MPI library to get the MPI version. The dummy
867 intelmpi modulefile is used to allow one to indicate that the Intel MPI library
868 is desired. The automated module handling mode can again allow the switch
869 functionality work properly, as in
870
871 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/foo-switch.out
872 :language: console
873
874 Here we note a deficiency in the switch support as compared to :ref:`Flavours_strategy`. In the last example
875 after loading ``intel/2019`` and foo, we have the non-MPI build of foo as expected. However, upon
876 subsequently loading the openmpi module, we still have the non-MPI version of foo loaded, as evidenced
877 by the output of the dummy foo command. I.e., the foo package was *not* automatically reloaded, as
878 there was no prereq in the foo modulefile on an MPI library (as in the non-MPI build there is no MPI
879 library to prereq). Also note that module list does not really inform one of this fact.
880
881 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/foo-defaults.out
882 :language: console
883
884 Above, we see once more that the compiler can be defaulted, but that the
885 defaulting mechanism is not smart enough to default the version of foo based
886 on the compiler loaded (or defaulted to).
887
888 The situation with bar is basically the same; with a compiler and simd
889 module loaded, the environment for the appropriate build of bar is loaded
890 when you module load bar.
891
892 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/bar-loads.out
893 :language: console
894
895 And an error is generated if there is no build for that combination of
896 compiler and simd. The automatic handling of modules again allows the
897 switch command to work as expected:
898
899 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/bar-switch.out
900 :language: console
901
902 and both the simd level and compiler can be defaulted, but one still has
903 to choose a version of bar which supports the defaults.
904
905 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/bar-defaults.out
906 :language: console
907
908 Summary of Homebrewed flavors strategy
909 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
910
911 * The :ref:`Automated module handling <v42-automated-module-handling-mode>` feature (introduced in Environment Modules
912 ``4.2``) allows for the switching out of a dependency (e.g. a compiler)
913 to cause the reload of all modulefiles depending on it. Without the
914 automated module handling (as is default for 4.x, and the only option for
915 3.x), the switching of a compiler only changes the compiler and leaves the
916 modulefiles that depend on the compiler unchanged.
917 * The various Tcl procedures make
918 it somewhat easy to determine which compiler, MPI, etc. modules have been
919 loaded and set the paths appropriately. Not as elegant or easy to use
920 as :ref:`Flavours_strategy`, but not difficult
921 * Like the Flavours package, the module avail output is concise, only e.g.
922 giving package name and version rather than listing a separate modulefile
923 for each combination of package, version, compiler+version,
924 MPI library+version, etc.
925 * Modules will fail with an error message if user tries to load a package
926 which was not built for the values of the dependency packages loaded.
927 * With :ref:`v42-automated-module-handling-mode`, the dependencies of a module
928 asked for load could automatically be loaded. But it requires that these
929 dependencies are clearly specified with mention of the versions for which
930 a build is available.
931 * It does not include a mechanism for more intelligent defaulting. I.e., if
932 an user requests to load a package without specifying the version desired,
933 the version will be defaulted to the latest version (or whatever
934 the ``.modulerc`` file specifies) without regard for which versions
935 support the versions of the compiler and other prereq-ed packages the user
936 has loaded.
937 * Note that future releases of Environment Modules will introduce additional
938 mechanisms to the :ref:`v42-automated-module-handling-mode`, which will
939 improve the user experience on such *Homebrewed flavors* setup.
940
941
942 .. _Modulerc-based_strategy:
943
944 Modulerc-based Strategy
945 -----------------------
946
947 The previous two strategies used additional code in the modulefile to
948 determine which compiler, etc. was loaded and adjust the values for
949 ``PATH``, etc. accordingly. The *Modulerc-based* strategy instead uses
950 ``.modulerc`` files to direct the module command to the proper modulefile
951 depending on what compiler, etc. was previously loaded. Because of
952 this, there are a number of differences in behavior and what is seen
953 by the user, most notably many, many more modulefiles. Whether this is good
954 or bad is a matter of taste.
955
956 Implementation
957 ^^^^^^^^^^^^^^
958
959 Whereas the :ref:`Homebrewed_flavors_strategy` had the modulefile invoke
960 a Tcl procedure to determine which, if any, version of a module like a compiler
961 was loaded and then adjust paths, the *Modulerc-based* strategy instead
962 uses the same Tcl procedures to default the modulefile which will be loaded.
963 This implies that there is a distinct modulefile for every build of the package,
964 and an immediate consequence is that this strategy has many more modulefiles
965 than the others. We make use of the techniques in the cookbook
966 :ref:`tips-for-code-reuse` to reduce the total amount of code; the actual modulefiles
967 for each build are typically small stubfiles defining a couple of Tcl variables
968 and then sourcing a ``common`` script (unique to each package) which does all
969 the real work.
970
971 The modules will be named with components for the different dependencies,
972 so the one for openmpi version ``4.0`` built with gcc version ``9.1.0`` would
973 be ``openmpi/4.0/gcc/9.1.0``; similarly the module for foo version ``1.1``
974 built for pgi version ``18.4`` and mvapich ``2.1`` would be
975 ``foo/1.1/pgi/18.4/mvapich/2.1``.
976
977 The ``.modulerc`` files themselves are not trivial, but these
978 can generally be written in a generic fashion, usable by multiple packages,
979 and can just be symlinked to the appropriate locations.
980 We define five such files which can be linked as ``.modulerc``:
981
982 Two of these files can be linked in the module tree
983 at various places as ``.modulerc`` for defaulting the compiler. One to
984 default to the family portion of the compiler (e.g. gcc, intel, or pgi),
985 and one for the version. For the family portion of the compiler,
986 we have the file ``modulerc.select_compiler_family`` as below:
987
988 .. literalinclude:: ../../example/compiler-etc-dependencies/modrc_common/modulerc.select_compiler_family
989 :language: tcl
990
991 The file starts by sourcing a set of useful Tcl procedures. For the purpose
992 of the example for this cookbook this is done based on the ``MOD_GIT_ROOTDIR``
993 environment variable. If you were to use this in production, it is
994 recommended that the Tcl procedures be placed in a site configuration script
995 and exposed to modulefiles via the techniques described in the cookbook
996 :ref:`expose-procs-to-modulefiles`. At the minimum, it is
997 recommended to hardcode the path to the ``common_utilities.tcl`` file.
998
999 The modulerc script then determines the directory it is in using the
1000 Tcl variable ``ModulesCurrentModulefile``. It then uses the
1001 ``GetLoadedCompiler`` Tcl procedure (which was discussed in the section
1002 on the :ref:`Homebrewed_flavors_strategy`. We then parse the resulting module
1003 name into family and version pieces (we will discuss the procedure
1004 ``GetPackageFamilyVersion`` later; for now suffice to say it takes
1005 a module name and returns a Tcl list with family and version).
1006
1007 We do some trickery to support the use of either gcc or gnu as family names for
1008 the GNU compiler suite, and then see if there is a directory or modulefile
1009 underneath the directory containing the ``.modulerc`` file, and if so defaults
1010 to it. For this purpose, we use the Tcl procedure ``FirstChildModuleInList``
1011 (defined in ``common_utilities.tcl``). This and a related procedure are
1012 defined as:
1013
1014 .. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/ChildModules.tcl
1015 :language: tcl
1016 :lines: 4-
1017
1018 and basically make use of the Tcl variable ``ModulesCurrentModulefile``. They
1019 are limited, however, in that they will not work properly if the module is
1020 split across multiple modulepaths. That is, ``ChildModuleExists`` and
1021 ``FirstChildModuleInList`` will only detect children in the same directory
1022 as the modulerc file they are called from; e.g., if two paths in the ``MODULEPATH``
1023 both have directories for openmpi, one with an intel subdirectory and one
1024 with a pgi subdirectory and the modulerc invoking ``ChildModuleExists``, the
1025 ChildModuleExists procedure will not see the intel directory.
1026
1027 The ``modulerc.select_compiler_version`` file is similar,
1028
1029 .. literalinclude:: ../../example/compiler-etc-dependencies/modrc_common/modulerc.select_compiler_version
1030 :language: tcl
1031
1032 Again, we source the ``common_utitilies.tcl`` file and use
1033 ``ModulesCurrentModulefile`` to get the directory in which the ``.modulerc``
1034 script resides. It then uses ``GetLoadedCompiler`` to determine what
1035 if any compiler was loaded. If one was, we split into family and version,
1036 and ensure that the directory containing the ``.modulerc`` file matches the
1037 family name. If so, it checks if there is a directory or modulefile in that
1038 directory matching the version name, and if so defaults to it.
1039
1040 For both of the ``.modulerc`` scripts we examined, we note that in case of
1041 errors, etc., the script just exits without defaulting. One might be
1042 tempted to have the ``.modulerc`` script actually return an error in such
1043 cases (perhaps using ``module-info mode`` to only have it output during
1044 load operations). This, however, runs into issues:
1045
1046 * For Environment Modules 3.x: for some reason, when modulecmd processes
1047 ``.modulerc`` scripts, ``module-info mode load`` always returns true. Because
1048 of this, users would get spurious errors when doing a ``module avail``
1049 if the ``.modulerc`` scripts error-ed, because they all get processed regardless
1050 of what compiler is loaded.
1051 * For versions 4.x: the modulecmd process tends to process all ``.modulerc``
1052 files underneath the package root during a load, which will similarly
1053 generate spurious errors if the ``.modulerc`` scripts throw errors.
1054
1055 Because of this, if an user explicitly requests a modulefile that conflicts
1056 with the loaded compiler (e.g. the user does a ``module load pgi/19.4``
1057 followed by a ``module load openmpi/4.0/intel/2019``), the modulefile needs
1058 to detect this and error appropriately. A similar situation can happen if
1059 one of the ``.modulerc`` scripts fail to default, presumably because there is no
1060 appropriate build for the requested package; the modulecmd will default to
1061 something, but likely not what is wanted. To facilitate this, we define
1062 the Tcl procedure ``LoadedCompilerMatches``:
1063
1064 .. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/LoadedCompMatches.tcl
1065 :language: tcl
1066 :lines: 4-
1067
1068 The procedure takes a string for the family and version of the compiler
1069 that the modulefile expects, and ensures that if a compiler is loaded, they
1070 match. If they match, the procedure just returns, and if not, it spits out
1071 an error indicating a compiler mismatch. It also takes optional boolean flags:
1072 ``requireIt`` to determine if the procedure should ``prereq`` the compiler
1073 before returning, and ``loadIt`` to determine if, when no compiler was
1074 previously loaded, whether the procedure should load it (using
1075 ``RequireCompiler`` so it can properly handle the default compiler specially
1076 if needed). The ``requireIt`` flag should generally be set for Environment
1077 Modules 4.x (as that will allow the automatic handling of modules to
1078 properly recognize dependencies, even though as we will see that does not
1079 work too well for this strategy as yet), and should not be set for versions
1080 3.x (as the loading of modules with ``loadIt`` does not occur until *after*
1081 the ``prereq``, causing module loads to fail).
1082
1083 The resulting modulefile for something depending only on the compiler,
1084 using mvapich as an example, then would look like:
1085
1086 .. literalinclude:: ../../example/compiler-etc-dependencies/modulerc4/mvapich/common
1087 :language: tcl
1088
1089 Basically, the modulefile knows what compiler it wants (in the above example,
1090 that is set by the stubfile and passed as the Tcl variable ``compilerTag`` into the common
1091 script above), and then calls the ``LoadedCompilerMatches`` procedure
1092 above to ensure the loaded compiler matches what the modulefile wants (and
1093 to load it if no compiler is loaded).
1094
1095 So the mvapich directory in the ``MODULEPATH`` consists of subdirectories for
1096 each version of mvapich supported (e.g. ``2.1`` and ``2.3.1``). In each of the
1097 version subdirectories, we symlink ``modulerc.select_compiler_family`` as
1098 ``.modulerc``, and create an additional layer of subdirectories, one for
1099 each compiler family for which the version of mvapich is built (e.g.
1100 gcc, intel, pgi). In each compiler family directory under each mvapich
1101 version, we symlink ``modulerc.select_compiler_version`` as ``.modulerc``,
1102 and create a stub modulefile named for the compiler version. The stub
1103 modulefile then defines some Tcl variables for the version of mvapich
1104 and the compiler family/version, and sources the ``common`` file above.
1105 E.g., for ``mvapich/2.3.1/intel/2019``, the stubfile would look like
1106
1107 .. literalinclude:: ../../example/compiler-etc-dependencies/modulerc4/mvapich/2.3.1/intel/2019
1108 :language: tcl
1109
1110 So the mvapich directory in ``MODULEPATH`` would have a structure like
1111
1112 .. code-block:: console
1113
1114 mvapich
1115 ├── 2.1
1116 │   ├── gcc
1117 │   │   ├── 8.2.0
1118 │   │   ├── 9.1.0
1119 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1120 │   ├── intel
1121 │   │   ├── 2018
1122 │   │   ├── 2019
1123 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1124 │   ├── .modulerc -> symlink to modulerc.select_compiler_family
1125 │   └── pgi
1126 │   ├── 18.4
1127 │   ├── 19.4
1128 │   └── .modulerc -> symlink to modulerc.select_compiler_version
1129 ├── 2.3.1
1130 │   ├── gcc
1131 │   │   ├── 9.1.0
1132 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1133 │   ├── intel
1134 │   │   ├── 2019
1135 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1136 │   ├── .modulerc -> symlink to modulerc.select_compiler_family
1137 │   └── pgi
1138 │   ├── 19.4
1139 │   └── .modulerc -> symlink to modulerc.select_compiler_version
1140 └── common
1141
1142 With this directory structure, an user can load a compiler and then
1143 module load a specific version of mvapich, and if a build of that version of
1144 mvapich exists for that compiler, the environment will be set up properly,
1145 and otherwise an error reported. However, if the user module loads
1146 mvapich without specifying a version, it will simply default the version
1147 to the latest version of mvapich available, regardless of whether there
1148 is a build of that version of mvapich for the loaded compiler. Ideally,
1149 we would like it so that
1150 if one issues a module load of a package without specifying a version, it
1151 should load the latest version available that is compatible with any loaded
1152 compilers or other modules.
1153
1154 A simple change can enable that. We add a symlink to
1155 ``modulerc.select_compiler_family`` as ``.modulerc`` directly under the
1156 mvapich directory, and create an additional subdirectory under the mvapich
1157 directory for each compiler family. We symlink
1158 ``modulerc.select_compiler_version`` as ``.modulerc`` under each compiler
1159 family subdirectory, along with a subdirectory for each version of that
1160 compiler family that a version of mvapich is built for. We then symlink
1161 the various stub modulefiles under the ``mvapich/VERSION`` into the
1162 corresponding ``mvapich/COMPILER_FAMILY/COMPILER_VERSION`` directory, with
1163 the symlink's name being the mvapich version number. So the mvapich
1164 directory tree will now look like:
1165
1166 .. code-block:: console
1167
1168 mvapich
1169 ├── 2.1
1170 │   ├── gcc
1171 │   │   ├── 8.2.0
1172 │   │   ├── 9.1.0
1173 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1174 │   ├── intel
1175 │   │   ├── 2018
1176 │   │   ├── 2019
1177 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1178 │   ├── .modulerc -> symlink to modulerc.select_compiler_family
1179 │   └── pgi
1180 │   ├── 18.4
1181 │   ├── 19.4
1182 │   └── .modulerc -> symlink to modulerc.select_compiler_version
1183 ├── 2.3.1
1184 │   ├── gcc
1185 │   │   ├── 9.1.0
1186 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1187 │   ├── intel
1188 │   │   ├── 2019
1189 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1190 │   ├── .modulerc -> symlink to modulerc.select_compiler_family
1191 │   └── pgi
1192 │   ├── 19.4
1193 │   └── .modulerc -> symlink to modulerc.select_compiler_version
1194 ├── common
1195 ├── gcc
1196 │   ├── 8.2.0
1197 │   │   └── 2.1 -> symlink to ../../2.1/gcc/8.2.0
1198 │   ├── 9.1.0
1199 │   │   ├── 2.1 -> symlink to ../../2.1/gcc/9.1.0
1200 │   │   └── 2.3.1 -> symlink to ../../2.3.1/gcc/9.1.0
1201 │   └── .modulerc -> symlink to modulerc.select_compiler_version
1202 ├── intel
1203 │   ├── 2018
1204 │   │   └── 2.1 -> symlink to ../../2.1/intel/2018
1205 │   ├── 2019
1206 │   │   ├── 2.1 -> symlink to ../../2.1/intel/2019
1207 │   │   └── 2.3.1 -> symlink to ../../2.3.1/intel/2019
1208 │   └── .modulerc -> symlink to modulerc.select_compiler_version
1209 ├── .modulerc -> symlink to modulerc.select_compiler_family
1210 └── pgi
1211 ├── 18.4
1212 │   └── 2.1 -> symlink to ../../2.1/pgi/18.4
1213 ├── 19.4
1214 │   ├── 2.1 -> symlink to ../../2.1/pgi/19.4
1215 │   └── 2.3.1 -> symlink to ../../2.3.1/pgi/19.4
1216 └── .modulerc -> symlink to modulerc.select_compiler_version
1217
1218 When one attempts to module load mvapich without specifying a version,
1219 the ``.modulerc`` file will default to the family of the loaded compiler (or
1220 the default compiler if none loaded). It then descends into the corresponding
1221 compiler family directory, where the ``.modulerc`` there will default to the
1222 version of the loaded or defaulted compiler. After descending into the
1223 compiler version directory, there is no ``.modulerc``, so the standard modulecmd
1224 defaulting mechanism will select the latest version.
1225
1226 We now have two ways to refer to the same build of mvapich, namely
1227 one using the traditional form ::
1228
1229 PACKAGE/PKGVERSION/COMPILER_FAMILY/COMPILER_VERSION
1230 (e.g. mvapich/2.3.1/pgi/19.4)
1231
1232 and a new one giving the compiler family and version immediately
1233 after package name, with the version of the package coming last ::
1234
1235 PACKAGE/COMPILER_FAMILY/COMPILER_VERSION/PKGVERSION
1236 (e.g. mvapich/pgi/19.4/2.3.1)
1237
1238 These both refer to the same build of the package (e.g. version ``2.3.1``
1239 of mvapich, built for version ``19.4`` of the PGI compiler suite).
1240 Indeed, through the judicious use of symlinks, these actually refer to the
1241 modulefile on disk, but with two distinct names depending on the path
1242 used to get to it. In general, we allow for
1243 modules to be named with multiple dependency packages and/or flags if
1244 doing so would make it easier for a user to default to the correct package.
1245
1246 This is the primary reason for the procedure ``GetPackageFamilyVersion`` ---
1247 it splits a module name into components and takes the first component
1248 as the family name. It then examines the second component, and if it
1249 matches the name of a known package (like a compiler family), it uses
1250 the last component as the version. Otherwise, the second component is
1251 assumed to be the version. The code is as follows:
1252
1253 .. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/GetFamVer.tcl
1254 :language: tcl
1255 :lines: 4-
1256
1257 Similarly, there are two corresponding ``.modulerc`` scripts for defaulting
1258 the MPI library: one to default the family (e.g. openmpi, mvapich, intelmpi)
1259 and one to default the version. The family version, shown below:
1260
1261 .. literalinclude:: ../../example/compiler-etc-dependencies/modrc_common/modulerc.select_mpi_family
1262 :language: tcl
1263
1264 is similar to the corresponding compiler version, but contains additional
1265 logic to handle the intelmpi case. We treat intelmpi specially, because in
1266 our example here we assume the environment for Intel MPI is setup properly
1267 when the user loads the intel module. If no MPI library is explicitly loaded,
1268 but the intel module is loaded, than we look for a subdirectory named
1269 intelmpi (or one of its aliases) and default to it if it exists. Otherwise,
1270 if no MPI module is loaded, it looks for and will default to a directory
1271 or modulefile named nompi if it exists.
1272
1273 The ``modulerc.select_mpi_version`` script is also similar to its compiler
1274 counterpart,
1275
1276 .. literalinclude:: ../../example/compiler-etc-dependencies/modrc_common/modulerc.select_mpi_version
1277 :language: tcl
1278
1279 It checks if any MPI library was explicitly loaded, and if so it checks
1280 if the family of the loaded MPI module matches the name of the parent directory
1281 of this ``.modulerc`` file. If so, it checks for the existence of a subdirectory
1282 matching the version, and if found it defaults to it. There is no special
1283 handling needed for the ``nompi`` case, as since there is no version attached
1284 to the MPI library in the nompi case, this ``.modulerc`` is not present there.
1285 Similarly, for the case of Intel MPI when the Intel compiler suite is loaded, we
1286 expect the Intel MPI version to be that from the Intel compiler suite, so
1287 no version or this ``.modulerc`` is needed.
1288
1289 As the MPI modules depend themselves on the compiler, it is assumed that
1290 any package depending on the MPI libraries also depend on the compiler,
1291 and so will have a structure like the one described below for foo
1292
1293 .. code-block:: console
1294
1295 foo
1296 ├── 1.1
1297 │   ├── gcc
1298 │   │   ├── 8.2.0
1299 │   │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
1300 │   │   │   ├── mvapich
1301 │   │   │   │   ├── 2.1
1302 │   │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1303 │   │   │   ├── nompi
1304 │   │   │   └── openmpi
1305 │   │   │   ├── 3.1
1306 │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1307 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1308 │   ├── intel
1309 │   │   ├── 2018
1310 │   │   │   ├── intelmpi
1311 │   │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
1312 │   │   │   ├── mvapich
1313 │   │   │   │   ├── 2.1
1314 │   │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1315 │   │   │   ├── nompi
1316 │   │   │   └── openmpi
1317 │   │   │   ├── 3.1
1318 │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1319 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1320 │   ├── .modulerc -> symlink to modulerc.select_compiler_family
1321 │   └── pgi
1322 │   ├── 18.4
1323 │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
1324 │   │   ├── mvapich
1325 │   │   │   ├── 2.1
1326 │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1327 │   │   ├── nompi
1328 │   │   └── openmpi
1329 │   │   ├── 3.1
1330 │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1331 │   └── .modulerc -> symlink to modulerc.select_compiler_version
1332 ├── 2.4
1333 │   ├── gcc
1334 │   │   ├── 9.1.0
1335 │   │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
1336 │   │   │   ├── mvapich
1337 │   │   │   │   ├── 2.3.1
1338 │   │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1339 │   │   │   ├── nompi
1340 │   │   │   └── openmpi
1341 │   │   │   ├── 4.0
1342 │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1343 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1344 │   ├── intel
1345 │   │   ├── 2019
1346 │   │   │   ├── intelmpi
1347 │   │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
1348 │   │   │   ├── mvapich
1349 │   │   │   │   ├── 2.3.1
1350 │   │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1351 │   │   │   ├── nompi
1352 │   │   │   └── openmpi
1353 │   │   │   ├── 4.0
1354 │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1355 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1356 │   ├── .modulerc -> symlink to modulerc.select_compiler_family
1357 │   └── pgi
1358 │   ├── 19.4
1359 │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
1360 │   │   ├── nompi
1361 │   │   └── openmpi
1362 │   │   ├── 3.1
1363 │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1364 │   └── .modulerc -> symlink to modulerc.select_compiler_version
1365 ├── common
1366 ├── gcc
1367 │   ├── 8.2.0
1368 │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
1369 │   │   ├── mvapich
1370 │   │   │   ├── 2.1
1371 │   │   │   │   └── 1.1 -> ../../../../1.1/gcc/8.2.0/mvapich/2.1
1372 │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1373 │   │   ├── nompi
1374 │   │   │   └── 1.1 -> ../../../1.1/gcc/8.2.0/nompi
1375 │   │   └── openmpi
1376 │   │   ├── 3.1
1377 │   │   │   └── 1.1 -> ../../../../1.1/gcc/8.2.0/openmpi/3.1
1378 │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1379 │   ├── 9.1.0
1380 │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
1381 │   │   ├── mvapich
1382 │   │   │   ├── 2.3.1
1383 │   │   │   │   └── 2.4 -> ../../../../2.4/gcc/9.1.0/mvapich/2.3.1
1384 │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1385 │   │   ├── nompi
1386 │   │   │   └── 2.4 -> ../../../2.4/gcc/9.1.0/nompi
1387 │   │   └── openmpi
1388 │   │   ├── 4.0
1389 │   │   │   └── 2.4 -> ../../../../2.4/gcc/9.1.0/openmpi/4.0
1390 │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1391 │   └── .modulerc -> symlink to modulerc.select_compiler_version
1392 ├── intel
1393 │   ├── 2018
1394 │   │   ├── intelmpi
1395 │   │   │   └── 1.1 -> ../../../1.1/intel/2018/intelmpi
1396 │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
1397 │   │   ├── mvapich
1398 │   │   │   ├── 2.1
1399 │   │   │   │   └── 1.1 -> ../../../../1.1/intel/2018/mvapich/2.1
1400 │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1401 │   │   ├── nompi
1402 │   │   │   └── 1.1 -> ../../../1.1/intel/2018/nompi
1403 │   │   └── openmpi
1404 │   │   ├── 3.1
1405 │   │   │   └── 1.1 -> ../../../../1.1/intel/2018/openmpi/3.1
1406 │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1407 │   ├── 2019
1408 │   │   ├── intelmpi
1409 │   │   │   └── 2.4 -> ../../../2.4/intel/2019/intelmpi
1410 │   │   ├── .modulerc -> symlink to modulerc.select_mpi_family
1411 │   │   ├── mvapich
1412 │   │   │   ├── 2.3.1
1413 │   │   │   │   └── 2.4 -> ../../../../2.4/intel/2019/mvapich/2.3.1
1414 │   │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1415 │   │   ├── nompi
1416 │   │   │   └── 2.4 -> ../../../2.4/intel/2019/nompi
1417 │   │   └── openmpi
1418 │   │   ├── 4.0
1419 │   │   │   └── 2.4 -> ../../../../2.4/intel/2019/openmpi/4.0
1420 │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1421 │   └── .modulerc -> symlink to modulerc.select_compiler_version
1422 ├── .modulerc -> symlink to modulerc.select_compiler_family
1423 └── pgi
1424 ├── 18.4
1425 │   ├── .modulerc -> symlink to modulerc.select_mpi_family
1426 │   ├── mvapich
1427 │   │   ├── 2.1
1428 │   │   │   └── 1.1 -> ../../../../1.1/pgi/18.4/mvapich/2.1
1429 │   │   └── .modulerc -> symlink to modulerc.select_mpi_version
1430 │   ├── nompi
1431 │   │   └── 1.1 -> ../../../1.1/pgi/18.4/nompi
1432 │   └── openmpi
1433 │   ├── 3.1
1434 │   │   └── 1.1 -> ../../../../1.1/pgi/18.4/openmpi/3.1
1435 │   └── .modulerc -> symlink to modulerc.select_mpi_version
1436 ├── 19.4
1437 │   ├── .modulerc -> symlink to modulerc.select_mpi_family
1438 │   ├── nompi
1439 │   │   └── 2.4 -> ../../../2.4/pgi/19.4/nompi
1440 │   └── openmpi
1441 │   ├── 3.1
1442 │   │   └── 2.4 -> ../../../../2.4/pgi/19.4/openmpi/3.1
1443 │   └── .modulerc -> symlink to modulerc.select_mpi_version
1444 └── .modulerc -> symlink to modulerc.select_compiler_version
1445
1446 Although the directory tree above is somewhat lengthy, it is similar
1447 to the case for a module depending only on the compiler, but expanded
1448 to handle MPI dependencies as well. There is a subdirectory under ``foo``
1449 for each version of foo. Each of these subdirectories have further subdirectories
1450 for the compiler families for which that version of foo was built, and under
1451 the subdirectories for each compiler family are subdirectories for each version
1452 of that compiler family for which foo was built. Underneath these
1453 are stub modulefiles for nompi (and for intel compilers, intelmpi) builds of
1454 the package, and subdirectories for each MPI family, containing stub modulefiles
1455 for each version of the MPI family the package was built for.
1456
1457 Additionally, there are subdirectories directly under ``foo`` for each compiler
1458 family the package was built for, with each having subdirectories for the
1459 version of the compiler. Beneath each of those are one or two more layers
1460 of subdirectories for the MPI family and version (the version layer is omitted
1461 in the nompi or intelmpi cases), underneath which are symlinks to the corresponding
1462 stub modulefile from the ``foo/FOOVERSION`` path.
1463
1464 By placing the appropriate ``.modulerc`` files (actually, symlinks to the correct
1465 modulerc file), when the user enters a partial modulename the modulecmd will
1466 descend this directory tree based on the previously loaded compiler and MPI
1467 library to get to correct build of the package, if it exists. If no MPI library
1468 was previously loaded, it will search for an intelmpi build if an intel compiler
1469 was loaded, or a nompi build if no intelmpi build found or a non-intel compiler
1470 was used. If only MPI builds were made, it will default (using standard
1471 module defaulting rules) to one of the MPI builds, and the appropriate MPI
1472 library will be loaded when the foo module gets loaded (via the ``LoadedMPIMatches``
1473 Tcl procedure described below). If a compiler and MPI library were loaded but no
1474 build for that combination exists, again a modulefile will be defaulted to
1475 (using standard module defaulting rules), but an error will be generated by
1476 the one of the ``LoadedCompilerMatches`` or ``LoadedMPIMatches`` calls in the
1477 foo modulefile (see below).
1478
1479 The common code of the modulefile is fairly standard, as shown below
1480
1481 .. literalinclude:: ../../example/compiler-etc-dependencies/modulerc4/foo/common
1482 :language: tcl
1483
1484 The main difference from a standard modulefile is the inclusion of the
1485 invocations of ``LoadedCompilerMatches`` and ``LoadedMpiMatches``. This
1486 is needed to ensure that the compiler and MPI for the modulefile being
1487 processed are compatible with any which are loaded. The ``LoadedMpiMatches``
1488 procedure is defined by:
1489
1490 .. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/LoadedMpiMatches.tcl
1491 :language: tcl
1492 :lines: 4-
1493
1494 Basically, it determines what if any MPI library was previously module loaded.
1495 If one was loaded, it ensures the family and version match what was expected,
1496 and if not exit with an error. The use of the ``GetPackageFamilyVersion``
1497 procedure is important, because as discussed previously packages can be
1498 represented by more than one name (e.g. the modulefile for version ``4.0`` of
1499 openmpi for ``gcc/9.1.0`` could be named ``openmpi/4.0/gcc/9.1.0`` or
1500 ``openmpi/gcc/9.1.0/4.0``) and we need to ensure either is recognized.
1501
1502 Although we only showed cases for compilers and MPI libraries, the techniques
1503 above can be adapted to other cases as well. For simple cases, where everything
1504 falls under a single package name, one can use ``GetTagOfModuleLoaded`` to
1505 determine the version of the package loaded in the ``.modulerc`` and in the
1506 modulefile to ensure the version matches.
1507
1508 For bar, we added variants on CPU vectorization support. Rather than add
1509 a simd modulefile, we just add variants to the bar modules. So the bar
1510 modulefile tree would look like:
1511
1512 .. code-block:: console
1513
1514 bar
1515 ├── 4.7
1516 │   ├── gcc
1517 │   │   ├── 8.2.0
1518 │   │   │   ├── avx
1519 │   │   │   ├── .modulerc -> symlink to modulerc.default_lowest_simd
1520 │   │   │   └── sse4.1
1521 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1522 │   └── .modulerc -> symlink to modulerc.select_compiler_family
1523 ├── 5.4
1524 │   ├── gcc
1525 │   │   ├── 9.1.0
1526 │   │   │   ├── avx
1527 │   │   │   └── avx2
1528 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1529 │   └── .modulerc -> symlink to modulerc.select_compiler_family
1530 ├── avx
1531 │   ├── gcc
1532 │   │   ├── 8.2.0
1533 │   │   │   └── 4.7 -> ../../../4.7/gcc/8.2.0/avx
1534 │   │   ├── 9.1.0
1535 │   │   │   └── 5.4 -> ../../../5.4/gcc/9.1.0/avx
1536 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1537 │   └── .modulerc -> symlink to modulerc.select_compiler_family
1538 ├── avx2
1539 │   ├── gcc
1540 │   │   ├── 9.1.0
1541 │   │   │   └── 5.4 -> ../../../5.4/gcc/9.1.0/avx2
1542 │   │   └── .modulerc -> symlink to modulerc.select_compiler_version
1543 │   └── .modulerc -> symlink to modulerc.select_compiler_family
1544 ├── common
1545 ├── gcc
1546 │   ├── 8.2.0
1547 │   │   ├── avx
1548 │   │   │   └── 4.7 -> ../../../4.7/gcc/8.2.0/avx
1549 │   │   ├── .modulerc -> symlink to modulerc.default_lowest_simd
1550 │   │   └── sse4.1
1551 │   │   └── 4.7 -> ../../../4.7/gcc/8.2.0/sse4.1
1552 │   ├── 9.1.0
1553 │   │   ├── avx
1554 │   │   │   └── 5.4 -> ../../../5.4/gcc/9.1.0/avx
1555 │   │   ├── avx2
1556 │   │   │   └── 5.4 -> ../../../5.4/gcc/9.1.0/avx2
1557 │   │   └── .modulerc -> symlink to modulerc.default_lowest_simd
1558 │   └── .modulerc -> symlink to modulerc.select_compiler_version
1559 ├── .modulerc -> symlink to modulerc.select_compiler_family
1560 └── sse4.1
1561 ├── gcc
1562 │   ├── 8.2.0
1563 │   │   └── 4.7 -> ../../../4.7/gcc/8.2.0/sse4.1
1564 │   └── .modulerc -> symlink to modulerc.select_compiler_version
1565 └── .modulerc -> symlink to modulerc.select_compiler_family
1566
1567 Here we added yet another alternate set of module names. So the module
1568 for bar version ``5.4`` built with gcc version ``9.1.0`` and ``avx2`` support can be
1569 called:
1570
1571 * ``bar/5.4/gcc/9.1.0/avx2``
1572 * ``bar/gcc/9.1.0/avx2/5.4``
1573 * ``bar/avx2/gcc/9.1.0/5.4``
1574
1575 Due to this multiplicity of names, if an user does a module load bar, the
1576 ``.modulerc`` immediately under bar will cause modulecmd to default along the
1577 path for the loaded (or defaulted) compiler, and then the
1578 ``modulerc.default_lowest_simd`` will default to the lowest simd level.
1579 If the user module loads bar/VERSION, it will find the build for that version
1580 of bar with the appropriate compiler (and again, default to the lowest simd
1581 level). And if the user module loads ``bar/avx`` or another simd level, then
1582 it will load the latest version of bar built for the loaded compiler and
1583 simd specified.
1584
1585 The ``modulerc.default_lowest_simd`` script looks like:
1586
1587 .. literalinclude:: ../../example/compiler-etc-dependencies/modrc_common/modulerc.default_lowest_simd
1588 :language: tcl
1589
1590 It will default to the lowest SIMD level, but could easily be adapted to
1591 do something else.
1592
1593 Examples
1594 ^^^^^^^^
1595
1596 We now look at the example modulefiles for the *Modulerc-based* strategy.
1597 As noted previously, the best choice of whether to set the ``requireIt`` flag
1598 to ``LoadedCompilerMatches`` and ``LoadedMPIMatches`` (instructing them to
1599 to a ``prereq`` on the requesting compiler/MPI module) depends on whether
1600 one is using a 3.x or 4.x version of Environment Modules. Due to this,
1601 we provide two modulefile trees, one for 3.x and one for 4.x; they are
1602 basically identical except for that matter. This leads to slightly
1603 different instructions on how to use the examples, depending on which
1604 version of Environment Modules is being used, namely:
1605
1606 #. Set (and export) ``MOD_GIT_ROOTDIR`` to where you git-cloned the modules source
1607 #. Do a ``module purge``
1608 #. If using Environment Modules 3.x, set your ``MODULEPATH`` to ``$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc3``
1609 #. If using Environment Modules 4.x, set your ``MODULEPATH`` to ``$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4``
1610
1611 As with the previous cases, we start with a ``module avail`` command, and here
1612 we see the first big difference:
1613
1614 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/modavail.out
1615 :language: console
1616
1617 Unlike the previous cases wherein only package names and versions were shown
1618 (because a single modulefile handled all the builds for that package and
1619 version), here we see a listing for every build of a package and version.
1620 Indeed, we not only see one such listing, but multiple listings per
1621 build in many cases (e.g. ``openmpi/3.1/intel/2018`` and
1622 ``openmpi/intel/2018/3.1``).
1623
1624 While admittedly the output of a ``module avail`` command without
1625 specifying any package is rather overwhelming, when a package is specified
1626 the output tends to be more reasonable, informing one of which builds
1627 of the package are available. This strategy deliberately opts for the
1628 presentation of more rather than less information.
1629
1630 The standard functionality of selecting the correct build of a package
1631 based on the loaded compiler, e.g.
1632
1633 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/ompi-loads1.out
1634 :language: console
1635
1636 works as expected (and fails with the expected error when one attempts to
1637 load a module not compatible with the loaded compiler).
1638 The only significant difference between the previously
1639 examined strategies is that the module list command provides information
1640 about what variant of each package is loaded.
1641
1642 The ``module switch`` command, however, does not work as well as one would
1643 like. While it indeed switches the specified module, it does not
1644 successfully reload the modules which depend on the replaced module, even
1645 with the :ref:`Automated module handling <v42-automated-module-handling-mode>`
1646 feature enabled. As currently implemented,
1647 the automated module handling feature attempts to reload dependent modules
1648 using the fully qualified module name, and as in this strategy the fully
1649 qualified modulename includes the information about the module that was switched
1650 out (e.g. ``pgi/19.4`` in the example below), it will be incompatible with the
1651 replacement module (e.g. ``intel/2019`` in example below). It
1652 is hoped a future version of modulecmd will allow for reloading based on the
1653 name specified when the module was loaded. But as things currently stand,
1654 the automatic module handling will throw an error attempting to reload the
1655 depend module, resulting in the the dependent modules being unloaded.
1656
1657 Without
1658 automated module handling mode (i.e. for older Environment Modules or without
1659 the --auto flag), the dependent modules remain loaded and there is inconsistency
1660 in the loaded modules. But at least module list clearly shows such.
1661
1662 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/ompi-switch.out
1663 :language: console
1664
1665 The defaulting of modules is more successful, however, as seen below:
1666
1667 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/ompi-defaults.out
1668 :language: console
1669
1670 Here it not only defaults to the default compiler, but if one tries to load
1671 openmpi without specifying a version, it will find and load the latest version
1672 compatible with the loaded compiler.
1673
1674 The situation is similar for foo:
1675
1676 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/foo-loads.out
1677 :language: console
1678
1679 As expected, the correct version of foo is loaded depending on the previously
1680 loaded compiler and MPI libraries. Note however that there is no dummy
1681 module for Intel MPI, and that when the intel compiler is loaded but no MPI
1682 library is explicitly loaded, module load foo/2.4 gets the Intel MPI build.
1683 Such only occurs if there is an Intel MPI build in the tree of modulefiles;
1684 otherwise a nompi build will be loaded if present. If both versions are
1685 present, as in this case, one needs to give the explicit
1686 ``foo/2.4/intel/2019/nompi`` specification.
1687
1688 Again, the ``module switch`` command only succeeds in switching the
1689 specified module, and does not successfully reload the modules depending
1690 on the switched module. With automated module handling mode, there will be errors
1691 reported and the dependent modules will be unloaded; without it the dependent
1692 modules will not get unloaded, and there will be inconsistent dependencies
1693 (but at least module list will show such).
1694
1695 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/foo-switch.out
1696 :language: console
1697
1698 The defaulting of modules works relatively well, as shown below:
1699
1700 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/foo-defaults.out
1701 :language: console
1702
1703 If one attempts to load foo without specifying a version or having previously loaded a compiler
1704 module, the compiler will be defaulted (to ``gcc/8.2.0`` compiler, as that is what
1705 we declared via the ``GetDefaultCompiler`` Tcl procedure), and the latest
1706 version of foo compatible with that compiler (and no MPI) will be loaded.
1707
1708 The situation with bar is similar. We do not have a dummy simd module,
1709 so the builds with different CPU vectorization support are specified by
1710 appending ``/avx``, etc. to the bar package name. As with previous strategies,
1711 if one attempts to load a simd variant which was not built for the compiler
1712 loaded, an error will occur.
1713
1714 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/bar-loads.out
1715 :language: console
1716
1717 Summary of Modulerc-based strategy
1718 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1719
1720 * The modulefiles are fairly standard. The only extra logic needed
1721 is to ensure that any loaded compiler/MPI or other dependency matches
1722 what the modulefile expects, and that can be handled fairly easily
1723 with the addition of one or two calls to Tcl procedures.
1724 * The ``.modulerc`` files do the work of routing partial specifications to
1725 the correct modulefile, and follow a fairly standard pattern. For the
1726 most part, these are generic/independent of the package, and typically
1727 can be written once and symlinked to the appropriate places in the
1728 directory tree.
1729 * This strategy involves the use of many more modulefiles than the
1730 previously examined strategies, having at least one modulefile per
1731 build, and in some cases multiple per build. The output of the module
1732 avail command without any module specified is rather overwhelming,
1733 although when a module is specified, module avail will provide information
1734 about what builds are available.
1735 * Because each build has a specific module, the module list command shows
1736 exactly what builds of the various modules are loaded.
1737 * The module switch command does not work very well. In
1738 particular, when one switches out a module on which other modules depend,
1739 the dependent modules are not successfully reloaded. Even with the
1740 :ref:`Automated module handling <v42-automated-module-handling-mode>`
1741 feature (introduced in ``4.2``), the dependent
1742 modules are unloaded but do not get reloaded properly since currently
1743 this feature attempts to reload based on the fully qualified name of the
1744 loaded module (which includes a dependency on the module switched out),
1745 not the specification used when the module was loaded.
1746 * In general, when the user tries to load a module based on a partially
1747 specified modulename, the ``.modulerc`` files handle it well, and the
1748 latest version of the module consistent with the specification and
1749 previously loaded modules will be used. If no compiler was previously
1750 loaded, will use the default compiler, or the latest compiler if nothing
1751 matches the default compiler.
1752 * Modules will fail with an error message if user tries to load a package
1753 which was not built for the values of the dependency packages loaded.
1754
1755
1756 .. _Modulepath-based_strategy:
1757
1758 Modulepath-based Strategy
1759 -------------------------
1760
1761 This strategy makes use of the ability of modules to support multiple
1762 directories in the ``MODULEPATH`` environment variable. Every time a module is loaded
1763 on which other modules might depend, a new path is added to ``MODULEPATH``
1764 containing the modulefiles which depend on the newly added module.
1765
1766 This is basically similar to the strategy that is used in the hierarchical
1767 modulefile approach of Lmod_.
1768
1769 .. _Lmod: https://github.com/TACC/Lmod
1770
1771 Implementation
1772 ^^^^^^^^^^^^^^
1773
1774 The :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` and
1775 :ref:`Modulerc-based <Modulerc-based_strategy>` strategies were based on
1776 modifying the modulefiles of modules which depended on other modules.
1777 The *Modulepath-based* strategy, in contrast, is based on modifying the
1778 modulefiles for modules which other modules might depend on.
1779
1780 To begin with, we set ``MODULEPATH`` to a ``Core`` directory containing
1781 modulefiles for modules which do not depend on any other modules.
1782 The modulefiles for compilers will typically be put in here.
1783 For this example, we opt not to use a dummy simd module (for reasons
1784 explained later), but instead add ``avx``, ``avx2``, ``sse4.1`` variants to the
1785 bar modules, but otherwise they might belong here. Other modules which
1786 do not depend on compilers, etc. could be place in here as well, e.g.
1787 applications which do not expose libraries, etc. So the directory
1788 structure would look like:
1789
1790 .. code-block:: console
1791
1792 Core
1793 ├── gcc
1794 │   ├── 8.2.0
1795 │   ├── 9.1.0
1796 │   └── common
1797 ├── intel
1798 │   ├── 2018
1799 │   ├── 2019
1800 │   └── common
1801 └── pgi
1802 ├── 18.4
1803 ├── 19.4
1804 └── common
1805
1806 A typical common file for the gcc compiler would be something like
1807
1808 .. literalinclude:: ../../example/compiler-etc-dependencies/modulepath/Core/gcc/common
1809 :language: tcl
1810
1811 The most interesting aspect is the ``module use`` at the end. We add
1812 to ``MODULEPATH`` a directory under ``Compiler`` for this specific compiler
1813 and version. Other compilers/versions would add their own specific path
1814 to ``MODULEPATH``.
1815
1816 In each of these compiler specific directories, modulefile directory
1817 trees exist for the supported MPI libraries and bar. We also include
1818 a foo directory containing the non-MPI variants of foo. It would look
1819 like:
1820
1821 .. code-block:: console
1822
1823 Compiler
1824 ├── gcc
1825 │   ├── 8.2.0
1826 │   │   ├── bar
1827 │   │   │   ├── 4.7
1828 │   │   │   │   ├── avx
1829 │   │   │   │   ├── .modulerc -> symlink to modulerc.default_lowest_simd
1830 │   │   │   │   └── sse4.1
1831 │   │   │   └── common -> symlink to bar/common
1832 │   │   ├── foo
1833 │   │   │   ├── 1.1
1834 │   │   │   └── common -> symlink to foo/common
1835 │   │   ├── mvapich
1836 │   │   │   ├── 2.1
1837 │   │   │   └── common -> symlink to mvapich/common
1838 │   │   └── openmpi
1839 │   │   ├── 3.1
1840 │   │   └── common -> symlink to openmpi/common
1841 │   └── 9.1.0
1842 │   ├── bar
1843 │   │   ├── 5.4
1844 │   │   │   ├── avx
1845 │   │   │   ├── avx2
1846 │   │   │   └── .modulerc -> symlink to modulerc.default_lowest_simd
1847 │   │   └── common -> symlink to bar/common
1848 │   ├── foo
1849 │   │   ├── 2.4
1850 │   │   └── common -> symlink to foo/common
1851 │   ├── mvapich
1852 │   │   ├── 2.1
1853 │   │   ├── 2.3.1
1854 │   │   └── common -> symlink to mvapich/common
1855 │   └── openmpi
1856 │   ├── 3.1
1857 │   ├── 4.0
1858 │   └── common -> symlink to openmpi/common
1859 ├── intel
1860 │   ├── 2018
1861 │   │   └── ... modules depending on compiler=intel/2018
1862 │   └── 2019
1863 │   └── ... modules depending on compiler=intel/2019
1864 └── pgi
1865 ├── 18.4
1866 │   └── ... modules depending on compiler=pgi/18.4
1867 └── 19.4
1868 └── ... modules depending on compiler=pgi/18.4
1869
1870 For brevity, we only show the directory structure for the modules depending on
1871 the two versions of gcc in detail; the other compilers will have similar
1872 structures.
1873
1874 Basically, there is a separate modulepath for each compiler, containing only the
1875 modulefiles depending on that specific build of the compiler (and not also depending
1876 on something else, like MPI library).
1877 This certainly enforces the consistency of loaded modules; one could not load
1878 a specific version of gcc (say ``gcc/9.1.0``) and an incompatible version of foo (e.g. ``foo/1.1``),
1879 because all of the foo modulefiles are in compiler specific module trees and there is
1880 no ``foo/1.1`` in the ``gcc/9.1.0`` module tree. Conflict statements in the compiler modulefiles
1881 will prevent one from loading multiple compilers, thereby preventing multiple compiler
1882 specific modulepaths (unless the user explicitly does a ``module use`` or similar, and
1883 there is only so far one can go in preventing users from shooting themselves in the foot).
1884
1885 The modulefiles for foo and bar are fairly straightforward; we have a common file which
1886 does the heavy lifting (and since this can be made independent of version and compiler, this
1887 is actually a symlink to a package specific common file in a ``common`` directory external
1888 to the modulepath trees).
1889 We then add some small stubfiles which set variables for the specific build information, which
1890 mostly are determined by their position in the tree structure (e.g. the stubfiles under
1891 ``Compiler/gcc/9.1.0`` are all going to set ``compilerTag`` to ``gcc/9.1.0``, etc), and
1892 then invoke the common file.
1893
1894 The modulefiles for openmpi are largely similar. E.g., for ``gcc/9.1.0`` we have a small
1895 stubfile like the following (for version ``4.0``)
1896
1897 .. literalinclude:: ../../example/compiler-etc-dependencies/modulepath/Compiler/gcc/9.1.0/openmpi/4.0
1898 :language: tcl
1899
1900 which defines the version and compilerTag variables for the OpenMPI version and
1901 compiler version, and then invokes the common script
1902
1903 .. literalinclude:: ../../example/compiler-etc-dependencies/modulepath/Compiler/gcc/9.1.0/openmpi/common
1904 :language: tcl
1905
1906 which does the usual stuff (define a help function, whatis string, and sets assorted environment
1907 variables like ``PATH``, ``LIBRARY_PATH``, etc. for using OpenMPI), and then adds another directory
1908 to the ``MODULEPATH``. This new directory, under ``CompilerMPI`` is for modulefiles which
1909 depend on the compiler AND the MPI library. (It is assumed that all modulefiles depending on
1910 MPI libraries also depend on the compiler). This new branch in the modulepath would have
1911 a structure like
1912
1913 .. code-block:: console
1914
1915 CompilerMPI
1916 ├── gcc
1917 │   ├── 8.2.0
1918 │   │   ├── mvapich
1919 │   │   │   └── 2.1
1920 │   │   │   └── foo
1921 │   │   │   ├── 1.1
1922 │   │   │   └── common -> symlink to foo/common
1923 │   │   └── openmpi
1924 │   │   └── 3.1
1925 │   │   └── foo
1926 │   │   ├── 1.1
1927 │   │   └── common -> symlink to foo/common
1928 │   └── 9.1.0
1929 │   ├── mvapich
1930 │   │   └── 2.3.1
1931 │   │   └── ... modules depending on gcc/9.1.0 and mvapich/2.3.1
1932 │   └── openmpi
1933 │   └── 4.0
1934 │   └── ... modules depending on gcc/9.1.0 and openmpi/4.0
1935 ├── intel
1936 │   ├── 2018
1937 │   │   ├── intelmpi
1938 │   │   │   └── default
1939 │   │   │   └── ... modules depending on intel/2018 and its included MPI lib
1940 │   │   ├── mvapich
1941 │   │   │   └── 2.1
1942 │   │   │   └── ... modules depending on intel/2018 and mvapich/2.1
1943 │   │   └── openmpi
1944 │   │   └── 3.1
1945 │   │   └── ... modules depending on intel/2018 and openmpi/3.1
1946 │   └── 2019
1947 │   ├── intelmpi
1948 │   │   └── default
1949 │   │   └── ... modules depending on intel/2019 and its included MPI lib
1950 │   ├── mvapich
1951 │   │   └── 2.3.1
1952 │   │   └── ... modules depending on intel/2019 and mvapich/2.3.1
1953 │   └── openmpi
1954 │   └── 4.0
1955 │   └── ... modules depending on intel/2019 and openmpi/4.0
1956 └── pgi
1957 ├── 18.4
1958 │   ├── mvapich
1959 │   │   └── 2.1
1960 │   │   └── ... modules depending on pgi/18.4 and mvapich/2.1
1961 │   └── openmpi
1962 │   └── 3.1
1963 │   └── ... modules depending on pgi/18.4 and openmpi/3.1
1964 └── 19.4
1965 └── openmpi
1966 ├── 3.1
1967 │   └── ... modules depending on pgi/19.4 and openmpi/3.1
1968 └── 4.0
1969 └── ... modules depending on pgi/19.4 and openmpi/3.1
1970
1971 Basically, there is a separate modulepath for each combination of compiler and
1972 MPI library, listing all modulefiles depending on that compiler and MPI library.
1973 The modulefiles underneath each directory simply do what is needed to load that
1974 version of the package built with the specified compiler and MPI library.
1975
1976 This process could be continued further, as needed. E.g., if you had packages which
1977 depended on NetCDF, and you had multiple builds of NetCDF for a given compiler/MPI
1978 combination, you could add another modulepath tree, e.g. ``CompilerMPINetCDF``,
1979 branching on each compiler, MPI, and NetCDF triplet. But things can also get
1980 unwieldy if carried too far.
1981
1982 One could also add modulepath trees for disjoint features like SIMD level,
1983 but this turns out to be a bit tricky. E.g., if you had a simd branch as well
1984 as a compiler branch, with both simd and compiler appearing in Core, things are
1985 fine for the Simd and Compiler trees. However, if there were to be modules depending
1986 on both, e.g. a CompilerSimd branch, then because compiler and simd are disjoint
1987 and could be loaded in either order, the modulefiles for both would need to handle
1988 adding the CompilerSimd branch depending on whether the other was previously loaded.
1989
1990 Examples
1991 ^^^^^^^^
1992
1993 We now look at the example usage for the *Modulepath-based* strategy.
1994 To use these examples, you must:
1995
1996 #. Set (and export) ``MOD_GIT_ROOTDIR`` to where you git-cloned the modules source
1997 #. Do a ``module purge``
1998 #. Set your ``MODULEPATH`` to ``$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Core``
1999
2000 Note that we set the ``MODULEPATH`` to the Core subdirectory; the Core branch is for those modulefiles
2001 that do not depend on compiler or MPI library, and that should be available from the start.
2002
2003 As with the previous cases, we start with a ``module avail`` command, and here
2004 we see the first big difference:
2005
2006 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/modavail.out
2007 :language: console
2008
2009 When we first do a ``module avail``, we only see the modulefiles for the compilers.
2010 The MPI libraries, foo, bar, etc. all depend on at least the compiler, and so
2011 will not become "available" until a compiler (and possibly MPI library as well) is loaded.
2012 In a production environment there would likely be other modulefiles available in Core
2013 (i.e. an application which is only used as a standalone application and does not
2014 provide an API/libraries to link against would likely be placed in Core), but in our
2015 simple example, only the compilers appear in Core. As we load a compiler and then an
2016 MPI library, additional modulefiles appear available. In a production environment, one
2017 might wish to set things up so that a compiler (and maybe MPI library) modulefile is
2018 automatically loaded when the user logs in.
2019
2020 The downside of this is that it becomes difficult for an user to know what software
2021 is available, at least via the module command. I.e., if application foobar is only built
2022 for a single compiler/MPI combination, it will not show up in module avail unless that
2023 specific compiler/MPI combination were previously loaded. Lmod_ adds a ``module spider``
2024 command which allows the user to list all packages installed for any compiler/MPI
2025 combination, and can be used to also find out which compiler/MPI combinations are needed
2026 to access a specific modulefile, but Environment Modules does not have a comparable function
2027 at this time. If you were to use this strategy in a production environment, you would
2028 likely need to generate lists of available packages (and their compiler/MPI/etc dependencies)
2029 in a web page or other documentation area which can be frequently updated.
2030
2031 The standard functionality of selecting the correct build of a package
2032 based on the loaded compiler, e.g.
2033
2034 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/ompi-loads1.out
2035 :language: console
2036
2037 works as expected. As shown in the last command above, when requesting a module not
2038 built for the loaded compiler (e.g. ``openmpi/4.0`` when ``gcc/8.2.0`` is loaded), the module command
2039 simply returns ``Unable to locate a modulefile`` as there is no corresponding modulefile
2040 in the current list of MODULEPATHS.
2041
2042 The functionality of the ``module switch`` command depends on version and/or settings
2043 of Environment Modules. For version 3.x, or 4.x with the :ref:`Automated module handling <v42-automated-module-handling-mode>`
2044 feature disabled, it does not work well, modules which depend on the switched out module
2045 are not reloaded, leading to inconsistent environments. But for 4.x versions with
2046 the automated module handling feature enabled (as indicated by the --auto after the
2047 switch, although that can be made the default), it works as expected. Modules depending
2048 on the module being switched out get unloaded and reloaded if possible, as shown below:
2049
2050 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/ompi-switch.out
2051 :language: console
2052
2053 The modulefiles themselves basically have no support for defaulting a compiler; the modulefiles for
2054 the various MPI libraries are simply not even available until a modulefile for a compiler is loaded, as
2055 seen below:
2056
2057 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/ompi-defaults.out
2058 :language: console
2059
2060 However, this could be somewhat mitigated by having the modulefile for the default compiler automatically
2061 loaded for the user upon login (e.g. in their dot files).
2062
2063 The situation is similar for foo:
2064
2065 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/foo-loads.out
2066 :language: console
2067
2068 As expected, the correct version of foo is loaded depending on the previously
2069 loaded compiler and MPI libraries. Here again we use an ``intelmpi`` modulefile
2070 to indicate when we wish to use Intel MPI libraries, even though they are enabled
2071 by the ``intel`` module. The ``intelmpi`` modulefile basically just adds the modulepath
2072 for the intel-compiler and intelmpi dependent modules.
2073
2074 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/foo-switch.out
2075 :language: console
2076
2077 Note again the same deficiency in the switch report as the :ref:`Homebrewed_flavors_strategy`
2078 had; in the last case above wherein we loaded foo with ``intel/2019`` loaded
2079 but no MPI module. As expected, a non-MPI build of foo was loaded. However, when an
2080 openmpi module is subsequently loaded, foo does not get reloaded and we still have
2081 the non-MPI build of foo, as evidenced by the output of the foo command. And that
2082 this fact is not obvious from the module list output.
2083
2084 The behavior when defaulting is nicer. Without any compiler or MPI libraries
2085 Here we note a deficiency in the switch support as compared to :ref:`Flavours <Flavours_strategy>`. In the last example
2086 after loading ``intel/2019`` and foo, we have the non-MPI build of foo as expected. However, upon
2087 subsequently loading the openmpi module, we still have the non-MPI version of foo loaded, as evidenced
2088 by the output of the dummy foo command. I.e., the foo package was *not* automatically reloaded, as
2089 there was no prereq in the foo modulefile on an MPI library (as in the non-MPI build there is no MPI
2090 library to prereq). Also note that module list does not really inform one of this fact.
2091
2092 The ability to default partial modulenames in this strategy is mixed. Without any compiler loaded,
2093 most modulefiles are not even visible/available. However, once a compiler is loaded, trying to
2094 load a module without specifying the version will end up loading the latest version compatible
2095 with the loaded compiler (as no incompatible versions are visible) as seen below:
2096
2097 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/foo-defaults.out
2098 :language: console
2099
2100 This is better than in the :ref:`Flavours <Flavours_strategy>` or :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` strategies. If one were to use
2101 this strategy in production, it is recommended to have a default compiler module (and maybe even
2102 an MPI library, etc) automatically loaded when the user logs in, thereby allowing for a reasonable defaulting
2103 ability (and more reasonable module avail output).
2104
2105 The situation with bar is similar. We do not have a dummy simd module,
2106 so the builds with different CPU vectorization support are specified by
2107 appending ``/avx``, etc. to the bar package name. As with previous strategies,
2108 if one attempts to load a simd variant which was not built for the compiler
2109 loaded, an error will occur.
2110
2111 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/bar-loads.out
2112 :language: console
2113
2114 Defaulting is handled well, as shown
2115
2116 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/bar-defaults.out
2117 :language: console
2118
2119 In particular, one case specify ``bar/avx2`` or ``bar/avx`` or ``bar/sse4.1`` and the
2120 latest version of bar consistent with the specification and any previously
2121 loaded compiler (or default compiler) will be loaded
2122
2123 Summary of Modulepath-based strategy
2124 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2125
2126 * It is quite difficult for an user to get an inconsistent environment,
2127 at least when automated module handling mode is used. The modulepaths are
2128 managed so that only modulefiles consistent with the previously loaded
2129 compiler/MPI library are visible.
2130 * With automated module handling mode, the module switch command is very nicely
2131 supported.
2132 * The module command does not lend itself to readily seeing all the packages
2133 which are installed on the system, nor seeing for which compiler/MPI combinations
2134 a certain package is built. The module avail command just does not show
2135 any modules not compatible with the currently loaded compiler and/or MPI
2136 libraries. Lmod needed to add a ``module spider`` command to address this,
2137 but no such functionality currently exists in Environment Modules.
2138 If one were to use this in production, you would need to provide something
2139 similar to the Lmod spider sub-command, or at least provide frequently
2140 updated web pages or similar with this information.
2141 * This strategy involves the use of many more modulepaths than the
2142 previously examined strategies, having at least one modulepath per
2143 compiler installed, and typically also one for each compiler/MPI library
2144 combination. If you add additional layers of dependency, things get even
2145 more complicated, which can be limiting. It is especially problematic if
2146 you have two disjoint dependencies -- i.e. neither dependencies is dependent on
2147 the other; in this case you might load one or the other, or both (in either
2148 order). If there are modulefiles dependent on both of these dependencies, the
2149 last one loaded (and *only* the last one) needs to handle adding the appropriate
2150 modulepath for the modulefiles depending on both.
2151 * In general, a module will fail to load with an error message
2152 if any dependent module is not already loaded. The error will actually
2153 state that the module cannot be found, as the modulepath containing it will
2154 not be added to the ``MODULEPATH`` until the modules depended on are loaded.
2155 So if you need to default a compiler, etc., you will need to have the
2156 user's initialization scripts automatically load the appropriate
2157 modulefile upon login.
2158 * However, what we have been referring to as "more intelligent defaulting" is
2159 effectively supported. I.e., if a user requests to load a package without
2160 specifying the version desired, the version will be defaulted to the latest
2161 version *compatible* with the currently loaded compiler and other dependencies.
2162 This is because the incompatible versions are not visible in the module tree.
2163
2164
2165 Comparison of Strategies
2166 ------------------------
2167
2168 All of the strategies discussed above have their peculiar strengths and weaknesses.
2169 The decision of which strategy to use will depend on how these strengths and weaknesses
2170 impact your design goals. It is advised to play around in the sandbox environments
2171 a bit, as actual use tends to help make clear which downsides you are willing to accept
2172 and which you are not.
2173
2174 With the exception of the :ref:`Flavours <Flavours_strategy>`, which works best with Environment Modules
2175 version 3.x, all of the other strategies work as well or better
2176 on the newer 4.x versions of Environment Modules. This difference is most visible
2177 in the discussion of features around the module switch command. So in the following
2178 discussions we will assume Environment Modules version 3.x for the Flavours strategies,
2179 as that is the version it works best with. For all other strategies, we assume
2180 Environment Modules version 4.x, with automated module handling mode enabled, as these
2181 strategies work best in that scenario.
2182
2183 The :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>`,
2184 :ref:`Modulerc-based <Modulerc-based_strategy>`, and :ref:`Modulepath-based <Modulepath-based_strategy>` strategies all
2185 require a significant amount of "assembly" in order to get them working for a
2186 production environment; the example scripts and procedures provided here should
2187 help significantly, but should not be considered as a polished product. You will
2188 likely need to customize and extend for your environment. The
2189 :ref:`Flavours_strategy`, on the other hand, has been packaged to present as a finished product
2190 and so requires less "assembly" to get working, but does not appear to be actively
2191 maintained, so the reduced up-front work might be neutralized by the costs of
2192 self support.
2193
2194 Basic Dependency Handling
2195 ^^^^^^^^^^^^^^^^^^^^^^^^^
2196
2197 All of the strategies discussed support a basic level of dependency
2198 handling. If a user attempts to load a package, they get the build of the
2199 package appropriate for the previously loaded compiler or other dependencies,
2200 or, if no appropriate build is found, an error message indicating such.
2201 E.g, if the user does:
2202
2203 .. code-block:: tcl
2204
2205 module load gcc/8.2.0
2206 module load foo/1.1
2207
2208 their environment will be set up to run foo version ``1.1`` built
2209 with gcc version ``8.2.0``.
2210
2211 All of the strategies discussed meet this criterion, with both 3.x and 4.x versions
2212 of Environment Modules.
2213
2214 Advanced Dependency Handling (e.g. the module switch sub-command)
2215 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2216
2217 Things are more complicated when we allow for the modules upon which
2218 other loaded modules might depend to be changed. This generally involves
2219 the module switch command.
2220
2221 While all of the strategies handle well the easy case, when we switch
2222 a module upon which no other loaded modules depend, the trick comes
2223 when one or more currently loaded modules depend on the module being
2224 switched out. E.g., if we assume that module foo depends on the
2225 previously loaded compiler, then ideally, we would like for a sequence like:
2226
2227 .. code-block:: tcl
2228
2229 module load intel/2019
2230 module load foo
2231 module switch intel pgi/18.4
2232
2233 to have the same effective outcome as if the user issued the commands:
2234
2235 .. code-block:: tcl
2236
2237 module load pgi/18.4
2238 module load foo
2239
2240 I.e., switching out the compiler
2241 will cause foo to be reloaded with the new compiler dependency. And ideally
2242 in this example we would have ``pgi/18.4`` loaded along with the latest version
2243 of foo compatible with that compiler.
2244
2245 This is handled reasonably well with the :ref:`Flavours <Flavours_strategy>` (using Environment Modules
2246 3.x) and the :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` and
2247 :ref:`Modulepath-based <Modulepath-based_strategy>` (both using Environment
2248 Modules 4.x) strategies. For the :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` and
2249 :ref:`Modulepath-based <Modulepath-based_strategy>`
2250 strategies, this relies on the :ref:`Automated module handling <v42-automated-module-handling-mode>` feature (As of
2251 version ``4``, this is disabled by default. It can be enabled in ``modulecmd.tcl``,
2252 or by running ``module config auto_handling 1``, or by adding the ``--auto`` flag
2253 to the modules command.)
2254
2255 For these strategies, swapping out a compiler or other module upon which a loaded
2256 module depends will cause the the module with a dependency to be unloaded, the
2257 module being switched being swapped out, and the module that had a dependency being
2258 reloaded.
2259
2260 Our example sequence above actually exposes a subtle issue. For the :ref:`Flavours <Flavours_strategy>`
2261 and :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` strategies, the sequence of loading ``pgi/18.4`` followed
2262 by foo (using our example software library) would fail, as foo would be defaulted to
2263 version ``2.4`` and as there was no build of ``foo/2.4`` for ``pgi/18.4``, the module load foo
2264 would fail. And indeed, the switch sequence above would not be completely successful
2265 under either strategy: for the :ref:`Homebrewed_flavors_strategy`, the compiler would be
2266 swapped out and the reload of foo would fail with an error (so effectively
2267 equivalent to the second sequence). For the :ref:`Flavours_strategy`, it detects that
2268 it would be unable to reload foo, and the switch command fails (i.e. the compiler
2269 remains ``intel/2019``).
2270
2271 For the :ref:`Modulepath-based_strategy`, the second sequence (loading ``pgi/18.4`` then foo) would
2272 actually work, as the modulepath exposed by loading ``pgi/18.4`` only has versions of
2273 foo that are compatible with ``pgi/18.4``. But the switch command does not work
2274 as well as would be desired. This is because that as currently implemented, when the
2275 automated module handling code does the reload of the module that was unloaded
2276 due to dependencies, it attempts to reload based on the fully qualified name of the
2277 module that was loaded, and not based on the (possibly partial) name specified
2278 when the module was originally loaded. So, after the module load ``intel/2019``,
2279 the module load foo will result in ``foo/2.4`` being loaded. When we switch out ``intel/2019``
2280 for ``pgi/18.4``, foo gets unloaded, but after the compiler swap, it tries to reload
2281 ``foo/2.4``. As there is no build of ``foo/2.4`` for ``pgi/18.4`` (and thus no modulefile in
2282 the current modulepath), the module is not found and is unable to be loaded.
2283 As in the :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` case, the compiler is switched but foo is left unloaded
2284 (with an appropriate warning to that effect).
2285
2286 This same issue prevents the :ref:`Automated module handling <v42-automated-module-handling-mode>` feature as currently implemented
2287 from being very useful with the :ref:`Modulerc-based <Modulerc-based_strategy>`. Basically, whenever you have a module
2288 loaded that depends on another module, switching out the module depended upon will
2289 fail to reload the other module. This is true even in less pathological cases, like:
2290
2291 .. code-block:: tcl
2292
2293 module load intel/2019
2294 module load foo
2295 module switch intel pgi/19.4
2296
2297 which succeeds under the other three strategies (as ``foo/2.4`` *is* built for ``pgi/19.4``).
2298 This is because when foo is initially loaded in the above scenario, the actual modulename
2299 loaded is ``foo/intel/2019/2.4``. With the switch command, foo gets unloaded, the compilers
2300 are switched, and the automated modules handling code tries to reload ``foo/intel/2019/2.4``.
2301 That modulename clearly depends on ``intel/2019``, not ``pgi/19.4``, and so will fail to load
2302 (with an error message). The net result is that the compiler gets switched, but any
2303 modules depending on the compiler (or whatever module being switched) get unloaded
2304 (with a warning to that effect). It is hoped that a future version of the
2305 :ref:`Automated module handling <v42-automated-module-handling-mode>` feature will have an option to reload the dependent
2306 modules using the modulename they were loaded with; this should allow for the
2307 switch command to work well in the :ref:`Modulerc-based_strategy`, as well as make
2308 edges case (like the ``pgi/18.4`` example discussed above) work better with the
2309 :ref:`Modulepath-based_strategy`.
2310
2311 There is another edge case which only the :ref:`Flavours_strategy` handles well. Consider a
2312 scenario like:
2313
2314 .. code-block:: tcl
2315
2316 module purge
2317 module load intel/2019
2318 module load foo
2319 foo
2320 # this should be foo/2.4 built for intel/2019 without MPI
2321 module load openmpi
2322 foo
2323 # What build is this, with or without MPI?
2324
2325 With the :ref:`Flavours_strategy`, the final foo will be built for ``intel/2019``
2326 with openmpi support; i.e. Flavours supports the concept of *optional* prerequisites, and foo has
2327 an optional prerequisite on the MPI libraries, so when openmpi is loaded, any previously loaded modules
2328 which depend on it (through a regular or optional prerequisite) get reloaded.
2329
2330 All of the other strategies rely on automated module handling mode for automatic reloads, and
2331 in this case no reload of foo will be initiated as foo does not have
2332 MPI libraries listed as a prerequisite (otherwise a no MPI version could not be loaded). This is especially
2333 problematic for the :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` and
2334 :ref:`Modulepath-based <Modulepath-based_strategy>` strategies, since it is not easily to
2335 tell from a module list or similar command which version of foo the environment is set for (for the
2336 :ref:`Modulerc-based_strategy`, the full modulename from module list will indicate that).
2337
2338 However, despite some potential issues with some edge cases, the :ref:`Flavours <Flavours_strategy>`,
2339 :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>`, and
2340 :ref:`Modulepath-based <Modulepath-based_strategy>` strategies all handle this challenge well.
2341 The :ref:`Modulerc-based <Modulerc-based_strategy>`
2342 approach has a poor showing (although at least the modules depending on the module switched
2343 out will be unloaded, so the set of loaded modules is consistent).
2344
2345 Note also that this topic shows the most dependence on the version of Environment
2346 Modules. For the :ref:`Flavours_strategy`, the switching of a module upon which
2347 other modules depended does not work (the wrapper script returned weird errors). The
2348 other strategies all rely on the :ref:`Automated module handling <v42-automated-module-handling-mode>` feature to enable the unload
2349 and reload of modules which depend on the module being switched out. Thus for 3.x versions
2350 of Environment Modules, the switch of a compiler, etc. will only result in the compiler being replaced, and any
2351 modules which depend on the compiler will not be reloaded. This means they will still be
2352 pointing to versions built for the old compiler, leading to an inconsistent set of
2353 modules being loaded. This is particularly bad in the
2354 :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` case, as a module
2355 load will not even inform one of that fact. Even if automated module handling mode is disabled
2356 on version 4.x of Environment Modules situation is better as loaded environment consistency
2357 :ref:`is assured<v42-prereq-constraints-consistency>`: the switch of a compiler will be denied
2358 since a dependent module is detected loaded.
2359
2360 Visibility into what packages are available
2361 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2362
2363 Another set of criteria to weigh has to do with visibility into the available
2364 packages. We are interested in
2365
2366 #. Viewing all of the packages installed on a system
2367 #. Determining what versions of a specific package are available for a
2368 given compiler/MPI/etc combination.
2369 #. Seeing what compiler/MPI/etc combinations a specific version of a
2370 package
2371 #. For packages that we have currently loaded, determining which compiler/MPI/etc
2372 they were built for.
2373
2374 In terms of seeing all of packages that are available, the :ref:`Flavours <Flavours_strategy>` and
2375 :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` strategies
2376 are probably the best. The ``module avail`` command will list all modulefiles available, which will
2377 typically just be a list of packagename/version for each installed version. On a production system
2378 with many packages installed, even this can be a bit overwhelming to the user.
2379
2380 The ``module avail`` command will also list all modulefiles for every installed package in the
2381 :ref:`Modulerc-based_strategy`, but here every build of every version of every package will have a distinct modulefile. I.e.,
2382 you will not just have ``foo/1.1`` and ``foo/2.4`` listings, but ``foo/1.1/gcc/8.2.0/nompi``, ``foo/1.1/gcc/8.2.0/mvapich/2.1``,
2383 ``foo/1.1/gcc/8.2.0/openmpi/3.1``, ``foo/1.1/intel/2018/nompi``, ``foo/1.1/intel/2018/intelmpi``, etc. modulefiles.
2384 Actually, the situation is even worse than that, as in this strategy there are often multiple
2385 modulefiles for the same build in order to ease navigation of the module tree, e.g. we could have
2386 ``foo/1.1/gcc/8.2.0/nompi`` and ``foo/gcc/8.2.0/nompi/1.1`` representing the same build of the same package.
2387 Whereas an unqualified ``module avail`` in the :ref:`Flavours <Flavours_strategy>` or
2388 :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` strategies can be a bit overwhelming
2389 in a large environment, with the :ref:`Modulerc-based_strategy` it can become practically unusable.
2390
2391 While an unqualified ``module avail`` in the :ref:`Flavours <Flavours_strategy>`,
2392 :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>`, and especially :ref:`Modulerc-based <Modulerc-based_strategy>` strategies
2393 can inundate the user with modulenames, the :ref:`Modulepath-based_strategy` has the opposite problem.
2394 With the :ref:`Modulepath-based_strategy` strategy, the modulefiles are split across multiple, often mutually incompatible, modulepaths, so the module avail
2395 command will never return a list of all modulefiles installed, only those available given the previously loaded
2396 compiler/MPI libraries/etc. E.g., if a package foobar is only installed for a particular compiler/MPI combination,
2397 it will not appear in any module avail listing unless that particular compiler and MPI were previously loaded.
2398
2399 While the number of modulefiles listed in an unqualified ``module avail`` command in the :ref:`Modulerc-based_strategy`
2400 is unwieldy, if one adds a partial package specification argument, the number of modulefiles returned is greatly
2401 reduced. This smaller list can provide information about which compilers/MPI/etc. are compatible with
2402 a specific version of a package. For example, to see the compilers for which ``foo/2.4`` is built, we can
2403 do something like:
2404
2405 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/foo-avail1.out
2406 :language: console
2407
2408 Similarly, to see the builds of foo using gcc compilers, one can do something like:
2409
2410 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/foo-avail2.out
2411 :language: console
2412
2413 The other strategies do not readily provide that information. The next best case is the
2414 :ref:`Modulepath-based_strategy`, because here at least a module avail will tell you what versions
2415 of a package are compatible with the currently loaded compiler/MPI/etc, e.g.:
2416
2417 .. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/foo-avail1.out
2418 :language: console
2419
2420 But the :ref:`Flavours <Flavours_strategy>` and :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>`
2421 strategies do not readily show what versions of packages are built
2422 for which compilers, etc. You would need to load a compiler/MPI/etc combination, and then try loading
2423 a particular version of a package.
2424
2425 Lastly, with the :ref:`Modulerc-based_strategy`, the module list command explicitly shows information
2426 about the compiler/MPI/etc used to build the loaded modules, as that is part of the full modulename.
2427 This information is not explicitly visible in the other three strategies, but that is usually not
2428 a problem. As long as the various user commands result in a set of modules wherein every module depending
2429 on either a compiler or MPI library, etc. is built for the currently loaded compiler/MPI library/etc.,
2430 that information is redundant.
2431
2432 The only cases where this is potentially an issue are the sort of edge cases described in the previous
2433 section, e.g. if one were to do something like:
2434
2435 .. code-block:: tcl
2436
2437 module purge
2438 module load intel/2019
2439 module load foo
2440 foo
2441 # this should be foo/2.4 built for intel/2019 without MPI
2442 module load openmpi
2443 foo
2444 # What build is this, with or without MPI?
2445
2446 In the above example, for the :ref:`Flavours <Flavours_strategy>`,
2447 :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>`, and :ref:`Modulepath-based <Modulepath-based_strategy>` strategies a
2448 module list at the end would simply list something like:
2449
2450 .. code-block:: shell
2451
2452 Currently Loaded Modulefiles:
2453 1) intel/2019 2) foo/2.4 3) openmpi/4.0
2454
2455 For the :ref:`Flavours_strategy`, foo would be built with MPI support, but for the other
2456 two, foo would still be the non-MPI build, which is not readily apparent from the
2457 above output (although possibly could be inferred by the ordering of the modules).
2458
2459 Ease of Navigating the Module Tree
2460 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2461
2462 The final criterion to be discussed is the ease of navigating the module tree.
2463 Ideally, we would like it so that when an user gives a partial modulename the
2464 system resolves in a sensible manner. Typically a partial modulename will be
2465 a package name without a version, in which case ideally the latest version of
2466 the package (compatible with previously loaded compiler, etc), should be loaded.
2467 And indeed, for a simple case like:
2468
2469 .. code-block:: tcl
2470
2471 module purge
2472 module load intel
2473 module load foo
2474
2475 all the strategies do just that. The intel module will default to ``intel/2019``,
2476 and foo to version ``2.4`` built for ``intel/2019``. However, with even a slight modification
2477 things begin to diverge, e.g.:
2478
2479 .. code-block:: tcl
2480
2481 module purge
2482 module load intel/2018
2483 module load foo
2484
2485 In this second case, the :ref:`Flavours <Flavours_strategy>` and
2486 :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` strategies still default
2487 the version of foo to version ``2.4``, despite there being no build of ``foo/2.4`` for ``intel/2018``.
2488 That is because they do not do intelligent defaulting, and will simply default
2489 the version of foo to the highest version they see
2490 even though it is not compatible with the loaded compiler.
2491 Thus the load of foo will fail for :ref:`Flavours <Flavours_strategy>` and
2492 :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` strategies.
2493 However, for the :ref:`Modulerc-based <Modulerc-based_strategy>` and
2494 :ref:`Modulepath-based <Modulepath-based_strategy>` strategies, the module load of foo
2495 will successfully load foo version ``1.1`` built for ``intel/2018``. For the :ref:`Modulepath-based_strategy`,
2496 this is because the only foo modules currently visible are compatible with the
2497 loaded ``intel/2018`` compiler, and it simply defaults to the highest version seen. For
2498 the :ref:`Modulerc-based_strategy`, the ``.modulerc`` file directly under foo directs it down the
2499 subdirectory for the intel compiler family, which has a ``.modulerc`` defaulting to the 2018 subdirectory
2500 (as the currently loaded compiler version), which has a ``.modulerc`` defaulting to the nompi
2501 subdirectory (the MPI family as no MPI library was loaded). At this point, there is not
2502 another ``.modulerc`` file, so the latest version will be defaulted to, and
2503 ``foo/intel/2018/nompi/1.1`` module will be loaded.
2504
2505 The situation is analogous when there are chained dependencies (e.g. compiler and MPI
2506 libraries); the :ref:`Flavours <Flavours_strategy>` and :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>`
2507 strategies will default to the latest
2508 version of the package they know about, regardless of whether it is compatible with the
2509 loaded environment. If it is compatible, it will be loaded and all is well, if not the
2510 load command will fail with an error message. Not horrible; the set of loaded modules is
2511 still consistent, but not terribly user-friendly either.
2512
2513 For the :ref:`Modulerc-based <Modulerc-based_strategy>` and :ref:`Modulepath-based <Modulepath-based_strategy>`
2514 strategies, the latest build compatible with the various dependencies will be loaded (or if no build is
2515 compatible, the load will fail with an appropriate error).
2516
2517 For the :ref:`Flavours <Flavours_strategy>`, :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>`,
2518 and :ref:`Modulepath-based <Modulepath-based_strategy>` strategies, typically modulenames
2519 only consist of the package name and version components, so generally only partial modulenames
2520 seen are packages without a version. The :ref:`Modulerc-based_strategy` has much longer modulenames,
2521 and therefore a greater variety of partial modulenames, which can allow for interesting
2522 possibilities. For example, one could do ``module load foo/pgi``, which (assuming no compiler
2523 was previously loaded), would load the latest version of foo built with the latest version of
2524 the pgi compiler. The module command will start looking for a modulefile in the
2525 ``foo/pgi`` subdirectory of the foo module structure,
2526 where the ``.modulerc`` file will attempt to default to the version of pgi which was loaded, but as
2527 no compiler was loaded, the modulerc file will terminate without setting a default, and the
2528 modulecmd will default to the latest version, and descend into that directory (``19.4``). Here,
2529 the ``.modulerc`` will attempt to default to the MPI family, but since none is loaded (as that
2530 would require a compiler to have been loaded), it defaults to the nompi directory, and then
2531 to the latest version in the nompi subdirectory. So the module ``foo/pgi/19.4/nompi/2.4`` will
2532 get loaded, which will also cause ``pgi/19.4`` to get loaded. Should we have issued that module
2533 load after having loaded a non-pgi compiler, the same module file would get loaded, but it
2534 would detect an incompatible previously loaded compiler and abort with an error. Should a
2535 pgi compiler have been previously loaded, we would have gotten the latest version of foo built
2536 for that compiler (and possibly with any previously loaded MPI library).
2537
2538 The above is supported because we create a ``PACKAGE/COMPILER_FAMILY/COMPILER_VERSION/.../PKGVERSION``
2539 tree in the package's modulefile directory for packages depending on the compiler. Possibly
2540 with additional components to the modulename for MPI libraries, etc. These are created
2541 to allow for the more basic ``module load foo`` case, but because the module path is
2542 descended component by component, it works if you jump in in the middle as well
2543 (and will fail if you try something stupid like :
2544
2545 .. code-block:: tcl
2546
2547 module load intel/2019
2548 module load foo/pgi
2549
2550 But a more interesting case is when there are somewhat disjoint options on the builds.
2551 Consider the case of the bar package from the examples, which in addition to the compiler
2552 and builds supporting different SIMD levels. For the :ref:`Flavours <Flavours_strategy>` and
2553 :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>`
2554 strategies, we handled this with a pseudo-simd package and modules. This worked, and allowed
2555 for one to select a build that matched both compiler and SIMD level, but you likely need to
2556 manually select a bar version compatible with both of these, as these strategies will not
2557 do intelligent defaulting of versions.
2558
2559 For the :ref:`Modulepath-based_strategy`, we added the SIMD label as a final component to the
2560 modulename, but the defaulting was somewhat simple (e.g. always default to lowest SIMD).
2561
2562 With the :ref:`Modulerc-based_strategy`, we added an additional naming scheme for the bar modulefiles,
2563 so they can start with bar/BARVERSION, or bar/COMPILER_FAMILY, or bar/SIMD_LEVEL. This not
2564 only makes it easier to view which versions of bar for which compilers are built with support
2565 for a given SIMD level (e.g. module avail bar/avx2), but also one can do something like:
2566
2567 .. code-block:: tcl
2568
2569 module load pgi/19.4
2570 module load bar/avx
2571
2572 to load the latest version of bar built for ``pgi/19.4`` with AVX support. While there is
2573 overhead associated with adding a new naming scheme like that (you add another set of
2574 modulefiles to bar and a new subdirectory tree), this can be done on a per package
2575 basis and when done judiciously it can be convenient for the users.
2576
2577
2578 Conclusions
2579 -----------
2580
2581 We have presented four strategies for dealing with modulefiles for packages
2582 with multiple builds depending on compiler, MPI, and/or other factors. All
2583 four strategies can deal with the basic requirement of loading of the correct
2584 build of a package depending on the previously loaded dependencies, or failing
2585 with a reasonable error message if no such build is available. They all have
2586 their own strengths and weaknesses. This document tried to present these
2587 strategies objectively and has been made to help you to evaluate how to handle
2588 this issue at your site.
2589
2590 .. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: