| 1 |
.. _compiler-etc-dependencies: |
| 2 |
|
| 3 |
Handling Compiler and other Package Dependencies |
| 4 |
================================================ |
| 5 |
|
| 6 |
When creating a collection of software (applications and libraries) |
| 7 |
for users to use, there is the problem of ensuring that the |
| 8 |
user is using the correct builds of everything. Generally, |
| 9 |
if an user is attempting to compile code making use of the system |
| 10 |
software collection, you want to ensure that the user is compiling |
| 11 |
their code with the same compiler that was used to compile the library. |
| 12 |
This tends to be particularly true of C++ and Fortran code using modules, |
| 13 |
and parallel codes using MPI libraries. |
| 14 |
|
| 15 |
As a result, in environments supporting multiple compilers, |
| 16 |
software libraries often end up with multiple installs of libraries |
| 17 |
and applications of the same version, depending on the compiler |
| 18 |
and other libraries used to build them. Sometimes there are |
| 19 |
even additional installs for variants with different threading |
| 20 |
models, number formats, level of vectorization support, etc. |
| 21 |
This cookbook describes various strategies for handling the modulefiles |
| 22 |
to support all these different builds for each package. |
| 23 |
|
| 24 |
For each strategy, we will provide an overview of how it works, |
| 25 |
and then show how an user might interact with it, usually a similar |
| 26 |
sequence for each case. We wish to explore how, and how well, each strategy |
| 27 |
succeeds in handling the multiple builds of the same version of a package, |
| 28 |
including |
| 29 |
|
| 30 |
#. Basic dependency handling: seeing how well the strategy supports |
| 31 |
the loading of the correct build of a package depending on the previously |
| 32 |
loaded dependencies. And if no appropriate build is available, they should |
| 33 |
error accordingly. |
| 34 |
#. The ``module switch`` command and more advanced dependency handling: how |
| 35 |
well the strategy supports more advanced cases. E.g. a case wherein several |
| 36 |
modules are loaded and the user replaces a module upon which other modules |
| 37 |
currently loaded depend. In general, how well the strategy prevents the |
| 38 |
user's set of loaded modules from being incompatible. |
| 39 |
#. Visibility into what packages are available. This includes being able |
| 40 |
to readily see all of the packages installed, seeing what versions of packages |
| 41 |
are available for a given compiler/MPI/etc combination, and seeing for |
| 42 |
which compiler/MPI/etc combinations a specific version of a package is available. |
| 43 |
#. How easily the user can navigate the modules for the builds. This |
| 44 |
includes how well partial modulenames (e.g. omitting version, etc) are |
| 45 |
handled by the different strategies. |
| 46 |
|
| 47 |
We then try to summarize the strengths, weaknesses, and other attributes of |
| 48 |
each strategy. We also try to discuss differences in using on older (3.x) |
| 49 |
and newer (4.x) Environment Modules versions. |
| 50 |
|
| 51 |
In addition to displaying examples for each strategy in this |
| 52 |
document, we have set up a the test environment as a playground in |
| 53 |
which you can explore. |
| 54 |
|
| 55 |
|
| 56 |
Contents |
| 57 |
-------- |
| 58 |
|
| 59 |
#. `Overview of Examples`_ |
| 60 |
#. :ref:`Flavours_strategy` |
| 61 |
#. :ref:`Homebrewed_flavors_strategy` |
| 62 |
#. :ref:`Modulerc-based_strategy` |
| 63 |
#. :ref:`Modulepath-based_strategy` |
| 64 |
#. `Comparison of Strategies`_ |
| 65 |
|
| 66 |
|
| 67 |
Overview of Examples |
| 68 |
-------------------- |
| 69 |
|
| 70 |
The examples are a bit more elaborate than in some other cookbooks, so |
| 71 |
the directory structure under ``doc/example/compiler-etc-dependencies`` |
| 72 |
is similarly more complicated. |
| 73 |
|
| 74 |
Example Software Library |
| 75 |
^^^^^^^^^^^^^^^^^^^^^^^^ |
| 76 |
|
| 77 |
For the purpose of the examples and the playground, we have created |
| 78 |
a fake example software library, rooted at the subdirectory |
| 79 |
``doc/example/compiler-etc-dependencies/dummy-sw-root`` beneath where |
| 80 |
you placed the modules source files. This software tree is intended |
| 81 |
to represent some of the features you might see in a real software |
| 82 |
tree, which supports various compiler and MPI libraries, and that has |
| 83 |
been added to over time, and not always in the most systematic way. |
| 84 |
|
| 85 |
The example software library does not contain any real code; there are |
| 86 |
dummy scripts for e.g. ``gcc``, ``mpirun``, etc. which just echo then |
| 87 |
name of the code and what version, compiler, etc. it was supposed to be |
| 88 |
built for, which is handy to show in the examples that the modulefiles |
| 89 |
are working as expected. It also shows how such a directory tree might |
| 90 |
be laid out --- the details of the layout will affect some of the code |
| 91 |
in the modulefiles, etc. The directory structure can be altered to |
| 92 |
fit your standards, but would require some minor modification to the |
| 93 |
modulefiles, etc. |
| 94 |
Note that there are also a bunch of subdirectories named ``1`` |
| 95 |
containing symlinks, these are for the strategy using the Flavours add-on and are |
| 96 |
discussed in that section. |
| 97 |
|
| 98 |
The software in the example software library consists of: |
| 99 |
|
| 100 |
* GNU compiler versions ``8.2.0`` and ``9.1.0`` |
| 101 |
* Intel Parallel Studio suite versions ``2018`` and ``2019`` (includes compilers, MPI and MKL) |
| 102 |
* PGI compiler suite versions ``18.4`` and ``19.4`` |
| 103 |
* OpenMPI version ``4.0``, built for: |
| 104 |
|
| 105 |
* ``gcc/9.1.0`` |
| 106 |
* ``intel/2019`` |
| 107 |
* ``pgi/19.4`` |
| 108 |
|
| 109 |
* OpenMPI version ``3.1``, built for: |
| 110 |
|
| 111 |
* ``gcc`` versions ``8.2.0`` and ``9.1.0`` |
| 112 |
* ``intel`` versions ``2018`` and ``2019`` |
| 113 |
* ``pgi`` versions ``18.4`` and ``19.4`` |
| 114 |
|
| 115 |
* mvapich version 2.3.1, built for: |
| 116 |
|
| 117 |
* ``gcc/9.1.0`` |
| 118 |
* ``intel/2019`` |
| 119 |
* ``pgi/19.4`` |
| 120 |
|
| 121 |
* mvapich version ``2.1``, built for: |
| 122 |
|
| 123 |
* ``gcc`` versions ``8.2.0`` and ``9.1.0`` |
| 124 |
* ``intel`` versions ``2018`` and ``2019`` |
| 125 |
* ``pgi`` versions ``18.4`` and ``19.4`` |
| 126 |
|
| 127 |
* foo version ``2.4``, built for: |
| 128 |
|
| 129 |
* ``gcc/9.1.0`` and ``openmpi/4.0`` |
| 130 |
* ``gcc/9.1.0`` and ``mvapich/2.3.1`` |
| 131 |
* ``gcc/9.1.0`` and no MPI |
| 132 |
* ``intel/2019`` and ``openmpi/4.0`` |
| 133 |
* ``intel/2019`` and ``mvapich/2.3.1`` |
| 134 |
* ``intel/2019`` and ``intelmpi`` |
| 135 |
* ``intel/2019`` and no MPI |
| 136 |
* ``pgi/19.4`` and ``openmpi/3.1`` |
| 137 |
* ``pgi/19.4`` and no MPI |
| 138 |
|
| 139 |
* foo version ``1.1``, built for: |
| 140 |
|
| 141 |
* ``gcc/8.2.0`` and ``openmpi/3.1`` |
| 142 |
* ``gcc/8.2.0`` and ``mvapich/2.1`` |
| 143 |
* ``gcc/8.2.0`` and no MPI |
| 144 |
* ``intel/2018`` and ``openmpi/3.1`` |
| 145 |
* ``intel/2018`` and ``mvapich/2.1`` |
| 146 |
* ``intel/2018`` and ``intelmpi`` |
| 147 |
* ``intel/2018`` and no MPI |
| 148 |
* ``pgi/18.4`` and ``openmpi/3.1`` |
| 149 |
* ``pgi/18.4`` and ``mvapich/2.1`` |
| 150 |
* ``pgi/18.4`` and no MPI |
| 151 |
|
| 152 |
* bar version ``5.4``, built with: |
| 153 |
|
| 154 |
* ``gcc/9.1.0`` and supporting ``avx2`` |
| 155 |
* ``gcc/9.1.0`` and supporting ``avx`` |
| 156 |
|
| 157 |
* bar version ``4.7``, built for: |
| 158 |
|
| 159 |
* ``gcc/8.2.0`` and supporting ``avx`` |
| 160 |
* ``gcc/8.2.0`` and supporting ``sse4.1`` |
| 161 |
|
| 162 |
I.e., we have 3 families of compiler suites with 2 different versions each. And |
| 163 |
two MPI families (openmpi and mvapich) with two versions each, with the |
| 164 |
most recent version only built with the latest compiler version of each family, |
| 165 |
and the older version built with both versions of each compiler family. |
| 166 |
In addition, it is assumed that the intel compiler suites include Intel's MPI |
| 167 |
library built for that compiler. The application foo depends on the compiler and optionally on MPI libraries |
| 168 |
and has two versions; the newer version *mostly* has builds for the latest compiler |
| 169 |
and MPI (for pgi it only supports the latest compiler and older openmpi), and the |
| 170 |
older version *mostly* has builds for the older compiler and MPI. The bar application |
| 171 |
depends on compiler and has variants depending on size of integers used in the API. |
| 172 |
|
| 173 |
We also assume that the ``gcc/8.2.0`` compiler is the system default; i.e. it is |
| 174 |
the compiler provided by default by the Linux distro used by the system, and |
| 175 |
therefore might potentially be available to users without loading any modules. |
| 176 |
|
| 177 |
More directories under doc/example/compiler-etc-dependencies |
| 178 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 179 |
|
| 180 |
The modulefiles for the different strategies do not play well with each |
| 181 |
other, in part because we use the same names for many of the modules |
| 182 |
between strategies. So in addition to the ``dummy-sw-root`` subdirectory, |
| 183 |
each strategy has its own modulepath tree subdirectory |
| 184 |
underneath ``doc/example/compiler-etc-dependencies``. There are some minor |
| 185 |
differences between the modulefiles for the :ref:`Modulerc-based_strategy` |
| 186 |
depending on whether Environment Modules 3.x or 4.x is being used, |
| 187 |
so we actually have two trees for that case (``modulerc3`` and ``modulerc4``). |
| 188 |
|
| 189 |
As there are a fair number of modulefiles, we make use of various tricks |
| 190 |
in the cookbook :ref:`tips-for-code-reuse` to minimize the amount of repeated |
| 191 |
code. In general, the actual modulefiles are small "stubfiles", setting one or a few |
| 192 |
Tcl variables, and then sourcing a ``common`` tcl file which does all the |
| 193 |
real work. Symlinks are used where possible to avoid duplicating files. |
| 194 |
The :ref:`Modulerc-based_strategy` also uses some complicated ``.modulerc`` files; these |
| 195 |
are fairly generic and to avoid redundancy are symlinked into the appropriate |
| 196 |
places in modulepath tree from the ``modrc_common`` directory. |
| 197 |
|
| 198 |
We also in some cases use Tcl procedures; for the sake of the |
| 199 |
examples these our sourced in the files as needed, but if one were to |
| 200 |
use the strategies needing such in production it would be better to follow |
| 201 |
the suggestions in :ref:`expose-procs-to-modulefiles` and |
| 202 |
place the required procedures in site config script. |
| 203 |
The various tcl procedures are placed in the ``tcllib`` sub-directory, outside |
| 204 |
of the modulepaths. These are actually broken up into multiple files |
| 205 |
for the purpose of this cookbook (so that smaller chunks of code can be |
| 206 |
looked at in this document). |
| 207 |
|
| 208 |
The ``example-sessions`` subdirectory contains various shell scripts |
| 209 |
used for the usage examples for each strategy (shell scripts are used |
| 210 |
because there are some slight variations required between the strategies), |
| 211 |
as well as the outputs of running such scripts. Subdirectories exist |
| 212 |
for each strategy, and beneath them for each of the two Environment |
| 213 |
Modules versions (``3.2.10`` and ``4.3.1``) used; for brevity not all of them |
| 214 |
are shown in this document, especially as the 3.x and 4.x differences |
| 215 |
are often small. In the example outputs, the Environment Modules |
| 216 |
version and the strategy being employed is indicated in the shell prompt |
| 217 |
(e.g. ``mod3-flavours`` or ``mod4 (modulerc)``). |
| 218 |
|
| 219 |
Using the playground environments |
| 220 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 221 |
|
| 222 |
Although we strive to provide a decent discussion in this cookbook, |
| 223 |
you are encouraged to try things out in the playground in order to |
| 224 |
get a better feel for things. |
| 225 |
|
| 226 |
Because we use some modulefile names (e.g. gcc, intel, pgi, openmpi, etc) |
| 227 |
that likely are present on your system as well, it is recommended |
| 228 |
that if you wish to explore the playground environment that you |
| 229 |
spawn a new shell, do a "module purge", and then set your ``MODULEPATH`` |
| 230 |
environment variable appropriately for the specific strategy. |
| 231 |
|
| 232 |
The `Flavours_strategy`, as will be discussed, requires some modifications |
| 233 |
to your Environment Modules installation. It is recommended that |
| 234 |
you you make a copy or new installation (for Flavours, a 3.x install |
| 235 |
works best), and then spawn a new shell and initialize the new Flavours |
| 236 |
install in that first. Flavours code is not provided with this cookbook. |
| 237 |
|
| 238 |
Some of the modulefiles, etc. require knowledge of where they were |
| 239 |
installed. To avoid requiring you to update lines in numerous |
| 240 |
files, we require you to set the environment variable |
| 241 |
``MOD_GIT_ROOTDIR`` to location where the modules git working directory |
| 242 |
was cloned. E.g., if you issued the command |
| 243 |
``git clone https://github.com/envmodules/modules.git ~/modules.test`` |
| 244 |
you should set ``MOD_GIT_ROOTDIR`` to ``~/modules.test``. Please |
| 245 |
ensure it is exported (use ``setenv`` in csh and related shells, |
| 246 |
or ``export`` in Bourne derived shells like bash). This is just |
| 247 |
a hack to make the examples work better; if you opt to use one of |
| 248 |
these strategies in production, you will want to hard code some |
| 249 |
relevant paths; the comments in the modulefiles will describe |
| 250 |
what needs to be done. |
| 251 |
|
| 252 |
Some more detail on setting up the playground is given at the |
| 253 |
start of the Examples section for each strategy. |
| 254 |
|
| 255 |
.. _Flavours_strategy: |
| 256 |
|
| 257 |
Flavours Strategy |
| 258 |
----------------- |
| 259 |
|
| 260 |
The Flavours strategy uses the `Flavours extension`_ to Environment Modules. |
| 261 |
Unlike the other strategies discussed, this requires the separate download and |
| 262 |
installation of an extension to Environment Modules. |
| 263 |
|
| 264 |
.. _Flavours extension: https://sourceforge.net/projects/flavours/ |
| 265 |
|
| 266 |
Installation and Implementation |
| 267 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 268 |
|
| 269 |
More details can be found at the website for this extension, but to install this |
| 270 |
you basically just need to: |
| 271 |
|
| 272 |
#. Clone the git repo somewhere (``git clone https://git.code.sf.net/p/flavours/code flavours-code``) |
| 273 |
#. Rename the standard Environment Modules ``modulecmd`` file (in the ``bin`` subdirectory |
| 274 |
under the installation root) to ``modulecmd.wrapped``. (It is recommended that you do this |
| 275 |
in a copy of your production installation, or better yet, in a new install of the 3.x Environment |
| 276 |
Modules (as Flavours has been developed for Modules 3.x)) |
| 277 |
#. Copy the ``modulecmd.wrapper`` file from Flavours to the ``bin`` subdirectory above. Make sure |
| 278 |
the ``modulecmd.wrapper`` file is executable. |
| 279 |
#. Symlink ``modulecmd.wrapper`` to ``modulecmd`` |
| 280 |
#. Edit ``modulecmd.wrapper`` where indicated to give fully qualified path to ``modulecmd.wrapped`` |
| 281 |
#. Copy the ``flavours.tcl`` and ``pkgIndex.tcl`` files to some (possibly new) directory under |
| 282 |
the modules installation root, and set ``TCLLIBPATH`` to that directory (you probably will want to |
| 283 |
add that to the various modules init scripts) |
| 284 |
|
| 285 |
The ``module`` command invokes ``modulecmd``, which in this case is results |
| 286 |
in the Flavours wrapper bash script ``modulecmd.wrapper`` being invoked. This |
| 287 |
calls the renamed standard ``modulecmd.wrapped`` command. This wrapper |
| 288 |
command catches and processes certain output from the modulefile evaluation |
| 289 |
intended for its consumption. |
| 290 |
|
| 291 |
The modulefiles themselves make use of various commands in the Tcl module ``flavours``. |
| 292 |
Many of these are just flavours variants of standard modulefile commands, e.g. |
| 293 |
``flavours prepend-path`` versus ``prepend-path``. Some important flavours |
| 294 |
commands: |
| 295 |
|
| 296 |
* ``package require flavours``: This loads the Tcl package flavours, and should occur |
| 297 |
near the top of your modulefile |
| 298 |
* ``flavours init``: This initializes the flavours package, and should be the first |
| 299 |
of the flavours commands issued. Typically call right after the package load. |
| 300 |
* ``flavours prereq``: Like the standard ``prereq`` command, this declares a prerequisite. |
| 301 |
But it also does quite a bit more, as is discussed further below. |
| 302 |
* ``flavours root``: This is used to set the root for where the package is actually |
| 303 |
installed. This is used when generating the ``flavours path`` |
| 304 |
* ``flavours revision``: seem intended to allow for changes in the path format in future |
| 305 |
versions of ``flavours``. It is used in constructing the final path to the package. |
| 306 |
* ``flavours conflict``: This is similar to the standard conflict command, but enhanced |
| 307 |
to recognize the flavours prereqs above. |
| 308 |
* ``flavours commit``: This should be called after the ``root``, ``revision``, and ``prereq`` |
| 309 |
sub-commands of ``flavours`` are called, and before any of the ``path`` sub-commands. |
| 310 |
It seems to be responsible for taking all those values to above and constructing the |
| 311 |
path to the package. |
| 312 |
* ``flavours path``: This returns a string with the path to the specific build of the package. |
| 313 |
* ``flavours prepend-path``, ``flavour append-path``: These work much like the standard |
| 314 |
``prepend-path`` and ``append-path``, except that the value being prepended/appended |
| 315 |
to the environment variable has the path (as returned by ``flavours path``) prepended |
| 316 |
to it with the appropriate directory separator. E.g., to add to the ``PATH`` variable the bin subdirectory |
| 317 |
of the root directory where the specific build was installed, use ``flavours prepend-path PATH bin`` |
| 318 |
* ``flavours cleanup``: This should be called after all ``flavours`` sub-commands are finished |
| 319 |
and before exiting the script to ensure proper cleanup. Among other things, it ensures |
| 320 |
that any packages that depend on this package will get reloaded if this package is switched out. |
| 321 |
|
| 322 |
The ``flavours prereq`` command accepts the new ``-class`` parameter, allowing |
| 323 |
it to require a class of packages; e.g. one could use ``-class compiler`` to indicate |
| 324 |
that it has a prereq on a compiler (any of the modules ``gnu``, ``intel``, or ``pgi``). |
| 325 |
The allowable classes, and the package basenames that are in each class, is defined in |
| 326 |
``flavours.tcl`` in the Tcl associative array ``_class``. The ones shipped by default are |
| 327 |
|
| 328 |
* compiler: consisting of gnu, intel, and pgi |
| 329 |
* mpi: consisting of openmpi, mvapich2, mvapich, intelmpi |
| 330 |
* linalg: consisting of mkl, atlas, acml, netlib |
| 331 |
|
| 332 |
You will likely want to adjust these if you go with flavours in production. |
| 333 |
|
| 334 |
The ``flavours prereq`` command also accepts the parameter ``-optional``, which declares |
| 335 |
optional prerequisites. Although it sounds a little oxymoronic, this comes into play |
| 336 |
with the secondary purpose of the command in declaring the components of the path, as |
| 337 |
discussed below. If a prereq is not optional, the modulefile will complain if nothing |
| 338 |
satisfying the prereq has been module loaded previously. If the prereq is optional, |
| 339 |
the modulefile will not complain if it was not loaded, but will use the prereq in |
| 340 |
constructing the path to the build of the package if it was loaded. |
| 341 |
|
| 342 |
The ``flavours prereq`` command also defines the components which will comprise the final |
| 343 |
path to the directory containing the specific build of the package. The order of the |
| 344 |
prereq commands controls the order of the components in the path. |
| 345 |
|
| 346 |
The modulefile will check that all non-optional ``flavours prereq`` commands are satisfied, |
| 347 |
and then construct a path to the installation root for this build of the package using |
| 348 |
the packages satisfying the prereqs. The resultant path is composed of: |
| 349 |
|
| 350 |
* the value from ``flavours root`` |
| 351 |
* directory separator (``/``) |
| 352 |
* the value from ``flavours revision`` |
| 353 |
* directory separator (``/``) |
| 354 |
* a ``prefix`` created by concatenating the package names satisfying the prereqs, in order. |
| 355 |
The package name and version will be separated by a hyphen (-), as will the different |
| 356 |
components. |
| 357 |
|
| 358 |
So if ``flavours root`` was set to ``/local/software/foo/1.7``, ``revision`` to 1, and the |
| 359 |
package had prereqs compiler and mpi, and ``gnu/9.1.0`` and ``openmpi/4.0`` were loaded, the |
| 360 |
resulting path would be ``/local/software/foo/1.7/1/gnu-9.1.0-openmpi-4.0``. |
| 361 |
|
| 362 |
The modulefile actually will test for the existence of that directory, and if not found will |
| 363 |
return an error to the that the package was not built for that combination of prereqs. |
| 364 |
You either need to install your packages using the above directory schema, or create |
| 365 |
symlinks linking that scheme to where you actually install the packages. |
| 366 |
|
| 367 |
Examples |
| 368 |
^^^^^^^^ |
| 369 |
|
| 370 |
We now look at the example modulefiles for flavours. To use the examples, |
| 371 |
you must |
| 372 |
|
| 373 |
#. Have `Flavours extension`_ installed. **NOTE** these examples will NOT work without |
| 374 |
the Flavours installed. |
| 375 |
#. Set (and export) ``MOD_GIT_ROOTDIR`` to where you git-cloned the modules source |
| 376 |
#. Do a ``module purge``, and then set your ``MODULEPATH`` |
| 377 |
to ``$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/flavours`` |
| 378 |
|
| 379 |
We start with the ``module avail`` command: |
| 380 |
|
| 381 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/flavours/modules4/modavail.out |
| 382 |
:language: console |
| 383 |
|
| 384 |
We note that we only see the package names and versions; e.g. ``foo/2.4``, without any mention |
| 385 |
of the compilers and MPI libraries for which it is built. This terser type was an intentional |
| 386 |
design goal of the authors. Also of note are the intelmpi and simd packages. The Flavours |
| 387 |
approach relies on seeing what modules have been loaded previously in order to determine what |
| 388 |
'flavor' of the requested package should be loaded. To support the different builds of ``bar`` |
| 389 |
which depend on the CPU vectorization commands supported, we need to add a "dummy" package ``simd``. |
| 390 |
The module definition is quite trivial; a simple stub file like |
| 391 |
|
| 392 |
.. literalinclude:: ../../example/compiler-etc-dependencies/flavours/simd/avx |
| 393 |
:language: tcl |
| 394 |
|
| 395 |
and the main content in the ``common`` file: |
| 396 |
|
| 397 |
.. literalinclude:: ../../example/compiler-etc-dependencies/flavours/simd/common |
| 398 |
:language: tcl |
| 399 |
|
| 400 |
Basically it just declares a help procedure and whatis text. This way, an user can |
| 401 |
load the appropriate simd module to control which variant of bar they will get. The |
| 402 |
only interesting aspect is that near the beginning of the file we do a |
| 403 |
``package require flavours`` and ``flavours init``, and add a ``flavours cleanup`` |
| 404 |
near the bottom. The lines at the beginning instruct Tcl command to load the Flavours |
| 405 |
package, and then initialize the package. |
| 406 |
The ``flavours cleanup`` is required so that if the simd module is switched out, |
| 407 |
any modulefiles that depend on it get reloaded. |
| 408 |
|
| 409 |
In our example, we assumed that the Intel MPI libraries are automatically set up properly if one |
| 410 |
were to load the ``intel`` module, and we assumed the Intel MPI libraries were not supported |
| 411 |
for either the GNU or PGI compilers. However, we also wished to allow for ``foo`` to |
| 412 |
be used without any MPI support. So we need a way to distinguish if someone wants to |
| 413 |
use an Intel compiler build of ``foo`` without MPI or with the Intel MPI libraries. Our |
| 414 |
choice for this example was to require one to explicitly module load ``intelmpi`` if one |
| 415 |
wished to use the Intel MPI variant --- we do not bother with a real version number because |
| 416 |
assuming the version is determined by the version of ``intel`` (the Intel Parallel Studio version). |
| 417 |
So the intelmpi modulefile is similar to the simd modulefiles, a dummy modulefile. Again, |
| 418 |
it includes the ``flavours init`` and ``flavours cleanup`` wrapping to ensure proper reloading |
| 419 |
of dependent modules should it be switched out. |
| 420 |
|
| 421 |
If you were to support Intel MPI for non-intel compilers, you could create your intelmpi |
| 422 |
modulefiles as usual, and then add a ``default`` or ``intel`` "dummy" version to use the |
| 423 |
version that is part of the ``intel`` Parallel Studio. Or you could separate the intelmpi |
| 424 |
bits from the ``intel`` modulefile so both non-intel and intel compilers need to explicitly |
| 425 |
module load intelmpi. |
| 426 |
|
| 427 |
The modulefiles for the various compilers are all pretty much standard, except for the |
| 428 |
same three ``flavours`` lines as the simd modulefile: ``package require flavours``, |
| 429 |
``flavours init``, and ``flavours cleanup``. These are required to |
| 430 |
ensure dependent modulefiles get reloaded if the compiler is switched out. |
| 431 |
We also note that the modulefile for the GNU Compiler Collection is referred to as ``gnu``, not ``gcc`` |
| 432 |
(this is due to how the ``compiler`` class is defined in ``flavours.tcl``, and we did not bother |
| 433 |
to change that for the purposes of this cookbook). |
| 434 |
|
| 435 |
With the openmpi and mvapich MPI libraries, things start to get interesting. These all |
| 436 |
should setup the environment for a different build depending on the compiler loaded. The |
| 437 |
real work is done in the ``common`` tcl file, as shown below: |
| 438 |
|
| 439 |
.. literalinclude:: ../../example/compiler-etc-dependencies/flavours/openmpi/common |
| 440 |
:language: tcl |
| 441 |
|
| 442 |
Like the previous cases, the file starts with the Tcl command to load the |
| 443 |
package, followed by the ``flavours init`` command. |
| 444 |
|
| 445 |
The ``flavours prereq`` command states that this package requires a compiler to have been previously |
| 446 |
loaded, and that the path to the specific build to use will depend on that. We note the use |
| 447 |
of the ``-class`` parameter; the exact definition of the compiler class is in the ``compiler`` |
| 448 |
field of the Tcl associative hash ``_class`` defined in ``flavours.tcl``. |
| 449 |
|
| 450 |
The ``flavours root`` sets the root directory of where the builds for this package is installed. |
| 451 |
We use the ``MOD_GIT_ROOTDIR`` environment variable for convenience in this example, but in production |
| 452 |
you would generally hardcode a path. The result of all the directives is that the build will be |
| 453 |
found in a path named after the compiler (since in this case there is only one ``flavour prereq``); |
| 454 |
e.g. for gcc version ``9.1.0``, we expect to find the build in ``$swroot/openmpi/4.0/1/gnu-9.1.0``. |
| 455 |
If you do not use that naming convention for your installation directories, you can use symlinks |
| 456 |
to fake it. |
| 457 |
|
| 458 |
The ``flavours path`` command in the ``setenv MPI_DIR`` statement sets MPI_DIR to the aforementioned |
| 459 |
build path. The ``flavours prepend-path`` commands prepend to the environment variable specified |
| 460 |
by the first argument the result of prepending the ``flavours path`` to their second argument. E.g., |
| 461 |
the first such, assuming openmpi version ``4.0`` was requested and ``gnu/9.1.0`` loaded, would be basically |
| 462 |
the same as a standard Modules command: |
| 463 |
|
| 464 |
.. code-block:: tcl |
| 465 |
|
| 466 |
prepend-path PATH $swroot/openmpi/4.0/1/gnu-9.1.0/bin |
| 467 |
|
| 468 |
The following shows how this would appear to the user: |
| 469 |
|
| 470 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/flavours/modules4/ompi-loads1.out |
| 471 |
:language: console |
| 472 |
|
| 473 |
Here we note that once a compiler is loaded, the ``PATH`` and the other environment |
| 474 |
variables are set appropriately to point to the bin dir for the particular build of |
| 475 |
``openmpi/4.0``, as evidenced by the output of our dummy mpirun command. At the end, we attempt |
| 476 |
to load ``openmpi/4.0`` for ``gnu/8.2.0``, and receive an error because our dummy SW library does not contain |
| 477 |
a matching build. This is determined from the ``flavours path``; if the |
| 478 |
path does not exist (in this example ``$swroot/openmpi/4.0/1/gnu-8.2.0``) it will |
| 479 |
abort in this fashion. |
| 480 |
|
| 481 |
In the above, we have explicitly unloaded openmpi, switched the compilers, and |
| 482 |
then reloaded openmpi. A nice feature of the `Flavours extension`_ is that it can handle the |
| 483 |
switching out of compilers or other modulefiles which other modulefiles depend on, |
| 484 |
as: |
| 485 |
|
| 486 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/flavours/modules3/ompi-switch.out |
| 487 |
:language: console |
| 488 |
|
| 489 |
Note that when we switched between the pgi and intel compilers above, Flavours |
| 490 |
automatically "unloaded" and "reloaded" the openmpi module. This happens in |
| 491 |
the ``flavours cleanup`` portion of the compiler modulefiles, and is due to |
| 492 |
``openmpi`` declaring a ``flavours prereq`` on the compiler class. |
| 493 |
|
| 494 |
.. note:: The above behavior with ``switch`` was done with version ``3.2.10`` |
| 495 |
of Environment Modules; it does *not* appear to work with ``4.3.1``. |
| 496 |
|
| 497 |
Note that when we further tried to replace version ``2019`` of the intel compiler with the ``2018`` version, |
| 498 |
the module switch of the compilers failed because ``openmpi/4.0`` was not built with |
| 499 |
``intel/2018``. Since the user never explicitly requested version ``4.0`` of openmpi (it was |
| 500 |
defaulted in the initial load as the latest version of openmpi available for ``pgi/19.4``), |
| 501 |
it would have been nicer had the attempted reload of openmpi allowed it t:o default to the ``3.1`` |
| 502 |
version (as the latest version available for ``intel/2018``). Nevertheless, it behaved well |
| 503 |
in this situation; the module switch failed with a reasonable error message and the resulting |
| 504 |
set of modules was still consistent. |
| 505 |
|
| 506 |
We also note that if we attempt to load openmpi without having previously loading a compiler, |
| 507 |
we will get an error: |
| 508 |
|
| 509 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/flavours/modules4/ompi-defaults.out |
| 510 |
:language: console |
| 511 |
|
| 512 |
In particular, there is no support for a "default" compiler; if e.g. you wished to make the |
| 513 |
distribution supplied gcc the default compiler, you will need to have the initializations |
| 514 |
scripts automatically do a module load of that compiler (possibly |
| 515 |
a dummy modulefile like simd/intelmpi if the compiler is already in the user's path) |
| 516 |
in your user's start up dot files or similar. We also note that there is no additional |
| 517 |
intelligence in the version defaulting --- in the last example, we have ``gnu/8.2.0`` loaded and if |
| 518 |
we try to load openmpi without specifying a version, it defaults to version ``4.0`` |
| 519 |
as that is the latest version of openmpi without regard for the fact that there |
| 520 |
is no build of openmpi version ``4.0`` for ``gnu/8.2.0`` (but there is such for ``openmpi/3.1``). |
| 521 |
|
| 522 |
The situation for ``foo`` is more complicated, as it depends both on the compiler and |
| 523 |
optionally on the MPI library. But with Flavours, the modulefile is only slightly more |
| 524 |
complicated, e.g. for the common file is: |
| 525 |
|
| 526 |
.. literalinclude:: ../../example/compiler-etc-dependencies/flavours/foo/common |
| 527 |
:language: tcl |
| 528 |
|
| 529 |
Basically, the main difference is the addition of the |
| 530 |
line ``flavours prereq -optional -class mpi``. |
| 531 |
This instructs Flavours that there is an additional, optional prereq. The |
| 532 |
order of the prereq lines matter, as that controls the resultant ``flavors path``. |
| 533 |
With the current configuration, assuming ``gnu/9.1.0`` and ``openmpi/4.0`` were loaded, |
| 534 |
the path would become ``$swroot/foo/2.4/1/gnu-9.1.0-openmpi-4.0``. If the order were |
| 535 |
reversed, the ``openmpi-4.0`` would precede the ``gnu-9.1.0``. Because the MPI requirement |
| 536 |
is optional, if ``gnu/9.1.0`` was loaded and no MPI library loaded, the path would |
| 537 |
evaluate to ``$swroot/foo/2.4/1/gnu-9.1.0``. |
| 538 |
|
| 539 |
We show how it works below: |
| 540 |
|
| 541 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/flavours/modules4/foo-loads.out |
| 542 |
:language: console |
| 543 |
|
| 544 |
So basically, if the user loads a compiler, the |
| 545 |
the environment variables (``PATH``, etc) are set up for the correct build of foo. |
| 546 |
If no MPI library was loaded, a version of foo built without MPI will be loaded, |
| 547 |
otherwise, a version of foo built with the loaded MPI library will be loaded. |
| 548 |
This is shown by the output of the ``foo`` command. Note |
| 549 |
also how we use the dummy ``intelmpi`` package to indicate a desire for the |
| 550 |
intelmpi enabled version. |
| 551 |
|
| 552 |
The 3.x version of Environment Modules supports using the switch command on |
| 553 |
either the compiler or MPI library, and will result in reloading of foo and the MPI |
| 554 |
library. |
| 555 |
|
| 556 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/flavours/modules3/foo-switch.out |
| 557 |
:language: console |
| 558 |
|
| 559 |
In particular note the final case, wherein we load ``intel/2019`` then foo, and get the |
| 560 |
version of foo built without MPI. When we subsequently load openmpi, foo is reloaded |
| 561 |
to be the openmpi version (this is because the hooks to reload foo are in the ``flavours cleanup`` |
| 562 |
part of the openmpi modulefile, and foo declared its optional dependency on MPI). |
| 563 |
Also, we don't bother showing it, but if you were to attempt to load foo without |
| 564 |
at least a compiler loaded, it would display an error. |
| 565 |
|
| 566 |
Our final example for flavours is the ``bar`` command. Here in addition to the |
| 567 |
compiler dependency, we have versions for different SIMD vectorization supported. |
| 568 |
Again, the difference in the modulefile is small, e.g. |
| 569 |
|
| 570 |
.. literalinclude:: ../../example/compiler-etc-dependencies/flavours/bar/common |
| 571 |
:language: tcl |
| 572 |
|
| 573 |
Basically, the optional ``flavours prereq`` on the mpi class from the ``foo`` package |
| 574 |
is replaced by a (mandatory) ``flavours prereq`` on the ``simd`` dummy package. |
| 575 |
We note that Flavours package knows nothing about our ``simd`` dummy package until |
| 576 |
we add it as a prereq for bar. (This is in contrast to the compiler and mpi classes). |
| 577 |
Usage would be like: |
| 578 |
|
| 579 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/flavours/modules4/bar-loads.out |
| 580 |
:language: console |
| 581 |
|
| 582 |
Here we note that as both the compiler and simd prereqs are non-optional, it complains |
| 583 |
unless both have been previously loaded. When both have been loaded, the ``PATH`` and |
| 584 |
other environment variables are set appropriately for the requested build; and if |
| 585 |
it does not exist and error is produced. |
| 586 |
|
| 587 |
Summary of Flavours |
| 588 |
^^^^^^^^^^^^^^^^^^^ |
| 589 |
|
| 590 |
* It is an external extension to Environment Modules, requiring additional installation steps. |
| 591 |
* The git repository appears to have been last updated in 2013; although which means |
| 592 |
that it has not been updated for Environment Modules 4.x, a simple experimentation |
| 593 |
indicates that it still works, with the exception of automatic reloading of a module if any of |
| 594 |
the modules it depends on are switched. |
| 595 |
However, the `Flavours extension`_ does not appear to be actively supported. |
| 596 |
* The Flavours package (using Environment Modules 3.x) fully supports the ``module switch`` syntax, |
| 597 |
with the switching out of a dependency (e.g. a compiler) causing the reload of all modulefiles depending on it. |
| 598 |
(however the test of this feature failed when using Environment Modules 4.x.) |
| 599 |
* The syntax for modulefiles is elegant, and one can easily extend the basic compiler dependency |
| 600 |
modulefile to add additional dependencies. Even for packages/dummy packages that the Flavours |
| 601 |
extension knows nothing about (e.g. simd in the above example). |
| 602 |
* The Flavours package will provide a shorter module avail output, only e.g. giving package name and |
| 603 |
version and not listing a separate modulefile for each combination of package, version, compiler+version, |
| 604 |
MPI library+version, etc. |
| 605 |
* The Flavours package will fail with an error message if user tries to load a package which was |
| 606 |
not built for the values of the dependency packages loaded. |
| 607 |
* The Flavours package will fail to load with an error message if any dependent package is not |
| 608 |
already loaded. In particular, it will not attempt to default these. So if such defaults are |
| 609 |
desired, you will need to have initialization scripts automatically load the appropriate modules. |
| 610 |
* The Flavours package does not include any mechanism for more intelligent defaulting. I.e., if an |
| 611 |
user requests to load a package without specifying the version desired, the version will be defaulted |
| 612 |
to the latest version (or whatever the ``.modulerc`` file specifies) without regard for which versions |
| 613 |
support the versions of the compiler and other prereq-ed packages the user has loaded. While one |
| 614 |
could write custom ``.modulerc`` files for such, Flavours does not provide any tools for simplifying |
| 615 |
such. |
| 616 |
|
| 617 |
|
| 618 |
.. _Homebrewed_flavors_strategy: |
| 619 |
|
| 620 |
Homebrewed Flavors Strategy |
| 621 |
--------------------------- |
| 622 |
|
| 623 |
Although the `Flavours extension`_ described above has an elegance about it, |
| 624 |
one can achieve much of the same functionality in modulefiles using standard |
| 625 |
Environment Modules and Tcl commands. This can be facilitated by the definition |
| 626 |
of some useful Tcl procedures. For lack of a better name, we will refer to |
| 627 |
this strategy as *Homebrewed flavors*. |
| 628 |
|
| 629 |
Implementation |
| 630 |
^^^^^^^^^^^^^^ |
| 631 |
|
| 632 |
This strategy just makes use of standard Environment Modules and Tcl procedures |
| 633 |
to query what modules of a given type are loaded and to construct the path to the |
| 634 |
software package accordingly. To avoid needless (and error prone) repetition of |
| 635 |
code, we collect these into several Tcl procedures of our own. Ideally, these |
| 636 |
should be placed in a site configuration Tcl file and exposed to modulefiles |
| 637 |
as explained in the cookbook :ref:`expose-procs-to-modulefiles`. |
| 638 |
However, to avoid the need for that in these examples, we instead have placed them |
| 639 |
into a file and use the ``MOD_GIT_ROOTDIR`` to locate and source that file in the |
| 640 |
relevant modulefiles. (Actually, we have a single tcl file that is sourced both |
| 641 |
for this and some other strategies, and it sources several files so that we can |
| 642 |
break up the discussion of the the Tcl procedures. All of that is just for the |
| 643 |
purposes of this cookbook; normally you just put the procedures you need in the |
| 644 |
one site config file). |
| 645 |
|
| 646 |
We discuss the various Tcl procedures here, as they are what provide most of the |
| 647 |
functionality. We start with the routines for generic loaded modules: |
| 648 |
|
| 649 |
.. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/LoadedModules.tcl |
| 650 |
:language: tcl |
| 651 |
:lines: 4- |
| 652 |
|
| 653 |
This defines the two Tcl procedures: |
| 654 |
|
| 655 |
* **GetLoadedModules** : this returns the list of loaded modules, from the ``LOADEDMODULES`` |
| 656 |
* **GetTagOfModuleLoaded** : this takes as argument the base name of a package, |
| 657 |
and returns the first full spec for the matching package, or an empty |
| 658 |
string if no matching package found. |
| 659 |
|
| 660 |
The Tcl procedure **GetTagOfModuleLoaded** can be used to find out what version |
| 661 |
of a given package is loaded, and is enough for many packages. However, for |
| 662 |
compilers, and similar, a bit more is needed. For compilers: |
| 663 |
|
| 664 |
.. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/CompilerUtils.tcl |
| 665 |
:language: tcl |
| 666 |
:lines: 4- |
| 667 |
|
| 668 |
We defined four procedures above: |
| 669 |
|
| 670 |
* **GetDefaultCompiler** : this simply returns the name of our default compiler, which for |
| 671 |
this example is ``gcc/8.2.0`` |
| 672 |
* **RequireCompiler** : this simply does a module load on the specified compiler. |
| 673 |
It is kept as a separate procedure just in case you wish to intercept and prevent |
| 674 |
the loading of the default compiler (e.g. because no modulefile exists for it). |
| 675 |
In our example, there is a modulefile for it and so it is just a wrapper for ``module load``. |
| 676 |
* **GetKnownCompilerFamilies** : this simply returns a Tcl list of known |
| 677 |
compiler families. |
| 678 |
* **GetLoadedCompiler**: this is the procedure that does the main work, and is |
| 679 |
described in detail below. |
| 680 |
|
| 681 |
The **GetLoadedCompiler** procedure basically checks if any packages matching |
| 682 |
the names in **GetKnownCompilerFamilies** have been previously loaded. If so, |
| 683 |
it returns the modulefile specification for the first one found, and returns. |
| 684 |
If not, if ``pathDefault`` is set and there is a recognized compiler name and version |
| 685 |
in the last two components of the module specification, it will return |
| 686 |
that compiler. Otherwise, if the optional flag ``useDefault`` is set, it will return the |
| 687 |
value from **GetDefaultCompiler**. If all else fails, returns the empty string. |
| 688 |
|
| 689 |
If the optional parameter ``loadIt`` is set, if a compiler was defaulted (i.e. |
| 690 |
not returned because it was already loaded), the procedure will call ``RequireCompiler`` |
| 691 |
to module load it. |
| 692 |
|
| 693 |
If the optional parameter ``requireIt`` is set, we invoke ``prereq`` on the compiler |
| 694 |
found before returning. |
| 695 |
|
| 696 |
A similar set of procedures exist for the MPI libraries, namely: |
| 697 |
|
| 698 |
.. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/MpiUtils.tcl |
| 699 |
:language: tcl |
| 700 |
:lines: 4- |
| 701 |
|
| 702 |
The three procedures here are analogues of the compiler versions: |
| 703 |
|
| 704 |
* **RequireMPI** : this basically does a module load of the specified MPI library. It |
| 705 |
has some added logic so that it will not do a module load if the MPI library is ``nompi``. |
| 706 |
Also, if the optional parameter ``noLoadIntel`` is set, if the MPI library is ``intelmpi`` |
| 707 |
(or a variant of that name) and the loaded compiler in ``intel``, we assume that no |
| 708 |
additional module needs to be loaded. For this strategy, we want to load ``intelmpi`` |
| 709 |
modules, because, just like in the :ref:`Flavours_strategy`, we need to provide dummy |
| 710 |
``intelmpi`` modules to allow one to request the use of the Intel MPI library. |
| 711 |
* **GetKnownMpiFamilies** : this returns a list of known MPI library family names. |
| 712 |
Used in **GetLoadedMPI** |
| 713 |
* **GetLoadedMPI** : This is the analogue of **GetLoadedCompiler**. If an MPI library is |
| 714 |
loaded, it will return the name of that module. If the optional ``requireIt`` flag |
| 715 |
is set, it will do a ``prereq`` on the MPI library before returning. The first optional |
| 716 |
argument, ``useIntel``, indicates whether this module should return ``intelmpi`` if no |
| 717 |
MPI library is loaded but an Intel compiler is loaded. |
| 718 |
|
| 719 |
The modulefiles for the compilers are basically standard; unlike the |
| 720 |
:ref:`Flavours_strategy` there is nothing special needed in these. Likewise |
| 721 |
for the dummy ``simd`` and ``intelmpi`` modules (the latter is only this |
| 722 |
basic because we assume intelmpi is only available if an Intel compiler |
| 723 |
is loaded. If one allowed for intelmpi with other compilers, it would |
| 724 |
more closely resemble the other MPI libraries). |
| 725 |
|
| 726 |
We also define some Tcl procedures for generating warning and error messages, namely |
| 727 |
|
| 728 |
.. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/WarningUtils.tcl |
| 729 |
:language: tcl |
| 730 |
:lines: 4- |
| 731 |
|
| 732 |
These procedures: |
| 733 |
|
| 734 |
* **PrintIfLoading**: will print supplied text to stderr only when in ``load`` mode |
| 735 |
* **PrintLoadInfo**: will print supplied text as informational text, but only when trying to load a module |
| 736 |
* **PrintLoadWarning**: will print supplied text as warning text, but only when trying to load a module |
| 737 |
* **PrintLoadError**: will print supplied text as error text and abort, but only when trying to load a module |
| 738 |
|
| 739 |
The gist of this is that we might wish to print errors if an user tries to |
| 740 |
load an incompatible modulefile, but do not wish to print errors if they are merely |
| 741 |
doing a help, display, or whatis command. |
| 742 |
|
| 743 |
The interesting bit begins with the openmpi and mvapich modulefiles. These |
| 744 |
both depend on the compiler, we show the main part of the openmpi modulefile |
| 745 |
below: |
| 746 |
|
| 747 |
.. literalinclude:: ../../example/compiler-etc-dependencies/homebrewed/openmpi/common |
| 748 |
:language: tcl |
| 749 |
|
| 750 |
We begin by sourcing the ``common_utilities`` file which defined the previously |
| 751 |
described Tcl procedures. Normally it is recommended that you put those |
| 752 |
procedures in a site config Tcl script and expose them to the modulefiles |
| 753 |
using the techniques described in the cookbook :ref:`expose-procs-to-modulefiles`. |
| 754 |
Even if you opt against that and decide to source a Tcl file, it is recommended |
| 755 |
to hard code the path. |
| 756 |
|
| 757 |
The next interesting bit comes when we set the local Tcl variable ``ctag`` |
| 758 |
by calling the ``GetLoadedCompiler`` procedure. We allow the procedure to |
| 759 |
default the compiler, and because we have a default compiler defined we |
| 760 |
should always get a value. (If no default compiler was defined, one would |
| 761 |
have to handle the error if no compiler was loaded/defaulted.) We then use |
| 762 |
the value of ``ctag`` to set the path to the build of the package. To ensure |
| 763 |
that the package is built for this compiler, we do a quick check that the |
| 764 |
package installation path exists. |
| 765 |
|
| 766 |
The modulefile for ``foo`` is a bit more complex: |
| 767 |
|
| 768 |
.. literalinclude:: ../../example/compiler-etc-dependencies/homebrewed/foo/common |
| 769 |
:language: tcl |
| 770 |
|
| 771 |
The main difference between this modulefile, depending on both compiler and |
| 772 |
optionally MPI, and the openmpi modulefile above, is that in addition to |
| 773 |
detecting which compiler is loaded, we call ``GetLoadedMPI`` to determine |
| 774 |
the MPI library which was loaded, and use both of them in constructing the |
| 775 |
prefix to the installed foo. |
| 776 |
|
| 777 |
Examples |
| 778 |
^^^^^^^^ |
| 779 |
|
| 780 |
We now look at the example modulefiles for the *Homebrewed flavors* strategy. |
| 781 |
To use the examples, you must |
| 782 |
#. Set (and export) ``MOD_GIT_ROOTDIR`` to where you git-cloned the modules source |
| 783 |
#. Do a ``module purge``, and then set your ``MODULEPATH`` to: |
| 784 |
|
| 785 |
.. code-block:: shell |
| 786 |
|
| 787 |
$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/homebrewed |
| 788 |
|
| 789 |
The *Homebrewed flavors* strategy behaves much like the :ref:`Flavours_strategy` |
| 790 |
in practice. The module avail command, |
| 791 |
|
| 792 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/modavail.out |
| 793 |
:language: console |
| 794 |
|
| 795 |
looks basically the same, showing the a concise listing of packages and |
| 796 |
versions without information on the compilers and MPI libraries they were |
| 797 |
built with. |
| 798 |
|
| 799 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/ompi-loads1.out |
| 800 |
:language: console |
| 801 |
|
| 802 |
Again, once a compiler is loaded, loading openmpi will set the ``PATH``, etc. for |
| 803 |
the correct build of openmpi, as evidenced by the output of the dummy |
| 804 |
mpirun command. Notice that the ``module list`` command only shows |
| 805 |
the version of openmpi loaded, and contains no information about what |
| 806 |
compiler it was built with (you just have to assume it matches the loaded |
| 807 |
compiler). And again we note that if one attempts to load a version |
| 808 |
of openmpi (e.g. ``4.0``) that was not built for the specified compiler (e.g. |
| 809 |
``gcc/8.2.0``), an error is generated. |
| 810 |
|
| 811 |
Unlike in :ref:`Flavours_strategy`, we did not put any code in the modulefiles to cause |
| 812 |
dependent modulefiles to be reloaded if a module they depend on gets switched |
| 813 |
out. However, starting with Environment Modules ``4.2``, a feature called |
| 814 |
:ref:`Automated module handling <v42-automated-module-handling-mode>` was added. Without this feature, attempting to |
| 815 |
switch out a module upon which other modules depended could be problematic, |
| 816 |
as evidenced in this sequence below (using Environment Modules ``3.2.10`` and |
| 817 |
so without automated module handling mode): |
| 818 |
|
| 819 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules3/ompi-switch.out |
| 820 |
:language: console |
| 821 |
|
| 822 |
Here we note that we were able to switch out the pgi compiler for the intel compiler, |
| 823 |
but the openmpi module was not reloaded and the environment is still set for openmpi |
| 824 |
compiled with the pgi compiler, and that this inconsistency is not readily determined |
| 825 |
from the module list command. |
| 826 |
|
| 827 |
Environment Modules 4.x, even with automated module handling disabled, is better --- |
| 828 |
in a command sequence as above the module switch from pgi to intel would fail due to the |
| 829 |
prereq module. However, with the :ref:`Automated module handling <v42-automated-module-handling-mode>` |
| 830 |
enabled (this feature is disabled by default but could be enabled on a per-command basis with the ``--auto`` |
| 831 |
flag), things work much better, as evidenced below: |
| 832 |
|
| 833 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/ompi-switch.out |
| 834 |
:language: console |
| 835 |
|
| 836 |
When we switch out the pgi compiler for the ``intel/2019`` compiler, the openmpi |
| 837 |
module is automatically unloaded before the compiler switch and reload afterwards, |
| 838 |
so we end up with the correct build of openmpi for the newly loaded ``intel/2019`` |
| 839 |
compiler. If one then switches out ``intel/2019`` replacing it with ``intel/2018``, |
| 840 |
the openmpi module is first unloaded, the compilers are switched, and as there |
| 841 |
is no ``openmpi/4.0`` build for ``intel/2018``, a warning is given and the openmpi module |
| 842 |
is left unloaded. Since the user never specifically requested version ``4.0`` of openmpi |
| 843 |
(it was defaulted in the original module load of openmpi as that was the latest version |
| 844 |
available for ``pgi/19.4``), it would have been nicer if on the switch of intel compiler |
| 845 |
versions the reload only attempted a ``module load openmpi`` instead of ``module load openmpi/4.0``, |
| 846 |
but nevertheless this well behaved. The openmpi module is dropped with a warning and the |
| 847 |
user has a consistent set of modules loaded. |
| 848 |
|
| 849 |
We note that the modulefile is able to |
| 850 |
default the compiler, so when we attempt to load openmpi without having |
| 851 |
previously loaded a compiler, as in |
| 852 |
|
| 853 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/ompi-defaults.out |
| 854 |
:language: console |
| 855 |
|
| 856 |
it will default to the default compiler, ``gcc/8.2.0``. Note however, that if |
| 857 |
one does not specify version ``3.1`` of openmpi, it will still default to ``4.0`` |
| 858 |
and fail to load as there is no build of ``openmpi/4.0`` for ``gcc/8.2.0``. |
| 859 |
|
| 860 |
The situation is similar for foo: |
| 861 |
|
| 862 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/foo-loads.out |
| 863 |
:language: console |
| 864 |
|
| 865 |
Again, one can load a compiler without an MPI library to get the non-MPI version |
| 866 |
of foo, or a compiler and MPI library to get the MPI version. The dummy |
| 867 |
intelmpi modulefile is used to allow one to indicate that the Intel MPI library |
| 868 |
is desired. The automated module handling mode can again allow the switch |
| 869 |
functionality work properly, as in |
| 870 |
|
| 871 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/foo-switch.out |
| 872 |
:language: console |
| 873 |
|
| 874 |
Here we note a deficiency in the switch support as compared to :ref:`Flavours_strategy`. In the last example |
| 875 |
after loading ``intel/2019`` and foo, we have the non-MPI build of foo as expected. However, upon |
| 876 |
subsequently loading the openmpi module, we still have the non-MPI version of foo loaded, as evidenced |
| 877 |
by the output of the dummy foo command. I.e., the foo package was *not* automatically reloaded, as |
| 878 |
there was no prereq in the foo modulefile on an MPI library (as in the non-MPI build there is no MPI |
| 879 |
library to prereq). Also note that module list does not really inform one of this fact. |
| 880 |
|
| 881 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/foo-defaults.out |
| 882 |
:language: console |
| 883 |
|
| 884 |
Above, we see once more that the compiler can be defaulted, but that the |
| 885 |
defaulting mechanism is not smart enough to default the version of foo based |
| 886 |
on the compiler loaded (or defaulted to). |
| 887 |
|
| 888 |
The situation with bar is basically the same; with a compiler and simd |
| 889 |
module loaded, the environment for the appropriate build of bar is loaded |
| 890 |
when you module load bar. |
| 891 |
|
| 892 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/bar-loads.out |
| 893 |
:language: console |
| 894 |
|
| 895 |
And an error is generated if there is no build for that combination of |
| 896 |
compiler and simd. The automatic handling of modules again allows the |
| 897 |
switch command to work as expected: |
| 898 |
|
| 899 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/bar-switch.out |
| 900 |
:language: console |
| 901 |
|
| 902 |
and both the simd level and compiler can be defaulted, but one still has |
| 903 |
to choose a version of bar which supports the defaults. |
| 904 |
|
| 905 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/homebrewed/modules4/bar-defaults.out |
| 906 |
:language: console |
| 907 |
|
| 908 |
Summary of Homebrewed flavors strategy |
| 909 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 910 |
|
| 911 |
* The :ref:`Automated module handling <v42-automated-module-handling-mode>` feature (introduced in Environment Modules |
| 912 |
``4.2``) allows for the switching out of a dependency (e.g. a compiler) |
| 913 |
to cause the reload of all modulefiles depending on it. Without the |
| 914 |
automated module handling (as is default for 4.x, and the only option for |
| 915 |
3.x), the switching of a compiler only changes the compiler and leaves the |
| 916 |
modulefiles that depend on the compiler unchanged. |
| 917 |
* The various Tcl procedures make |
| 918 |
it somewhat easy to determine which compiler, MPI, etc. modules have been |
| 919 |
loaded and set the paths appropriately. Not as elegant or easy to use |
| 920 |
as :ref:`Flavours_strategy`, but not difficult |
| 921 |
* Like the Flavours package, the module avail output is concise, only e.g. |
| 922 |
giving package name and version rather than listing a separate modulefile |
| 923 |
for each combination of package, version, compiler+version, |
| 924 |
MPI library+version, etc. |
| 925 |
* Modules will fail with an error message if user tries to load a package |
| 926 |
which was not built for the values of the dependency packages loaded. |
| 927 |
* With :ref:`v42-automated-module-handling-mode`, the dependencies of a module |
| 928 |
asked for load could automatically be loaded. But it requires that these |
| 929 |
dependencies are clearly specified with mention of the versions for which |
| 930 |
a build is available. |
| 931 |
* It does not include a mechanism for more intelligent defaulting. I.e., if |
| 932 |
an user requests to load a package without specifying the version desired, |
| 933 |
the version will be defaulted to the latest version (or whatever |
| 934 |
the ``.modulerc`` file specifies) without regard for which versions |
| 935 |
support the versions of the compiler and other prereq-ed packages the user |
| 936 |
has loaded. |
| 937 |
* Note that future releases of Environment Modules will introduce additional |
| 938 |
mechanisms to the :ref:`v42-automated-module-handling-mode`, which will |
| 939 |
improve the user experience on such *Homebrewed flavors* setup. |
| 940 |
|
| 941 |
|
| 942 |
.. _Modulerc-based_strategy: |
| 943 |
|
| 944 |
Modulerc-based Strategy |
| 945 |
----------------------- |
| 946 |
|
| 947 |
The previous two strategies used additional code in the modulefile to |
| 948 |
determine which compiler, etc. was loaded and adjust the values for |
| 949 |
``PATH``, etc. accordingly. The *Modulerc-based* strategy instead uses |
| 950 |
``.modulerc`` files to direct the module command to the proper modulefile |
| 951 |
depending on what compiler, etc. was previously loaded. Because of |
| 952 |
this, there are a number of differences in behavior and what is seen |
| 953 |
by the user, most notably many, many more modulefiles. Whether this is good |
| 954 |
or bad is a matter of taste. |
| 955 |
|
| 956 |
Implementation |
| 957 |
^^^^^^^^^^^^^^ |
| 958 |
|
| 959 |
Whereas the :ref:`Homebrewed_flavors_strategy` had the modulefile invoke |
| 960 |
a Tcl procedure to determine which, if any, version of a module like a compiler |
| 961 |
was loaded and then adjust paths, the *Modulerc-based* strategy instead |
| 962 |
uses the same Tcl procedures to default the modulefile which will be loaded. |
| 963 |
This implies that there is a distinct modulefile for every build of the package, |
| 964 |
and an immediate consequence is that this strategy has many more modulefiles |
| 965 |
than the others. We make use of the techniques in the cookbook |
| 966 |
:ref:`tips-for-code-reuse` to reduce the total amount of code; the actual modulefiles |
| 967 |
for each build are typically small stubfiles defining a couple of Tcl variables |
| 968 |
and then sourcing a ``common`` script (unique to each package) which does all |
| 969 |
the real work. |
| 970 |
|
| 971 |
The modules will be named with components for the different dependencies, |
| 972 |
so the one for openmpi version ``4.0`` built with gcc version ``9.1.0`` would |
| 973 |
be ``openmpi/4.0/gcc/9.1.0``; similarly the module for foo version ``1.1`` |
| 974 |
built for pgi version ``18.4`` and mvapich ``2.1`` would be |
| 975 |
``foo/1.1/pgi/18.4/mvapich/2.1``. |
| 976 |
|
| 977 |
The ``.modulerc`` files themselves are not trivial, but these |
| 978 |
can generally be written in a generic fashion, usable by multiple packages, |
| 979 |
and can just be symlinked to the appropriate locations. |
| 980 |
We define five such files which can be linked as ``.modulerc``: |
| 981 |
|
| 982 |
Two of these files can be linked in the module tree |
| 983 |
at various places as ``.modulerc`` for defaulting the compiler. One to |
| 984 |
default to the family portion of the compiler (e.g. gcc, intel, or pgi), |
| 985 |
and one for the version. For the family portion of the compiler, |
| 986 |
we have the file ``modulerc.select_compiler_family`` as below: |
| 987 |
|
| 988 |
.. literalinclude:: ../../example/compiler-etc-dependencies/modrc_common/modulerc.select_compiler_family |
| 989 |
:language: tcl |
| 990 |
|
| 991 |
The file starts by sourcing a set of useful Tcl procedures. For the purpose |
| 992 |
of the example for this cookbook this is done based on the ``MOD_GIT_ROOTDIR`` |
| 993 |
environment variable. If you were to use this in production, it is |
| 994 |
recommended that the Tcl procedures be placed in a site configuration script |
| 995 |
and exposed to modulefiles via the techniques described in the cookbook |
| 996 |
:ref:`expose-procs-to-modulefiles`. At the minimum, it is |
| 997 |
recommended to hardcode the path to the ``common_utilities.tcl`` file. |
| 998 |
|
| 999 |
The modulerc script then determines the directory it is in using the |
| 1000 |
Tcl variable ``ModulesCurrentModulefile``. It then uses the |
| 1001 |
``GetLoadedCompiler`` Tcl procedure (which was discussed in the section |
| 1002 |
on the :ref:`Homebrewed_flavors_strategy`. We then parse the resulting module |
| 1003 |
name into family and version pieces (we will discuss the procedure |
| 1004 |
``GetPackageFamilyVersion`` later; for now suffice to say it takes |
| 1005 |
a module name and returns a Tcl list with family and version). |
| 1006 |
|
| 1007 |
We do some trickery to support the use of either gcc or gnu as family names for |
| 1008 |
the GNU compiler suite, and then see if there is a directory or modulefile |
| 1009 |
underneath the directory containing the ``.modulerc`` file, and if so defaults |
| 1010 |
to it. For this purpose, we use the Tcl procedure ``FirstChildModuleInList`` |
| 1011 |
(defined in ``common_utilities.tcl``). This and a related procedure are |
| 1012 |
defined as: |
| 1013 |
|
| 1014 |
.. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/ChildModules.tcl |
| 1015 |
:language: tcl |
| 1016 |
:lines: 4- |
| 1017 |
|
| 1018 |
and basically make use of the Tcl variable ``ModulesCurrentModulefile``. They |
| 1019 |
are limited, however, in that they will not work properly if the module is |
| 1020 |
split across multiple modulepaths. That is, ``ChildModuleExists`` and |
| 1021 |
``FirstChildModuleInList`` will only detect children in the same directory |
| 1022 |
as the modulerc file they are called from; e.g., if two paths in the ``MODULEPATH`` |
| 1023 |
both have directories for openmpi, one with an intel subdirectory and one |
| 1024 |
with a pgi subdirectory and the modulerc invoking ``ChildModuleExists``, the |
| 1025 |
ChildModuleExists procedure will not see the intel directory. |
| 1026 |
|
| 1027 |
The ``modulerc.select_compiler_version`` file is similar, |
| 1028 |
|
| 1029 |
.. literalinclude:: ../../example/compiler-etc-dependencies/modrc_common/modulerc.select_compiler_version |
| 1030 |
:language: tcl |
| 1031 |
|
| 1032 |
Again, we source the ``common_utitilies.tcl`` file and use |
| 1033 |
``ModulesCurrentModulefile`` to get the directory in which the ``.modulerc`` |
| 1034 |
script resides. It then uses ``GetLoadedCompiler`` to determine what |
| 1035 |
if any compiler was loaded. If one was, we split into family and version, |
| 1036 |
and ensure that the directory containing the ``.modulerc`` file matches the |
| 1037 |
family name. If so, it checks if there is a directory or modulefile in that |
| 1038 |
directory matching the version name, and if so defaults to it. |
| 1039 |
|
| 1040 |
For both of the ``.modulerc`` scripts we examined, we note that in case of |
| 1041 |
errors, etc., the script just exits without defaulting. One might be |
| 1042 |
tempted to have the ``.modulerc`` script actually return an error in such |
| 1043 |
cases (perhaps using ``module-info mode`` to only have it output during |
| 1044 |
load operations). This, however, runs into issues: |
| 1045 |
|
| 1046 |
* For Environment Modules 3.x: for some reason, when modulecmd processes |
| 1047 |
``.modulerc`` scripts, ``module-info mode load`` always returns true. Because |
| 1048 |
of this, users would get spurious errors when doing a ``module avail`` |
| 1049 |
if the ``.modulerc`` scripts error-ed, because they all get processed regardless |
| 1050 |
of what compiler is loaded. |
| 1051 |
* For versions 4.x: the modulecmd process tends to process all ``.modulerc`` |
| 1052 |
files underneath the package root during a load, which will similarly |
| 1053 |
generate spurious errors if the ``.modulerc`` scripts throw errors. |
| 1054 |
|
| 1055 |
Because of this, if an user explicitly requests a modulefile that conflicts |
| 1056 |
with the loaded compiler (e.g. the user does a ``module load pgi/19.4`` |
| 1057 |
followed by a ``module load openmpi/4.0/intel/2019``), the modulefile needs |
| 1058 |
to detect this and error appropriately. A similar situation can happen if |
| 1059 |
one of the ``.modulerc`` scripts fail to default, presumably because there is no |
| 1060 |
appropriate build for the requested package; the modulecmd will default to |
| 1061 |
something, but likely not what is wanted. To facilitate this, we define |
| 1062 |
the Tcl procedure ``LoadedCompilerMatches``: |
| 1063 |
|
| 1064 |
.. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/LoadedCompMatches.tcl |
| 1065 |
:language: tcl |
| 1066 |
:lines: 4- |
| 1067 |
|
| 1068 |
The procedure takes a string for the family and version of the compiler |
| 1069 |
that the modulefile expects, and ensures that if a compiler is loaded, they |
| 1070 |
match. If they match, the procedure just returns, and if not, it spits out |
| 1071 |
an error indicating a compiler mismatch. It also takes optional boolean flags: |
| 1072 |
``requireIt`` to determine if the procedure should ``prereq`` the compiler |
| 1073 |
before returning, and ``loadIt`` to determine if, when no compiler was |
| 1074 |
previously loaded, whether the procedure should load it (using |
| 1075 |
``RequireCompiler`` so it can properly handle the default compiler specially |
| 1076 |
if needed). The ``requireIt`` flag should generally be set for Environment |
| 1077 |
Modules 4.x (as that will allow the automatic handling of modules to |
| 1078 |
properly recognize dependencies, even though as we will see that does not |
| 1079 |
work too well for this strategy as yet), and should not be set for versions |
| 1080 |
3.x (as the loading of modules with ``loadIt`` does not occur until *after* |
| 1081 |
the ``prereq``, causing module loads to fail). |
| 1082 |
|
| 1083 |
The resulting modulefile for something depending only on the compiler, |
| 1084 |
using mvapich as an example, then would look like: |
| 1085 |
|
| 1086 |
.. literalinclude:: ../../example/compiler-etc-dependencies/modulerc4/mvapich/common |
| 1087 |
:language: tcl |
| 1088 |
|
| 1089 |
Basically, the modulefile knows what compiler it wants (in the above example, |
| 1090 |
that is set by the stubfile and passed as the Tcl variable ``compilerTag`` into the common |
| 1091 |
script above), and then calls the ``LoadedCompilerMatches`` procedure |
| 1092 |
above to ensure the loaded compiler matches what the modulefile wants (and |
| 1093 |
to load it if no compiler is loaded). |
| 1094 |
|
| 1095 |
So the mvapich directory in the ``MODULEPATH`` consists of subdirectories for |
| 1096 |
each version of mvapich supported (e.g. ``2.1`` and ``2.3.1``). In each of the |
| 1097 |
version subdirectories, we symlink ``modulerc.select_compiler_family`` as |
| 1098 |
``.modulerc``, and create an additional layer of subdirectories, one for |
| 1099 |
each compiler family for which the version of mvapich is built (e.g. |
| 1100 |
gcc, intel, pgi). In each compiler family directory under each mvapich |
| 1101 |
version, we symlink ``modulerc.select_compiler_version`` as ``.modulerc``, |
| 1102 |
and create a stub modulefile named for the compiler version. The stub |
| 1103 |
modulefile then defines some Tcl variables for the version of mvapich |
| 1104 |
and the compiler family/version, and sources the ``common`` file above. |
| 1105 |
E.g., for ``mvapich/2.3.1/intel/2019``, the stubfile would look like |
| 1106 |
|
| 1107 |
.. literalinclude:: ../../example/compiler-etc-dependencies/modulerc4/mvapich/2.3.1/intel/2019 |
| 1108 |
:language: tcl |
| 1109 |
|
| 1110 |
So the mvapich directory in ``MODULEPATH`` would have a structure like |
| 1111 |
|
| 1112 |
.. code-block:: console |
| 1113 |
|
| 1114 |
mvapich |
| 1115 |
├── 2.1 |
| 1116 |
│ ├── gcc |
| 1117 |
│ │ ├── 8.2.0 |
| 1118 |
│ │ ├── 9.1.0 |
| 1119 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1120 |
│ ├── intel |
| 1121 |
│ │ ├── 2018 |
| 1122 |
│ │ ├── 2019 |
| 1123 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1124 |
│ ├── .modulerc -> symlink to modulerc.select_compiler_family |
| 1125 |
│ └── pgi |
| 1126 |
│ ├── 18.4 |
| 1127 |
│ ├── 19.4 |
| 1128 |
│ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1129 |
├── 2.3.1 |
| 1130 |
│ ├── gcc |
| 1131 |
│ │ ├── 9.1.0 |
| 1132 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1133 |
│ ├── intel |
| 1134 |
│ │ ├── 2019 |
| 1135 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1136 |
│ ├── .modulerc -> symlink to modulerc.select_compiler_family |
| 1137 |
│ └── pgi |
| 1138 |
│ ├── 19.4 |
| 1139 |
│ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1140 |
└── common |
| 1141 |
|
| 1142 |
With this directory structure, an user can load a compiler and then |
| 1143 |
module load a specific version of mvapich, and if a build of that version of |
| 1144 |
mvapich exists for that compiler, the environment will be set up properly, |
| 1145 |
and otherwise an error reported. However, if the user module loads |
| 1146 |
mvapich without specifying a version, it will simply default the version |
| 1147 |
to the latest version of mvapich available, regardless of whether there |
| 1148 |
is a build of that version of mvapich for the loaded compiler. Ideally, |
| 1149 |
we would like it so that |
| 1150 |
if one issues a module load of a package without specifying a version, it |
| 1151 |
should load the latest version available that is compatible with any loaded |
| 1152 |
compilers or other modules. |
| 1153 |
|
| 1154 |
A simple change can enable that. We add a symlink to |
| 1155 |
``modulerc.select_compiler_family`` as ``.modulerc`` directly under the |
| 1156 |
mvapich directory, and create an additional subdirectory under the mvapich |
| 1157 |
directory for each compiler family. We symlink |
| 1158 |
``modulerc.select_compiler_version`` as ``.modulerc`` under each compiler |
| 1159 |
family subdirectory, along with a subdirectory for each version of that |
| 1160 |
compiler family that a version of mvapich is built for. We then symlink |
| 1161 |
the various stub modulefiles under the ``mvapich/VERSION`` into the |
| 1162 |
corresponding ``mvapich/COMPILER_FAMILY/COMPILER_VERSION`` directory, with |
| 1163 |
the symlink's name being the mvapich version number. So the mvapich |
| 1164 |
directory tree will now look like: |
| 1165 |
|
| 1166 |
.. code-block:: console |
| 1167 |
|
| 1168 |
mvapich |
| 1169 |
├── 2.1 |
| 1170 |
│ ├── gcc |
| 1171 |
│ │ ├── 8.2.0 |
| 1172 |
│ │ ├── 9.1.0 |
| 1173 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1174 |
│ ├── intel |
| 1175 |
│ │ ├── 2018 |
| 1176 |
│ │ ├── 2019 |
| 1177 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1178 |
│ ├── .modulerc -> symlink to modulerc.select_compiler_family |
| 1179 |
│ └── pgi |
| 1180 |
│ ├── 18.4 |
| 1181 |
│ ├── 19.4 |
| 1182 |
│ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1183 |
├── 2.3.1 |
| 1184 |
│ ├── gcc |
| 1185 |
│ │ ├── 9.1.0 |
| 1186 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1187 |
│ ├── intel |
| 1188 |
│ │ ├── 2019 |
| 1189 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1190 |
│ ├── .modulerc -> symlink to modulerc.select_compiler_family |
| 1191 |
│ └── pgi |
| 1192 |
│ ├── 19.4 |
| 1193 |
│ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1194 |
├── common |
| 1195 |
├── gcc |
| 1196 |
│ ├── 8.2.0 |
| 1197 |
│ │ └── 2.1 -> symlink to ../../2.1/gcc/8.2.0 |
| 1198 |
│ ├── 9.1.0 |
| 1199 |
│ │ ├── 2.1 -> symlink to ../../2.1/gcc/9.1.0 |
| 1200 |
│ │ └── 2.3.1 -> symlink to ../../2.3.1/gcc/9.1.0 |
| 1201 |
│ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1202 |
├── intel |
| 1203 |
│ ├── 2018 |
| 1204 |
│ │ └── 2.1 -> symlink to ../../2.1/intel/2018 |
| 1205 |
│ ├── 2019 |
| 1206 |
│ │ ├── 2.1 -> symlink to ../../2.1/intel/2019 |
| 1207 |
│ │ └── 2.3.1 -> symlink to ../../2.3.1/intel/2019 |
| 1208 |
│ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1209 |
├── .modulerc -> symlink to modulerc.select_compiler_family |
| 1210 |
└── pgi |
| 1211 |
├── 18.4 |
| 1212 |
│ └── 2.1 -> symlink to ../../2.1/pgi/18.4 |
| 1213 |
├── 19.4 |
| 1214 |
│ ├── 2.1 -> symlink to ../../2.1/pgi/19.4 |
| 1215 |
│ └── 2.3.1 -> symlink to ../../2.3.1/pgi/19.4 |
| 1216 |
└── .modulerc -> symlink to modulerc.select_compiler_version |
| 1217 |
|
| 1218 |
When one attempts to module load mvapich without specifying a version, |
| 1219 |
the ``.modulerc`` file will default to the family of the loaded compiler (or |
| 1220 |
the default compiler if none loaded). It then descends into the corresponding |
| 1221 |
compiler family directory, where the ``.modulerc`` there will default to the |
| 1222 |
version of the loaded or defaulted compiler. After descending into the |
| 1223 |
compiler version directory, there is no ``.modulerc``, so the standard modulecmd |
| 1224 |
defaulting mechanism will select the latest version. |
| 1225 |
|
| 1226 |
We now have two ways to refer to the same build of mvapich, namely |
| 1227 |
one using the traditional form :: |
| 1228 |
|
| 1229 |
PACKAGE/PKGVERSION/COMPILER_FAMILY/COMPILER_VERSION |
| 1230 |
(e.g. mvapich/2.3.1/pgi/19.4) |
| 1231 |
|
| 1232 |
and a new one giving the compiler family and version immediately |
| 1233 |
after package name, with the version of the package coming last :: |
| 1234 |
|
| 1235 |
PACKAGE/COMPILER_FAMILY/COMPILER_VERSION/PKGVERSION |
| 1236 |
(e.g. mvapich/pgi/19.4/2.3.1) |
| 1237 |
|
| 1238 |
These both refer to the same build of the package (e.g. version ``2.3.1`` |
| 1239 |
of mvapich, built for version ``19.4`` of the PGI compiler suite). |
| 1240 |
Indeed, through the judicious use of symlinks, these actually refer to the |
| 1241 |
modulefile on disk, but with two distinct names depending on the path |
| 1242 |
used to get to it. In general, we allow for |
| 1243 |
modules to be named with multiple dependency packages and/or flags if |
| 1244 |
doing so would make it easier for a user to default to the correct package. |
| 1245 |
|
| 1246 |
This is the primary reason for the procedure ``GetPackageFamilyVersion`` --- |
| 1247 |
it splits a module name into components and takes the first component |
| 1248 |
as the family name. It then examines the second component, and if it |
| 1249 |
matches the name of a known package (like a compiler family), it uses |
| 1250 |
the last component as the version. Otherwise, the second component is |
| 1251 |
assumed to be the version. The code is as follows: |
| 1252 |
|
| 1253 |
.. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/GetFamVer.tcl |
| 1254 |
:language: tcl |
| 1255 |
:lines: 4- |
| 1256 |
|
| 1257 |
Similarly, there are two corresponding ``.modulerc`` scripts for defaulting |
| 1258 |
the MPI library: one to default the family (e.g. openmpi, mvapich, intelmpi) |
| 1259 |
and one to default the version. The family version, shown below: |
| 1260 |
|
| 1261 |
.. literalinclude:: ../../example/compiler-etc-dependencies/modrc_common/modulerc.select_mpi_family |
| 1262 |
:language: tcl |
| 1263 |
|
| 1264 |
is similar to the corresponding compiler version, but contains additional |
| 1265 |
logic to handle the intelmpi case. We treat intelmpi specially, because in |
| 1266 |
our example here we assume the environment for Intel MPI is setup properly |
| 1267 |
when the user loads the intel module. If no MPI library is explicitly loaded, |
| 1268 |
but the intel module is loaded, than we look for a subdirectory named |
| 1269 |
intelmpi (or one of its aliases) and default to it if it exists. Otherwise, |
| 1270 |
if no MPI module is loaded, it looks for and will default to a directory |
| 1271 |
or modulefile named nompi if it exists. |
| 1272 |
|
| 1273 |
The ``modulerc.select_mpi_version`` script is also similar to its compiler |
| 1274 |
counterpart, |
| 1275 |
|
| 1276 |
.. literalinclude:: ../../example/compiler-etc-dependencies/modrc_common/modulerc.select_mpi_version |
| 1277 |
:language: tcl |
| 1278 |
|
| 1279 |
It checks if any MPI library was explicitly loaded, and if so it checks |
| 1280 |
if the family of the loaded MPI module matches the name of the parent directory |
| 1281 |
of this ``.modulerc`` file. If so, it checks for the existence of a subdirectory |
| 1282 |
matching the version, and if found it defaults to it. There is no special |
| 1283 |
handling needed for the ``nompi`` case, as since there is no version attached |
| 1284 |
to the MPI library in the nompi case, this ``.modulerc`` is not present there. |
| 1285 |
Similarly, for the case of Intel MPI when the Intel compiler suite is loaded, we |
| 1286 |
expect the Intel MPI version to be that from the Intel compiler suite, so |
| 1287 |
no version or this ``.modulerc`` is needed. |
| 1288 |
|
| 1289 |
As the MPI modules depend themselves on the compiler, it is assumed that |
| 1290 |
any package depending on the MPI libraries also depend on the compiler, |
| 1291 |
and so will have a structure like the one described below for foo |
| 1292 |
|
| 1293 |
.. code-block:: console |
| 1294 |
|
| 1295 |
foo |
| 1296 |
├── 1.1 |
| 1297 |
│ ├── gcc |
| 1298 |
│ │ ├── 8.2.0 |
| 1299 |
│ │ │ ├── .modulerc -> symlink to modulerc.select_mpi_family |
| 1300 |
│ │ │ ├── mvapich |
| 1301 |
│ │ │ │ ├── 2.1 |
| 1302 |
│ │ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1303 |
│ │ │ ├── nompi |
| 1304 |
│ │ │ └── openmpi |
| 1305 |
│ │ │ ├── 3.1 |
| 1306 |
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1307 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1308 |
│ ├── intel |
| 1309 |
│ │ ├── 2018 |
| 1310 |
│ │ │ ├── intelmpi |
| 1311 |
│ │ │ ├── .modulerc -> symlink to modulerc.select_mpi_family |
| 1312 |
│ │ │ ├── mvapich |
| 1313 |
│ │ │ │ ├── 2.1 |
| 1314 |
│ │ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1315 |
│ │ │ ├── nompi |
| 1316 |
│ │ │ └── openmpi |
| 1317 |
│ │ │ ├── 3.1 |
| 1318 |
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1319 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1320 |
│ ├── .modulerc -> symlink to modulerc.select_compiler_family |
| 1321 |
│ └── pgi |
| 1322 |
│ ├── 18.4 |
| 1323 |
│ │ ├── .modulerc -> symlink to modulerc.select_mpi_family |
| 1324 |
│ │ ├── mvapich |
| 1325 |
│ │ │ ├── 2.1 |
| 1326 |
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1327 |
│ │ ├── nompi |
| 1328 |
│ │ └── openmpi |
| 1329 |
│ │ ├── 3.1 |
| 1330 |
│ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1331 |
│ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1332 |
├── 2.4 |
| 1333 |
│ ├── gcc |
| 1334 |
│ │ ├── 9.1.0 |
| 1335 |
│ │ │ ├── .modulerc -> symlink to modulerc.select_mpi_family |
| 1336 |
│ │ │ ├── mvapich |
| 1337 |
│ │ │ │ ├── 2.3.1 |
| 1338 |
│ │ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1339 |
│ │ │ ├── nompi |
| 1340 |
│ │ │ └── openmpi |
| 1341 |
│ │ │ ├── 4.0 |
| 1342 |
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1343 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1344 |
│ ├── intel |
| 1345 |
│ │ ├── 2019 |
| 1346 |
│ │ │ ├── intelmpi |
| 1347 |
│ │ │ ├── .modulerc -> symlink to modulerc.select_mpi_family |
| 1348 |
│ │ │ ├── mvapich |
| 1349 |
│ │ │ │ ├── 2.3.1 |
| 1350 |
│ │ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1351 |
│ │ │ ├── nompi |
| 1352 |
│ │ │ └── openmpi |
| 1353 |
│ │ │ ├── 4.0 |
| 1354 |
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1355 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1356 |
│ ├── .modulerc -> symlink to modulerc.select_compiler_family |
| 1357 |
│ └── pgi |
| 1358 |
│ ├── 19.4 |
| 1359 |
│ │ ├── .modulerc -> symlink to modulerc.select_mpi_family |
| 1360 |
│ │ ├── nompi |
| 1361 |
│ │ └── openmpi |
| 1362 |
│ │ ├── 3.1 |
| 1363 |
│ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1364 |
│ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1365 |
├── common |
| 1366 |
├── gcc |
| 1367 |
│ ├── 8.2.0 |
| 1368 |
│ │ ├── .modulerc -> symlink to modulerc.select_mpi_family |
| 1369 |
│ │ ├── mvapich |
| 1370 |
│ │ │ ├── 2.1 |
| 1371 |
│ │ │ │ └── 1.1 -> ../../../../1.1/gcc/8.2.0/mvapich/2.1 |
| 1372 |
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1373 |
│ │ ├── nompi |
| 1374 |
│ │ │ └── 1.1 -> ../../../1.1/gcc/8.2.0/nompi |
| 1375 |
│ │ └── openmpi |
| 1376 |
│ │ ├── 3.1 |
| 1377 |
│ │ │ └── 1.1 -> ../../../../1.1/gcc/8.2.0/openmpi/3.1 |
| 1378 |
│ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1379 |
│ ├── 9.1.0 |
| 1380 |
│ │ ├── .modulerc -> symlink to modulerc.select_mpi_family |
| 1381 |
│ │ ├── mvapich |
| 1382 |
│ │ │ ├── 2.3.1 |
| 1383 |
│ │ │ │ └── 2.4 -> ../../../../2.4/gcc/9.1.0/mvapich/2.3.1 |
| 1384 |
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1385 |
│ │ ├── nompi |
| 1386 |
│ │ │ └── 2.4 -> ../../../2.4/gcc/9.1.0/nompi |
| 1387 |
│ │ └── openmpi |
| 1388 |
│ │ ├── 4.0 |
| 1389 |
│ │ │ └── 2.4 -> ../../../../2.4/gcc/9.1.0/openmpi/4.0 |
| 1390 |
│ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1391 |
│ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1392 |
├── intel |
| 1393 |
│ ├── 2018 |
| 1394 |
│ │ ├── intelmpi |
| 1395 |
│ │ │ └── 1.1 -> ../../../1.1/intel/2018/intelmpi |
| 1396 |
│ │ ├── .modulerc -> symlink to modulerc.select_mpi_family |
| 1397 |
│ │ ├── mvapich |
| 1398 |
│ │ │ ├── 2.1 |
| 1399 |
│ │ │ │ └── 1.1 -> ../../../../1.1/intel/2018/mvapich/2.1 |
| 1400 |
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1401 |
│ │ ├── nompi |
| 1402 |
│ │ │ └── 1.1 -> ../../../1.1/intel/2018/nompi |
| 1403 |
│ │ └── openmpi |
| 1404 |
│ │ ├── 3.1 |
| 1405 |
│ │ │ └── 1.1 -> ../../../../1.1/intel/2018/openmpi/3.1 |
| 1406 |
│ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1407 |
│ ├── 2019 |
| 1408 |
│ │ ├── intelmpi |
| 1409 |
│ │ │ └── 2.4 -> ../../../2.4/intel/2019/intelmpi |
| 1410 |
│ │ ├── .modulerc -> symlink to modulerc.select_mpi_family |
| 1411 |
│ │ ├── mvapich |
| 1412 |
│ │ │ ├── 2.3.1 |
| 1413 |
│ │ │ │ └── 2.4 -> ../../../../2.4/intel/2019/mvapich/2.3.1 |
| 1414 |
│ │ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1415 |
│ │ ├── nompi |
| 1416 |
│ │ │ └── 2.4 -> ../../../2.4/intel/2019/nompi |
| 1417 |
│ │ └── openmpi |
| 1418 |
│ │ ├── 4.0 |
| 1419 |
│ │ │ └── 2.4 -> ../../../../2.4/intel/2019/openmpi/4.0 |
| 1420 |
│ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1421 |
│ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1422 |
├── .modulerc -> symlink to modulerc.select_compiler_family |
| 1423 |
└── pgi |
| 1424 |
├── 18.4 |
| 1425 |
│ ├── .modulerc -> symlink to modulerc.select_mpi_family |
| 1426 |
│ ├── mvapich |
| 1427 |
│ │ ├── 2.1 |
| 1428 |
│ │ │ └── 1.1 -> ../../../../1.1/pgi/18.4/mvapich/2.1 |
| 1429 |
│ │ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1430 |
│ ├── nompi |
| 1431 |
│ │ └── 1.1 -> ../../../1.1/pgi/18.4/nompi |
| 1432 |
│ └── openmpi |
| 1433 |
│ ├── 3.1 |
| 1434 |
│ │ └── 1.1 -> ../../../../1.1/pgi/18.4/openmpi/3.1 |
| 1435 |
│ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1436 |
├── 19.4 |
| 1437 |
│ ├── .modulerc -> symlink to modulerc.select_mpi_family |
| 1438 |
│ ├── nompi |
| 1439 |
│ │ └── 2.4 -> ../../../2.4/pgi/19.4/nompi |
| 1440 |
│ └── openmpi |
| 1441 |
│ ├── 3.1 |
| 1442 |
│ │ └── 2.4 -> ../../../../2.4/pgi/19.4/openmpi/3.1 |
| 1443 |
│ └── .modulerc -> symlink to modulerc.select_mpi_version |
| 1444 |
└── .modulerc -> symlink to modulerc.select_compiler_version |
| 1445 |
|
| 1446 |
Although the directory tree above is somewhat lengthy, it is similar |
| 1447 |
to the case for a module depending only on the compiler, but expanded |
| 1448 |
to handle MPI dependencies as well. There is a subdirectory under ``foo`` |
| 1449 |
for each version of foo. Each of these subdirectories have further subdirectories |
| 1450 |
for the compiler families for which that version of foo was built, and under |
| 1451 |
the subdirectories for each compiler family are subdirectories for each version |
| 1452 |
of that compiler family for which foo was built. Underneath these |
| 1453 |
are stub modulefiles for nompi (and for intel compilers, intelmpi) builds of |
| 1454 |
the package, and subdirectories for each MPI family, containing stub modulefiles |
| 1455 |
for each version of the MPI family the package was built for. |
| 1456 |
|
| 1457 |
Additionally, there are subdirectories directly under ``foo`` for each compiler |
| 1458 |
family the package was built for, with each having subdirectories for the |
| 1459 |
version of the compiler. Beneath each of those are one or two more layers |
| 1460 |
of subdirectories for the MPI family and version (the version layer is omitted |
| 1461 |
in the nompi or intelmpi cases), underneath which are symlinks to the corresponding |
| 1462 |
stub modulefile from the ``foo/FOOVERSION`` path. |
| 1463 |
|
| 1464 |
By placing the appropriate ``.modulerc`` files (actually, symlinks to the correct |
| 1465 |
modulerc file), when the user enters a partial modulename the modulecmd will |
| 1466 |
descend this directory tree based on the previously loaded compiler and MPI |
| 1467 |
library to get to correct build of the package, if it exists. If no MPI library |
| 1468 |
was previously loaded, it will search for an intelmpi build if an intel compiler |
| 1469 |
was loaded, or a nompi build if no intelmpi build found or a non-intel compiler |
| 1470 |
was used. If only MPI builds were made, it will default (using standard |
| 1471 |
module defaulting rules) to one of the MPI builds, and the appropriate MPI |
| 1472 |
library will be loaded when the foo module gets loaded (via the ``LoadedMPIMatches`` |
| 1473 |
Tcl procedure described below). If a compiler and MPI library were loaded but no |
| 1474 |
build for that combination exists, again a modulefile will be defaulted to |
| 1475 |
(using standard module defaulting rules), but an error will be generated by |
| 1476 |
the one of the ``LoadedCompilerMatches`` or ``LoadedMPIMatches`` calls in the |
| 1477 |
foo modulefile (see below). |
| 1478 |
|
| 1479 |
The common code of the modulefile is fairly standard, as shown below |
| 1480 |
|
| 1481 |
.. literalinclude:: ../../example/compiler-etc-dependencies/modulerc4/foo/common |
| 1482 |
:language: tcl |
| 1483 |
|
| 1484 |
The main difference from a standard modulefile is the inclusion of the |
| 1485 |
invocations of ``LoadedCompilerMatches`` and ``LoadedMpiMatches``. This |
| 1486 |
is needed to ensure that the compiler and MPI for the modulefile being |
| 1487 |
processed are compatible with any which are loaded. The ``LoadedMpiMatches`` |
| 1488 |
procedure is defined by: |
| 1489 |
|
| 1490 |
.. literalinclude:: ../../example/compiler-etc-dependencies/tcllib/LoadedMpiMatches.tcl |
| 1491 |
:language: tcl |
| 1492 |
:lines: 4- |
| 1493 |
|
| 1494 |
Basically, it determines what if any MPI library was previously module loaded. |
| 1495 |
If one was loaded, it ensures the family and version match what was expected, |
| 1496 |
and if not exit with an error. The use of the ``GetPackageFamilyVersion`` |
| 1497 |
procedure is important, because as discussed previously packages can be |
| 1498 |
represented by more than one name (e.g. the modulefile for version ``4.0`` of |
| 1499 |
openmpi for ``gcc/9.1.0`` could be named ``openmpi/4.0/gcc/9.1.0`` or |
| 1500 |
``openmpi/gcc/9.1.0/4.0``) and we need to ensure either is recognized. |
| 1501 |
|
| 1502 |
Although we only showed cases for compilers and MPI libraries, the techniques |
| 1503 |
above can be adapted to other cases as well. For simple cases, where everything |
| 1504 |
falls under a single package name, one can use ``GetTagOfModuleLoaded`` to |
| 1505 |
determine the version of the package loaded in the ``.modulerc`` and in the |
| 1506 |
modulefile to ensure the version matches. |
| 1507 |
|
| 1508 |
For bar, we added variants on CPU vectorization support. Rather than add |
| 1509 |
a simd modulefile, we just add variants to the bar modules. So the bar |
| 1510 |
modulefile tree would look like: |
| 1511 |
|
| 1512 |
.. code-block:: console |
| 1513 |
|
| 1514 |
bar |
| 1515 |
├── 4.7 |
| 1516 |
│ ├── gcc |
| 1517 |
│ │ ├── 8.2.0 |
| 1518 |
│ │ │ ├── avx |
| 1519 |
│ │ │ ├── .modulerc -> symlink to modulerc.default_lowest_simd |
| 1520 |
│ │ │ └── sse4.1 |
| 1521 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1522 |
│ └── .modulerc -> symlink to modulerc.select_compiler_family |
| 1523 |
├── 5.4 |
| 1524 |
│ ├── gcc |
| 1525 |
│ │ ├── 9.1.0 |
| 1526 |
│ │ │ ├── avx |
| 1527 |
│ │ │ └── avx2 |
| 1528 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1529 |
│ └── .modulerc -> symlink to modulerc.select_compiler_family |
| 1530 |
├── avx |
| 1531 |
│ ├── gcc |
| 1532 |
│ │ ├── 8.2.0 |
| 1533 |
│ │ │ └── 4.7 -> ../../../4.7/gcc/8.2.0/avx |
| 1534 |
│ │ ├── 9.1.0 |
| 1535 |
│ │ │ └── 5.4 -> ../../../5.4/gcc/9.1.0/avx |
| 1536 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1537 |
│ └── .modulerc -> symlink to modulerc.select_compiler_family |
| 1538 |
├── avx2 |
| 1539 |
│ ├── gcc |
| 1540 |
│ │ ├── 9.1.0 |
| 1541 |
│ │ │ └── 5.4 -> ../../../5.4/gcc/9.1.0/avx2 |
| 1542 |
│ │ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1543 |
│ └── .modulerc -> symlink to modulerc.select_compiler_family |
| 1544 |
├── common |
| 1545 |
├── gcc |
| 1546 |
│ ├── 8.2.0 |
| 1547 |
│ │ ├── avx |
| 1548 |
│ │ │ └── 4.7 -> ../../../4.7/gcc/8.2.0/avx |
| 1549 |
│ │ ├── .modulerc -> symlink to modulerc.default_lowest_simd |
| 1550 |
│ │ └── sse4.1 |
| 1551 |
│ │ └── 4.7 -> ../../../4.7/gcc/8.2.0/sse4.1 |
| 1552 |
│ ├── 9.1.0 |
| 1553 |
│ │ ├── avx |
| 1554 |
│ │ │ └── 5.4 -> ../../../5.4/gcc/9.1.0/avx |
| 1555 |
│ │ ├── avx2 |
| 1556 |
│ │ │ └── 5.4 -> ../../../5.4/gcc/9.1.0/avx2 |
| 1557 |
│ │ └── .modulerc -> symlink to modulerc.default_lowest_simd |
| 1558 |
│ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1559 |
├── .modulerc -> symlink to modulerc.select_compiler_family |
| 1560 |
└── sse4.1 |
| 1561 |
├── gcc |
| 1562 |
│ ├── 8.2.0 |
| 1563 |
│ │ └── 4.7 -> ../../../4.7/gcc/8.2.0/sse4.1 |
| 1564 |
│ └── .modulerc -> symlink to modulerc.select_compiler_version |
| 1565 |
└── .modulerc -> symlink to modulerc.select_compiler_family |
| 1566 |
|
| 1567 |
Here we added yet another alternate set of module names. So the module |
| 1568 |
for bar version ``5.4`` built with gcc version ``9.1.0`` and ``avx2`` support can be |
| 1569 |
called: |
| 1570 |
|
| 1571 |
* ``bar/5.4/gcc/9.1.0/avx2`` |
| 1572 |
* ``bar/gcc/9.1.0/avx2/5.4`` |
| 1573 |
* ``bar/avx2/gcc/9.1.0/5.4`` |
| 1574 |
|
| 1575 |
Due to this multiplicity of names, if an user does a module load bar, the |
| 1576 |
``.modulerc`` immediately under bar will cause modulecmd to default along the |
| 1577 |
path for the loaded (or defaulted) compiler, and then the |
| 1578 |
``modulerc.default_lowest_simd`` will default to the lowest simd level. |
| 1579 |
If the user module loads bar/VERSION, it will find the build for that version |
| 1580 |
of bar with the appropriate compiler (and again, default to the lowest simd |
| 1581 |
level). And if the user module loads ``bar/avx`` or another simd level, then |
| 1582 |
it will load the latest version of bar built for the loaded compiler and |
| 1583 |
simd specified. |
| 1584 |
|
| 1585 |
The ``modulerc.default_lowest_simd`` script looks like: |
| 1586 |
|
| 1587 |
.. literalinclude:: ../../example/compiler-etc-dependencies/modrc_common/modulerc.default_lowest_simd |
| 1588 |
:language: tcl |
| 1589 |
|
| 1590 |
It will default to the lowest SIMD level, but could easily be adapted to |
| 1591 |
do something else. |
| 1592 |
|
| 1593 |
Examples |
| 1594 |
^^^^^^^^ |
| 1595 |
|
| 1596 |
We now look at the example modulefiles for the *Modulerc-based* strategy. |
| 1597 |
As noted previously, the best choice of whether to set the ``requireIt`` flag |
| 1598 |
to ``LoadedCompilerMatches`` and ``LoadedMPIMatches`` (instructing them to |
| 1599 |
to a ``prereq`` on the requesting compiler/MPI module) depends on whether |
| 1600 |
one is using a 3.x or 4.x version of Environment Modules. Due to this, |
| 1601 |
we provide two modulefile trees, one for 3.x and one for 4.x; they are |
| 1602 |
basically identical except for that matter. This leads to slightly |
| 1603 |
different instructions on how to use the examples, depending on which |
| 1604 |
version of Environment Modules is being used, namely: |
| 1605 |
|
| 1606 |
#. Set (and export) ``MOD_GIT_ROOTDIR`` to where you git-cloned the modules source |
| 1607 |
#. Do a ``module purge`` |
| 1608 |
#. If using Environment Modules 3.x, set your ``MODULEPATH`` to ``$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc3`` |
| 1609 |
#. If using Environment Modules 4.x, set your ``MODULEPATH`` to ``$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulerc4`` |
| 1610 |
|
| 1611 |
As with the previous cases, we start with a ``module avail`` command, and here |
| 1612 |
we see the first big difference: |
| 1613 |
|
| 1614 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/modavail.out |
| 1615 |
:language: console |
| 1616 |
|
| 1617 |
Unlike the previous cases wherein only package names and versions were shown |
| 1618 |
(because a single modulefile handled all the builds for that package and |
| 1619 |
version), here we see a listing for every build of a package and version. |
| 1620 |
Indeed, we not only see one such listing, but multiple listings per |
| 1621 |
build in many cases (e.g. ``openmpi/3.1/intel/2018`` and |
| 1622 |
``openmpi/intel/2018/3.1``). |
| 1623 |
|
| 1624 |
While admittedly the output of a ``module avail`` command without |
| 1625 |
specifying any package is rather overwhelming, when a package is specified |
| 1626 |
the output tends to be more reasonable, informing one of which builds |
| 1627 |
of the package are available. This strategy deliberately opts for the |
| 1628 |
presentation of more rather than less information. |
| 1629 |
|
| 1630 |
The standard functionality of selecting the correct build of a package |
| 1631 |
based on the loaded compiler, e.g. |
| 1632 |
|
| 1633 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/ompi-loads1.out |
| 1634 |
:language: console |
| 1635 |
|
| 1636 |
works as expected (and fails with the expected error when one attempts to |
| 1637 |
load a module not compatible with the loaded compiler). |
| 1638 |
The only significant difference between the previously |
| 1639 |
examined strategies is that the module list command provides information |
| 1640 |
about what variant of each package is loaded. |
| 1641 |
|
| 1642 |
The ``module switch`` command, however, does not work as well as one would |
| 1643 |
like. While it indeed switches the specified module, it does not |
| 1644 |
successfully reload the modules which depend on the replaced module, even |
| 1645 |
with the :ref:`Automated module handling <v42-automated-module-handling-mode>` |
| 1646 |
feature enabled. As currently implemented, |
| 1647 |
the automated module handling feature attempts to reload dependent modules |
| 1648 |
using the fully qualified module name, and as in this strategy the fully |
| 1649 |
qualified modulename includes the information about the module that was switched |
| 1650 |
out (e.g. ``pgi/19.4`` in the example below), it will be incompatible with the |
| 1651 |
replacement module (e.g. ``intel/2019`` in example below). It |
| 1652 |
is hoped a future version of modulecmd will allow for reloading based on the |
| 1653 |
name specified when the module was loaded. But as things currently stand, |
| 1654 |
the automatic module handling will throw an error attempting to reload the |
| 1655 |
depend module, resulting in the the dependent modules being unloaded. |
| 1656 |
|
| 1657 |
Without |
| 1658 |
automated module handling mode (i.e. for older Environment Modules or without |
| 1659 |
the --auto flag), the dependent modules remain loaded and there is inconsistency |
| 1660 |
in the loaded modules. But at least module list clearly shows such. |
| 1661 |
|
| 1662 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/ompi-switch.out |
| 1663 |
:language: console |
| 1664 |
|
| 1665 |
The defaulting of modules is more successful, however, as seen below: |
| 1666 |
|
| 1667 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/ompi-defaults.out |
| 1668 |
:language: console |
| 1669 |
|
| 1670 |
Here it not only defaults to the default compiler, but if one tries to load |
| 1671 |
openmpi without specifying a version, it will find and load the latest version |
| 1672 |
compatible with the loaded compiler. |
| 1673 |
|
| 1674 |
The situation is similar for foo: |
| 1675 |
|
| 1676 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/foo-loads.out |
| 1677 |
:language: console |
| 1678 |
|
| 1679 |
As expected, the correct version of foo is loaded depending on the previously |
| 1680 |
loaded compiler and MPI libraries. Note however that there is no dummy |
| 1681 |
module for Intel MPI, and that when the intel compiler is loaded but no MPI |
| 1682 |
library is explicitly loaded, module load foo/2.4 gets the Intel MPI build. |
| 1683 |
Such only occurs if there is an Intel MPI build in the tree of modulefiles; |
| 1684 |
otherwise a nompi build will be loaded if present. If both versions are |
| 1685 |
present, as in this case, one needs to give the explicit |
| 1686 |
``foo/2.4/intel/2019/nompi`` specification. |
| 1687 |
|
| 1688 |
Again, the ``module switch`` command only succeeds in switching the |
| 1689 |
specified module, and does not successfully reload the modules depending |
| 1690 |
on the switched module. With automated module handling mode, there will be errors |
| 1691 |
reported and the dependent modules will be unloaded; without it the dependent |
| 1692 |
modules will not get unloaded, and there will be inconsistent dependencies |
| 1693 |
(but at least module list will show such). |
| 1694 |
|
| 1695 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/foo-switch.out |
| 1696 |
:language: console |
| 1697 |
|
| 1698 |
The defaulting of modules works relatively well, as shown below: |
| 1699 |
|
| 1700 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/foo-defaults.out |
| 1701 |
:language: console |
| 1702 |
|
| 1703 |
If one attempts to load foo without specifying a version or having previously loaded a compiler |
| 1704 |
module, the compiler will be defaulted (to ``gcc/8.2.0`` compiler, as that is what |
| 1705 |
we declared via the ``GetDefaultCompiler`` Tcl procedure), and the latest |
| 1706 |
version of foo compatible with that compiler (and no MPI) will be loaded. |
| 1707 |
|
| 1708 |
The situation with bar is similar. We do not have a dummy simd module, |
| 1709 |
so the builds with different CPU vectorization support are specified by |
| 1710 |
appending ``/avx``, etc. to the bar package name. As with previous strategies, |
| 1711 |
if one attempts to load a simd variant which was not built for the compiler |
| 1712 |
loaded, an error will occur. |
| 1713 |
|
| 1714 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/bar-loads.out |
| 1715 |
:language: console |
| 1716 |
|
| 1717 |
Summary of Modulerc-based strategy |
| 1718 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1719 |
|
| 1720 |
* The modulefiles are fairly standard. The only extra logic needed |
| 1721 |
is to ensure that any loaded compiler/MPI or other dependency matches |
| 1722 |
what the modulefile expects, and that can be handled fairly easily |
| 1723 |
with the addition of one or two calls to Tcl procedures. |
| 1724 |
* The ``.modulerc`` files do the work of routing partial specifications to |
| 1725 |
the correct modulefile, and follow a fairly standard pattern. For the |
| 1726 |
most part, these are generic/independent of the package, and typically |
| 1727 |
can be written once and symlinked to the appropriate places in the |
| 1728 |
directory tree. |
| 1729 |
* This strategy involves the use of many more modulefiles than the |
| 1730 |
previously examined strategies, having at least one modulefile per |
| 1731 |
build, and in some cases multiple per build. The output of the module |
| 1732 |
avail command without any module specified is rather overwhelming, |
| 1733 |
although when a module is specified, module avail will provide information |
| 1734 |
about what builds are available. |
| 1735 |
* Because each build has a specific module, the module list command shows |
| 1736 |
exactly what builds of the various modules are loaded. |
| 1737 |
* The module switch command does not work very well. In |
| 1738 |
particular, when one switches out a module on which other modules depend, |
| 1739 |
the dependent modules are not successfully reloaded. Even with the |
| 1740 |
:ref:`Automated module handling <v42-automated-module-handling-mode>` |
| 1741 |
feature (introduced in ``4.2``), the dependent |
| 1742 |
modules are unloaded but do not get reloaded properly since currently |
| 1743 |
this feature attempts to reload based on the fully qualified name of the |
| 1744 |
loaded module (which includes a dependency on the module switched out), |
| 1745 |
not the specification used when the module was loaded. |
| 1746 |
* In general, when the user tries to load a module based on a partially |
| 1747 |
specified modulename, the ``.modulerc`` files handle it well, and the |
| 1748 |
latest version of the module consistent with the specification and |
| 1749 |
previously loaded modules will be used. If no compiler was previously |
| 1750 |
loaded, will use the default compiler, or the latest compiler if nothing |
| 1751 |
matches the default compiler. |
| 1752 |
* Modules will fail with an error message if user tries to load a package |
| 1753 |
which was not built for the values of the dependency packages loaded. |
| 1754 |
|
| 1755 |
|
| 1756 |
.. _Modulepath-based_strategy: |
| 1757 |
|
| 1758 |
Modulepath-based Strategy |
| 1759 |
------------------------- |
| 1760 |
|
| 1761 |
This strategy makes use of the ability of modules to support multiple |
| 1762 |
directories in the ``MODULEPATH`` environment variable. Every time a module is loaded |
| 1763 |
on which other modules might depend, a new path is added to ``MODULEPATH`` |
| 1764 |
containing the modulefiles which depend on the newly added module. |
| 1765 |
|
| 1766 |
This is basically similar to the strategy that is used in the hierarchical |
| 1767 |
modulefile approach of Lmod_. |
| 1768 |
|
| 1769 |
.. _Lmod: https://github.com/TACC/Lmod |
| 1770 |
|
| 1771 |
Implementation |
| 1772 |
^^^^^^^^^^^^^^ |
| 1773 |
|
| 1774 |
The :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` and |
| 1775 |
:ref:`Modulerc-based <Modulerc-based_strategy>` strategies were based on |
| 1776 |
modifying the modulefiles of modules which depended on other modules. |
| 1777 |
The *Modulepath-based* strategy, in contrast, is based on modifying the |
| 1778 |
modulefiles for modules which other modules might depend on. |
| 1779 |
|
| 1780 |
To begin with, we set ``MODULEPATH`` to a ``Core`` directory containing |
| 1781 |
modulefiles for modules which do not depend on any other modules. |
| 1782 |
The modulefiles for compilers will typically be put in here. |
| 1783 |
For this example, we opt not to use a dummy simd module (for reasons |
| 1784 |
explained later), but instead add ``avx``, ``avx2``, ``sse4.1`` variants to the |
| 1785 |
bar modules, but otherwise they might belong here. Other modules which |
| 1786 |
do not depend on compilers, etc. could be place in here as well, e.g. |
| 1787 |
applications which do not expose libraries, etc. So the directory |
| 1788 |
structure would look like: |
| 1789 |
|
| 1790 |
.. code-block:: console |
| 1791 |
|
| 1792 |
Core |
| 1793 |
├── gcc |
| 1794 |
│ ├── 8.2.0 |
| 1795 |
│ ├── 9.1.0 |
| 1796 |
│ └── common |
| 1797 |
├── intel |
| 1798 |
│ ├── 2018 |
| 1799 |
│ ├── 2019 |
| 1800 |
│ └── common |
| 1801 |
└── pgi |
| 1802 |
├── 18.4 |
| 1803 |
├── 19.4 |
| 1804 |
└── common |
| 1805 |
|
| 1806 |
A typical common file for the gcc compiler would be something like |
| 1807 |
|
| 1808 |
.. literalinclude:: ../../example/compiler-etc-dependencies/modulepath/Core/gcc/common |
| 1809 |
:language: tcl |
| 1810 |
|
| 1811 |
The most interesting aspect is the ``module use`` at the end. We add |
| 1812 |
to ``MODULEPATH`` a directory under ``Compiler`` for this specific compiler |
| 1813 |
and version. Other compilers/versions would add their own specific path |
| 1814 |
to ``MODULEPATH``. |
| 1815 |
|
| 1816 |
In each of these compiler specific directories, modulefile directory |
| 1817 |
trees exist for the supported MPI libraries and bar. We also include |
| 1818 |
a foo directory containing the non-MPI variants of foo. It would look |
| 1819 |
like: |
| 1820 |
|
| 1821 |
.. code-block:: console |
| 1822 |
|
| 1823 |
Compiler |
| 1824 |
├── gcc |
| 1825 |
│ ├── 8.2.0 |
| 1826 |
│ │ ├── bar |
| 1827 |
│ │ │ ├── 4.7 |
| 1828 |
│ │ │ │ ├── avx |
| 1829 |
│ │ │ │ ├── .modulerc -> symlink to modulerc.default_lowest_simd |
| 1830 |
│ │ │ │ └── sse4.1 |
| 1831 |
│ │ │ └── common -> symlink to bar/common |
| 1832 |
│ │ ├── foo |
| 1833 |
│ │ │ ├── 1.1 |
| 1834 |
│ │ │ └── common -> symlink to foo/common |
| 1835 |
│ │ ├── mvapich |
| 1836 |
│ │ │ ├── 2.1 |
| 1837 |
│ │ │ └── common -> symlink to mvapich/common |
| 1838 |
│ │ └── openmpi |
| 1839 |
│ │ ├── 3.1 |
| 1840 |
│ │ └── common -> symlink to openmpi/common |
| 1841 |
│ └── 9.1.0 |
| 1842 |
│ ├── bar |
| 1843 |
│ │ ├── 5.4 |
| 1844 |
│ │ │ ├── avx |
| 1845 |
│ │ │ ├── avx2 |
| 1846 |
│ │ │ └── .modulerc -> symlink to modulerc.default_lowest_simd |
| 1847 |
│ │ └── common -> symlink to bar/common |
| 1848 |
│ ├── foo |
| 1849 |
│ │ ├── 2.4 |
| 1850 |
│ │ └── common -> symlink to foo/common |
| 1851 |
│ ├── mvapich |
| 1852 |
│ │ ├── 2.1 |
| 1853 |
│ │ ├── 2.3.1 |
| 1854 |
│ │ └── common -> symlink to mvapich/common |
| 1855 |
│ └── openmpi |
| 1856 |
│ ├── 3.1 |
| 1857 |
│ ├── 4.0 |
| 1858 |
│ └── common -> symlink to openmpi/common |
| 1859 |
├── intel |
| 1860 |
│ ├── 2018 |
| 1861 |
│ │ └── ... modules depending on compiler=intel/2018 |
| 1862 |
│ └── 2019 |
| 1863 |
│ └── ... modules depending on compiler=intel/2019 |
| 1864 |
└── pgi |
| 1865 |
├── 18.4 |
| 1866 |
│ └── ... modules depending on compiler=pgi/18.4 |
| 1867 |
└── 19.4 |
| 1868 |
└── ... modules depending on compiler=pgi/18.4 |
| 1869 |
|
| 1870 |
For brevity, we only show the directory structure for the modules depending on |
| 1871 |
the two versions of gcc in detail; the other compilers will have similar |
| 1872 |
structures. |
| 1873 |
|
| 1874 |
Basically, there is a separate modulepath for each compiler, containing only the |
| 1875 |
modulefiles depending on that specific build of the compiler (and not also depending |
| 1876 |
on something else, like MPI library). |
| 1877 |
This certainly enforces the consistency of loaded modules; one could not load |
| 1878 |
a specific version of gcc (say ``gcc/9.1.0``) and an incompatible version of foo (e.g. ``foo/1.1``), |
| 1879 |
because all of the foo modulefiles are in compiler specific module trees and there is |
| 1880 |
no ``foo/1.1`` in the ``gcc/9.1.0`` module tree. Conflict statements in the compiler modulefiles |
| 1881 |
will prevent one from loading multiple compilers, thereby preventing multiple compiler |
| 1882 |
specific modulepaths (unless the user explicitly does a ``module use`` or similar, and |
| 1883 |
there is only so far one can go in preventing users from shooting themselves in the foot). |
| 1884 |
|
| 1885 |
The modulefiles for foo and bar are fairly straightforward; we have a common file which |
| 1886 |
does the heavy lifting (and since this can be made independent of version and compiler, this |
| 1887 |
is actually a symlink to a package specific common file in a ``common`` directory external |
| 1888 |
to the modulepath trees). |
| 1889 |
We then add some small stubfiles which set variables for the specific build information, which |
| 1890 |
mostly are determined by their position in the tree structure (e.g. the stubfiles under |
| 1891 |
``Compiler/gcc/9.1.0`` are all going to set ``compilerTag`` to ``gcc/9.1.0``, etc), and |
| 1892 |
then invoke the common file. |
| 1893 |
|
| 1894 |
The modulefiles for openmpi are largely similar. E.g., for ``gcc/9.1.0`` we have a small |
| 1895 |
stubfile like the following (for version ``4.0``) |
| 1896 |
|
| 1897 |
.. literalinclude:: ../../example/compiler-etc-dependencies/modulepath/Compiler/gcc/9.1.0/openmpi/4.0 |
| 1898 |
:language: tcl |
| 1899 |
|
| 1900 |
which defines the version and compilerTag variables for the OpenMPI version and |
| 1901 |
compiler version, and then invokes the common script |
| 1902 |
|
| 1903 |
.. literalinclude:: ../../example/compiler-etc-dependencies/modulepath/Compiler/gcc/9.1.0/openmpi/common |
| 1904 |
:language: tcl |
| 1905 |
|
| 1906 |
which does the usual stuff (define a help function, whatis string, and sets assorted environment |
| 1907 |
variables like ``PATH``, ``LIBRARY_PATH``, etc. for using OpenMPI), and then adds another directory |
| 1908 |
to the ``MODULEPATH``. This new directory, under ``CompilerMPI`` is for modulefiles which |
| 1909 |
depend on the compiler AND the MPI library. (It is assumed that all modulefiles depending on |
| 1910 |
MPI libraries also depend on the compiler). This new branch in the modulepath would have |
| 1911 |
a structure like |
| 1912 |
|
| 1913 |
.. code-block:: console |
| 1914 |
|
| 1915 |
CompilerMPI |
| 1916 |
├── gcc |
| 1917 |
│ ├── 8.2.0 |
| 1918 |
│ │ ├── mvapich |
| 1919 |
│ │ │ └── 2.1 |
| 1920 |
│ │ │ └── foo |
| 1921 |
│ │ │ ├── 1.1 |
| 1922 |
│ │ │ └── common -> symlink to foo/common |
| 1923 |
│ │ └── openmpi |
| 1924 |
│ │ └── 3.1 |
| 1925 |
│ │ └── foo |
| 1926 |
│ │ ├── 1.1 |
| 1927 |
│ │ └── common -> symlink to foo/common |
| 1928 |
│ └── 9.1.0 |
| 1929 |
│ ├── mvapich |
| 1930 |
│ │ └── 2.3.1 |
| 1931 |
│ │ └── ... modules depending on gcc/9.1.0 and mvapich/2.3.1 |
| 1932 |
│ └── openmpi |
| 1933 |
│ └── 4.0 |
| 1934 |
│ └── ... modules depending on gcc/9.1.0 and openmpi/4.0 |
| 1935 |
├── intel |
| 1936 |
│ ├── 2018 |
| 1937 |
│ │ ├── intelmpi |
| 1938 |
│ │ │ └── default |
| 1939 |
│ │ │ └── ... modules depending on intel/2018 and its included MPI lib |
| 1940 |
│ │ ├── mvapich |
| 1941 |
│ │ │ └── 2.1 |
| 1942 |
│ │ │ └── ... modules depending on intel/2018 and mvapich/2.1 |
| 1943 |
│ │ └── openmpi |
| 1944 |
│ │ └── 3.1 |
| 1945 |
│ │ └── ... modules depending on intel/2018 and openmpi/3.1 |
| 1946 |
│ └── 2019 |
| 1947 |
│ ├── intelmpi |
| 1948 |
│ │ └── default |
| 1949 |
│ │ └── ... modules depending on intel/2019 and its included MPI lib |
| 1950 |
│ ├── mvapich |
| 1951 |
│ │ └── 2.3.1 |
| 1952 |
│ │ └── ... modules depending on intel/2019 and mvapich/2.3.1 |
| 1953 |
│ └── openmpi |
| 1954 |
│ └── 4.0 |
| 1955 |
│ └── ... modules depending on intel/2019 and openmpi/4.0 |
| 1956 |
└── pgi |
| 1957 |
├── 18.4 |
| 1958 |
│ ├── mvapich |
| 1959 |
│ │ └── 2.1 |
| 1960 |
│ │ └── ... modules depending on pgi/18.4 and mvapich/2.1 |
| 1961 |
│ └── openmpi |
| 1962 |
│ └── 3.1 |
| 1963 |
│ └── ... modules depending on pgi/18.4 and openmpi/3.1 |
| 1964 |
└── 19.4 |
| 1965 |
└── openmpi |
| 1966 |
├── 3.1 |
| 1967 |
│ └── ... modules depending on pgi/19.4 and openmpi/3.1 |
| 1968 |
└── 4.0 |
| 1969 |
└── ... modules depending on pgi/19.4 and openmpi/3.1 |
| 1970 |
|
| 1971 |
Basically, there is a separate modulepath for each combination of compiler and |
| 1972 |
MPI library, listing all modulefiles depending on that compiler and MPI library. |
| 1973 |
The modulefiles underneath each directory simply do what is needed to load that |
| 1974 |
version of the package built with the specified compiler and MPI library. |
| 1975 |
|
| 1976 |
This process could be continued further, as needed. E.g., if you had packages which |
| 1977 |
depended on NetCDF, and you had multiple builds of NetCDF for a given compiler/MPI |
| 1978 |
combination, you could add another modulepath tree, e.g. ``CompilerMPINetCDF``, |
| 1979 |
branching on each compiler, MPI, and NetCDF triplet. But things can also get |
| 1980 |
unwieldy if carried too far. |
| 1981 |
|
| 1982 |
One could also add modulepath trees for disjoint features like SIMD level, |
| 1983 |
but this turns out to be a bit tricky. E.g., if you had a simd branch as well |
| 1984 |
as a compiler branch, with both simd and compiler appearing in Core, things are |
| 1985 |
fine for the Simd and Compiler trees. However, if there were to be modules depending |
| 1986 |
on both, e.g. a CompilerSimd branch, then because compiler and simd are disjoint |
| 1987 |
and could be loaded in either order, the modulefiles for both would need to handle |
| 1988 |
adding the CompilerSimd branch depending on whether the other was previously loaded. |
| 1989 |
|
| 1990 |
Examples |
| 1991 |
^^^^^^^^ |
| 1992 |
|
| 1993 |
We now look at the example usage for the *Modulepath-based* strategy. |
| 1994 |
To use these examples, you must: |
| 1995 |
|
| 1996 |
#. Set (and export) ``MOD_GIT_ROOTDIR`` to where you git-cloned the modules source |
| 1997 |
#. Do a ``module purge`` |
| 1998 |
#. Set your ``MODULEPATH`` to ``$MOD_GIT_ROOTDIR/doc/example/compiler-etc-dependencies/modulepath/Core`` |
| 1999 |
|
| 2000 |
Note that we set the ``MODULEPATH`` to the Core subdirectory; the Core branch is for those modulefiles |
| 2001 |
that do not depend on compiler or MPI library, and that should be available from the start. |
| 2002 |
|
| 2003 |
As with the previous cases, we start with a ``module avail`` command, and here |
| 2004 |
we see the first big difference: |
| 2005 |
|
| 2006 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/modavail.out |
| 2007 |
:language: console |
| 2008 |
|
| 2009 |
When we first do a ``module avail``, we only see the modulefiles for the compilers. |
| 2010 |
The MPI libraries, foo, bar, etc. all depend on at least the compiler, and so |
| 2011 |
will not become "available" until a compiler (and possibly MPI library as well) is loaded. |
| 2012 |
In a production environment there would likely be other modulefiles available in Core |
| 2013 |
(i.e. an application which is only used as a standalone application and does not |
| 2014 |
provide an API/libraries to link against would likely be placed in Core), but in our |
| 2015 |
simple example, only the compilers appear in Core. As we load a compiler and then an |
| 2016 |
MPI library, additional modulefiles appear available. In a production environment, one |
| 2017 |
might wish to set things up so that a compiler (and maybe MPI library) modulefile is |
| 2018 |
automatically loaded when the user logs in. |
| 2019 |
|
| 2020 |
The downside of this is that it becomes difficult for an user to know what software |
| 2021 |
is available, at least via the module command. I.e., if application foobar is only built |
| 2022 |
for a single compiler/MPI combination, it will not show up in module avail unless that |
| 2023 |
specific compiler/MPI combination were previously loaded. Lmod_ adds a ``module spider`` |
| 2024 |
command which allows the user to list all packages installed for any compiler/MPI |
| 2025 |
combination, and can be used to also find out which compiler/MPI combinations are needed |
| 2026 |
to access a specific modulefile, but Environment Modules does not have a comparable function |
| 2027 |
at this time. If you were to use this strategy in a production environment, you would |
| 2028 |
likely need to generate lists of available packages (and their compiler/MPI/etc dependencies) |
| 2029 |
in a web page or other documentation area which can be frequently updated. |
| 2030 |
|
| 2031 |
The standard functionality of selecting the correct build of a package |
| 2032 |
based on the loaded compiler, e.g. |
| 2033 |
|
| 2034 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/ompi-loads1.out |
| 2035 |
:language: console |
| 2036 |
|
| 2037 |
works as expected. As shown in the last command above, when requesting a module not |
| 2038 |
built for the loaded compiler (e.g. ``openmpi/4.0`` when ``gcc/8.2.0`` is loaded), the module command |
| 2039 |
simply returns ``Unable to locate a modulefile`` as there is no corresponding modulefile |
| 2040 |
in the current list of MODULEPATHS. |
| 2041 |
|
| 2042 |
The functionality of the ``module switch`` command depends on version and/or settings |
| 2043 |
of Environment Modules. For version 3.x, or 4.x with the :ref:`Automated module handling <v42-automated-module-handling-mode>` |
| 2044 |
feature disabled, it does not work well, modules which depend on the switched out module |
| 2045 |
are not reloaded, leading to inconsistent environments. But for 4.x versions with |
| 2046 |
the automated module handling feature enabled (as indicated by the --auto after the |
| 2047 |
switch, although that can be made the default), it works as expected. Modules depending |
| 2048 |
on the module being switched out get unloaded and reloaded if possible, as shown below: |
| 2049 |
|
| 2050 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/ompi-switch.out |
| 2051 |
:language: console |
| 2052 |
|
| 2053 |
The modulefiles themselves basically have no support for defaulting a compiler; the modulefiles for |
| 2054 |
the various MPI libraries are simply not even available until a modulefile for a compiler is loaded, as |
| 2055 |
seen below: |
| 2056 |
|
| 2057 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/ompi-defaults.out |
| 2058 |
:language: console |
| 2059 |
|
| 2060 |
However, this could be somewhat mitigated by having the modulefile for the default compiler automatically |
| 2061 |
loaded for the user upon login (e.g. in their dot files). |
| 2062 |
|
| 2063 |
The situation is similar for foo: |
| 2064 |
|
| 2065 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/foo-loads.out |
| 2066 |
:language: console |
| 2067 |
|
| 2068 |
As expected, the correct version of foo is loaded depending on the previously |
| 2069 |
loaded compiler and MPI libraries. Here again we use an ``intelmpi`` modulefile |
| 2070 |
to indicate when we wish to use Intel MPI libraries, even though they are enabled |
| 2071 |
by the ``intel`` module. The ``intelmpi`` modulefile basically just adds the modulepath |
| 2072 |
for the intel-compiler and intelmpi dependent modules. |
| 2073 |
|
| 2074 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/foo-switch.out |
| 2075 |
:language: console |
| 2076 |
|
| 2077 |
Note again the same deficiency in the switch report as the :ref:`Homebrewed_flavors_strategy` |
| 2078 |
had; in the last case above wherein we loaded foo with ``intel/2019`` loaded |
| 2079 |
but no MPI module. As expected, a non-MPI build of foo was loaded. However, when an |
| 2080 |
openmpi module is subsequently loaded, foo does not get reloaded and we still have |
| 2081 |
the non-MPI build of foo, as evidenced by the output of the foo command. And that |
| 2082 |
this fact is not obvious from the module list output. |
| 2083 |
|
| 2084 |
The behavior when defaulting is nicer. Without any compiler or MPI libraries |
| 2085 |
Here we note a deficiency in the switch support as compared to :ref:`Flavours <Flavours_strategy>`. In the last example |
| 2086 |
after loading ``intel/2019`` and foo, we have the non-MPI build of foo as expected. However, upon |
| 2087 |
subsequently loading the openmpi module, we still have the non-MPI version of foo loaded, as evidenced |
| 2088 |
by the output of the dummy foo command. I.e., the foo package was *not* automatically reloaded, as |
| 2089 |
there was no prereq in the foo modulefile on an MPI library (as in the non-MPI build there is no MPI |
| 2090 |
library to prereq). Also note that module list does not really inform one of this fact. |
| 2091 |
|
| 2092 |
The ability to default partial modulenames in this strategy is mixed. Without any compiler loaded, |
| 2093 |
most modulefiles are not even visible/available. However, once a compiler is loaded, trying to |
| 2094 |
load a module without specifying the version will end up loading the latest version compatible |
| 2095 |
with the loaded compiler (as no incompatible versions are visible) as seen below: |
| 2096 |
|
| 2097 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/foo-defaults.out |
| 2098 |
:language: console |
| 2099 |
|
| 2100 |
This is better than in the :ref:`Flavours <Flavours_strategy>` or :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` strategies. If one were to use |
| 2101 |
this strategy in production, it is recommended to have a default compiler module (and maybe even |
| 2102 |
an MPI library, etc) automatically loaded when the user logs in, thereby allowing for a reasonable defaulting |
| 2103 |
ability (and more reasonable module avail output). |
| 2104 |
|
| 2105 |
The situation with bar is similar. We do not have a dummy simd module, |
| 2106 |
so the builds with different CPU vectorization support are specified by |
| 2107 |
appending ``/avx``, etc. to the bar package name. As with previous strategies, |
| 2108 |
if one attempts to load a simd variant which was not built for the compiler |
| 2109 |
loaded, an error will occur. |
| 2110 |
|
| 2111 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/bar-loads.out |
| 2112 |
:language: console |
| 2113 |
|
| 2114 |
Defaulting is handled well, as shown |
| 2115 |
|
| 2116 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/bar-defaults.out |
| 2117 |
:language: console |
| 2118 |
|
| 2119 |
In particular, one case specify ``bar/avx2`` or ``bar/avx`` or ``bar/sse4.1`` and the |
| 2120 |
latest version of bar consistent with the specification and any previously |
| 2121 |
loaded compiler (or default compiler) will be loaded |
| 2122 |
|
| 2123 |
Summary of Modulepath-based strategy |
| 2124 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 2125 |
|
| 2126 |
* It is quite difficult for an user to get an inconsistent environment, |
| 2127 |
at least when automated module handling mode is used. The modulepaths are |
| 2128 |
managed so that only modulefiles consistent with the previously loaded |
| 2129 |
compiler/MPI library are visible. |
| 2130 |
* With automated module handling mode, the module switch command is very nicely |
| 2131 |
supported. |
| 2132 |
* The module command does not lend itself to readily seeing all the packages |
| 2133 |
which are installed on the system, nor seeing for which compiler/MPI combinations |
| 2134 |
a certain package is built. The module avail command just does not show |
| 2135 |
any modules not compatible with the currently loaded compiler and/or MPI |
| 2136 |
libraries. Lmod needed to add a ``module spider`` command to address this, |
| 2137 |
but no such functionality currently exists in Environment Modules. |
| 2138 |
If one were to use this in production, you would need to provide something |
| 2139 |
similar to the Lmod spider sub-command, or at least provide frequently |
| 2140 |
updated web pages or similar with this information. |
| 2141 |
* This strategy involves the use of many more modulepaths than the |
| 2142 |
previously examined strategies, having at least one modulepath per |
| 2143 |
compiler installed, and typically also one for each compiler/MPI library |
| 2144 |
combination. If you add additional layers of dependency, things get even |
| 2145 |
more complicated, which can be limiting. It is especially problematic if |
| 2146 |
you have two disjoint dependencies -- i.e. neither dependencies is dependent on |
| 2147 |
the other; in this case you might load one or the other, or both (in either |
| 2148 |
order). If there are modulefiles dependent on both of these dependencies, the |
| 2149 |
last one loaded (and *only* the last one) needs to handle adding the appropriate |
| 2150 |
modulepath for the modulefiles depending on both. |
| 2151 |
* In general, a module will fail to load with an error message |
| 2152 |
if any dependent module is not already loaded. The error will actually |
| 2153 |
state that the module cannot be found, as the modulepath containing it will |
| 2154 |
not be added to the ``MODULEPATH`` until the modules depended on are loaded. |
| 2155 |
So if you need to default a compiler, etc., you will need to have the |
| 2156 |
user's initialization scripts automatically load the appropriate |
| 2157 |
modulefile upon login. |
| 2158 |
* However, what we have been referring to as "more intelligent defaulting" is |
| 2159 |
effectively supported. I.e., if a user requests to load a package without |
| 2160 |
specifying the version desired, the version will be defaulted to the latest |
| 2161 |
version *compatible* with the currently loaded compiler and other dependencies. |
| 2162 |
This is because the incompatible versions are not visible in the module tree. |
| 2163 |
|
| 2164 |
|
| 2165 |
Comparison of Strategies |
| 2166 |
------------------------ |
| 2167 |
|
| 2168 |
All of the strategies discussed above have their peculiar strengths and weaknesses. |
| 2169 |
The decision of which strategy to use will depend on how these strengths and weaknesses |
| 2170 |
impact your design goals. It is advised to play around in the sandbox environments |
| 2171 |
a bit, as actual use tends to help make clear which downsides you are willing to accept |
| 2172 |
and which you are not. |
| 2173 |
|
| 2174 |
With the exception of the :ref:`Flavours <Flavours_strategy>`, which works best with Environment Modules |
| 2175 |
version 3.x, all of the other strategies work as well or better |
| 2176 |
on the newer 4.x versions of Environment Modules. This difference is most visible |
| 2177 |
in the discussion of features around the module switch command. So in the following |
| 2178 |
discussions we will assume Environment Modules version 3.x for the Flavours strategies, |
| 2179 |
as that is the version it works best with. For all other strategies, we assume |
| 2180 |
Environment Modules version 4.x, with automated module handling mode enabled, as these |
| 2181 |
strategies work best in that scenario. |
| 2182 |
|
| 2183 |
The :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>`, |
| 2184 |
:ref:`Modulerc-based <Modulerc-based_strategy>`, and :ref:`Modulepath-based <Modulepath-based_strategy>` strategies all |
| 2185 |
require a significant amount of "assembly" in order to get them working for a |
| 2186 |
production environment; the example scripts and procedures provided here should |
| 2187 |
help significantly, but should not be considered as a polished product. You will |
| 2188 |
likely need to customize and extend for your environment. The |
| 2189 |
:ref:`Flavours_strategy`, on the other hand, has been packaged to present as a finished product |
| 2190 |
and so requires less "assembly" to get working, but does not appear to be actively |
| 2191 |
maintained, so the reduced up-front work might be neutralized by the costs of |
| 2192 |
self support. |
| 2193 |
|
| 2194 |
Basic Dependency Handling |
| 2195 |
^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 2196 |
|
| 2197 |
All of the strategies discussed support a basic level of dependency |
| 2198 |
handling. If a user attempts to load a package, they get the build of the |
| 2199 |
package appropriate for the previously loaded compiler or other dependencies, |
| 2200 |
or, if no appropriate build is found, an error message indicating such. |
| 2201 |
E.g, if the user does: |
| 2202 |
|
| 2203 |
.. code-block:: tcl |
| 2204 |
|
| 2205 |
module load gcc/8.2.0 |
| 2206 |
module load foo/1.1 |
| 2207 |
|
| 2208 |
their environment will be set up to run foo version ``1.1`` built |
| 2209 |
with gcc version ``8.2.0``. |
| 2210 |
|
| 2211 |
All of the strategies discussed meet this criterion, with both 3.x and 4.x versions |
| 2212 |
of Environment Modules. |
| 2213 |
|
| 2214 |
Advanced Dependency Handling (e.g. the module switch sub-command) |
| 2215 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 2216 |
|
| 2217 |
Things are more complicated when we allow for the modules upon which |
| 2218 |
other loaded modules might depend to be changed. This generally involves |
| 2219 |
the module switch command. |
| 2220 |
|
| 2221 |
While all of the strategies handle well the easy case, when we switch |
| 2222 |
a module upon which no other loaded modules depend, the trick comes |
| 2223 |
when one or more currently loaded modules depend on the module being |
| 2224 |
switched out. E.g., if we assume that module foo depends on the |
| 2225 |
previously loaded compiler, then ideally, we would like for a sequence like: |
| 2226 |
|
| 2227 |
.. code-block:: tcl |
| 2228 |
|
| 2229 |
module load intel/2019 |
| 2230 |
module load foo |
| 2231 |
module switch intel pgi/18.4 |
| 2232 |
|
| 2233 |
to have the same effective outcome as if the user issued the commands: |
| 2234 |
|
| 2235 |
.. code-block:: tcl |
| 2236 |
|
| 2237 |
module load pgi/18.4 |
| 2238 |
module load foo |
| 2239 |
|
| 2240 |
I.e., switching out the compiler |
| 2241 |
will cause foo to be reloaded with the new compiler dependency. And ideally |
| 2242 |
in this example we would have ``pgi/18.4`` loaded along with the latest version |
| 2243 |
of foo compatible with that compiler. |
| 2244 |
|
| 2245 |
This is handled reasonably well with the :ref:`Flavours <Flavours_strategy>` (using Environment Modules |
| 2246 |
3.x) and the :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` and |
| 2247 |
:ref:`Modulepath-based <Modulepath-based_strategy>` (both using Environment |
| 2248 |
Modules 4.x) strategies. For the :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` and |
| 2249 |
:ref:`Modulepath-based <Modulepath-based_strategy>` |
| 2250 |
strategies, this relies on the :ref:`Automated module handling <v42-automated-module-handling-mode>` feature (As of |
| 2251 |
version ``4``, this is disabled by default. It can be enabled in ``modulecmd.tcl``, |
| 2252 |
or by running ``module config auto_handling 1``, or by adding the ``--auto`` flag |
| 2253 |
to the modules command.) |
| 2254 |
|
| 2255 |
For these strategies, swapping out a compiler or other module upon which a loaded |
| 2256 |
module depends will cause the the module with a dependency to be unloaded, the |
| 2257 |
module being switched being swapped out, and the module that had a dependency being |
| 2258 |
reloaded. |
| 2259 |
|
| 2260 |
Our example sequence above actually exposes a subtle issue. For the :ref:`Flavours <Flavours_strategy>` |
| 2261 |
and :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` strategies, the sequence of loading ``pgi/18.4`` followed |
| 2262 |
by foo (using our example software library) would fail, as foo would be defaulted to |
| 2263 |
version ``2.4`` and as there was no build of ``foo/2.4`` for ``pgi/18.4``, the module load foo |
| 2264 |
would fail. And indeed, the switch sequence above would not be completely successful |
| 2265 |
under either strategy: for the :ref:`Homebrewed_flavors_strategy`, the compiler would be |
| 2266 |
swapped out and the reload of foo would fail with an error (so effectively |
| 2267 |
equivalent to the second sequence). For the :ref:`Flavours_strategy`, it detects that |
| 2268 |
it would be unable to reload foo, and the switch command fails (i.e. the compiler |
| 2269 |
remains ``intel/2019``). |
| 2270 |
|
| 2271 |
For the :ref:`Modulepath-based_strategy`, the second sequence (loading ``pgi/18.4`` then foo) would |
| 2272 |
actually work, as the modulepath exposed by loading ``pgi/18.4`` only has versions of |
| 2273 |
foo that are compatible with ``pgi/18.4``. But the switch command does not work |
| 2274 |
as well as would be desired. This is because that as currently implemented, when the |
| 2275 |
automated module handling code does the reload of the module that was unloaded |
| 2276 |
due to dependencies, it attempts to reload based on the fully qualified name of the |
| 2277 |
module that was loaded, and not based on the (possibly partial) name specified |
| 2278 |
when the module was originally loaded. So, after the module load ``intel/2019``, |
| 2279 |
the module load foo will result in ``foo/2.4`` being loaded. When we switch out ``intel/2019`` |
| 2280 |
for ``pgi/18.4``, foo gets unloaded, but after the compiler swap, it tries to reload |
| 2281 |
``foo/2.4``. As there is no build of ``foo/2.4`` for ``pgi/18.4`` (and thus no modulefile in |
| 2282 |
the current modulepath), the module is not found and is unable to be loaded. |
| 2283 |
As in the :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` case, the compiler is switched but foo is left unloaded |
| 2284 |
(with an appropriate warning to that effect). |
| 2285 |
|
| 2286 |
This same issue prevents the :ref:`Automated module handling <v42-automated-module-handling-mode>` feature as currently implemented |
| 2287 |
from being very useful with the :ref:`Modulerc-based <Modulerc-based_strategy>`. Basically, whenever you have a module |
| 2288 |
loaded that depends on another module, switching out the module depended upon will |
| 2289 |
fail to reload the other module. This is true even in less pathological cases, like: |
| 2290 |
|
| 2291 |
.. code-block:: tcl |
| 2292 |
|
| 2293 |
module load intel/2019 |
| 2294 |
module load foo |
| 2295 |
module switch intel pgi/19.4 |
| 2296 |
|
| 2297 |
which succeeds under the other three strategies (as ``foo/2.4`` *is* built for ``pgi/19.4``). |
| 2298 |
This is because when foo is initially loaded in the above scenario, the actual modulename |
| 2299 |
loaded is ``foo/intel/2019/2.4``. With the switch command, foo gets unloaded, the compilers |
| 2300 |
are switched, and the automated modules handling code tries to reload ``foo/intel/2019/2.4``. |
| 2301 |
That modulename clearly depends on ``intel/2019``, not ``pgi/19.4``, and so will fail to load |
| 2302 |
(with an error message). The net result is that the compiler gets switched, but any |
| 2303 |
modules depending on the compiler (or whatever module being switched) get unloaded |
| 2304 |
(with a warning to that effect). It is hoped that a future version of the |
| 2305 |
:ref:`Automated module handling <v42-automated-module-handling-mode>` feature will have an option to reload the dependent |
| 2306 |
modules using the modulename they were loaded with; this should allow for the |
| 2307 |
switch command to work well in the :ref:`Modulerc-based_strategy`, as well as make |
| 2308 |
edges case (like the ``pgi/18.4`` example discussed above) work better with the |
| 2309 |
:ref:`Modulepath-based_strategy`. |
| 2310 |
|
| 2311 |
There is another edge case which only the :ref:`Flavours_strategy` handles well. Consider a |
| 2312 |
scenario like: |
| 2313 |
|
| 2314 |
.. code-block:: tcl |
| 2315 |
|
| 2316 |
module purge |
| 2317 |
module load intel/2019 |
| 2318 |
module load foo |
| 2319 |
foo |
| 2320 |
# this should be foo/2.4 built for intel/2019 without MPI |
| 2321 |
module load openmpi |
| 2322 |
foo |
| 2323 |
# What build is this, with or without MPI? |
| 2324 |
|
| 2325 |
With the :ref:`Flavours_strategy`, the final foo will be built for ``intel/2019`` |
| 2326 |
with openmpi support; i.e. Flavours supports the concept of *optional* prerequisites, and foo has |
| 2327 |
an optional prerequisite on the MPI libraries, so when openmpi is loaded, any previously loaded modules |
| 2328 |
which depend on it (through a regular or optional prerequisite) get reloaded. |
| 2329 |
|
| 2330 |
All of the other strategies rely on automated module handling mode for automatic reloads, and |
| 2331 |
in this case no reload of foo will be initiated as foo does not have |
| 2332 |
MPI libraries listed as a prerequisite (otherwise a no MPI version could not be loaded). This is especially |
| 2333 |
problematic for the :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` and |
| 2334 |
:ref:`Modulepath-based <Modulepath-based_strategy>` strategies, since it is not easily to |
| 2335 |
tell from a module list or similar command which version of foo the environment is set for (for the |
| 2336 |
:ref:`Modulerc-based_strategy`, the full modulename from module list will indicate that). |
| 2337 |
|
| 2338 |
However, despite some potential issues with some edge cases, the :ref:`Flavours <Flavours_strategy>`, |
| 2339 |
:ref:`Homebrewed flavors <Homebrewed_flavors_strategy>`, and |
| 2340 |
:ref:`Modulepath-based <Modulepath-based_strategy>` strategies all handle this challenge well. |
| 2341 |
The :ref:`Modulerc-based <Modulerc-based_strategy>` |
| 2342 |
approach has a poor showing (although at least the modules depending on the module switched |
| 2343 |
out will be unloaded, so the set of loaded modules is consistent). |
| 2344 |
|
| 2345 |
Note also that this topic shows the most dependence on the version of Environment |
| 2346 |
Modules. For the :ref:`Flavours_strategy`, the switching of a module upon which |
| 2347 |
other modules depended does not work (the wrapper script returned weird errors). The |
| 2348 |
other strategies all rely on the :ref:`Automated module handling <v42-automated-module-handling-mode>` feature to enable the unload |
| 2349 |
and reload of modules which depend on the module being switched out. Thus for 3.x versions |
| 2350 |
of Environment Modules, the switch of a compiler, etc. will only result in the compiler being replaced, and any |
| 2351 |
modules which depend on the compiler will not be reloaded. This means they will still be |
| 2352 |
pointing to versions built for the old compiler, leading to an inconsistent set of |
| 2353 |
modules being loaded. This is particularly bad in the |
| 2354 |
:ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` case, as a module |
| 2355 |
load will not even inform one of that fact. Even if automated module handling mode is disabled |
| 2356 |
on version 4.x of Environment Modules situation is better as loaded environment consistency |
| 2357 |
:ref:`is assured<v42-prereq-constraints-consistency>`: the switch of a compiler will be denied |
| 2358 |
since a dependent module is detected loaded. |
| 2359 |
|
| 2360 |
Visibility into what packages are available |
| 2361 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 2362 |
|
| 2363 |
Another set of criteria to weigh has to do with visibility into the available |
| 2364 |
packages. We are interested in |
| 2365 |
|
| 2366 |
#. Viewing all of the packages installed on a system |
| 2367 |
#. Determining what versions of a specific package are available for a |
| 2368 |
given compiler/MPI/etc combination. |
| 2369 |
#. Seeing what compiler/MPI/etc combinations a specific version of a |
| 2370 |
package |
| 2371 |
#. For packages that we have currently loaded, determining which compiler/MPI/etc |
| 2372 |
they were built for. |
| 2373 |
|
| 2374 |
In terms of seeing all of packages that are available, the :ref:`Flavours <Flavours_strategy>` and |
| 2375 |
:ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` strategies |
| 2376 |
are probably the best. The ``module avail`` command will list all modulefiles available, which will |
| 2377 |
typically just be a list of packagename/version for each installed version. On a production system |
| 2378 |
with many packages installed, even this can be a bit overwhelming to the user. |
| 2379 |
|
| 2380 |
The ``module avail`` command will also list all modulefiles for every installed package in the |
| 2381 |
:ref:`Modulerc-based_strategy`, but here every build of every version of every package will have a distinct modulefile. I.e., |
| 2382 |
you will not just have ``foo/1.1`` and ``foo/2.4`` listings, but ``foo/1.1/gcc/8.2.0/nompi``, ``foo/1.1/gcc/8.2.0/mvapich/2.1``, |
| 2383 |
``foo/1.1/gcc/8.2.0/openmpi/3.1``, ``foo/1.1/intel/2018/nompi``, ``foo/1.1/intel/2018/intelmpi``, etc. modulefiles. |
| 2384 |
Actually, the situation is even worse than that, as in this strategy there are often multiple |
| 2385 |
modulefiles for the same build in order to ease navigation of the module tree, e.g. we could have |
| 2386 |
``foo/1.1/gcc/8.2.0/nompi`` and ``foo/gcc/8.2.0/nompi/1.1`` representing the same build of the same package. |
| 2387 |
Whereas an unqualified ``module avail`` in the :ref:`Flavours <Flavours_strategy>` or |
| 2388 |
:ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` strategies can be a bit overwhelming |
| 2389 |
in a large environment, with the :ref:`Modulerc-based_strategy` it can become practically unusable. |
| 2390 |
|
| 2391 |
While an unqualified ``module avail`` in the :ref:`Flavours <Flavours_strategy>`, |
| 2392 |
:ref:`Homebrewed flavors <Homebrewed_flavors_strategy>`, and especially :ref:`Modulerc-based <Modulerc-based_strategy>` strategies |
| 2393 |
can inundate the user with modulenames, the :ref:`Modulepath-based_strategy` has the opposite problem. |
| 2394 |
With the :ref:`Modulepath-based_strategy` strategy, the modulefiles are split across multiple, often mutually incompatible, modulepaths, so the module avail |
| 2395 |
command will never return a list of all modulefiles installed, only those available given the previously loaded |
| 2396 |
compiler/MPI libraries/etc. E.g., if a package foobar is only installed for a particular compiler/MPI combination, |
| 2397 |
it will not appear in any module avail listing unless that particular compiler and MPI were previously loaded. |
| 2398 |
|
| 2399 |
While the number of modulefiles listed in an unqualified ``module avail`` command in the :ref:`Modulerc-based_strategy` |
| 2400 |
is unwieldy, if one adds a partial package specification argument, the number of modulefiles returned is greatly |
| 2401 |
reduced. This smaller list can provide information about which compilers/MPI/etc. are compatible with |
| 2402 |
a specific version of a package. For example, to see the compilers for which ``foo/2.4`` is built, we can |
| 2403 |
do something like: |
| 2404 |
|
| 2405 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/foo-avail1.out |
| 2406 |
:language: console |
| 2407 |
|
| 2408 |
Similarly, to see the builds of foo using gcc compilers, one can do something like: |
| 2409 |
|
| 2410 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulerc/modules4/foo-avail2.out |
| 2411 |
:language: console |
| 2412 |
|
| 2413 |
The other strategies do not readily provide that information. The next best case is the |
| 2414 |
:ref:`Modulepath-based_strategy`, because here at least a module avail will tell you what versions |
| 2415 |
of a package are compatible with the currently loaded compiler/MPI/etc, e.g.: |
| 2416 |
|
| 2417 |
.. literalinclude:: ../../example/compiler-etc-dependencies/example-sessions/modulepath/modules4/foo-avail1.out |
| 2418 |
:language: console |
| 2419 |
|
| 2420 |
But the :ref:`Flavours <Flavours_strategy>` and :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` |
| 2421 |
strategies do not readily show what versions of packages are built |
| 2422 |
for which compilers, etc. You would need to load a compiler/MPI/etc combination, and then try loading |
| 2423 |
a particular version of a package. |
| 2424 |
|
| 2425 |
Lastly, with the :ref:`Modulerc-based_strategy`, the module list command explicitly shows information |
| 2426 |
about the compiler/MPI/etc used to build the loaded modules, as that is part of the full modulename. |
| 2427 |
This information is not explicitly visible in the other three strategies, but that is usually not |
| 2428 |
a problem. As long as the various user commands result in a set of modules wherein every module depending |
| 2429 |
on either a compiler or MPI library, etc. is built for the currently loaded compiler/MPI library/etc., |
| 2430 |
that information is redundant. |
| 2431 |
|
| 2432 |
The only cases where this is potentially an issue are the sort of edge cases described in the previous |
| 2433 |
section, e.g. if one were to do something like: |
| 2434 |
|
| 2435 |
.. code-block:: tcl |
| 2436 |
|
| 2437 |
module purge |
| 2438 |
module load intel/2019 |
| 2439 |
module load foo |
| 2440 |
foo |
| 2441 |
# this should be foo/2.4 built for intel/2019 without MPI |
| 2442 |
module load openmpi |
| 2443 |
foo |
| 2444 |
# What build is this, with or without MPI? |
| 2445 |
|
| 2446 |
In the above example, for the :ref:`Flavours <Flavours_strategy>`, |
| 2447 |
:ref:`Homebrewed flavors <Homebrewed_flavors_strategy>`, and :ref:`Modulepath-based <Modulepath-based_strategy>` strategies a |
| 2448 |
module list at the end would simply list something like: |
| 2449 |
|
| 2450 |
.. code-block:: shell |
| 2451 |
|
| 2452 |
Currently Loaded Modulefiles: |
| 2453 |
1) intel/2019 2) foo/2.4 3) openmpi/4.0 |
| 2454 |
|
| 2455 |
For the :ref:`Flavours_strategy`, foo would be built with MPI support, but for the other |
| 2456 |
two, foo would still be the non-MPI build, which is not readily apparent from the |
| 2457 |
above output (although possibly could be inferred by the ordering of the modules). |
| 2458 |
|
| 2459 |
Ease of Navigating the Module Tree |
| 2460 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 2461 |
|
| 2462 |
The final criterion to be discussed is the ease of navigating the module tree. |
| 2463 |
Ideally, we would like it so that when an user gives a partial modulename the |
| 2464 |
system resolves in a sensible manner. Typically a partial modulename will be |
| 2465 |
a package name without a version, in which case ideally the latest version of |
| 2466 |
the package (compatible with previously loaded compiler, etc), should be loaded. |
| 2467 |
And indeed, for a simple case like: |
| 2468 |
|
| 2469 |
.. code-block:: tcl |
| 2470 |
|
| 2471 |
module purge |
| 2472 |
module load intel |
| 2473 |
module load foo |
| 2474 |
|
| 2475 |
all the strategies do just that. The intel module will default to ``intel/2019``, |
| 2476 |
and foo to version ``2.4`` built for ``intel/2019``. However, with even a slight modification |
| 2477 |
things begin to diverge, e.g.: |
| 2478 |
|
| 2479 |
.. code-block:: tcl |
| 2480 |
|
| 2481 |
module purge |
| 2482 |
module load intel/2018 |
| 2483 |
module load foo |
| 2484 |
|
| 2485 |
In this second case, the :ref:`Flavours <Flavours_strategy>` and |
| 2486 |
:ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` strategies still default |
| 2487 |
the version of foo to version ``2.4``, despite there being no build of ``foo/2.4`` for ``intel/2018``. |
| 2488 |
That is because they do not do intelligent defaulting, and will simply default |
| 2489 |
the version of foo to the highest version they see |
| 2490 |
even though it is not compatible with the loaded compiler. |
| 2491 |
Thus the load of foo will fail for :ref:`Flavours <Flavours_strategy>` and |
| 2492 |
:ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` strategies. |
| 2493 |
However, for the :ref:`Modulerc-based <Modulerc-based_strategy>` and |
| 2494 |
:ref:`Modulepath-based <Modulepath-based_strategy>` strategies, the module load of foo |
| 2495 |
will successfully load foo version ``1.1`` built for ``intel/2018``. For the :ref:`Modulepath-based_strategy`, |
| 2496 |
this is because the only foo modules currently visible are compatible with the |
| 2497 |
loaded ``intel/2018`` compiler, and it simply defaults to the highest version seen. For |
| 2498 |
the :ref:`Modulerc-based_strategy`, the ``.modulerc`` file directly under foo directs it down the |
| 2499 |
subdirectory for the intel compiler family, which has a ``.modulerc`` defaulting to the 2018 subdirectory |
| 2500 |
(as the currently loaded compiler version), which has a ``.modulerc`` defaulting to the nompi |
| 2501 |
subdirectory (the MPI family as no MPI library was loaded). At this point, there is not |
| 2502 |
another ``.modulerc`` file, so the latest version will be defaulted to, and |
| 2503 |
``foo/intel/2018/nompi/1.1`` module will be loaded. |
| 2504 |
|
| 2505 |
The situation is analogous when there are chained dependencies (e.g. compiler and MPI |
| 2506 |
libraries); the :ref:`Flavours <Flavours_strategy>` and :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` |
| 2507 |
strategies will default to the latest |
| 2508 |
version of the package they know about, regardless of whether it is compatible with the |
| 2509 |
loaded environment. If it is compatible, it will be loaded and all is well, if not the |
| 2510 |
load command will fail with an error message. Not horrible; the set of loaded modules is |
| 2511 |
still consistent, but not terribly user-friendly either. |
| 2512 |
|
| 2513 |
For the :ref:`Modulerc-based <Modulerc-based_strategy>` and :ref:`Modulepath-based <Modulepath-based_strategy>` |
| 2514 |
strategies, the latest build compatible with the various dependencies will be loaded (or if no build is |
| 2515 |
compatible, the load will fail with an appropriate error). |
| 2516 |
|
| 2517 |
For the :ref:`Flavours <Flavours_strategy>`, :ref:`Homebrewed flavors <Homebrewed_flavors_strategy>`, |
| 2518 |
and :ref:`Modulepath-based <Modulepath-based_strategy>` strategies, typically modulenames |
| 2519 |
only consist of the package name and version components, so generally only partial modulenames |
| 2520 |
seen are packages without a version. The :ref:`Modulerc-based_strategy` has much longer modulenames, |
| 2521 |
and therefore a greater variety of partial modulenames, which can allow for interesting |
| 2522 |
possibilities. For example, one could do ``module load foo/pgi``, which (assuming no compiler |
| 2523 |
was previously loaded), would load the latest version of foo built with the latest version of |
| 2524 |
the pgi compiler. The module command will start looking for a modulefile in the |
| 2525 |
``foo/pgi`` subdirectory of the foo module structure, |
| 2526 |
where the ``.modulerc`` file will attempt to default to the version of pgi which was loaded, but as |
| 2527 |
no compiler was loaded, the modulerc file will terminate without setting a default, and the |
| 2528 |
modulecmd will default to the latest version, and descend into that directory (``19.4``). Here, |
| 2529 |
the ``.modulerc`` will attempt to default to the MPI family, but since none is loaded (as that |
| 2530 |
would require a compiler to have been loaded), it defaults to the nompi directory, and then |
| 2531 |
to the latest version in the nompi subdirectory. So the module ``foo/pgi/19.4/nompi/2.4`` will |
| 2532 |
get loaded, which will also cause ``pgi/19.4`` to get loaded. Should we have issued that module |
| 2533 |
load after having loaded a non-pgi compiler, the same module file would get loaded, but it |
| 2534 |
would detect an incompatible previously loaded compiler and abort with an error. Should a |
| 2535 |
pgi compiler have been previously loaded, we would have gotten the latest version of foo built |
| 2536 |
for that compiler (and possibly with any previously loaded MPI library). |
| 2537 |
|
| 2538 |
The above is supported because we create a ``PACKAGE/COMPILER_FAMILY/COMPILER_VERSION/.../PKGVERSION`` |
| 2539 |
tree in the package's modulefile directory for packages depending on the compiler. Possibly |
| 2540 |
with additional components to the modulename for MPI libraries, etc. These are created |
| 2541 |
to allow for the more basic ``module load foo`` case, but because the module path is |
| 2542 |
descended component by component, it works if you jump in in the middle as well |
| 2543 |
(and will fail if you try something stupid like : |
| 2544 |
|
| 2545 |
.. code-block:: tcl |
| 2546 |
|
| 2547 |
module load intel/2019 |
| 2548 |
module load foo/pgi |
| 2549 |
|
| 2550 |
But a more interesting case is when there are somewhat disjoint options on the builds. |
| 2551 |
Consider the case of the bar package from the examples, which in addition to the compiler |
| 2552 |
and builds supporting different SIMD levels. For the :ref:`Flavours <Flavours_strategy>` and |
| 2553 |
:ref:`Homebrewed flavors <Homebrewed_flavors_strategy>` |
| 2554 |
strategies, we handled this with a pseudo-simd package and modules. This worked, and allowed |
| 2555 |
for one to select a build that matched both compiler and SIMD level, but you likely need to |
| 2556 |
manually select a bar version compatible with both of these, as these strategies will not |
| 2557 |
do intelligent defaulting of versions. |
| 2558 |
|
| 2559 |
For the :ref:`Modulepath-based_strategy`, we added the SIMD label as a final component to the |
| 2560 |
modulename, but the defaulting was somewhat simple (e.g. always default to lowest SIMD). |
| 2561 |
|
| 2562 |
With the :ref:`Modulerc-based_strategy`, we added an additional naming scheme for the bar modulefiles, |
| 2563 |
so they can start with bar/BARVERSION, or bar/COMPILER_FAMILY, or bar/SIMD_LEVEL. This not |
| 2564 |
only makes it easier to view which versions of bar for which compilers are built with support |
| 2565 |
for a given SIMD level (e.g. module avail bar/avx2), but also one can do something like: |
| 2566 |
|
| 2567 |
.. code-block:: tcl |
| 2568 |
|
| 2569 |
module load pgi/19.4 |
| 2570 |
module load bar/avx |
| 2571 |
|
| 2572 |
to load the latest version of bar built for ``pgi/19.4`` with AVX support. While there is |
| 2573 |
overhead associated with adding a new naming scheme like that (you add another set of |
| 2574 |
modulefiles to bar and a new subdirectory tree), this can be done on a per package |
| 2575 |
basis and when done judiciously it can be convenient for the users. |
| 2576 |
|
| 2577 |
|
| 2578 |
Conclusions |
| 2579 |
----------- |
| 2580 |
|
| 2581 |
We have presented four strategies for dealing with modulefiles for packages |
| 2582 |
with multiple builds depending on compiler, MPI, and/or other factors. All |
| 2583 |
four strategies can deal with the basic requirement of loading of the correct |
| 2584 |
build of a package depending on the previously loaded dependencies, or failing |
| 2585 |
with a reasonable error message if no such build is available. They all have |
| 2586 |
their own strengths and weaknesses. This document tried to present these |
| 2587 |
strategies objectively and has been made to help you to evaluate how to handle |
| 2588 |
this issue at your site. |
| 2589 |
|
| 2590 |
.. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: |