| 1 |
.. _tips-for-code-reuse: |
| 2 |
|
| 3 |
Tips for Code Reuse in Modulefiles |
| 4 |
================================== |
| 5 |
|
| 6 |
Although Modules allow one to manage many packages and versions of packages |
| 7 |
on a system, managing the modulefiles themselves can become error prone, |
| 8 |
especially since the differences between modulefiles for different versions |
| 9 |
of the same package are often rather small. This can lead to issues with |
| 10 |
the consistency of the module definitions. |
| 11 |
|
| 12 |
E.g., let us say you have a package foo, version 1.0 with a corresponding |
| 13 |
modulefile. A bit later, foo version 2.0 comes out, and you add a modulefile |
| 14 |
for that, likely copying the 1.0 version and editing a few lines. Then a |
| 15 |
bug is discovered in the version 1.0 modulefile, e.g. an environment variable |
| 16 |
that would be useful in some cases was omitted, or a typo in the help |
| 17 |
function. The version 1.0 modulefile is fixed, but did you remember to |
| 18 |
fix the 2.0 modulefile which, since it was created by copying the version 1.0 |
| 19 |
modulefile, likely has the same error? |
| 20 |
|
| 21 |
This page includes some tips to reduce the amount of redundant code |
| 22 |
in your modulefiles (i.e. increase the amount of code reuse) so that |
| 23 |
they are easier to manage and more consistent. |
| 24 |
|
| 25 |
A Simple Example |
| 26 |
---------------- |
| 27 |
|
| 28 |
We return to our "foo" example. |
| 29 |
A fairly traditional set up would have a "foo" directory somewhere under your MODULEPATH with |
| 30 |
modulefiles 1.0 and 2.0 underneath it. Each of 1.0 and 2.0 will likely be largely the same, having |
| 31 |
basically the same help text, whatis, and variables set, with only some |
| 32 |
changes to some text and paths to reflect the version difference. Example |
| 33 |
code for this and other cases discussed in this document can be found in the |
| 34 |
"doc/example/tips-for-code-reuse" subdirectory of this package. The modulefiles |
| 35 |
for this simple example start in the "foo" subdirectory. For example, the |
| 36 |
version 1.0 modulefile might look something like |
| 37 |
|
| 38 |
.. include:: ../../example/tips-for-code-reuse/foo/1.0 |
| 39 |
:literal: |
| 40 |
|
| 41 |
Although the above is simple enough, there is still much redundancy between |
| 42 |
the version 1.0 and 2.0 files, and we do not bother showing the file for 2.0 |
| 43 |
because it is so similar (only 5 chars change between the 1.0 and 2.0 |
| 44 |
versions, basically all the "1.0" strings become "2.0"). The command |
| 45 |
"module avail foo" shows versions 1.0 and 2.0, and loading either will |
| 46 |
set the environmental variables PATH, MANPATH, and LD_LIBRARY_PATH appropriately, |
| 47 |
conflict with itself (so only one version of foo can be loaded at a time), |
| 48 |
and define the help and whatis texts. |
| 49 |
|
| 50 |
To refactor this to increase code reuse, we copy one of the original |
| 51 |
modulefiles to "common", and change all the references to the version number |
| 52 |
to a "$version" variable. To avoid confusion with the previous version, |
| 53 |
we change the package name from "foo" to "fooA", and the new versions can be |
| 54 |
found in the "fooA" subdirectory of "doc/example/tips-for-code-reuse". After |
| 55 |
adding some comments, etc, the common module becomes |
| 56 |
|
| 57 |
.. include:: ../../example/tips-for-code-reuse/fooA/common |
| 58 |
:literal: |
| 59 |
|
| 60 |
We then create a pair of small stub files to replace "1.0" and "2.0". These |
| 61 |
simply just set the version variable appropriately, and then "source" the |
| 62 |
"common" file. For "1.0", the file would look like |
| 63 |
|
| 64 |
.. include:: ../../example/tips-for-code-reuse/fooA/1.0 |
| 65 |
:literal: |
| 66 |
|
| 67 |
A similar file would be created for "2.0", just changing the version number |
| 68 |
in the "set version" line. |
| 69 |
The "set moduledir" line sets the Tcl variable moduledir to contain the name of |
| 70 |
the directory in which the final modulefile (i.e. "1.0") is located, so we |
| 71 |
can find the "common" file. |
| 72 |
|
| 73 |
The common file then handles all the work of defining the PATH, MANPATH, |
| 74 |
and LD_LIBRARY_PATH environmental variables, using the value of the Tcl |
| 75 |
variable "version" passed to it by the stub file. It also handles defining |
| 76 |
the help procedure and the whatis text, and the conflict with itself. |
| 77 |
|
| 78 |
Note that the "common" file does **NOT** start |
| 79 |
with the Modules magic "#%Module"; this will keep "common" from showing |
| 80 |
up in "module avail fooA". The "1.0" and "2.0" files do need to start with |
| 81 |
the magic "#%Module" tag so the module command will "see" them. Because |
| 82 |
of this, the "module avail fooA" command will just show the 1.0 and 2.0 |
| 83 |
versions as expected. |
| 84 |
|
| 85 |
The files for this example are in the "doc/example/tips-for-code-reuse/fooA" |
| 86 |
directory. |
| 87 |
|
| 88 |
A Simple Example, revisited |
| 89 |
--------------------------- |
| 90 |
|
| 91 |
The "fooA" case above is fairly common, and we can actually improve upon |
| 92 |
what we did above. To avoid confusion, we will repeat using the package |
| 93 |
name "fooB", and the files for this example will be in the |
| 94 |
"doc/example/tips-for-code-reuse/fooB" directory. |
| 95 |
|
| 96 |
Using introspection in the modulefiles, we can get the version |
| 97 |
number of fooB from the name of the modulefile. So we add code to the |
| 98 |
top of "common" to default the version variable from the modulename. |
| 99 |
|
| 100 |
We then copy the "1.0" stub modulefile to ".generic", and remove the line |
| 101 |
which sets the version. Because is starts with a leading ".", the generic |
| 102 |
modulefile will not be displayed in "module avail" and such. Since this |
| 103 |
generic modulefile is now version independent, we can replace "1.0" and "2.0" |
| 104 |
with symlinks to the .generic file. |
| 105 |
|
| 106 |
The new common file looks like |
| 107 |
|
| 108 |
.. include:: ../../example/tips-for-code-reuse/fooB/common |
| 109 |
:literal: |
| 110 |
|
| 111 |
The .generic stub file looks like |
| 112 |
|
| 113 |
.. include:: ../../example/tips-for-code-reuse/fooB/1.0 |
| 114 |
:literal: |
| 115 |
|
| 116 |
Again, the common file handles setting the environmental variables |
| 117 |
PATH, MANPATH, and LD_LIBRARY_PATH, using the Tcl variable version |
| 118 |
which was defaulted from modulename. This works even in the case |
| 119 |
where the user does not specify the full module path, and the modulecmd |
| 120 |
defaults the version (e.g. if the user types "module load fooB" without |
| 121 |
specifying the version of fooB.) The common file also handles defining |
| 122 |
the help procedure and whatis text. When the command "module avail fooB" is issued, |
| 123 |
the common file does not get listed (because it does not start with the magic |
| 124 |
"#%Module" tag), nor does the .generic file get listed (because it starts |
| 125 |
with a period (".")), but the two 1.0 and 2.0 symlinks do get listed, just |
| 126 |
as one wants. |
| 127 |
|
| 128 |
This replaces a file for each version of fooB with one common file doing |
| 129 |
all the work, one generic file, and a bunch of symlinks. And the .generic |
| 130 |
file is not even specific to fooB. Indeed, at our site, we put the |
| 131 |
generic modulefile in an "utilities" directory outside of the MODULEPATH, |
| 132 |
and just symlink it to ".generic" in each application directory, and then |
| 133 |
symlink the versions to this symlinked ".generic". So we end up with one |
| 134 |
"common" file for each application, one shared generic modulefile, and a |
| 135 |
bunch of symlinks. |
| 136 |
|
| 137 |
A More Complicated Example |
| 138 |
--------------------------- |
| 139 |
|
| 140 |
Although the simple cases like the "foo" example above are not uncommon, |
| 141 |
many packages are more complicated. We now consider the fictitious "bar" |
| 142 |
package (example modulefiles in "example/tips-for-code-reuse/bar" directory). |
| 143 |
This is admittedly a contrived example, but it displays some of the ways in |
| 144 |
which we can modify the above simple cases to handle the more complicated |
| 145 |
needs of some packages. Our "bar" package has the follow characteristics: |
| 146 |
|
| 147 |
* We have two versions of "bar" built, "1.0" and "2.0" |
| 148 |
* For each bar version, we have builds for three different threading models: nothreads, pthreads, and openmp. |
| 149 |
* We expect users to use commands specifying both the bar version and the threading model, e.g. "module load fooB/1.0/pthreads" or "module load fooB/2.0/openmp" |
| 150 |
* The path to the installation directories include both the version number of "bar" and the threading model |
| 151 |
* The package requires the environmental variable BAR_LICENSE_FILE to be set appropriately |
| 152 |
* All builds of bar version 1.0 use one license file which is different than that used by bar version2.0 |
| 153 |
* Bar 2.0 nothreads and openmp builds use the same license file, with a completely different name than that used by bar version 1.0. |
| 154 |
* The bar 2.0 pthreads build uses its very own license file with a very different filename than those used by bar version 1.0 or the other threading models of version 2.0. |
| 155 |
* The "nothreads" builds have a prerequisite on "fooB", with bar 1.0 wanting fooB version 1.1, and bar 2.0 wanting fooB version 3.2 |
| 156 |
|
| 157 |
Even with the above exceptions, there are still more similarities than |
| 158 |
differences between the various modulefile definitions. Like in the "fooA" |
| 159 |
case, we will have a "common" script that does almost all the work, and stub |
| 160 |
files for each of the variants, but in this case the stub files are expected |
| 161 |
to define some more variables. Our "common" file, which can be found in |
| 162 |
the "example/tips-for-code-reuse/bar" directory, looks like |
| 163 |
name "fooB", and the files for this example will be in the |
| 164 |
|
| 165 |
.. include:: ../../example/tips-for-code-reuse/bar/common |
| 166 |
:literal: |
| 167 |
|
| 168 |
Here we have extended the parameters which are allowed to be passed in from |
| 169 |
the build specific "stub" files; in addition to the version of "bar", the stub |
| 170 |
files are expected to have set the variables "threadingmodel" and optionally |
| 171 |
"licensefile". If "licensefile" was not set, the common script will default |
| 172 |
it based on the version of "bar". The rest of the "common" file is similar |
| 173 |
to the "fooA" case, except that we use both "version" and "threadingmodel" |
| 174 |
in setting the paths, and now define BAR_LICENSE_FILE based on "licensefile". |
| 175 |
Again, almost all the work is done in the common file, including setting the |
| 176 |
relevant environmental variables, defining the help procedure and whatis text, |
| 177 |
and preventing multiple versions of bar being loaded at the same time. |
| 178 |
|
| 179 |
We now create stubfiles for each of the three threading models underneath directories |
| 180 |
for each bar version. These can all be found under the |
| 181 |
"example/tips-for-code-reuse/bar" directory. The one for bar 1.0 and openmp |
| 182 |
looks like |
| 183 |
|
| 184 |
|
| 185 |
.. include:: ../../example/tips-for-code-reuse/bar/1.0/openmp |
| 186 |
:literal: |
| 187 |
|
| 188 |
The pthreads version is almost identical, just setting threadingmodel to |
| 189 |
"pthreads" instead of "openmp". The "openmp" build for bar version 2.0 |
| 190 |
is also very similar, differing only in value "version" is set to. In |
| 191 |
all three cases, these stubfiles are basically the same as those used in |
| 192 |
the "fooA" example; we set a couple of Tcl variables (version and threadingmodel), |
| 193 |
and then source the common file which does all the work. The licensefile |
| 194 |
variable is left unset so the common file will default it correctly based |
| 195 |
on the bar version. |
| 196 |
|
| 197 |
The stub file for the pthreads build for bar version 2.0 is as below: |
| 198 |
|
| 199 |
.. include:: ../../example/tips-for-code-reuse/bar/2.0/pthreads |
| 200 |
:literal: |
| 201 |
|
| 202 |
This version is similar to the previous three, with the addition of explicitly |
| 203 |
setting the "licensefile" variable. This value will be used in the common file |
| 204 |
to set BAR_LICENSE_FILE, rather than the default which would have been set in |
| 205 |
the common file had no value been provided. |
| 206 |
|
| 207 |
Finally, the two "nothreads" build also have a slightly different stub file than |
| 208 |
the other builds. The one for bar version 2.0 is shown below: |
| 209 |
|
| 210 |
.. include:: ../../example/tips-for-code-reuse/bar/2.0/nothreads |
| 211 |
:literal: |
| 212 |
|
| 213 |
Here we add a "prereq fooB/3.2" to require that the correct version of fooB |
| 214 |
was loaded. The bar version 1.0 stubfile is almost the same, just changing |
| 215 |
the setting of the version variable from 2.0 to 1.0, and changing the version |
| 216 |
of fooB needed as a prerequisite. |
| 217 |
|
| 218 |
So instead of having six somewhat lengthy modulefiles (~40 lines each), |
| 219 |
one for each combination of bar version and threading model, we put almost |
| 220 |
all the work into a single, slightly longer (55 lines) common file, with |
| 221 |
six small (8 lines or less) stub files which mostly just set variables |
| 222 |
to define the behavior of the common file. |
| 223 |
|
| 224 |
When customization is needed for a specific build, we showed two ways of |
| 225 |
accommodating such. First, for simple, more-or-less "one-off" cases, |
| 226 |
you can add some code to the |
| 227 |
appropriate "stubfiles" to handle the added complexity, like was done |
| 228 |
with the "prereq fooB" for the "nothreads" builds in the examples. |
| 229 |
|
| 230 |
Alternatively, one can add some additional logic to the common script |
| 231 |
to accommodate the complexity, like was done with the license file |
| 232 |
differences between builds. This is the recommended way to handle |
| 233 |
complexity which is likely to recur in multiple places. Generally, |
| 234 |
this is done by: |
| 235 |
|
| 236 |
* adding a parameter to be passed from the stub file to the common file and setting appropriately in the stub files (e.g. licensefile in bar/2.0/pthreads) |
| 237 |
|
| 238 |
* adding logic to the common file to default it appropriately (e.g. licensefile for the other builds) |
| 239 |
|
| 240 |
The above was only a simple example of what could be done, but it shows how |
| 241 |
modulefiles can leverage the fact that the modulefiles are evaluated in a |
| 242 |
full programming language to increase the amount of code reuse, which should |
| 243 |
reduce errors, improve consistency, and in general make things more manageable |
| 244 |
over the long run. |
| 245 |
|
| 246 |
The tips given above should work with both 3.x and 4.x versions of the Tcl |
| 247 |
based environmental packages. They have been explicitly tested on versions |
| 248 |
3.2.10 and 4.3.0. |
| 249 |
|
| 250 |
.. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: |