Debian Codegraph @

doc/source/cookbook/tips-for-code-reuse.rst

raw
1 .. _tips-for-code-reuse:
2
3 Tips for Code Reuse in Modulefiles
4 ==================================
5
6 Although Modules allow one to manage many packages and versions of packages
7 on a system, managing the modulefiles themselves can become error prone,
8 especially since the differences between modulefiles for different versions
9 of the same package are often rather small. This can lead to issues with
10 the consistency of the module definitions.
11
12 E.g., let us say you have a package foo, version 1.0 with a corresponding
13 modulefile. A bit later, foo version 2.0 comes out, and you add a modulefile
14 for that, likely copying the 1.0 version and editing a few lines. Then a
15 bug is discovered in the version 1.0 modulefile, e.g. an environment variable
16 that would be useful in some cases was omitted, or a typo in the help
17 function. The version 1.0 modulefile is fixed, but did you remember to
18 fix the 2.0 modulefile which, since it was created by copying the version 1.0
19 modulefile, likely has the same error?
20
21 This page includes some tips to reduce the amount of redundant code
22 in your modulefiles (i.e. increase the amount of code reuse) so that
23 they are easier to manage and more consistent.
24
25 A Simple Example
26 ----------------
27
28 We return to our "foo" example.
29 A fairly traditional set up would have a "foo" directory somewhere under your MODULEPATH with
30 modulefiles 1.0 and 2.0 underneath it. Each of 1.0 and 2.0 will likely be largely the same, having
31 basically the same help text, whatis, and variables set, with only some
32 changes to some text and paths to reflect the version difference. Example
33 code for this and other cases discussed in this document can be found in the
34 "doc/example/tips-for-code-reuse" subdirectory of this package. The modulefiles
35 for this simple example start in the "foo" subdirectory. For example, the
36 version 1.0 modulefile might look something like
37
38 .. include:: ../../example/tips-for-code-reuse/foo/1.0
39 :literal:
40
41 Although the above is simple enough, there is still much redundancy between
42 the version 1.0 and 2.0 files, and we do not bother showing the file for 2.0
43 because it is so similar (only 5 chars change between the 1.0 and 2.0
44 versions, basically all the "1.0" strings become "2.0"). The command
45 "module avail foo" shows versions 1.0 and 2.0, and loading either will
46 set the environmental variables PATH, MANPATH, and LD_LIBRARY_PATH appropriately,
47 conflict with itself (so only one version of foo can be loaded at a time),
48 and define the help and whatis texts.
49
50 To refactor this to increase code reuse, we copy one of the original
51 modulefiles to "common", and change all the references to the version number
52 to a "$version" variable. To avoid confusion with the previous version,
53 we change the package name from "foo" to "fooA", and the new versions can be
54 found in the "fooA" subdirectory of "doc/example/tips-for-code-reuse". After
55 adding some comments, etc, the common module becomes
56
57 .. include:: ../../example/tips-for-code-reuse/fooA/common
58 :literal:
59
60 We then create a pair of small stub files to replace "1.0" and "2.0". These
61 simply just set the version variable appropriately, and then "source" the
62 "common" file. For "1.0", the file would look like
63
64 .. include:: ../../example/tips-for-code-reuse/fooA/1.0
65 :literal:
66
67 A similar file would be created for "2.0", just changing the version number
68 in the "set version" line.
69 The "set moduledir" line sets the Tcl variable moduledir to contain the name of
70 the directory in which the final modulefile (i.e. "1.0") is located, so we
71 can find the "common" file.
72
73 The common file then handles all the work of defining the PATH, MANPATH,
74 and LD_LIBRARY_PATH environmental variables, using the value of the Tcl
75 variable "version" passed to it by the stub file. It also handles defining
76 the help procedure and the whatis text, and the conflict with itself.
77
78 Note that the "common" file does **NOT** start
79 with the Modules magic "#%Module"; this will keep "common" from showing
80 up in "module avail fooA". The "1.0" and "2.0" files do need to start with
81 the magic "#%Module" tag so the module command will "see" them. Because
82 of this, the "module avail fooA" command will just show the 1.0 and 2.0
83 versions as expected.
84
85 The files for this example are in the "doc/example/tips-for-code-reuse/fooA"
86 directory.
87
88 A Simple Example, revisited
89 ---------------------------
90
91 The "fooA" case above is fairly common, and we can actually improve upon
92 what we did above. To avoid confusion, we will repeat using the package
93 name "fooB", and the files for this example will be in the
94 "doc/example/tips-for-code-reuse/fooB" directory.
95
96 Using introspection in the modulefiles, we can get the version
97 number of fooB from the name of the modulefile. So we add code to the
98 top of "common" to default the version variable from the modulename.
99
100 We then copy the "1.0" stub modulefile to ".generic", and remove the line
101 which sets the version. Because is starts with a leading ".", the generic
102 modulefile will not be displayed in "module avail" and such. Since this
103 generic modulefile is now version independent, we can replace "1.0" and "2.0"
104 with symlinks to the .generic file.
105
106 The new common file looks like
107
108 .. include:: ../../example/tips-for-code-reuse/fooB/common
109 :literal:
110
111 The .generic stub file looks like
112
113 .. include:: ../../example/tips-for-code-reuse/fooB/1.0
114 :literal:
115
116 Again, the common file handles setting the environmental variables
117 PATH, MANPATH, and LD_LIBRARY_PATH, using the Tcl variable version
118 which was defaulted from modulename. This works even in the case
119 where the user does not specify the full module path, and the modulecmd
120 defaults the version (e.g. if the user types "module load fooB" without
121 specifying the version of fooB.) The common file also handles defining
122 the help procedure and whatis text. When the command "module avail fooB" is issued,
123 the common file does not get listed (because it does not start with the magic
124 "#%Module" tag), nor does the .generic file get listed (because it starts
125 with a period (".")), but the two 1.0 and 2.0 symlinks do get listed, just
126 as one wants.
127
128 This replaces a file for each version of fooB with one common file doing
129 all the work, one generic file, and a bunch of symlinks. And the .generic
130 file is not even specific to fooB. Indeed, at our site, we put the
131 generic modulefile in an "utilities" directory outside of the MODULEPATH,
132 and just symlink it to ".generic" in each application directory, and then
133 symlink the versions to this symlinked ".generic". So we end up with one
134 "common" file for each application, one shared generic modulefile, and a
135 bunch of symlinks.
136
137 A More Complicated Example
138 ---------------------------
139
140 Although the simple cases like the "foo" example above are not uncommon,
141 many packages are more complicated. We now consider the fictitious "bar"
142 package (example modulefiles in "example/tips-for-code-reuse/bar" directory).
143 This is admittedly a contrived example, but it displays some of the ways in
144 which we can modify the above simple cases to handle the more complicated
145 needs of some packages. Our "bar" package has the follow characteristics:
146
147 * We have two versions of "bar" built, "1.0" and "2.0"
148 * For each bar version, we have builds for three different threading models: nothreads, pthreads, and openmp.
149 * We expect users to use commands specifying both the bar version and the threading model, e.g. "module load fooB/1.0/pthreads" or "module load fooB/2.0/openmp"
150 * The path to the installation directories include both the version number of "bar" and the threading model
151 * The package requires the environmental variable BAR_LICENSE_FILE to be set appropriately
152 * All builds of bar version 1.0 use one license file which is different than that used by bar version2.0
153 * Bar 2.0 nothreads and openmp builds use the same license file, with a completely different name than that used by bar version 1.0.
154 * The bar 2.0 pthreads build uses its very own license file with a very different filename than those used by bar version 1.0 or the other threading models of version 2.0.
155 * The "nothreads" builds have a prerequisite on "fooB", with bar 1.0 wanting fooB version 1.1, and bar 2.0 wanting fooB version 3.2
156
157 Even with the above exceptions, there are still more similarities than
158 differences between the various modulefile definitions. Like in the "fooA"
159 case, we will have a "common" script that does almost all the work, and stub
160 files for each of the variants, but in this case the stub files are expected
161 to define some more variables. Our "common" file, which can be found in
162 the "example/tips-for-code-reuse/bar" directory, looks like
163 name "fooB", and the files for this example will be in the
164
165 .. include:: ../../example/tips-for-code-reuse/bar/common
166 :literal:
167
168 Here we have extended the parameters which are allowed to be passed in from
169 the build specific "stub" files; in addition to the version of "bar", the stub
170 files are expected to have set the variables "threadingmodel" and optionally
171 "licensefile". If "licensefile" was not set, the common script will default
172 it based on the version of "bar". The rest of the "common" file is similar
173 to the "fooA" case, except that we use both "version" and "threadingmodel"
174 in setting the paths, and now define BAR_LICENSE_FILE based on "licensefile".
175 Again, almost all the work is done in the common file, including setting the
176 relevant environmental variables, defining the help procedure and whatis text,
177 and preventing multiple versions of bar being loaded at the same time.
178
179 We now create stubfiles for each of the three threading models underneath directories
180 for each bar version. These can all be found under the
181 "example/tips-for-code-reuse/bar" directory. The one for bar 1.0 and openmp
182 looks like
183
184
185 .. include:: ../../example/tips-for-code-reuse/bar/1.0/openmp
186 :literal:
187
188 The pthreads version is almost identical, just setting threadingmodel to
189 "pthreads" instead of "openmp". The "openmp" build for bar version 2.0
190 is also very similar, differing only in value "version" is set to. In
191 all three cases, these stubfiles are basically the same as those used in
192 the "fooA" example; we set a couple of Tcl variables (version and threadingmodel),
193 and then source the common file which does all the work. The licensefile
194 variable is left unset so the common file will default it correctly based
195 on the bar version.
196
197 The stub file for the pthreads build for bar version 2.0 is as below:
198
199 .. include:: ../../example/tips-for-code-reuse/bar/2.0/pthreads
200 :literal:
201
202 This version is similar to the previous three, with the addition of explicitly
203 setting the "licensefile" variable. This value will be used in the common file
204 to set BAR_LICENSE_FILE, rather than the default which would have been set in
205 the common file had no value been provided.
206
207 Finally, the two "nothreads" build also have a slightly different stub file than
208 the other builds. The one for bar version 2.0 is shown below:
209
210 .. include:: ../../example/tips-for-code-reuse/bar/2.0/nothreads
211 :literal:
212
213 Here we add a "prereq fooB/3.2" to require that the correct version of fooB
214 was loaded. The bar version 1.0 stubfile is almost the same, just changing
215 the setting of the version variable from 2.0 to 1.0, and changing the version
216 of fooB needed as a prerequisite.
217
218 So instead of having six somewhat lengthy modulefiles (~40 lines each),
219 one for each combination of bar version and threading model, we put almost
220 all the work into a single, slightly longer (55 lines) common file, with
221 six small (8 lines or less) stub files which mostly just set variables
222 to define the behavior of the common file.
223
224 When customization is needed for a specific build, we showed two ways of
225 accommodating such. First, for simple, more-or-less "one-off" cases,
226 you can add some code to the
227 appropriate "stubfiles" to handle the added complexity, like was done
228 with the "prereq fooB" for the "nothreads" builds in the examples.
229
230 Alternatively, one can add some additional logic to the common script
231 to accommodate the complexity, like was done with the license file
232 differences between builds. This is the recommended way to handle
233 complexity which is likely to recur in multiple places. Generally,
234 this is done by:
235
236 * adding a parameter to be passed from the stub file to the common file and setting appropriately in the stub files (e.g. licensefile in bar/2.0/pthreads)
237
238 * adding logic to the common file to default it appropriately (e.g. licensefile for the other builds)
239
240 The above was only a simple example of what could be done, but it shows how
241 modulefiles can leverage the fact that the modulefiles are evaluated in a
242 full programming language to increase the amount of code reuse, which should
243 reduce errors, improve consistency, and in general make things more manageable
244 over the long run.
245
246 The tips given above should work with both 3.x and 4.x versions of the Tcl
247 based environmental packages. They have been explicitly tested on versions
248 3.2.10 and 4.3.0.
249
250 .. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: