Debian Codegraph @

doc/source/design/source-shell-script-in-modulefile.rst

raw
1 .. _source-shell-script-in-modulefile:
2
3 Source shell script in modulefile
4 =================================
5
6 Configuration
7 -------------
8
9 - No specific configuration
10
11
12 Specification
13 -------------
14
15 - Execute a shell script and include the environment changes it performs in modulefile
16
17 - Environment changes done by shell script are gathered and evaluated in modulefile context through corresponding modulefile commands
18 - Goal is to get the same environment after loading a modulefile using a :mfcmd:`source-sh` than if shell script targeted by the ``source-sh`` where directly sourced in shell session
19
20 - Environment changes tracked are (all environment elements ``module`` can handle):
21
22 - Value set to variable: transformed into ``setenv`` modulefile command
23 - Variable unset: transformed into ``unsetenv`` modulefile command
24 - Path element added to variable: transformed into ``append-path`` or ``prepend-path`` modulefile command
25 - Path element removed from variable: transformed into ``remove-path`` modulefile command
26 - Current working directory change: transformed into ``cd`` modulefile command
27 - Shell alias definition: transformed into ``set-alias``
28 - Shell alias unset: transformed into ``unset-alias``
29 - Shell function definition: transformed into ``set-function``
30 - Shell function unset: transformed into ``unset-function``
31 - Shell completion definition: transformed into ``complete``
32 - Shell completion unset: transformed into ``uncomplete``
33
34 - Depending on modulefile evaluation mode, ``source-sh`` has different behaviors:
35
36 - ``load``, ``refresh`` and ``scan``: execute script to gather its environment changes, transform them in modulefile commands and evaluate them
37 - ``unload``: undo environment changes made on load mode
38 - ``display``: execute script to gather its environment changes and report resulting command name and arguments for display
39 - ``help``, ``test`` and ``whatis``: no operation
40
41 - **Limitation**: code in modulefile cannot rely on the environment changes done in script targeted by a ``source-sh`` command
42
43 - For instance an environment variable set in shell script cannot be used to define another variable later in modulefile
44 - This will work on ``load``, ``unload`` and ``display`` modes, as script is run and/or analyzed and corresponding modulefile commands are evaluated in modulefile context
45 - But it will not work on the other modes, as script is not run and analyzed there
46
47 - To simplify processing as script need to be run and analyzed if not yet loaded, but if already loaded changes recorded in environment for tracking should be used instead
48 - To avoid a negative impact on performances on the ``whatis`` global evaluation
49
50 - Result of ``source-sh`` command evaluation is stored into the environment when modulefile is loaded
51
52 - To keep track of environment changes made by ``source-sh`` script evaluation
53 - In order to undo these changes when unloading modulefile and report corresponding modulefile commands when displaying modulefile
54 - Environment variable :envvar:`__MODULES_LMSOURCESH` is used for this need
55
56 - Using following syntax: ``mod&shell script arg|cmd|cmd|...&shell script|cmd:mod&shell script arg|cmd``
57 - Example value: ``foo/1&sh /tmp/source.sh|append-path PATH /tmp|cd /tmp``
58 - Characters used to structure information in ``__MODULES_LMSOURCESH`` (``:``, ``&`` and ``|``) are escaped
59
60 - Respectively to ``<EnvModEscPS>``, ``<EnvModEscS1>`` and ``<EnvModEscS2>``
61 - If found in environment changes to record
62
63 - Actual bodies of shell alias, shell functions and shell completions are not recorded in ``__MODULES_LMSOURCESH``, an empty body is recorded instead
64
65 - Example value: ``foo/1&sh /tmp/source.sh|set-alias alfoo {}|set-function funcfoo {}|complete bash foocmd {}``
66
67 - When unloading modulefile, content found for this modulefile in ``__MODULES_LMSOURCESH`` variable is evaluated to reverse environment changes
68
69 - When reaching a ``source-sh`` modulefile command, recorded content is evaluated through a modulefile Tcl interpreter in unload mode, to get the reversed effect
70
71 - When displaying modulefile
72
73 - If it is loaded
74
75 - the content found for this modulefile in ``__MODULES_LMSOURCESH`` variable is evaluated in display mode to report each resulting modulefile command
76 - script is evaluated to fetch shell alias, function and completion definitions which are not recorded in ``__MODULES_LMSOURCESH``
77
78 - If not loaded, script is evaluated to gather environment changes and report each resulting modulefile command
79
80 - Script targeted by a ``source-sh`` command has to be run and environment prior this run and after this run have to be compared to determine the environment changes the script performs
81
82 - The shell to use to run script has to be specified to ``source-sh``
83 - This shell will be run to execute the following sequence:
84
85 1. output current environment state
86 2. source the script with its args
87 3. output environment state after script source
88
89 - Script output is kept to return it in an error message in case its execution fails
90 - This comparison determines the environment changes performed by script which are then translated into corresponding modulefile commands
91 - Shell is run in a mode where neither its user nor its system-wide configuration is evaluated to avoid side effect
92 - Shell needs current environment to correctly evaluate script
93
94 - it seems desirable to run shell in an empty environment to get the full list of changes it does over the environment
95 - but the script may need the environment to be defined to correctly operate
96
97 - for instance it needs the ``PATH`` to be set to execute external commands
98
99 - so shell run inherits current environment to ensure script will be properly evaluated
100 - but as a consequence if an environment change performed by script is already done prior script run, this environment change will not be *seen*
101
102 - **Limitation**: a variable already set by another module will not be seen set sourced script
103
104 - if this module is unloaded and if it does not trigger the unload of the modulefile using ``source-sh``, variable will be unset as source-sh script has not increased variable reference counter
105
106 - **Note**: environment change done through ``source-sh`` will not preserve change order occurring in sourced script, as all changes are analyzed after sourcing script
107
108 - Most shell will not get the full environment content when spawned as a sub-process
109
110 - For instance shell aliases are not transmitted into sub-shells
111 - As a result the environment prior script source will most of the time have no shell alias defined
112
113 - **Note**: shells may have specific behaviors
114
115 - ``dash`` cannot pass arguments to the sourced script
116 - ``ksh93`` reads the full script prior sourcing it which may cause issue if a shell alias is defined in script then used in script
117
118 - **Limitation**: implementation does not currently support:
119
120 - the ``zsh`` shell when used as ``sh`` or as ``ksh``
121 - the ``mksh`` shell
122 - the BSD variant of the ``sh`` shell
123 - the Windows ``cmd`` shell
124
125 - **Note**: if sourced script produces shell alias or function, these alias or function may not be compatible with the current shell of the user
126
127 - Shell completion is defined for the shell used to evaluate script
128
129 - :mfcmd:`complete` commands are generated even if current user shell is not the same as the shell used to evaluate script
130 - but the completion definition will produce a no-operation on current user shell when :mfcmd:`complete` command is evaluated
131
132 - **Note**: the mechanism described here only applies for shell script as to understand the impact the script execution over the user environment, this environment need to be compared prior and after execution
133
134 - ``source-sh`` modulefile command relies of the ``sh-to-mod`` procedure of ``modulecmd.tcl``
135
136 - ``sh-to-mod`` procedure handles the evaluation of the targeted script and comparison of the environment prior and after this comparison
137 - ``sh-to-mod`` returns as a result the list of modulefile commands describing the environment changes made by the analyzed script
138
139 - :subcmd:`sh-to-mod` is a module sub-command, calling the ``sh-to-mod`` procedure and outputting resulting modulefile commands
140
141 - with a ``#%Module`` header
142 - to enable users to convert the environment changes made by script in modulefiles
143
144 - There is no need to also make ``sh-to-mod`` a modulefile command and ``source-sh`` a module sub-command
145
146 - Shell to use to source targeted script may be specified as a command name or as a path name
147
148 - When specified as a command name (e.g., ``bash``), command location will be determined based on currently set ``PATH``
149 - When specified as a path name (e.g., ``/bin/bash``), this path name determines command location
150
151 - Prior executing shell and making it sourcing script:
152
153 - Shell is checked to ensure an executable command matches it
154 - Script is checked to ensure it exists
155 - Those tests are done prior executing to avoid it if one check fails and provide a consistent error message whatever the shell used
156
157 - Environment changes to output as result should be enclosed and escaped
158
159 - Enclosed if they contains space character (like white-space, tab and newline characters)
160 - Escaped if they contains curly braces, as output is formatted as Tcl code, to avoid parsing issue
161
162 - If an error occurs during targeted script evaluation
163
164 - Error is thrown which leads to either a modulefile evaluation error or a module sub-command error
165 - Error and output messages reported by evaluated script will be reported along error by modulefile command or module sub-command
166
167 - To get prior and resulting environment state
168
169 - ``env`` command is not used to gather environment state through exec source execution
170
171 - it would simplify environment state parsing, as same command would be used for every shell
172 - but it is an external command, so requires extra processing and an additional requirement
173 - moreover it does not return shell functions in general, only exported Bash functions
174
175 - Shell builtin commands are used to query existing environment variables, aliases, functions, completions and current working directory
176
177 - which provides best processing efficiency
178 - but leads to specific output parsing for each shell
179
180 - A separator string ``%ModulesShToMod%`` is printed between each kind of environment item (variable, alias, etc) and also prior and after script evaluation
181
182 - to separate each output kind and then be able to split them for separate analysis
183
184 - De-duplication of path entries is applied for changes on path-like environment variables
185
186 - If the same path entry appears several times in the newly prepended entries for a variable, the first occurrence of this entry is kept others are dropped
187 - If the same path entry appears several times in the newly appended entries for a variable, the first occurrence of this entry is kept others are dropped
188 - De-duplication is not applied for path entries:
189
190 - appearing in both the new prepended entries and newly appended entries
191 - appearing in newly prepended entries or newly appended entries and in entries defined prior script evaluation
192
193 - An environment variable equaling to the path separator character (``:``) prior script evaluation is considered as undefined prior script evaluation to avoid misleading analysis
194
195 - Environment variables made for Modules private use are filtered-out from the environment changes produced
196
197 - ``LOADEDMODULES``, ``_LMFILES_`` and any variable prefixed by ``__MODULES_`` are concerned
198 - Changes relative to Modules state are ignored this way
199 - If script loads in turn a modulefile, environment changes reported will not report the loaded module but only the environment changes it does
200 - Modules configuration variable (prefixed by ``MODULES_``) are still taken into account
201
202 - There may be several shell completion changes found for the same command on particular shell like fish
203
204 - It produces multiple ``complete`` modulefile commands
205 - But in ``__MODULES_LMSOURCESH`` variable, a single entry is recorded as a single ``uncomplete`` command should be generated when unloading modulefile
206 - If completion definition is partial for a command when loading modulefile, on fish shell previous completion definition is cleared right before setting those defined in modulefile
207 - When a completion definition difference is spotted after script evaluation, new completion definition is set which means previous definition is cleared (either intrinsically by shell or for fish due to the definition clear specifically added)
208
209 - Fish shell allows nested definition of shell functions
210
211 - A separator string ``%ModulesSubShToMod%`` is printed between each function definition not to get main and nested functions mixed up
212 - Such sub-separator is only applied on fish shell
213
214 - ``bash-eval`` shell is added in version 5.4
215
216 - Instead of sourcing a bash shell script, execution output of this script is evaluated
217
218 - ``--ignore`` option is added in version 5.4
219
220 - This option accepts a list of shell element separated by ``:`` character among:
221
222 - ``envvar``
223 - ``alias``
224 - ``function``
225 - ``chdir``
226 - ``complete``
227
228 - Shell elements changed by sourced script and listed on ``--ignore`` option are ignored (no modulefile command relative to these changes are produced)
229
230 - **FUTURE**: this feature may be extended to translate environment changes made by tools like Spack, Lmod or pkg-config. It may provide this way bridges between different realms.
231
232 .. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: