Debian Codegraph @

doc/source/design/source-shell-script-in-modulefile.rst

raw
1 .. _source-shell-script-in-modulefile:
2
3 Source shell script in modulefile
4 =================================
5
6 Configuration
7 -------------
8
9 - No specific configuration
10
11
12 Specification
13 -------------
14
15 - Execute a shell script and include the environment changes it performs in modulefile
16
17 - Environment changes done by shell script are gathered and evaluated in modulefile context through corresponding modulefile commands
18 - Goal is to get the same environment after loading a modulefile using a :mfcmd:`source-sh` than if shell script targeted by the ``source-sh`` where directly sourced in shell session
19
20 - Environment changes tracked are (all environment elements ``module`` can handle):
21
22 - Value set to variable: transformed into ``setenv`` modulefile command
23 - Variable unset: transformed into ``unsetenv`` modulefile command
24 - Path element added to variable: transformed into ``append-path`` or ``prepend-path`` modulefile command
25 - Path element removed from variable: transformed into ``remove-path`` modulefile command
26 - Current working directory change: transformed into ``cd`` modulefile command
27 - Shell alias definition: transformed into ``set-alias``
28 - Shell alias unset: transformed into ``unset-alias``
29 - Shell function definition: transformed into ``set-function``
30 - Shell function unset: transformed into ``unset-function``
31 - Shell completion definition: transformed into ``complete``
32 - Shell completion unset: transformed into ``uncomplete``
33
34 - Depending on modulefile evaluation mode, ``source-sh`` has different behaviors:
35
36 - ``load``, ``refresh`` and ``scan``: execute script to gather its environment changes, transform them in modulefile commands and evaluate them
37 - ``unload``: undo environment changes made on load mode
38 - ``display``: execute script to gather its environment changes and report resulting command name and arguments for display
39 - ``help``, ``test`` and ``whatis``: no operation
40
41 - **Limitation**: code in modulefile cannot rely on the environment changes done in script targeted by a ``source-sh`` command
42
43 - For instance an environment variable set in shell script cannot be used to define another variable later in modulefile
44 - This will work on ``load``, ``unload`` and ``display`` modes, as script is run and/or analyzed and corresponding modulefile commands are evaluated in modulefile context
45 - But it will not work on the other modes, as script is not run and analyzed there
46
47 - To simplify processing as script need to be run and analyzed if not yet loaded, but if already loaded changes recorded in environment for tracking should be used instead
48 - To avoid a negative impact on performances on the ``whatis`` global evaluation
49
50 - Result of ``source-sh`` command evaluation is stored into the environment when modulefile is loaded
51
52 - To keep track of environment changes made by ``source-sh`` script evaluation
53 - In order to undo these changes when unloading modulefile and report corresponding modulefile commands when displaying modulefile
54 - Environment variable :envvar:`__MODULES_LMSOURCESH` is used for this need
55
56 - Using following syntax: ``mod&shell script arg|cmd|cmd|...&shell script|cmd:mod&shell script arg|cmd``
57 - Example value: ``foo/1&sh /tmp/source.sh|append-path PATH /tmp|cd /tmp``
58 - Characters used to structure information in ``__MODULES_LMSOURCESH`` (``:``, ``&`` and ``|``) are escaped
59
60 - Respectively to ``<EnvModEscPS>``, ``<EnvModEscS1>`` and ``<EnvModEscS2>``
61 - If found in environment changes to record
62
63 - Actual bodies of shell alias, shell functions and shell completions are not recorded in ``__MODULES_LMSOURCESH``, an empty body is recorded instead
64
65 - Example value: ``foo/1&sh /tmp/source.sh|set-alias alfoo {}|set-function funcfoo {}|complete bash foocmd {}``
66
67 - When unloading modulefile, content found for this modulefile in ``__MODULES_LMSOURCESH`` variable is evaluated to reverse environment changes
68
69 - When reaching a ``source-sh`` modulefile command, recorded content is evaluated through a modulefile Tcl interpreter in unload mode, to get the reversed effect
70
71 - When displaying modulefile
72
73 - If it is loaded
74
75 - the content found for this modulefile in ``__MODULES_LMSOURCESH`` variable is evaluated in display mode to report each resulting modulefile command
76 - script is evaluated to fetch shell alias, function and completion definitions which are not recorded in ``__MODULES_LMSOURCESH``
77
78 - If not loaded, script is evaluated to gather environment changes and report each resulting modulefile command
79
80 - Script targeted by a ``source-sh`` command has to be run and environment prior this run and after this run have to be compared to determine the environment changes the script performs
81
82 - The shell to use to run script has to be specified to ``source-sh``
83 - This shell will be run to execute the following sequence:
84
85 1. output current environment state
86 2. source the script with its args
87 3. output environment state after script source
88
89 - Script output is kept to return it in an error message in case its execution fails
90 - This comparison determines the environment changes performed by script which are then translated into corresponding modulefile commands
91 - Shell is run in a mode where neither its user nor its system-wide configuration is evaluated to avoid side effect
92 - Shell needs current environment to correctly evaluate script
93
94 - it seems desirable to run shell in an empty environment to get the full list of changes it does over the environment
95 - but the script may need the environment to be defined to correctly operate
96
97 - for instance it needs the ``PATH`` to be set to execute external commands
98
99 - so shell run inherits current environment to ensure script will be properly evaluated
100 - but as a consequence if an environment change performed by script is already done prior script run, this environment change will not be *seen*
101
102 - **Limitation**: a variable already set by another module will not be seen set sourced script
103
104 - if this module is unloaded and if it does not trigger the unload of the modulefile using ``source-sh``, variable will be unset as source-sh script has not increased variable reference counter
105
106 - **Limitation**: Script will not get access to shell aliases, functions or completions defined in interactive mode or shell profile configurations
107
108 - **Note**: environment change done through ``source-sh`` will not preserve change order occurring in sourced script, as all changes are analyzed after sourcing script
109
110 - Most shell will not get the full environment content when spawned as a sub-process
111
112 - For instance shell aliases are not transmitted into sub-shells
113 - As a result the environment prior script source will most of the time have no shell alias defined
114
115 - **Note**: shells may have specific behaviors
116
117 - ``dash`` cannot pass arguments to the sourced script
118 - ``ksh93`` reads the full script prior sourcing it which may cause issue if a shell alias is defined in script then used in script
119
120 - **Limitation**: implementation does not currently support:
121
122 - the ``zsh`` shell when used as ``sh`` or as ``ksh``
123 - the ``mksh`` shell
124 - the BSD variant of the ``sh`` shell
125 - the Windows ``cmd`` shell
126
127 - **Note**: if sourced script produces shell alias or function, these alias or function may not be compatible with the current shell of the user
128
129 - Shell completion is defined for the shell used to evaluate script
130
131 - :mfcmd:`complete` commands are generated even if current user shell is not the same as the shell used to evaluate script
132 - but the completion definition will produce a no-operation on current user shell when :mfcmd:`complete` command is evaluated
133
134 - **Note**: the mechanism described here only applies for shell script as to understand the impact the script execution over the user environment, this environment need to be compared prior and after execution
135
136 - ``source-sh`` modulefile command relies of the ``sh-to-mod`` procedure of ``modulecmd.tcl``
137
138 - ``sh-to-mod`` procedure handles the evaluation of the targeted script and comparison of the environment prior and after this comparison
139 - ``sh-to-mod`` returns as a result the list of modulefile commands describing the environment changes made by the analyzed script
140
141 - :subcmd:`sh-to-mod` is a module sub-command, calling the ``sh-to-mod`` procedure and outputting resulting modulefile commands
142
143 - with a ``#%Module`` header
144 - to enable users to convert the environment changes made by script in modulefiles
145
146 - There is no need to also make ``sh-to-mod`` a modulefile command and ``source-sh`` a module sub-command
147
148 - Shell to use to source targeted script may be specified as a command name or as a path name
149
150 - When specified as a command name (e.g., ``bash``), command location will be determined based on currently set ``PATH``
151 - When specified as a path name (e.g., ``/bin/bash``), this path name determines command location
152
153 - Prior executing shell and making it sourcing script:
154
155 - Shell is checked to ensure an executable command matches it
156 - Script is checked to ensure it exists
157 - Those tests are done prior executing to avoid it if one check fails and provide a consistent error message whatever the shell used
158
159 - Environment changes to output as result should be enclosed and escaped
160
161 - Enclosed if they contains space character (like white-space, tab and newline characters)
162 - Escaped if they contains curly braces, as output is formatted as Tcl code, to avoid parsing issue
163
164 - If an error occurs during targeted script evaluation
165
166 - Error is thrown which leads to either a modulefile evaluation error or a module sub-command error
167 - Error and output messages reported by evaluated script will be reported along error by modulefile command or module sub-command
168
169 - To get prior and resulting environment state
170
171 - ``env`` command is not used to gather environment state through exec source execution
172
173 - it would simplify environment state parsing, as same command would be used for every shell
174 - but it is an external command, so requires extra processing and an additional requirement
175 - moreover it does not return shell functions in general, only exported Bash functions
176
177 - Shell builtin commands are used to query existing environment variables, aliases, functions, completions and current working directory
178
179 - which provides best processing efficiency
180 - but leads to specific output parsing for each shell
181
182 - A separator string ``%ModulesShToMod%`` is printed between each kind of environment item (variable, alias, etc) and also prior and after script evaluation
183
184 - to separate each output kind and then be able to split them for separate analysis
185
186 - Path entries found in the newly prepended entries or in the newly appended entries for a variable are never de-duplicated: every occurrence is kept and passed on the corresponding ``prepend-path`` or ``append-path`` command
187
188 - The ``--duplicates`` option is added to a ``prepend-path`` or ``append-path`` command when one of the entries it adds:
189
190 - is already found in the variable value prior script evaluation
191 - is found several times in the newly prepended entries or in the newly appended entries
192 - is found in both the newly prepended entries and the newly appended entries
193
194 - Such an entry is genuinely meant to end up at this specific position in the resulting value
195 - Without this option, ``prepend-path`` and ``append-path``'s default de-duplication behavior would silently absorb the entry, and the ``path_entry_reorder`` configuration option could relocate it, which would not correctly reproduce the change made by the script
196
197 - An environment variable equaling to the path separator character (``:``) prior script evaluation is considered as undefined prior script evaluation to avoid misleading analysis
198
199 - Path-like variable change is recorded as a ``setenv`` command instead of ``prepend-path`` and ``append-path`` commands when the newly prepended and newly appended parts would each need a different delimiter character to be expressed
200
201 - Prior and after variable values are compared to find a common part between them: what is found before this common part is expressed as a ``prepend-path`` command and what is found after it as an ``append-path`` command
202 - Each of these two commands may need its own delimiter character, determined from the character found in the after value right before (for the prepended part) or right after (for the appended part) the common part, when this character is not the variable path separator
203 - When the delimiter character determined for the prepended part differs from the one determined for the appended part, describing the change as a ``prepend-path`` and an ``append-path`` command would not correctly reproduce the resulting value, as each command would independently split and de-duplicate the full variable value using its own delimiter
204 - A ``setenv`` command directly setting the after value is generated instead in this situation, to always accurately reflect the change made by the script
205
206 - Environment variables made for Modules private use are filtered-out from the environment changes produced
207
208 - ``LOADEDMODULES``, ``_LMFILES_`` and any variable prefixed by ``__MODULES_`` are concerned
209 - Changes relative to Modules state are ignored this way
210 - If script loads in turn a modulefile, environment changes reported will not report the loaded module but only the environment changes it does
211 - Modules configuration variable (prefixed by ``MODULES_``) are still taken into account
212
213 - There may be several shell completion changes found for the same command on particular shell like fish
214
215 - It produces multiple ``complete`` modulefile commands
216 - But in ``__MODULES_LMSOURCESH`` variable, a single entry is recorded as a single ``uncomplete`` command should be generated when unloading modulefile
217 - If completion definition is partial for a command when loading modulefile, on fish shell previous completion definition is cleared right before setting those defined in modulefile
218 - When a completion definition difference is spotted after script evaluation, new completion definition is set which means previous definition is cleared (either intrinsically by shell or for fish due to the definition clear specifically added)
219
220 - Fish shell allows nested definition of shell functions
221
222 - A separator string ``%ModulesSubShToMod%`` is printed between each function definition not to get main and nested functions mixed up
223 - Such sub-separator is only applied on fish shell
224
225 - ``bash-eval`` shell is added in version 5.4
226
227 - Instead of sourcing a bash shell script, execution output of this script is evaluated
228
229 - ``--ignore`` option is added in version 5.4
230
231 - This option accepts a list of shell element separated by ``:`` character among:
232
233 - ``envvar``
234 - ``alias``
235 - ``function``
236 - ``chdir``
237 - ``complete``
238
239 - Shell elements changed by sourced script and listed on ``--ignore`` option are ignored (no modulefile command relative to these changes are produced)
240
241 - **FUTURE**: this feature may be extended to translate environment changes made by tools like Spack, Lmod or pkg-config. It may provide this way bridges between different realms.
242
243 .. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: