.. _source-shell-script-in-modulefile: Source shell script in modulefile ================================= Configuration ------------- - No specific configuration Specification ------------- - Execute a shell script and include the environment changes it performs in modulefile - Environment changes done by shell script are gathered and evaluated in modulefile context through corresponding modulefile commands - Goal is to get the same environment after loading a modulefile using a :mfcmd:`source-sh` than if shell script targeted by the ``source-sh`` where directly sourced in shell session - Environment changes tracked are (all environment elements ``module`` can handle): - Value set to variable: transformed into ``setenv`` modulefile command - Variable unset: transformed into ``unsetenv`` modulefile command - Path element added to variable: transformed into ``append-path`` or ``prepend-path`` modulefile command - Path element removed from variable: transformed into ``remove-path`` modulefile command - Current working directory change: transformed into ``cd`` modulefile command - Shell alias definition: transformed into ``set-alias`` - Shell alias unset: transformed into ``unset-alias`` - Shell function definition: transformed into ``set-function`` - Shell function unset: transformed into ``unset-function`` - Shell completion definition: transformed into ``complete`` - Shell completion unset: transformed into ``uncomplete`` - Depending on modulefile evaluation mode, ``source-sh`` has different behaviors: - ``load``, ``refresh`` and ``scan``: execute script to gather its environment changes, transform them in modulefile commands and evaluate them - ``unload``: undo environment changes made on load mode - ``display``: execute script to gather its environment changes and report resulting command name and arguments for display - ``help``, ``test`` and ``whatis``: no operation - **Limitation**: code in modulefile cannot rely on the environment changes done in script targeted by a ``source-sh`` command - For instance an environment variable set in shell script cannot be used to define another variable later in modulefile - This will work on ``load``, ``unload`` and ``display`` modes, as script is run and/or analyzed and corresponding modulefile commands are evaluated in modulefile context - But it will not work on the other modes, as script is not run and analyzed there - To simplify processing as script need to be run and analyzed if not yet loaded, but if already loaded changes recorded in environment for tracking should be used instead - To avoid a negative impact on performances on the ``whatis`` global evaluation - Result of ``source-sh`` command evaluation is stored into the environment when modulefile is loaded - To keep track of environment changes made by ``source-sh`` script evaluation - In order to undo these changes when unloading modulefile and report corresponding modulefile commands when displaying modulefile - Environment variable :envvar:`__MODULES_LMSOURCESH` is used for this need - Using following syntax: ``mod&shell script arg|cmd|cmd|...&shell script|cmd:mod&shell script arg|cmd`` - Example value: ``foo/1&sh /tmp/source.sh|append-path PATH /tmp|cd /tmp`` - Characters used to structure information in ``__MODULES_LMSOURCESH`` (``:``, ``&`` and ``|``) are escaped - Respectively to ````, ```` and ```` - If found in environment changes to record - Actual bodies of shell alias, shell functions and shell completions are not recorded in ``__MODULES_LMSOURCESH``, an empty body is recorded instead - Example value: ``foo/1&sh /tmp/source.sh|set-alias alfoo {}|set-function funcfoo {}|complete bash foocmd {}`` - When unloading modulefile, content found for this modulefile in ``__MODULES_LMSOURCESH`` variable is evaluated to reverse environment changes - When reaching a ``source-sh`` modulefile command, recorded content is evaluated through a modulefile Tcl interpreter in unload mode, to get the reversed effect - When displaying modulefile - If it is loaded - the content found for this modulefile in ``__MODULES_LMSOURCESH`` variable is evaluated in display mode to report each resulting modulefile command - script is evaluated to fetch shell alias, function and completion definitions which are not recorded in ``__MODULES_LMSOURCESH`` - If not loaded, script is evaluated to gather environment changes and report each resulting modulefile command - Script targeted by a ``source-sh`` command has to be run and environment prior this run and after this run have to be compared to determine the environment changes the script performs - The shell to use to run script has to be specified to ``source-sh`` - This shell will be run to execute the following sequence: 1. output current environment state 2. source the script with its args 3. output environment state after script source - Script output is kept to return it in an error message in case its execution fails - This comparison determines the environment changes performed by script which are then translated into corresponding modulefile commands - Shell is run in a mode where neither its user nor its system-wide configuration is evaluated to avoid side effect - Shell needs current environment to correctly evaluate script - it seems desirable to run shell in an empty environment to get the full list of changes it does over the environment - but the script may need the environment to be defined to correctly operate - for instance it needs the ``PATH`` to be set to execute external commands - so shell run inherits current environment to ensure script will be properly evaluated - but as a consequence if an environment change performed by script is already done prior script run, this environment change will not be *seen* - **Limitation**: a variable already set by another module will not be seen set sourced script - if this module is unloaded and if it does not trigger the unload of the modulefile using ``source-sh``, variable will be unset as source-sh script has not increased variable reference counter - **Limitation**: Script will not get access to shell aliases, functions or completions defined in interactive mode or shell profile configurations - **Note**: environment change done through ``source-sh`` will not preserve change order occurring in sourced script, as all changes are analyzed after sourcing script - Most shell will not get the full environment content when spawned as a sub-process - For instance shell aliases are not transmitted into sub-shells - As a result the environment prior script source will most of the time have no shell alias defined - **Note**: shells may have specific behaviors - ``dash`` cannot pass arguments to the sourced script - ``ksh93`` reads the full script prior sourcing it which may cause issue if a shell alias is defined in script then used in script - **Limitation**: implementation does not currently support: - the ``zsh`` shell when used as ``sh`` or as ``ksh`` - the ``mksh`` shell - the BSD variant of the ``sh`` shell - the Windows ``cmd`` shell - **Note**: if sourced script produces shell alias or function, these alias or function may not be compatible with the current shell of the user - Shell completion is defined for the shell used to evaluate script - :mfcmd:`complete` commands are generated even if current user shell is not the same as the shell used to evaluate script - but the completion definition will produce a no-operation on current user shell when :mfcmd:`complete` command is evaluated - **Note**: the mechanism described here only applies for shell script as to understand the impact the script execution over the user environment, this environment need to be compared prior and after execution - ``source-sh`` modulefile command relies of the ``sh-to-mod`` procedure of ``modulecmd.tcl`` - ``sh-to-mod`` procedure handles the evaluation of the targeted script and comparison of the environment prior and after this comparison - ``sh-to-mod`` returns as a result the list of modulefile commands describing the environment changes made by the analyzed script - :subcmd:`sh-to-mod` is a module sub-command, calling the ``sh-to-mod`` procedure and outputting resulting modulefile commands - with a ``#%Module`` header - to enable users to convert the environment changes made by script in modulefiles - There is no need to also make ``sh-to-mod`` a modulefile command and ``source-sh`` a module sub-command - Shell to use to source targeted script may be specified as a command name or as a path name - When specified as a command name (e.g., ``bash``), command location will be determined based on currently set ``PATH`` - When specified as a path name (e.g., ``/bin/bash``), this path name determines command location - Prior executing shell and making it sourcing script: - Shell is checked to ensure an executable command matches it - Script is checked to ensure it exists - Those tests are done prior executing to avoid it if one check fails and provide a consistent error message whatever the shell used - Environment changes to output as result should be enclosed and escaped - Enclosed if they contains space character (like white-space, tab and newline characters) - Escaped if they contains curly braces, as output is formatted as Tcl code, to avoid parsing issue - If an error occurs during targeted script evaluation - Error is thrown which leads to either a modulefile evaluation error or a module sub-command error - Error and output messages reported by evaluated script will be reported along error by modulefile command or module sub-command - To get prior and resulting environment state - ``env`` command is not used to gather environment state through exec source execution - it would simplify environment state parsing, as same command would be used for every shell - but it is an external command, so requires extra processing and an additional requirement - moreover it does not return shell functions in general, only exported Bash functions - Shell builtin commands are used to query existing environment variables, aliases, functions, completions and current working directory - which provides best processing efficiency - but leads to specific output parsing for each shell - A separator string ``%ModulesShToMod%`` is printed between each kind of environment item (variable, alias, etc) and also prior and after script evaluation - to separate each output kind and then be able to split them for separate analysis - Path entries found in the newly prepended entries or in the newly appended entries for a variable are never de-duplicated: every occurrence is kept and passed on the corresponding ``prepend-path`` or ``append-path`` command - The ``--duplicates`` option is added to a ``prepend-path`` or ``append-path`` command when one of the entries it adds: - is already found in the variable value prior script evaluation - is found several times in the newly prepended entries or in the newly appended entries - is found in both the newly prepended entries and the newly appended entries - Such an entry is genuinely meant to end up at this specific position in the resulting value - Without this option, ``prepend-path`` and ``append-path``'s default de-duplication behavior would silently absorb the entry, and the ``path_entry_reorder`` configuration option could relocate it, which would not correctly reproduce the change made by the script - An environment variable equaling to the path separator character (``:``) prior script evaluation is considered as undefined prior script evaluation to avoid misleading analysis - Path-like variable change is recorded as a ``setenv`` command instead of ``prepend-path`` and ``append-path`` commands when the newly prepended and newly appended parts would each need a different delimiter character to be expressed - Prior and after variable values are compared to find a common part between them: what is found before this common part is expressed as a ``prepend-path`` command and what is found after it as an ``append-path`` command - Each of these two commands may need its own delimiter character, determined from the character found in the after value right before (for the prepended part) or right after (for the appended part) the common part, when this character is not the variable path separator - When the delimiter character determined for the prepended part differs from the one determined for the appended part, describing the change as a ``prepend-path`` and an ``append-path`` command would not correctly reproduce the resulting value, as each command would independently split and de-duplicate the full variable value using its own delimiter - A ``setenv`` command directly setting the after value is generated instead in this situation, to always accurately reflect the change made by the script - Environment variables made for Modules private use are filtered-out from the environment changes produced - ``LOADEDMODULES``, ``_LMFILES_`` and any variable prefixed by ``__MODULES_`` are concerned - Changes relative to Modules state are ignored this way - If script loads in turn a modulefile, environment changes reported will not report the loaded module but only the environment changes it does - Modules configuration variable (prefixed by ``MODULES_``) are still taken into account - There may be several shell completion changes found for the same command on particular shell like fish - It produces multiple ``complete`` modulefile commands - But in ``__MODULES_LMSOURCESH`` variable, a single entry is recorded as a single ``uncomplete`` command should be generated when unloading modulefile - If completion definition is partial for a command when loading modulefile, on fish shell previous completion definition is cleared right before setting those defined in modulefile - When a completion definition difference is spotted after script evaluation, new completion definition is set which means previous definition is cleared (either intrinsically by shell or for fish due to the definition clear specifically added) - Fish shell allows nested definition of shell functions - A separator string ``%ModulesSubShToMod%`` is printed between each function definition not to get main and nested functions mixed up - Such sub-separator is only applied on fish shell - ``bash-eval`` shell is added in version 5.4 - Instead of sourcing a bash shell script, execution output of this script is evaluated - ``--ignore`` option is added in version 5.4 - This option accepts a list of shell element separated by ``:`` character among: - ``envvar`` - ``alias`` - ``function`` - ``chdir`` - ``complete`` - Shell elements changed by sourced script and listed on ``--ignore`` option are ignored (no modulefile command relative to these changes are produced) - **FUTURE**: this feature may be extended to translate environment changes made by tools like Spack, Lmod or pkg-config. It may provide this way bridges between different realms. .. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: