| 1 |
.. _sync-remote-appdir: |
| 2 |
|
| 3 |
Sync remote application directories on first load |
| 4 |
================================================= |
| 5 |
|
| 6 |
Some workstations and laptops mount a network share holding all kind of |
| 7 |
software installations that may be used on this machine, alongside their own |
| 8 |
local, faster disk. Reading application files directly from this network |
| 9 |
share on every use works, but it is slower than reading them from local |
| 10 |
disk, and it stops working entirely when the machine goes offline. |
| 11 |
|
| 12 |
This recipe describes how to keep modulefiles pointing at application |
| 13 |
directories on a network share mounted under :file:`/remote_apps`, while |
| 14 |
transparently copying, with the `rsync`_ tool, the directory of a given |
| 15 |
application to :file:`/local_apps` the first time the corresponding module |
| 16 |
is loaded. Every later load of the same module finds its application |
| 17 |
directory already synced locally and skips the copy. |
| 18 |
|
| 19 |
.. _rsync: https://rsync.samba.org/ |
| 20 |
|
| 21 |
Implementation |
| 22 |
-------------- |
| 23 |
|
| 24 |
A :file:`.module_appdir_map` file, located at the root of the |
| 25 |
:file:`/remote_apps` network share, maps each module name and version to the |
| 26 |
basename of its application directory under :file:`/remote_apps`: |
| 27 |
|
| 28 |
.. literalinclude:: ../../example/sync-remote-appdir/.module_appdir_map |
| 29 |
:caption: /remote_apps/.module_appdir_map |
| 30 |
|
| 31 |
The :file:`.modulerc` file at the root of the modulepath reads this map file |
| 32 |
and applies the ``remote`` tag, with the :mfcmd:`module-tag` modulefile |
| 33 |
command, to every module listed in it whose application directory has not |
| 34 |
been synced locally yet -- tracked by the presence of a |
| 35 |
:file:`.<basename>.synced` marker file under :file:`/local_apps`: |
| 36 |
|
| 37 |
.. literalinclude:: ../../example/sync-remote-appdir/modulefiles/.modulerc |
| 38 |
:language: tcl |
| 39 |
:caption: modulefiles/.modulerc |
| 40 |
|
| 41 |
The actual sync is performed by a :mhook:`before-modulefile-eval` hook, |
| 42 |
registered with the ``add-hook`` siteconfig command (see the :ref:`hook-api` |
| 43 |
design notes and the *Hooks* section of :manpage:`module(1)` man page). The |
| 44 |
hook procedure returns immediately if the evaluation mode is not ``load``, |
| 45 |
or if the module being evaluated is not tagged ``remote`` -- checked with |
| 46 |
:mfcmd:`module-info tags<module-info>`, run in the modulefile Tcl |
| 47 |
interpreter reached through ``getCurrentModfileInterpName``, exactly as |
| 48 |
already documented for a hook procedure that needs to run modulefile |
| 49 |
commands. Otherwise, it looks up the application directory basename mapped |
| 50 |
to the module in :file:`.module_appdir_map`, and skips the sync if this |
| 51 |
directory was already marked as synced by a previous load. Otherwise it |
| 52 |
copies the application directory with ``rsync`` and, on success, touches |
| 53 |
the :file:`.<basename>.synced` marker file so later loads skip the copy. |
| 54 |
|
| 55 |
Because the modulepath root :file:`.modulerc` tags a module ``remote`` ahead |
| 56 |
of the sync actually happening, this tag would otherwise still be recorded |
| 57 |
as applying to the module once loaded, which is misleading once its |
| 58 |
application directory has just been synced locally. The ``remote`` tag is |
| 59 |
therefore added to the :mconfig:`non_exportable_tags` configuration option, |
| 60 |
introduced in Modules v5.7 together with the hook API, so it is dropped from |
| 61 |
the tag list persisted once a module is loaded, without affecting how it is |
| 62 |
reported beforehand, for instance on an :subcmd:`avail` listing. The |
| 63 |
``remote`` tag is also given its own abbreviation and color, so modules |
| 64 |
whose application directory has not been synced yet stand out on an |
| 65 |
:subcmd:`avail` or :subcmd:`spider` listing. These ``module config`` calls |
| 66 |
are set in |file etcdir_initrc|, evaluated once when the ``module`` shell |
| 67 |
function initializes with :subcmd:`autoinit`, the only context ``module |
| 68 |
config`` is usable from within a file evaluated by Modules itself: |
| 69 |
|
| 70 |
.. literalinclude:: ../../example/sync-remote-appdir/initrc |
| 71 |
:language: tcl |
| 72 |
:caption: initrc |
| 73 |
|
| 74 |
The sync itself is triggered by the hook procedure, defined and registered |
| 75 |
in :file:`siteconfig.tcl`: |
| 76 |
|
| 77 |
.. literalinclude:: ../../example/sync-remote-appdir/siteconfig.tcl |
| 78 |
:language: tcl |
| 79 |
:caption: siteconfig.tcl |
| 80 |
|
| 81 |
Since the hook fires before the modulefile itself is evaluated, and the sync |
| 82 |
is performed with a blocking ``exec`` call, ``module load`` waits for the |
| 83 |
copy to complete before the modulefile that relies on the now-local |
| 84 |
application directory gets evaluated. A hook procedure cannot abort the |
| 85 |
evaluation it wraps, so a sync failure is reported as an error but does not |
| 86 |
prevent the module from loading afterward, even though its application |
| 87 |
directory may still be missing locally in that case. |
| 88 |
|
| 89 |
**Compatible with Modules v5.7+** |
| 90 |
|
| 91 |
Installation |
| 92 |
------------ |
| 93 |
|
| 94 |
Create site-specific configuration directory if it does not exist yet: |
| 95 |
|
| 96 |
.. parsed-literal:: |
| 97 |
|
| 98 |
$ mkdir \ |etcdir| |
| 99 |
|
| 100 |
Copy the site-specific configuration script and initialization file of this |
| 101 |
recipe: |
| 102 |
|
| 103 |
.. parsed-literal:: |
| 104 |
|
| 105 |
$ cp example/sync-remote-appdir/siteconfig.tcl \ |etcdir|\ / |
| 106 |
$ cp example/sync-remote-appdir/initrc \ |etcdir|\ / |
| 107 |
|
| 108 |
.. note:: |
| 109 |
|
| 110 |
Defined location for the site-specific configuration script may vary from |
| 111 |
one installation to another. To determine the expected location for this |
| 112 |
file on your setup, check the value of the ``siteconfig`` configuration |
| 113 |
option: |
| 114 |
|
| 115 |
.. parsed-literal:: |
| 116 |
|
| 117 |
:ps:`$` module config siteconfig |
| 118 |
|
| 119 |
Adapt :file:`modulefiles/.modulerc` to your modulepath, and copy it at its |
| 120 |
root, next to the modulefiles it applies to. Finally, create |
| 121 |
:file:`/remote_apps/.module_appdir_map` on the network share, with one |
| 122 |
``<module_name_and_version> <software_install_directory_basename>`` entry |
| 123 |
per line for each application directory that should be synced this way. |
| 124 |
|
| 125 |
Usage example |
| 126 |
------------- |
| 127 |
|
| 128 |
The application directory of ``foo/2.1`` has not been synced locally yet, so |
| 129 |
it shows up tagged ``remote`` on an :subcmd:`avail` listing, using the |
| 130 |
abbreviation and color configured for this tag: |
| 131 |
|
| 132 |
.. parsed-literal:: |
| 133 |
|
| 134 |
:ps:`$` module avail foo |
| 135 |
--------------- :sgrdi:`/path/to/modulefiles` --------------- |
| 136 |
foo/2.1 <R> |
| 137 |
|
| 138 |
Key: |
| 139 |
<module-tag> <R>=remote |
| 140 |
|
| 141 |
Loading it triggers the sync, then proceeds with the load once the copy |
| 142 |
completes: |
| 143 |
|
| 144 |
.. parsed-literal:: |
| 145 |
|
| 146 |
:ps:`$` module load -v foo/2.1 |
| 147 |
Syncing 'foo-2.1-build3' application directory from remote share... |
| 148 |
Loading :sgrhi:`foo/2.1` |
| 149 |
|
| 150 |
Once loaded, the module no longer carries the ``remote`` tag, since its |
| 151 |
application directory now lives on local disk: |
| 152 |
|
| 153 |
.. parsed-literal:: |
| 154 |
|
| 155 |
:ps:`$` module list |
| 156 |
Currently Loaded Modulefiles: |
| 157 |
1) foo/2.1 |
| 158 |
|
| 159 |
A later load, after the module has been unloaded, finds the |
| 160 |
:file:`.foo-2.1-build3.synced` marker file and skips the sync entirely, |
| 161 |
and the module no longer shows up tagged ``remote`` on :subcmd:`avail` |
| 162 |
either: |
| 163 |
|
| 164 |
.. parsed-literal:: |
| 165 |
|
| 166 |
:ps:`$` module unload foo/2.1 |
| 167 |
:ps:`$` module avail foo |
| 168 |
--------------- :sgrdi:`/path/to/modulefiles` --------------- |
| 169 |
foo/2.1 |
| 170 |
:ps:`$` module load -v foo/2.1 |
| 171 |
Loading :sgrhi:`foo/2.1` |
| 172 |
|
| 173 |
.. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: |