Debian Codegraph @

doc/source/cookbook/sync-remote-appdir.rst

raw
1 .. _sync-remote-appdir:
2
3 Sync remote application directories on first load
4 =================================================
5
6 Some workstations and laptops mount a network share holding all kind of
7 software installations that may be used on this machine, alongside their own
8 local, faster disk. Reading application files directly from this network
9 share on every use works, but it is slower than reading them from local
10 disk, and it stops working entirely when the machine goes offline.
11
12 This recipe describes how to keep modulefiles pointing at application
13 directories on a network share mounted under :file:`/remote_apps`, while
14 transparently copying, with the `rsync`_ tool, the directory of a given
15 application to :file:`/local_apps` the first time the corresponding module
16 is loaded. Every later load of the same module finds its application
17 directory already synced locally and skips the copy.
18
19 .. _rsync: https://rsync.samba.org/
20
21 Implementation
22 --------------
23
24 A :file:`.module_appdir_map` file, located at the root of the
25 :file:`/remote_apps` network share, maps each module name and version to the
26 basename of its application directory under :file:`/remote_apps`:
27
28 .. literalinclude:: ../../example/sync-remote-appdir/.module_appdir_map
29 :caption: /remote_apps/.module_appdir_map
30
31 The :file:`.modulerc` file at the root of the modulepath reads this map file
32 and applies the ``remote`` tag, with the :mfcmd:`module-tag` modulefile
33 command, to every module listed in it whose application directory has not
34 been synced locally yet -- tracked by the presence of a
35 :file:`.<basename>.synced` marker file under :file:`/local_apps`:
36
37 .. literalinclude:: ../../example/sync-remote-appdir/modulefiles/.modulerc
38 :language: tcl
39 :caption: modulefiles/.modulerc
40
41 The actual sync is performed by a :mhook:`before-modulefile-eval` hook,
42 registered with the ``add-hook`` siteconfig command (see the :ref:`hook-api`
43 design notes and the *Hooks* section of :manpage:`module(1)` man page). The
44 hook procedure returns immediately if the evaluation mode is not ``load``,
45 or if the module being evaluated is not tagged ``remote`` -- checked with
46 :mfcmd:`module-info tags<module-info>`, run in the modulefile Tcl
47 interpreter reached through ``getCurrentModfileInterpName``, exactly as
48 already documented for a hook procedure that needs to run modulefile
49 commands. Otherwise, it looks up the application directory basename mapped
50 to the module in :file:`.module_appdir_map`, and skips the sync if this
51 directory was already marked as synced by a previous load. Otherwise it
52 copies the application directory with ``rsync`` and, on success, touches
53 the :file:`.<basename>.synced` marker file so later loads skip the copy.
54
55 Because the modulepath root :file:`.modulerc` tags a module ``remote`` ahead
56 of the sync actually happening, this tag would otherwise still be recorded
57 as applying to the module once loaded, which is misleading once its
58 application directory has just been synced locally. The ``remote`` tag is
59 therefore added to the :mconfig:`non_exportable_tags` configuration option,
60 introduced in Modules v5.7 together with the hook API, so it is dropped from
61 the tag list persisted once a module is loaded, without affecting how it is
62 reported beforehand, for instance on an :subcmd:`avail` listing. The
63 ``remote`` tag is also given its own abbreviation and color, so modules
64 whose application directory has not been synced yet stand out on an
65 :subcmd:`avail` or :subcmd:`spider` listing. These ``module config`` calls
66 are set in |file etcdir_initrc|, evaluated once when the ``module`` shell
67 function initializes with :subcmd:`autoinit`, the only context ``module
68 config`` is usable from within a file evaluated by Modules itself:
69
70 .. literalinclude:: ../../example/sync-remote-appdir/initrc
71 :language: tcl
72 :caption: initrc
73
74 The sync itself is triggered by the hook procedure, defined and registered
75 in :file:`siteconfig.tcl`:
76
77 .. literalinclude:: ../../example/sync-remote-appdir/siteconfig.tcl
78 :language: tcl
79 :caption: siteconfig.tcl
80
81 Since the hook fires before the modulefile itself is evaluated, and the sync
82 is performed with a blocking ``exec`` call, ``module load`` waits for the
83 copy to complete before the modulefile that relies on the now-local
84 application directory gets evaluated. A hook procedure cannot abort the
85 evaluation it wraps, so a sync failure is reported as an error but does not
86 prevent the module from loading afterward, even though its application
87 directory may still be missing locally in that case.
88
89 **Compatible with Modules v5.7+**
90
91 Installation
92 ------------
93
94 Create site-specific configuration directory if it does not exist yet:
95
96 .. parsed-literal::
97
98 $ mkdir \ |etcdir|
99
100 Copy the site-specific configuration script and initialization file of this
101 recipe:
102
103 .. parsed-literal::
104
105 $ cp example/sync-remote-appdir/siteconfig.tcl \ |etcdir|\ /
106 $ cp example/sync-remote-appdir/initrc \ |etcdir|\ /
107
108 .. note::
109
110 Defined location for the site-specific configuration script may vary from
111 one installation to another. To determine the expected location for this
112 file on your setup, check the value of the ``siteconfig`` configuration
113 option:
114
115 .. parsed-literal::
116
117 :ps:`$` module config siteconfig
118
119 Adapt :file:`modulefiles/.modulerc` to your modulepath, and copy it at its
120 root, next to the modulefiles it applies to. Finally, create
121 :file:`/remote_apps/.module_appdir_map` on the network share, with one
122 ``<module_name_and_version> <software_install_directory_basename>`` entry
123 per line for each application directory that should be synced this way.
124
125 Usage example
126 -------------
127
128 The application directory of ``foo/2.1`` has not been synced locally yet, so
129 it shows up tagged ``remote`` on an :subcmd:`avail` listing, using the
130 abbreviation and color configured for this tag:
131
132 .. parsed-literal::
133
134 :ps:`$` module avail foo
135 --------------- :sgrdi:`/path/to/modulefiles` ---------------
136 foo/2.1 <R>
137
138 Key:
139 <module-tag> <R>=remote
140
141 Loading it triggers the sync, then proceeds with the load once the copy
142 completes:
143
144 .. parsed-literal::
145
146 :ps:`$` module load -v foo/2.1
147 Syncing 'foo-2.1-build3' application directory from remote share...
148 Loading :sgrhi:`foo/2.1`
149
150 Once loaded, the module no longer carries the ``remote`` tag, since its
151 application directory now lives on local disk:
152
153 .. parsed-literal::
154
155 :ps:`$` module list
156 Currently Loaded Modulefiles:
157 1) foo/2.1
158
159 A later load, after the module has been unloaded, finds the
160 :file:`.foo-2.1-build3.synced` marker file and skips the sync entirely,
161 and the module no longer shows up tagged ``remote`` on :subcmd:`avail`
162 either:
163
164 .. parsed-literal::
165
166 :ps:`$` module unload foo/2.1
167 :ps:`$` module avail foo
168 --------------- :sgrdi:`/path/to/modulefiles` ---------------
169 foo/2.1
170 :ps:`$` module load -v foo/2.1
171 Loading :sgrhi:`foo/2.1`
172
173 .. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: