| 1 |
.. _hide-and-forbid-modules: |
| 2 |
|
| 3 |
Hide and forbid modules |
| 4 |
======================= |
| 5 |
|
| 6 |
:command:`module` is often used to give access to hundreds of different |
| 7 |
software to a large variety of users, each of them provided in several |
| 8 |
versions. In such situation users ends up facing thousands of modulefiles |
| 9 |
which could be confusing. |
| 10 |
|
| 11 |
On the other side, site's staff needs to decommission older software version |
| 12 |
to ensure for instance that only the best matches for their hardware resources |
| 13 |
remain. Such work needs specific care to avoid breaking the computing workflow |
| 14 |
of users. |
| 15 |
|
| 16 |
This recipe provides examples on how to hide or/and forbid modulefiles to |
| 17 |
better cope with software life cycles and help both user and staff to get |
| 18 |
their bearings in large environments. |
| 19 |
|
| 20 |
Implementation |
| 21 |
-------------- |
| 22 |
|
| 23 |
The :mfcmd:`module-hide` and :mfcmd:`module-forbid` modulefile commands block |
| 24 |
respectively the visibility or the load evaluation of the modulefiles they |
| 25 |
target. These two commands have a variety of options to inhibit their action |
| 26 |
or display specific messages. Combining both commands enables to hide and |
| 27 |
forbid specified modulefiles. |
| 28 |
|
| 29 |
**Compatible with Modules v4.6+** |
| 30 |
|
| 31 |
.. note:: |
| 32 |
|
| 33 |
Modules v4.7+ is required to use the ``--hidden-loaded`` option of |
| 34 |
:mfcmd:`module-hide` command. |
| 35 |
|
| 36 |
Usage examples |
| 37 |
-------------- |
| 38 |
|
| 39 |
The use cases below describe situations where hiding or/and forbidding use of |
| 40 |
modulefiles can be helpful. |
| 41 |
|
| 42 |
The following set of modules will be used in the examples of this recipe. Each |
| 43 |
use case will progressively add :mfcmd:`module-hide` and |
| 44 |
:mfcmd:`module-forbid` statements in modulepath's :file:`.modulerc` file to |
| 45 |
improve adapt module visibility or forbid access to them. |
| 46 |
|
| 47 |
.. code-block:: console |
| 48 |
|
| 49 |
$ module avail |
| 50 |
-- ../example/hide-and-forbid-modules/modulefiles -- |
| 51 |
appA/1.0 bioappA/1.0 chemappA/1.0 deplibA/1.0 |
| 52 |
appA/2.0 bioappA/2.0 chemappA/2.0 deplibA/2.0 |
| 53 |
appB/1.0 bioappB/1.0 chemappB/1.0 deplibB/1.0 |
| 54 |
appB/2.0 bioappB/2.0 chemappB/2.0 deplibB/2.0 |
| 55 |
|
| 56 |
Limiting view to the useful software only |
| 57 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 58 |
|
| 59 |
With a large number of available modules it is interesting to reduce user's |
| 60 |
visibility to only report the modules of interest. |
| 61 |
|
| 62 |
Some modules may require other to be loaded as dependency. This is the case |
| 63 |
for ``appA`` and ``appB`` which depend on ``deplibA`` or ``deplibB``. These |
| 64 |
two dependency libraries are not interesting by themselves so they could be |
| 65 |
hidden by default but they should stay findable for ``appA`` or ``appB`` |
| 66 |
modules to load their dependency. |
| 67 |
|
| 68 |
:mfcmd:`module-hide --soft<module-hide>` could be used to address this need. |
| 69 |
Designated module will be hidden unless searched. Following lines are added to |
| 70 |
modulepath's :file:`.modulerc` file |
| 71 |
|
| 72 |
.. code-block:: tcl |
| 73 |
|
| 74 |
# hide modules only loaded as dependency |
| 75 |
module-hide --soft deplibA |
| 76 |
module-hide --soft --hidden-loaded deplibB |
| 77 |
|
| 78 |
As a result ``deplibA`` and ``deplibB`` are not returned anymore on a global |
| 79 |
:subcmd:`avail` sub-command:: |
| 80 |
|
| 81 |
$ module avail |
| 82 |
-- ../example/hide-and-forbid-modules/modulefiles -- |
| 83 |
appA/1.0 appB/2.0 bioappB/1.0 chemappA/2.0 |
| 84 |
appA/2.0 bioappA/1.0 bioappB/2.0 chemappB/1.0 |
| 85 |
appB/1.0 bioappA/2.0 chemappA/1.0 chemappB/2.0 |
| 86 |
|
| 87 |
Yet these modules are still found when loading the modules of the application |
| 88 |
requiring them:: |
| 89 |
|
| 90 |
$ module load appA |
| 91 |
Loading appA/2.0 |
| 92 |
Loading requirement: deplibA/2.0 |
| 93 |
|
| 94 |
In some cases it is desirable to hide such dependency modules also when they |
| 95 |
are loaded. This could be achieved by using the ``--hidden-loaded`` option of |
| 96 |
the :mfcmd:`module-hide` command. Hidden loaded modules do not appear by |
| 97 |
default on :subcmd:`list` sub-command output unless :option:`--all` is set. |
| 98 |
Queries like :subcmd:`is-loaded` still detect such modules as loaded even if |
| 99 |
hidden. In addition loading or unloading informational messages related to |
| 100 |
these modules are not reported unless a :envvar:`verbosity |
| 101 |
mode<MODULES_VERBOSITY>` higher than ``verbose`` is configured. |
| 102 |
|
| 103 |
.. code-block:: console |
| 104 |
|
| 105 |
$ module load appB |
| 106 |
$ module list |
| 107 |
Currently Loaded Modulefiles: |
| 108 |
1) appB/2.0 |
| 109 |
$ module list --all |
| 110 |
Currently Loaded Modulefiles: |
| 111 |
1) deplibB/2.0 2) appB/2.0 |
| 112 |
$ module is-loaded deplibB |
| 113 |
$ echo $? |
| 114 |
0 |
| 115 |
|
| 116 |
Going further, among scientific applications some are only useful for a given |
| 117 |
scientific field. A site may provide many software covering many scientific |
| 118 |
fields but a user may only be concerned by one of these fields. In our example |
| 119 |
software are provided for biology users (*bioappA* and *bioappB*) and other |
| 120 |
software for chemistry users (*chemappA* and *chemappB*). |
| 121 |
|
| 122 |
Say every biology users are part of a ``bio`` Unix group and every chemistry |
| 123 |
users are member of a ``chem`` Unix group. Available software visibility could |
| 124 |
be improved by only reporting the *bio* software to the *bio* users and the |
| 125 |
*chem* software to the *chem* users. |
| 126 |
|
| 127 |
.. code-block:: tcl |
| 128 |
|
| 129 |
# hide modules not from user's scientific field |
| 130 |
module-hide --soft --not-group bio bioappA bioappB |
| 131 |
module-hide --soft --not-group chem chemappA chemappB |
| 132 |
|
| 133 |
With the above statements put in modulepath's :file:`.modulerc` file, the |
| 134 |
*bio* software are not seen anymore by *chem* users:: |
| 135 |
|
| 136 |
$ module avail |
| 137 |
-- ../example/hide-and-forbid-modules/modulefiles -- |
| 138 |
appA/1.0 appB/1.0 chemappA/1.0 chemappB/1.0 |
| 139 |
appA/2.0 appB/2.0 chemappA/2.0 chemappB/2.0 |
| 140 |
|
| 141 |
However as they are softly hidden, these modules can still be seen if queried |
| 142 |
or if ``--all`` option is used:: |
| 143 |
|
| 144 |
$ module avail bioappA |
| 145 |
-- ../example/hide-and-forbid-modules/modulefiles -- |
| 146 |
bioappA/1.0 bioappA/2.0 |
| 147 |
$ module avail --all |
| 148 |
-- ../example/hide-and-forbid-modules/modulefiles -- |
| 149 |
appA/1.0 bioappA/1.0 chemappA/1.0 deplibA/1.0 |
| 150 |
appA/2.0 bioappA/2.0 chemappA/2.0 deplibA/2.0 |
| 151 |
appB/1.0 bioappB/1.0 chemappB/1.0 deplibB/1.0 |
| 152 |
appB/2.0 bioappB/2.0 chemappB/2.0 deplibB/2.0 |
| 153 |
|
| 154 |
Software requiring administrative actions prior usage |
| 155 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 156 |
|
| 157 |
Some software may require an administrative step to get allowed to use them, |
| 158 |
like the signature of a user agreement. Such software should be reported among |
| 159 |
the available modules but should not be used prior the administrative step |
| 160 |
achieved. |
| 161 |
|
| 162 |
In our example, the ``chemappA`` application requires a user charter to be |
| 163 |
signed and then sent to the site staff. The access to the ``chemappA`` is |
| 164 |
controlled by the ``chemappA`` Unix group: a user needs to be part of this |
| 165 |
group to access and use the application. |
| 166 |
|
| 167 |
.. code-block:: tcl |
| 168 |
|
| 169 |
# forbid use of modules unless software user agreement is signed |
| 170 |
set msg {User agreement for ChemAppA application must be validated to use it |
| 171 |
1. Please connect to https://ChemAppA.example.org |
| 172 |
2. Read the Term of Use and sign it |
| 173 |
3. Send back this signed agreement to our service desk} |
| 174 |
module-forbid --not-group chemappA --message $msg chemappA |
| 175 |
|
| 176 |
With the above :mfcmd:`module-forbid` statement put in modulepath's |
| 177 |
:file:`.modulerc` file, the software is still visible but its load is denied |
| 178 |
unless if the user is part of the ``chemappA`` group. |
| 179 |
|
| 180 |
.. code-block:: console |
| 181 |
|
| 182 |
$ module avail chemappA |
| 183 |
-- ../example/hide-and-forbid-modules/modulefiles -- |
| 184 |
chemappA/1.0 chemappA/2.0 |
| 185 |
$ module load chemappA |
| 186 |
ERROR: Access to module 'chemappA/2.0' is denied |
| 187 |
User agreement for ChemAppA application must be validated to use it |
| 188 |
1. Please connect to https://ChemAppA.example.org |
| 189 |
2. Read the Term of Use and sign it |
| 190 |
3. Send back this signed agreement to our service desk |
| 191 |
|
| 192 |
A specific message is provided through the ``--message`` option to guide the |
| 193 |
user to complete the required administrative step. Once group membership is |
| 194 |
acquired, the module can be seamlessly loaded. |
| 195 |
|
| 196 |
.. code-block:: console |
| 197 |
|
| 198 |
$ id --groups --name |
| 199 |
chemappA chem |
| 200 |
$ module load -v chemappA |
| 201 |
Loading chemappA/2.0 |
| 202 |
|
| 203 |
.. note:: |
| 204 |
|
| 205 |
Do not forget to protect the access to the directory where the software is |
| 206 |
installed to really ensure that only allowed users can use it. |
| 207 |
|
| 208 |
Software limited to particular users |
| 209 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 210 |
|
| 211 |
Some applications may be restricted to a limited set of users. For instance |
| 212 |
because such application should not be disclosed or because it requires a |
| 213 |
license that is paid only by a few users. |
| 214 |
|
| 215 |
In our example, the ``appC`` application works with token-based licenses. The |
| 216 |
only users that can use this software are those that have paid for a license |
| 217 |
token. Other users should not access nor even see the availability of this |
| 218 |
application. Users that have bought a license token are added to the ``appC`` |
| 219 |
Unix group. |
| 220 |
|
| 221 |
.. code-block:: tcl |
| 222 |
|
| 223 |
# fully hide and forbid modules unless user owns a license token |
| 224 |
module-hide --hard --not-group appC appC |
| 225 |
set msg {Access is restricted to owners of license token} |
| 226 |
module-forbid --not-group appC --message $msg appC |
| 227 |
|
| 228 |
The above statements have been added in modulepath's :file:`.modulerc` file. |
| 229 |
The :mfcmd:`module-hide --hard<module-hide>` command is used to completely |
| 230 |
remove visibility for non-authorized users. |
| 231 |
|
| 232 |
.. code-block:: console |
| 233 |
|
| 234 |
$ module avail |
| 235 |
-- ../example/hide-and-forbid-modules/modulefiles -- |
| 236 |
appA/1.0 appB/1.0 chemappA/1.0 chemappB/1.0 |
| 237 |
appA/2.0 appB/2.0 chemappA/2.0 chemappB/2.0 |
| 238 |
$ module avail appC |
| 239 |
$ module load appC |
| 240 |
ERROR: Unable to locate a modulefile for 'appC' |
| 241 |
|
| 242 |
The :mfcmd:`module-forbid` statement added for ``appC`` helps to get a clear |
| 243 |
error message for the non-authorized users that are aware of the existence of |
| 244 |
the module name and version (instead of getting a modulefile location error). |
| 245 |
|
| 246 |
.. code-block:: console |
| 247 |
|
| 248 |
$ module load appC/2.0 |
| 249 |
ERROR: Access to module 'appC/2.0' is denied |
| 250 |
Access is restricted to owners of license token |
| 251 |
|
| 252 |
Alternatively such restrictions on modulefiles can be achieved by adapting |
| 253 |
file permission mode instead of adding statements in modulepath's |
| 254 |
:file:`.modulerc` file. |
| 255 |
|
| 256 |
.. code-block:: console |
| 257 |
|
| 258 |
$ chmod 640 ../example/hide-and-forbid-modules/modulefiles/appC/* |
| 259 |
$ chgrp appC ../example/hide-and-forbid-modules/modulefiles/appC/* |
| 260 |
|
| 261 |
However restricting file permission mode does not enable to authorize several |
| 262 |
Unix groups or users to access those modulefiles or to have specific error |
| 263 |
messages unlike when :mfcmd:`module-hide` and :mfcmd:`module-forbid` commands |
| 264 |
are used. |
| 265 |
|
| 266 |
.. note:: |
| 267 |
|
| 268 |
Do not forget to protect the access to the directory where the software is |
| 269 |
installed to really ensure that only the authorized users can use it. |
| 270 |
|
| 271 |
Software life cycle |
| 272 |
^^^^^^^^^^^^^^^^^^^ |
| 273 |
|
| 274 |
When providing a new software version, it may be interesting to have a test |
| 275 |
phase, for instance during one week, to make this new version checked by some |
| 276 |
some pilot users prior the general availability. |
| 277 |
|
| 278 |
In our example, we are currently November 13th and ``chempappB/2.0`` has been |
| 279 |
installed two days ago. A test phase is ongoing and users that are member of |
| 280 |
the ``pilot`` Unix group can already access this software prior its general |
| 281 |
availability set for November 18th. |
| 282 |
|
| 283 |
.. code-block:: tcl |
| 284 |
|
| 285 |
# test new version of chemappB prior general availability |
| 286 |
module-hide --before 2020-11-18 --not-group pilot chemappB/2.0 |
| 287 |
|
| 288 |
With the above statement added to modulepath's :file:`.modulerc` file, common |
| 289 |
users will not see the new module until November 18th. |
| 290 |
|
| 291 |
.. code-block:: console |
| 292 |
|
| 293 |
$ date |
| 294 |
Fri 13 Nov 2020 02:04:21 PM CET |
| 295 |
$ module avail chemappB |
| 296 |
-- ../example/hide-and-forbid-modules/modulefiles -- |
| 297 |
chemappB/1.0 |
| 298 |
|
| 299 |
At some point software need to be decommissioned to ensure for instance that |
| 300 |
buggy or under-optimized versions are not used anymore by users. |
| 301 |
|
| 302 |
In our example, ``appA/1.0`` needs to be removed as a newer version is |
| 303 |
available for *appA* and this new version fixes a lot of issues that were |
| 304 |
encountered with version 1.0. Decommission date for ``appA/1.0`` is planned on |
| 305 |
November 18th. |
| 306 |
|
| 307 |
.. code-block:: none |
| 308 |
|
| 309 |
# decommission of old version of appA |
| 310 |
set nearmsg {appA/1.0 will be decommissioned, please use appA/2.0} |
| 311 |
set msg {appA/1.0 is decommissioned, please use appA/2.0} |
| 312 |
module-hide --hard --after 2020-11-18 --not-group eol appA/1.0 |
| 313 |
module-forbid --after 2020-11-18 --not-group eol \ |
| 314 |
--nearly-message $nearmsg --message $msg appA/1.0 |
| 315 |
|
| 316 |
The above statements added to modulepath's :file:`.modulerc` file will remove |
| 317 |
visibility and access to the decommissioned module on November 18th. Only |
| 318 |
users part of the ``eol`` Unix group will still keep an access to the software |
| 319 |
(which is useful in case some users are unfortunately stuck on this specific |
| 320 |
version of *appA*). Specific messages are set to guide users toward selecting |
| 321 |
another version for *appA*. |
| 322 |
|
| 323 |
.. code-block:: console |
| 324 |
|
| 325 |
$ module avail appA |
| 326 |
-- ../example/hide-and-forbid-modules/modulefiles -- |
| 327 |
appA/1.0 appA/2.0 |
| 328 |
$ module load appA/1.0 |
| 329 |
Loading appA/1.0 |
| 330 |
WARNING: Access to module will be denied starting '2020-11-18' |
| 331 |
appA/1.0 will be decommissioned, please use appA/2.0 |
| 332 |
Loading requirement: deplibA/1.0 |
| 333 |
|
| 334 |
Prior decommission date, module will still be visible and loadable. However |
| 335 |
when loading the module a warning message will appear when the expiry date |
| 336 |
will be close. The :envvar:`nearly_forbidden_days<MODULES_NEARLY_FORBIDDEN_DAYS>` |
| 337 |
module configuration defines starting when such warning message should be |
| 338 |
reported prior decommission date (14 days by default). |
| 339 |
|
| 340 |
.. code-block:: console |
| 341 |
|
| 342 |
$ module config nearly_forbidden_days |
| 343 |
Modules Release 4.6.0 (2020-09-16) |
| 344 |
|
| 345 |
- Config. name ---------.- Value (set by if default overridden) --------------- |
| 346 |
nearly_forbidden_days 14 |
| 347 |
|
| 348 |
Once the decommissioned date is over, ``appA/1.0`` has disappeared unless for |
| 349 |
users member of the ``eol`` Unix group. |
| 350 |
|
| 351 |
.. code-block:: console |
| 352 |
|
| 353 |
$ date |
| 354 |
Thu 19 Nov 2020 02:46:27 PM CET |
| 355 |
$ module avail appA |
| 356 |
-- ../example/hide-and-forbid-modules/modulefiles -- |
| 357 |
appA/2.0 |
| 358 |
$ module load appA/1.0 |
| 359 |
ERROR: Access to module 'appA/1.0' is denied |
| 360 |
appA/1.0 is decommissioned, please use appA/2.0 |
| 361 |
|
| 362 |
.. note:: |
| 363 |
|
| 364 |
Do not forget to protect the access to the directory where the software is |
| 365 |
installed prior its general availability and after its decommissioned date. |
| 366 |
|
| 367 |
.. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: |