Debian Codegraph @

doc/source/cookbook/log-module-commands.rst

raw
1 .. _log-module-command:
2
3 Log module command
4 ==================
5
6 It is sometimes desired to better understand the ``module`` usage on the
7 system we manage. Especially to determine what are the modulefiles most used
8 and what are those never used that could be removed. This recipe describes a
9 way to track the ``module`` usage by logging each request made.
10
11
12 Implementation
13 --------------
14
15 Logging module commands and modulefile evaluations relies on two configuration
16 options introduced in version 5.5:
17
18 * :mconfig:`logger`, the command run to transmit messages to the log system
19 * :mconfig:`logged_events`, list of module event to log
20
21 :mconfig:`logger` option relies on the `logger`_ command by default, which is
22 usually available and already installed on most systems.
23
24 Following events are recognized by :mconfig:`logged_events` option:
25
26 * ``requested_cmd``: record module commands typed by users
27 * ``requested_eval``: record modulefile evaluations requested by users
28 * ``auto_eval``: record modulefile evaluations automatically triggered
29
30 Note that some messages can also be sent during modulefile evaluation by using
31 the ``log`` channel of the :mfcmd:`puts` command.
32
33 **Compatible with Modules v5.5+**
34
35 .. _logger: https://man7.org/linux/man-pages/man1/logger.1.html
36
37 Configuration
38 -------------
39
40 Verify the currently defined :mconfig:`logger` command points to an installed
41 and valid tool:
42
43 .. parsed-literal::
44
45 :ps:`$` module config logger
46 Modules Release 5.5.0 (2024-11-11)
47
48 - Config. name ---------.- Value (set by if default overridden) ---------------
49 logger /usr/bin/logger -t modules
50
51 The :mconfig:`logged_events` option is empty by default, so no content is sent
52 to logs by default.
53
54 Configure :mconfig:`logged_events` to record for instance the
55 ``requested_cmd`` and ``requested_eval`` events:
56
57 .. parsed-literal::
58
59 :ps:`$` module config logged_events +requested_cmd:requested_eval
60
61 With this setup, all module commands and modulefile evaluations directly asked
62 by users will be recorded into the log system. Automatically triggered
63 evaluations will not be recorded.
64
65 Usage example
66 -------------
67
68 Listing available modulefiles, then loading a bare modulefile (that sends
69 itself a message to logs), then another one with a dependency and purging
70 everything in the end:
71
72 .. parsed-literal::
73
74 :ps:`$` cat /path/to/modulefiles/foo/1.0
75 #%Module
76 puts log {some message sent to log}
77 :ps:`$` ml av
78 --------------- :sgrdi:`/path/to/modulefiles` ---------------
79 bar/1.0 foo/1.0 qux/1.0
80 :ps:`$` ml foo/1.0
81 :ps:`$` module load bar
82 Loading :sgrhi:`bar/1.0`
83 :sgrin:`Loading requirement`: qux/1.0
84 :ps:`$` module purge
85
86 Log entries can then be retrieved from system log files:
87
88 .. parsed-literal::
89
90 :ps:`$` journalctl -q -t modules
91 Apr 29 07:47:42 hostname modules[3777797]: user="username" command="avail" arguments=""
92 Apr 29 07:47:51 hostname modules[3777835]: user="username" command="ml" arguments="foo/1.0"
93 Apr 29 07:47:51 hostname modules[3777835]: some message sent to log
94 Apr 29 07:47:51 hostname modules[3777835]: user="username" mode="load" module="foo/1.0" specified="foo/1.0" modulefile="/path/to/modulefiles/foo/1.0" requested="1"
95 Apr 29 07:48:10 hostname modules[3777876]: user="username" command="load" arguments="bar"
96 Apr 29 07:48:10 hostname modules[3777876]: user="username" mode="load" module="bar/1.0" specified="bar" modulefile="/path/to/modulefiles/bar/1.0" requested="1"
97 Apr 29 07:48:17 hostname modules[3777914]: user="username" command="purge" arguments=""
98 Apr 29 07:48:17 hostname modules[3777914]: user="username" mode="unload" module="bar/1.0" specified="bar/1.0" modulefile="/path/to/modulefiles/bar/1.0" requested="1"
99 Apr 29 07:48:17 hostname modules[3777914]: some message sent to log
100 Apr 29 07:48:17 hostname modules[3777914]: user="username" mode="unload" module="foo/1.0" specified="foo/1.0" modulefile="/path/to/modulefiles/foo/1.0" requested="1"
101
102 Implementation on older versions
103 --------------------------------
104
105 Logging module commands is implemented by the use of a site-specific
106 configuration that traces every call to a modulefile evaluation.
107
108 .. literalinclude:: ../../example/log-module-commands/siteconfig.tcl
109 :language: tcl
110 :caption: siteconfig.tcl
111 :lines: 26-54
112 :lineno-start: 26
113
114 This code defines a ``logModfileInterp`` procedure which is set to be
115 evaluated after each evaluation of the ``execute-modulefile`` procedure with
116 the `trace`_ Tcl command. Thanks to the ``trace`` mechanism
117 ``logModfileInterp`` receives the arguments passed to ``execute-modulefile``.
118
119 The ``logModfileInterp`` procedure applies filter to only log *load* and
120 *unload* modulefile evaluations. It may be extended to the other evaluation
121 modes (*help*, *display*, *test*, *whatis* and *refresh*) by adapting the
122 following line:
123
124 .. literalinclude:: ../../example/log-module-commands/siteconfig.tcl
125 :language: tcl
126 :lines: 36-37
127 :lineno-start: 36
128
129 In the proposed code, log entries are formatted as a JSON record which is
130 convenient to push these logs in a search and analytics engine like
131 `Elasticsearch`_ or `Splunk`_. Such tools help to globally monitor the whole
132 set of log entries produced from thousands of computing nodes.
133
134 .. literalinclude:: ../../example/log-module-commands/siteconfig.tcl
135 :language: tcl
136 :lines: 47-49
137 :lineno-start: 47
138
139 The :command:`logger` command is run to generate the log message. This is done
140 through a specific ``execLogger`` procedure ensuring that the current user
141 environment does not confuse :command:`logger` with unexpected version of the
142 libraries it requires.
143
144 .. literalinclude:: ../../example/log-module-commands/siteconfig.tcl
145 :language: tcl
146 :lines: 13-24
147 :lineno-start: 13
148
149 Example code also defines a ``logModuleCmd`` procedure which is set to be
150 evaluated after each evaluation of the ``module`` and the ``ml`` procedures
151 with `trace`_ Tcl command.
152
153 .. literalinclude:: ../../example/log-module-commands/siteconfig.tcl
154 :language: tcl
155 :caption: siteconfig.tcl
156 :lines: 56-76
157 :lineno-start: 56
158
159 .. note::
160
161 This code example may be extended to log for instance additional
162 information in each message. The `upvar`_ Tcl command may be used to
163 retrieve variables from the calling context. However beware that the
164 internal code of Modules may change, so if you rely on internal variables
165 please re-check the code in the :file:`siteconfig.tcl` file deployed after
166 each upgrade of Modules.
167
168 **Compatible with Modules v4.2 to v5.4**
169
170 .. _trace: https://www.tcl-lang.org/man/tcl8.5/TclCmd/trace.html
171 .. _Elasticsearch: https://www.elastic.co/elasticsearch/
172 .. _Splunk: https://www.splunk.com/
173 .. _upvar: https://www.tcl-lang.org/man/tcl8.5/TclCmd/upvar.html
174
175 Installation
176 ^^^^^^^^^^^^
177
178 Create site-specific configuration directory if it does not exist yet:
179
180 .. parsed-literal::
181
182 $ mkdir \ |etcdir|
183
184 Then copy there the site-specific configuration script of this recipe:
185
186 .. parsed-literal::
187
188 $ cp example/log-module-commands/siteconfig.tcl \ |etcdir|\ /
189
190 .. note::
191
192 Defined location for the site-specific configuration script may vary from
193 one installation to another. To determine the expected location for this
194 file on your setup, check the value of the ``siteconfig`` option on Modules
195 version 4.3 or above:
196
197 .. parsed-literal::
198
199 :ps:`$` module config siteconfig
200
201 On older version of Modules, check the ``modulecmd.tcl`` script:
202
203 .. parsed-literal::
204
205 :ps:`$` grep '^set g_siteconfig ' $MODULES_CMD
206
207 Usage example
208 ^^^^^^^^^^^^^
209
210 Listing available modulefiles, then loading a bare modulefile, then another
211 one with a dependency and purging everything in the end:
212
213 .. parsed-literal::
214
215 :ps:`$` ml av
216 --------------- :sgrdi:`/path/to/modulefiles` ---------------
217 bar/1.0 foo/1.0 qux/1.0
218 :ps:`$` ml foo
219 :ps:`$` module load bar
220 Loading :sgrhi:`bar/1.0`
221 :sgrin:`Loading requirement`: qux/1.0
222 :ps:`$` module purge
223
224 A log entry can then be retrieved from system log files:
225
226 .. parsed-literal::
227
228 :ps:`$` journalctl -q -t module
229 Sep 12 20:24:01 hostname module[9925]: { "user": "username", "cmd": "ml", "args": "av" }
230 Sep 12 20:24:02 hostname module[9925]: { "user": "username", "mode": "load", "module": "foo/1.0", "auto": false }
231 Sep 12 20:24:02 hostname module[9925]: { "user": "username", "cmd": "ml", "args": "foo" }
232 Sep 12 20:24:03 hostname module[9925]: { "user": "username", "mode": "load", "module": "qux/1.0", "auto": true }
233 Sep 12 20:24:03 hostname module[9925]: { "user": "username", "mode": "load", "module": "bar/1.0", "auto": false }
234 Sep 12 20:24:03 hostname module[9925]: { "user": "username", "cmd": "module", "args": "load bar" }
235 Sep 12 20:24:04 hostname module[9925]: { "user": "username", "mode": "unload", "module": "bar/1.0" }
236 Sep 12 20:24:04 hostname module[9925]: { "user": "username", "mode": "unload", "module": "qux/1.0" }
237 Sep 12 20:24:04 hostname module[9925]: { "user": "username", "mode": "unload", "module": "foo/1.0" }
238 Sep 12 20:24:04 hostname module[9925]: { "user": "username", "cmd": "module", "args": "purge" }
239
240 .. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: