Debian Codegraph @

README

raw
1
2 .:. METALOG .:.
3
4
5
6 ------------------------ BLURB ------------------------
7
8
9 Metalog is a modern replacement for syslogd and klogd. The logged messages
10 can be dispatched according to their facility, urgency, program name and/or
11 Perl-compatible regular expressions. Log files can be automatically rotated
12 when they exceed a certain size or age. External shell scripts (e.g., mail)
13 can be launched when specific patterns are found.
14
15 Logs are written according to RFC3164, RFC5423 or a minimal format with
16 optional severity levels.
17
18 It can send its logs to remote syslog servers via UDP. It can gather info from
19 the kernel or /dev/log as well as from several Unix Domain Sockets. This allows
20 logging from chrooted or containerized applications.
21
22 Metalog is simple to configure, accepts unlimited number of rules and has
23 (switchable) memory bufferization for maximal performance.
24
25 http://metalog.sourceforge.net/
26
27 ------------------------ COMPILATION ------------------------
28
29
30 In order to compile Metalog, you need to have the PCRE2 library. It's used
31 to process Perl-compatible regular expressions.
32
33 libpcre2 comes with many distributions, but you can also download it from:
34 https://www.pcre.org/
35
36 In case you want to use the optional native gzip compression for rotated
37 log files you need to have zlib. It's very likely, that this it's already
38 installed, but you can also download it from:
39 https://www.zlib.net/
40
41 Then, compiling and installing Metalab isn't rocket science:
42
43 ./autogen.sh
44 ./configure
45 make
46 make install
47
48
49 ------------------------ CONFIGURATION ------------------------
50
51
52 A configuration file should be installed. Its default location is
53 /etc/metalog.conf (unless you tweaked --with-sysconfdir). You can find a
54 sample file in this directory, but it's certainly not perfect for your system
55 and your needs. So read on.
56
57 A configuration file has the following syntax:
58
59 [default values]
60
61 <section title>:
62 <section values>
63 ...
64
65 <section title 2>:
66 <section 2 values>
67 ...
68
69 [...]
70
71 To make it clear, here's an example:
72
73 maxsize = 100000
74 maxtime = 86400
75 maxfiles = 5
76
77 Kernel messages:
78 facility = "kern"
79 logdir = "/var/log/kernel"
80
81 Crond:
82 program = "crond"
83 logdir = "/var/log/crond"
84
85 A section defines several things:
86
87 - A title (useless for the software, it's just to make your configuration
88 file look better).
89
90 - Filters: you can define facilities, program names, urgency levels and
91 regular expressions. An incoming message will pass through all filters. If
92 all conditions are matching, actions defined for the section are performed.
93
94 - Actions: they are taken only when all previous conditions are met. Only
95 two actions are currently possible: write the message to a log file, and/or
96 launch an external script.
97
98 Here's a list of values that can be independently assigned for each section:
99
100 * minimum = <level>: only record a message if its urgency is inferior or
101 equal to <level>. Level '0' is the most critical one, while level '7' is for
102 debugging messages. 'minimum = 5' will strip all non-important messages. The
103 default minimum level is 7 (ie. keep all messages).
104
105 Example: record only critical messages to /var/log/important:
106
107 Critical messages:
108 facility = "*"
109 minimum = 1
110 logdir = "/var/log/important"
111
112 * maximum = <level>: don't log if the message level exceeds that value. By
113 default, maximum is the largest possible level.
114
115 * facility = <facility>: only record a message if the application that issued
116 it uses syslog facility <facility>.
117
118 * break = 0|1 (default=0): if set to 1 and a section is matched,
119 perform action, but don't consider any more possible section matches
120 below this one in the config file. This is useful for creating a config
121 where specific types of log messages are matched and dispatched, while a
122 "catch-all" section at the bottom of the config file handles the default
123 case. Using break=0 would cause messages to be potentially handled by
124 multiple sections, causing message duplication, while using break=1
125 (default) on the sections above the "catch-all" would avoid duplication.
126
127 Facility names are: "auth", "authpriv", "cron", "daemon", "ftp", "kern",
128 "lpr", "mail", "news", "security", "syslog", "user", "uucp", "local0",
129 "local1" ... "local7". All kernel messages are logged with facility "kern".
130
131 A section can have several "facility = ..." lines to match more than one
132 facility. If <facility> is "*", it'll match all the facilities.
133
134 Example: record all authentication messages to /var/log/auth:
135
136 Authentication messages:
137 facility = "auth"
138 facility = "authpriv"
139 logdir = "/var/log/auth"
140
141 * program = <program>: only record messages signed by a specific daemon or
142 program.
143
144 You can use this as a modern alternative to syslog facilities: use "*" as a
145 facility, and set a program name with that directive.
146
147 Kernel messages can be caught with 'program = "kernel"'.
148
149 Example: record Pure-FTPd messages in a directory, and in.ftpd messages in
150 another directory (although the facility is the same: FTP):
151
152 In.FTPd messages:
153 facility = "*"
154 program = "in.ftpd"
155 logdir = "/var/log/in.ftpd"
156
157 Pure-FTPd messages:
158 facility = "*"
159 program = "pure-ftpd"
160 logdir = "/var/log/pure-ftpd"
161
162 * program_regex = <regex>: similar to program, but not just a string equal
163 match, but a regular expression.
164
165 Example: this would match program fields with [postfix/smtp], [postfix/smtpd]
166 and [postfix/cleanup] etc.
167
168 All Postfix messages:
169 facility = "*"
170 program_regex = "postfix"
171 logdir = /var/log/postfix
172
173 * program_neg_regex = <regex>: log programs that do _not_ match the regex.
174
175 * regex = <regex>: a message must match the regular expression to pass that
176 filter. Multiple regexes are allowed for a single section. All Perl
177 extensions are allowed, and matching is case insensitive.
178
179 Example: record all authentication failures to /var/log/pwdfail:
180
181 Password failures:
182 regex = "(password|login|authentication)\s+(fail|invalid)"
183 regex = "(failed|invalid)\s+(password|login|authentication)"
184 regex = "ILLEGAL ROOT LOGIN"
185 logdir = "/var/log/pwdfail"
186
187 * neg_regex = <regex>: this is the opposite of the previous directive.
188 Logging will occur if the regex doesn't match. It can be useful to remove the
189 crap, and it can be freely mixed with "regex" directives. "regex" and
190 "neg_regex" directives are scanned in order.
191
192 Example:
193 facility = "mail"
194 neg_regex= "starting daemon"
195 logdir = "/var/log/mail"
196
197 * socket = <path to socketfile>: metalog will create a socket file on the
198 given path and listen on this Unix Domain Socket. That way applications
199 in a chroot jail can still send log messages that will be received by
200 metalog. Such a socket could also get bind mounted in LXC containers and
201 overmount /dev/log in there. Within the container then is no longer a own
202 logger necessary.
203 This option can be set in the default part of the config or within a section.
204 When used within a section it can we use as a selector for a specific log
205 file instead of flags like "program", "facility", "regex" and so on.
206
207 * maxsize = <size in bytes>: automatically rotate log files when their size
208 have exceeded that size. Messages are never truncated and no message can be
209 lost during a rotation. Every section can have a different maxsize value.
210
211 * maxfiles = <number of files>: how many files to keep after rotation. This
212 parameter can be set independently for each section. It defaults to 5.
213
214 * maxtime = <age in seconds>: automatically rotate log files when they are
215 older than this number of seconds. Every section can have a different
216 maxtime value. maxsize and maxtime can be combined together, so that rotation
217 occurs when any of these conditions occurs.
218
219 * compress = 0|1: With this active and when compiled with zlib support the
220 rotated log files will get compress with gzip.
221
222 * compress_delay = <number of files>: This will leave the amount of the
223 latest rotated log files uncompressed.
224
225 Example: rotate log files daily or when they are more than 1,000,000 bytes
226 long, keep only 3 history files with the first and second uncompressed and
227 the third one compressed:
228
229 maxsize = 1000000
230 maxtime = 86400
231 maxfiles = 3
232 compress = 1
233 compress_delay = 2
234
235 * log_format = <format>: The log entries can be written in different formats:
236 "legacy", "legacy_timestamp", "rfc3164" and "rfc5424".
237 The legacy ones are quite similar to the format defined in obsolete RFC3164.
238 legacy_timestamp is the default format.
239
240 * log_severity = <number>: In case a legacy format is used, this will
241 introduce the severity info to a log entry. In both RFC formats this is
242 already included and can't get disabled.
243
244 * showrepeats = 0|1: disable printing the "last message repeated N times"
245 summary messages. When showrepeats is set to 1, messages will be printed
246 directly to the log, without summarizing them based on number received (which
247 can cause problems for some log monitoring programs, as well as delaying
248 repeated messages if another unique message isn't received promptyly).
249 Defaults to 0 (i.e. repeated messages will be summarized)
250
251 * logdir = <directory>: record messages in the specified directory. If the
252 directory doesn't exist, it will be automatically created when the first
253 matching message will be logged (the parent directory has to exist, though).
254
255 * perms = <mode> permissions for the log directory. Defaults to 0700
256
257 Example: Let those in the group with the GID of the process read the log.
258 Don't forget to run metalog as the group.
259
260 perms = 0770
261
262 * command = <path/to/command>: run a program or a shell-script when all
263 conditions are met. This directive is not incompatible with logdir: a
264 message can be both logged and passed to an external command. When the
265 command is launched, the first argument is filled with the date the message
266 was received, the second argument is the program name, and the last one is
267 the text of the message itself. Environment variables aren't cleared.
268
269 Example: send a mail to root when authentication failures occur:
270
271 Mail password failures:
272 regex = "(password|login|authentication)\s+(fail|invalid)"
273 regex = "(failed|invalid)\s+(password|login|authentication)"
274 regex = "ILLEGAL ROOT LOGIN"
275 command = "/usr/local/sbin/pwdfail.sh"
276
277 "pwdfail.sh" can be a simple shell script like this one:
278
279 #! /bin/sh
280 echo "$3" | mail -s "Password failure (program: $2)" root
281
282 Don't forget to properly quote arguments to avoid security problems.
283
284 * postrotate_cmd = <path/to/script>: run a script after rotating
285 If specified, the postrotate_cmd is run after log-files are rotated.
286 The following parameters are provided, environments are not cleared:
287 date: date of the last message received
288 prog: program name of the last message received
289 file: file name with path of the rotated log-file
290
291 Example: compress rotated log files with bzip2
292
293 postrotate_cmd = /usr/local/bin/compress.sh
294
295 "/usr/local/bin/compress.sh" provides bzip2 compression:
296 #!/bin/sh
297 #*
298 #* Copyright (c) 2006 by Lukas Ruf ([email protected]),
299 #* Computer Engineering and Networks Laboratory (TIK),
300 #* Swiss Federal Institute of Technology (ETH) Zurich
301 #*
302
303 DATE="$1"
304 PRG="$2"
305 FILE="$3"
306
307 # tests to run bzip2
308 # parameter file is provided
309 # file exists
310 # file has a size greater than zero
311 if [ -n "${FILE}" -a -f "${FILE}" -a -s "${FILE}" ]; then
312 bzip2 -9 ${FILE}
313 fi
314
315 * configdir = <path to directory>: This optional directory can contain
316 more config files that will get interpreted at metalog start. Only file
317 names ending with ".conf" are interpreted. This option is only allowed once,
318 all repetitions will be ignored. The content of the additional config files
319 will be appended to the existing config in alphanumeric order of the file
320 names, to get a determined behaviour of metalog. The config files could
321 contain sections for specific services.
322
323 Example: Content of /etc/metalog.d/:
324
325 00_kernel.conf
326 10_metalog.conf
327 11_crond.conf
328 20_application_xy.conf
329 21_apache.conf
330
331
332 --------- CONFIGURATION: REMOTE SYSLOG SERVERS ------------
333
334
335 metalog can send log messages to syslog server(s) via UDP.
336
337 * remote_log = 0|1: Log messages will be sent to a remote syslog server.
338 If defined in the default settings section, all log files will be sent.
339 It can also be defined in the seperate sections, so remote logging can
340 get activated/deactivated for every single section.
341
342 * remote_host = <IP address or domain name>: This is the remote syslog
343 server.
344
345 * remote_port = <number>: This is the UDP the syslog server listens for
346 new log messages.
347
348 * remote_format = <format>: It makes sense to set the format to "rfc3164" or
349 "rfc5424", so the host name of the sender will be included in the message.
350
351 * remote_severity_level = <severity level>: This defines which log messages
352 will be sent to the remote syslog servers. It filters the less important
353 messages and so reduces the amount of messages sent.
354
355 Example: send all logs to a remote server:
356
357 remote_host = 10.1.0.1
358 remote_port = 514
359 remote_format = rfc5424
360 remote_severity_level = 3
361
362 This will send all log messages also to the remote syslog server, that have
363 a severity level of 3 [ERROR] or a higher (that have a lower number).
364
365 If several syslog servers should receive the logs, the above remote_*
366 commands can be replaced with remote_*[<name>], where <name> is a
367 alphanumeric value.
368
369 Example: This will send the logs to two servers: one called "main" and the
370 other one "backup".
371
372 remote_log[main] = 1
373 remote_host[main] = 10.1.0.1
374 remote_port[main] = 514
375 remote_format[main] = rfc5424
376 remote_severity_level[main] = 3
377
378 remote_log[backup] = 1
379 remote_host[backup] = 192.168.1.10
380 remote_port[backup] = 514
381 remote_format[backup] = rfc5424
382 remote_severity_level[backup] = 3
383
384
385 ------------------------ LOG FILES ------------------------
386
387
388 With the "logdir" directive, messages from a specific section are recorded
389 in a directory. In that directory, the following files are created:
390
391 - "current": this file contains the latest recorded messages. It can be
392 incomplete (ie. data is being written to the file when you are reading it).
393
394 - ".timestamp": the creation date of "current". Needed for the "maxtime"
395 directive.
396
397 - "log-<year>-<month>-<day>-<hour>:<minutes>:<seconds>": old logs,
398 chronologically sorted.
399
400 If you ever delete these files by hand for some obscure reasons, it's not a
401 bad idea to restart the daemon (and even better: to stop it before you mess
402 the directories).
403
404
405 ------------------------ RUNNING METALOG ------------------------
406
407
408 Kill "klogd" and "syslogd" first. Don't run Metalog while they are running.
409 Multiple programs listening for the same source of data is a silly idea.
410
411 Once these programs are killed (and you are *sure* they are killed), check
412 that /etc/metalog.conf is installed, and simply run:
413
414 /usr/local/sbin/metalog &
415
416 On GNU/Linux systems, two processes are spawned: one is the "MASTER", doing
417 much of the work. The other one (named "KERNEL") is needed for Linux and
418 replaces the "klogd" daemon. It is responsible for logging kernel messages.
419
420 By default, messages are immediately recorded into log files, and the I/O
421 cache is flushed to be sure that you don't loose any message if a fatal crash
422 or a power outage occurs.
423
424 But if you prefer speed, Metalog can work asynchronously, using the --async
425 switch. In this mode, to avoid disk I/O, and unlike traditional syslog
426 daemons, Metalog works in memory buffers, then flushes the data to disk by
427 blocks. It improves a lot overall performances.
428
429 If you temporarely want to switch to the asynchronous mode, send an USR2
430 signal to the process. Something like:
431
432 kill -USR2 $(cat /var/run/metalog.pid)
433
434 should do the trick.
435
436 Later, if you want to watch activity in real-time (like a good old "tail -f"
437 on a log file), you can disable buffering. Just send an USR1 signal to the
438 "MASTER" process. You can always re-enable buffering afterwards.
439
440
441 ------------------------ RUN-TIME OPTIONS ------------------------
442
443
444 Metalog accepts some run-time options:
445
446 - '-a' or '--async': improve performance by using buffers (but log files
447 won't get updated in real time).
448
449 - '-B' or '--daemonize': have the server start in background
450 (daemonization).
451
452 - '-c <xxx>' or '--consolelevel=<xxx>': set the console log level on
453 Linux. Valid values are from 0 to 7. The default is 7.
454
455 - '-C <configuration file>' or '--configfile=<configuration file>': use an
456 alternative configuration file.
457
458 - '-g <group>' or '--group=<group>': change the GID of the metalog process.
459 Created files will be owned by this group.
460
461 - '-h' or '--help': show help and version number.
462
463 - '-p <filename>' or '--pidfile=<filename>': set the name of the file
464 that will hold the PID number. It defaults to /var/run/metalog.pid
465
466 - '-S <pathname>' or '--socket=<pathname>': set the path that will
467 name the main syslog socket. It defaults to /dev/log
468
469 - '-s' or '--sync': start in synchronous mode, with no bufferization.
470
471 - '-t' or '--test-config': Exit after parsing the config file(s).
472 Returns 0 if config is ok.
473
474
475
476
477 -Frank DENIS "Jedi/Sector One" <[email protected]> .