Debian Codegraph @

Doc/howto/logging.rst

raw
1 .. _logging-howto:
2
3 =============
4 Logging HOWTO
5 =============
6
7 :Author: Vinay Sajip <vinay_sajip at red-dove dot com>
8
9 .. _logging-basic-tutorial:
10
11 .. currentmodule:: logging
12
13 This page contains tutorial information. For links to reference information and a
14 logging cookbook, please see :ref:`tutorial-ref-links`.
15
16 Basic Logging Tutorial
17 ----------------------
18
19 Logging is a means of tracking events that happen when some software runs. The
20 software's developer adds logging calls to their code to indicate that certain
21 events have occurred. An event is described by a descriptive message which can
22 optionally contain variable data (i.e. data that is potentially different for
23 each occurrence of the event). Events also have an importance which the
24 developer ascribes to the event; the importance can also be called the *level*
25 or *severity*.
26
27 When to use logging
28 ^^^^^^^^^^^^^^^^^^^
29
30 You can access logging functionality by creating a logger via ``logger =
31 logging.getLogger(__name__)``, and then calling the logger's :meth:`~Logger.debug`,
32 :meth:`~Logger.info`, :meth:`~Logger.warning`, :meth:`~Logger.error` and
33 :meth:`~Logger.critical` methods. To determine when to use logging, and to see
34 which logger methods to use when, see the table below. It states, for each of a
35 set of common tasks, the best tool to use for that task.
36
37 +-------------------------------------+--------------------------------------+
38 | Task you want to perform | The best tool for the task |
39 +=====================================+======================================+
40 | Display console output for ordinary | :func:`print` |
41 | usage of a command line script or | |
42 | program | |
43 +-------------------------------------+--------------------------------------+
44 | Report events that occur during | A logger's :meth:`~Logger.info` (or |
45 | normal operation of a program (e.g. | :meth:`~Logger.debug` method for very|
46 | for status monitoring or fault | detailed output for diagnostic |
47 | investigation) | purposes) |
48 +-------------------------------------+--------------------------------------+
49 | Issue a warning regarding a | :func:`warnings.warn` in library |
50 | particular runtime event | code if the issue is avoidable and |
51 | | the client application should be |
52 | | modified to eliminate the warning |
53 | | |
54 | | A logger's :meth:`~Logger.warning` |
55 | | method if there is nothing the client|
56 | | application can do about the |
57 | | situation, but the event should still|
58 | | be noted |
59 +-------------------------------------+--------------------------------------+
60 | Report an error regarding a | Raise an exception |
61 | particular runtime event | |
62 +-------------------------------------+--------------------------------------+
63 | Report suppression of an error | A logger's :meth:`~Logger.error`, |
64 | without raising an exception (e.g. | :meth:`~Logger.exception` or |
65 | error handler in a long-running | :meth:`~Logger.critical` method as |
66 | server process) | appropriate for the specific error |
67 | | and application domain |
68 +-------------------------------------+--------------------------------------+
69
70 The logger methods are named after the level or severity of the events
71 they are used to track. The standard levels and their applicability are
72 described below (in increasing order of severity):
73
74 .. tabularcolumns:: |l|L|
75
76 +--------------+---------------------------------------------+
77 | Level | When it's used |
78 +==============+=============================================+
79 | ``DEBUG`` | Detailed information, typically of interest |
80 | | only when diagnosing problems. |
81 +--------------+---------------------------------------------+
82 | ``INFO`` | Confirmation that things are working as |
83 | | expected. |
84 +--------------+---------------------------------------------+
85 | ``WARNING`` | An indication that something unexpected |
86 | | happened, or indicative of some problem in |
87 | | the near future (e.g. 'disk space low'). |
88 | | The software is still working as expected. |
89 +--------------+---------------------------------------------+
90 | ``ERROR`` | Due to a more serious problem, the software |
91 | | has not been able to perform some function. |
92 +--------------+---------------------------------------------+
93 | ``CRITICAL`` | A serious error, indicating that the program|
94 | | itself may be unable to continue running. |
95 +--------------+---------------------------------------------+
96
97 The default level is ``WARNING``, which means that only events of this severity and higher
98 will be tracked, unless the logging package is configured to do otherwise.
99
100 Events that are tracked can be handled in different ways. The simplest way of
101 handling tracked events is to print them to the console. Another common way
102 is to write them to a disk file.
103
104
105 .. _howto-minimal-example:
106
107 A simple example
108 ^^^^^^^^^^^^^^^^
109
110 A very simple example is::
111
112 import logging
113 logging.warning('Watch out!') # will print a message to the console
114 logging.info('I told you so') # will not print anything
115
116 If you type these lines into a script and run it, you'll see:
117
118 .. code-block:: none
119
120 WARNING:root:Watch out!
121
122 printed out on the console. The ``INFO`` message doesn't appear because the
123 default level is ``WARNING``. The printed message includes the indication of the
124 level and the description of the event provided in the logging call, i.e.
125 'Watch out!'. The actual output can be formatted quite flexibly if you need
126 that; formatting options will also be explained later.
127
128 Notice that in this example, we use functions directly on the ``logging``
129 module, like ``logging.debug``, rather than creating a logger and calling
130 functions on it. These functions operate on the root logger, but can be useful
131 as they will call :func:`~logging.basicConfig` for you if it has not been called yet, like in
132 this example. In larger programs you'll usually want to control the logging
133 configuration explicitly however - so for that reason as well as others, it's
134 better to create loggers and call their methods.
135
136 Logging to a file
137 ^^^^^^^^^^^^^^^^^
138
139 A very common situation is that of recording logging events in a file, so let's
140 look at that next. Be sure to try the following in a newly started Python
141 interpreter, and don't just continue from the session described above::
142
143 import logging
144 logger = logging.getLogger(__name__)
145 logging.basicConfig(filename='example.log', encoding='utf-8', level=logging.DEBUG)
146 logger.debug('This message should go to the log file')
147 logger.info('So should this')
148 logger.warning('And this, too')
149 logger.error('And non-ASCII stuff, too, like Øresund and Malmö')
150
151 .. versionchanged:: 3.9
152 The *encoding* argument was added. In earlier Python versions, or if not
153 specified, the encoding used is the default value used by :func:`open`. While
154 not shown in the above example, an *errors* argument can also now be passed,
155 which determines how encoding errors are handled. For available values and
156 the default, see the documentation for :func:`open`.
157
158 And now if we open the file and look at what we have, we should find the log
159 messages:
160
161 .. code-block:: none
162
163 DEBUG:__main__:This message should go to the log file
164 INFO:__main__:So should this
165 WARNING:__main__:And this, too
166 ERROR:__main__:And non-ASCII stuff, too, like Øresund and Malmö
167
168 This example also shows how you can set the logging level which acts as the
169 threshold for tracking. In this case, because we set the threshold to
170 ``DEBUG``, all of the messages were printed.
171
172 If you want to set the logging level from a command-line option such as:
173
174 .. code-block:: none
175
176 --log=INFO
177
178 and you have the value of the parameter passed for ``--log`` in some variable
179 *loglevel*, you can use::
180
181 getattr(logging, loglevel.upper())
182
183 to get the value which you'll pass to :func:`basicConfig` via the *level*
184 argument. You may want to error check any user input value, perhaps as in the
185 following example::
186
187 # assuming loglevel is bound to the string value obtained from the
188 # command line argument. Convert to upper case to allow the user to
189 # specify --log=DEBUG or --log=debug
190 numeric_level = getattr(logging, loglevel.upper(), None)
191 if not isinstance(numeric_level, int):
192 raise ValueError('Invalid log level: %s' % loglevel)
193 logging.basicConfig(level=numeric_level, ...)
194
195 The call to :func:`basicConfig` should come *before* any calls to a logger's
196 methods such as :meth:`~Logger.debug`, :meth:`~Logger.info`, etc. Otherwise,
197 that logging event may not be handled in the desired manner.
198
199 If you run the above script several times, the messages from successive runs
200 are appended to the file *example.log*. If you want each run to start afresh,
201 not remembering the messages from earlier runs, you can specify the *filemode*
202 argument, by changing the call in the above example to::
203
204 logging.basicConfig(filename='example.log', filemode='w', level=logging.DEBUG)
205
206 The output will be the same as before, but the log file is no longer appended
207 to, so the messages from earlier runs are lost.
208
209
210 Logging variable data
211 ^^^^^^^^^^^^^^^^^^^^^
212
213 To log variable data, use a format string for the event description message and
214 append the variable data as arguments. For example::
215
216 import logging
217 logging.warning('%s before you %s', 'Look', 'leap!')
218
219 will display:
220
221 .. code-block:: none
222
223 WARNING:root:Look before you leap!
224
225 As you can see, merging of variable data into the event description message
226 uses the old, %-style of string formatting. This is for backwards
227 compatibility: the logging package pre-dates newer formatting options such as
228 :meth:`str.format` and :class:`string.Template`. These newer formatting
229 options *are* supported, but exploring them is outside the scope of this
230 tutorial: see :ref:`formatting-styles` for more information.
231
232
233 Changing the format of displayed messages
234 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
235
236 To change the format which is used to display messages, you need to
237 specify the format you want to use::
238
239 import logging
240 logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
241 logging.debug('This message should appear on the console')
242 logging.info('So should this')
243 logging.warning('And this, too')
244
245 which would print:
246
247 .. code-block:: none
248
249 DEBUG:This message should appear on the console
250 INFO:So should this
251 WARNING:And this, too
252
253 Notice that the 'root' which appeared in earlier examples has disappeared. For
254 a full set of things that can appear in format strings, you can refer to the
255 documentation for :ref:`logrecord-attributes`, but for simple usage, you just
256 need the *levelname* (severity), *message* (event description, including
257 variable data) and perhaps to display when the event occurred. This is
258 described in the next section.
259
260
261 Displaying the date/time in messages
262 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
263
264 To display the date and time of an event, you would place '%(asctime)s' in
265 your format string::
266
267 import logging
268 logging.basicConfig(format='%(asctime)s %(message)s')
269 logging.warning('is when this event was logged.')
270
271 which should print something like this:
272
273 .. code-block:: none
274
275 2010-12-12 11:41:42,612 is when this event was logged.
276
277 The default format for date/time display (shown above) is like ISO8601 or
278 :rfc:`3339`. If you need more control over the formatting of the date/time, provide
279 a *datefmt* argument to ``basicConfig``, as in this example::
280
281 import logging
282 logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%m/%d/%Y %I:%M:%S %p')
283 logging.warning('is when this event was logged.')
284
285 which would display something like this:
286
287 .. code-block:: none
288
289 12/12/2010 11:46:36 AM is when this event was logged.
290
291 The format of the *datefmt* argument is the same as supported by
292 :func:`time.strftime`.
293
294
295 Next Steps
296 ^^^^^^^^^^
297
298 That concludes the basic tutorial. It should be enough to get you up and
299 running with logging. There's a lot more that the logging package offers, but
300 to get the best out of it, you'll need to invest a little more of your time in
301 reading the following sections. If you're ready for that, grab some of your
302 favourite beverage and carry on.
303
304 If your logging needs are simple, then use the above examples to incorporate
305 logging into your own scripts, and if you run into problems or don't understand
306 something, please post a question in the Help category of the `Python
307 discussion forum <https://discuss.python.org/c/help/7>`_ and you should receive
308 help before too long.
309
310 Still here? You can carry on reading the next few sections, which provide a
311 slightly more advanced/in-depth tutorial than the basic one above. After that,
312 you can take a look at the :ref:`logging-cookbook`.
313
314 .. _logging-advanced-tutorial:
315
316
317 Advanced Logging Tutorial
318 -------------------------
319
320 The logging library takes a modular approach and offers several categories
321 of components: loggers, handlers, filters, and formatters.
322
323 * Loggers expose the interface that application code directly uses.
324 * Handlers send the log records (created by loggers) to the appropriate
325 destination.
326 * Filters provide a finer grained facility for determining which log records
327 to output.
328 * Formatters specify the layout of log records in the final output.
329
330 Log event information is passed between loggers, handlers, filters and
331 formatters in a :class:`LogRecord` instance.
332
333 Logging is performed by calling methods on instances of the :class:`Logger`
334 class (hereafter called :dfn:`loggers`). Each instance has a name, and they are
335 conceptually arranged in a namespace hierarchy using dots (periods) as
336 separators. For example, a logger named 'scan' is the parent of loggers
337 'scan.text', 'scan.html' and 'scan.pdf'. Logger names can be anything you want,
338 and indicate the area of an application in which a logged message originates.
339
340 A good convention to use when naming loggers is to use a module-level logger,
341 in each module which uses logging, named as follows::
342
343 logger = logging.getLogger(__name__)
344
345 This means that logger names track the package/module hierarchy, and it's
346 intuitively obvious where events are logged just from the logger name.
347
348 The root of the hierarchy of loggers is called the root logger. That's the
349 logger used by the functions :func:`debug`, :func:`info`, :func:`warning`,
350 :func:`error` and :func:`critical`, which just call the same-named method of
351 the root logger. The functions and the methods have the same signatures. The
352 root logger's name is printed as 'root' in the logged output.
353
354 It is, of course, possible to log messages to different destinations. Support
355 is included in the package for writing log messages to files, HTTP GET/POST
356 locations, email via SMTP, generic sockets, queues, or OS-specific logging
357 mechanisms such as syslog or the Windows NT event log. Destinations are served
358 by :dfn:`handler` classes. You can create your own log destination class if
359 you have special requirements not met by any of the built-in handler classes.
360
361 By default, no destination is set for any logging messages. You can specify
362 a destination (such as console or file) by using :func:`basicConfig` as in the
363 tutorial examples. If you call the functions :func:`debug`, :func:`info`,
364 :func:`warning`, :func:`error` and :func:`critical`, they will check to see
365 if no destination is set; and if one is not set, they will set a destination
366 of the console (``sys.stderr``) and a default format for the displayed
367 message before delegating to the root logger to do the actual message output.
368
369 The default format set by :func:`basicConfig` for messages is:
370
371 .. code-block:: none
372
373 severity:logger name:message
374
375 You can change this by passing a format string to :func:`basicConfig` with the
376 *format* keyword argument. For all options regarding how a format string is
377 constructed, see :ref:`formatter-objects`.
378
379 Logging Flow
380 ^^^^^^^^^^^^
381
382 The flow of log event information in loggers and handlers is illustrated in the
383 following diagram.
384
385 .. only:: not html
386
387 .. image:: logging_flow.*
388
389 .. raw:: html
390 :file: logging_flow.svg
391
392 .. raw:: html
393
394 <script>
395 /*
396 * This snippet is needed to handle the case where a light or dark theme is
397 * chosen via the theme is selected in the page. We call the existing handler
398 * and then add a dark-theme class to the body when the dark theme is selected.
399 * The SVG styling (above) then does the rest.
400 *
401 * If the pydoc theme is updated to set the dark-theme class, this snippet
402 * won't be needed any more.
403 */
404 (function() {
405 var oldActivateTheme = activateTheme;
406
407 function updateBody(theme) {
408 let elem = document.body;
409
410 elem.classList.remove('dark-theme');
411 elem.classList.remove('light-theme');
412 if (theme === 'dark') {
413 elem.classList.add('dark-theme');
414 }
415 else if (theme === 'light') {
416 elem.classList.add('light-theme');
417 }
418 }
419
420 activateTheme = function(theme) {
421 oldActivateTheme(theme);
422 updateBody(theme);
423 };
424 /*
425 * If the page is refreshed, make sure we update the body - the overriding
426 * of activateTheme won't have taken effect yet.
427 */
428 updateBody(localStorage.getItem('currentTheme') || 'auto');
429 })();
430 </script>
431
432 Loggers
433 ^^^^^^^
434
435 :class:`Logger` objects have a threefold job. First, they expose several
436 methods to application code so that applications can log messages at runtime.
437 Second, logger objects determine which log messages to act upon based upon
438 severity (the default filtering facility) or filter objects. Third, logger
439 objects pass along relevant log messages to all interested log handlers.
440
441 The most widely used methods on logger objects fall into two categories:
442 configuration and message sending.
443
444 These are the most common configuration methods:
445
446 * :meth:`Logger.setLevel` specifies the lowest-severity log message a logger
447 will handle, where debug is the lowest built-in severity level and critical
448 is the highest built-in severity. For example, if the severity level is
449 INFO, the logger will handle only INFO, WARNING, ERROR, and CRITICAL messages
450 and will ignore DEBUG messages.
451
452 * :meth:`Logger.addHandler` and :meth:`Logger.removeHandler` add and remove
453 handler objects from the logger object. Handlers are covered in more detail
454 in :ref:`handler-basic`.
455
456 * :meth:`Logger.addFilter` and :meth:`Logger.removeFilter` add and remove filter
457 objects from the logger object. Filters are covered in more detail in
458 :ref:`filter`.
459
460 You don't need to always call these methods on every logger you create. See the
461 last two paragraphs in this section.
462
463 With the logger object configured, the following methods create log messages:
464
465 * :meth:`Logger.debug`, :meth:`Logger.info`, :meth:`Logger.warning`,
466 :meth:`Logger.error`, and :meth:`Logger.critical` all create log records with
467 a message and a level that corresponds to their respective method names. The
468 message is actually a format string, which may contain the standard string
469 substitution syntax of ``%s``, ``%d``, ``%f``, and so on. The
470 rest of their arguments is a list of objects that correspond with the
471 substitution fields in the message. With regard to ``**kwargs``, the
472 logging methods care only about a keyword of ``exc_info`` and use it to
473 determine whether to log exception information.
474
475 * :meth:`Logger.exception` creates a log message similar to
476 :meth:`Logger.error`. The difference is that :meth:`Logger.exception` dumps a
477 stack trace along with it. Call this method only from an exception handler.
478
479 * :meth:`Logger.log` takes a log level as an explicit argument. This is a
480 little more verbose for logging messages than using the log level convenience
481 methods listed above, but this is how to log at custom log levels.
482
483 :func:`getLogger` returns a reference to a logger instance with the specified
484 name if it is provided, or ``root`` if not. The names are period-separated
485 hierarchical structures. Multiple calls to :func:`getLogger` with the same name
486 will return a reference to the same logger object. Loggers that are further
487 down in the hierarchical list are children of loggers higher up in the list.
488 For example, given a logger with a name of ``foo``, loggers with names of
489 ``foo.bar``, ``foo.bar.baz``, and ``foo.bam`` are all descendants of ``foo``.
490
491 Loggers have a concept of *effective level*. If a level is not explicitly set
492 on a logger, the level of its parent is used instead as its effective level.
493 If the parent has no explicit level set, *its* parent is examined, and so on -
494 all ancestors are searched until an explicitly set level is found. The root
495 logger always has an explicit level set (``WARNING`` by default). When deciding
496 whether to process an event, the effective level of the logger is used to
497 determine whether the event is passed to the logger's handlers.
498
499 Child loggers propagate messages up to the handlers associated with their
500 ancestor loggers. Because of this, it is unnecessary to define and configure
501 handlers for all the loggers an application uses. It is sufficient to
502 configure handlers for a top-level logger and create child loggers as needed.
503 (You can, however, turn off propagation by setting the *propagate*
504 attribute of a logger to ``False``.)
505
506
507 .. _handler-basic:
508
509 Handlers
510 ^^^^^^^^
511
512 :class:`~logging.Handler` objects are responsible for dispatching the
513 appropriate log messages (based on the log messages' severity) to the handler's
514 specified destination. :class:`Logger` objects can add zero or more handler
515 objects to themselves with an :meth:`~Logger.addHandler` method. As an example
516 scenario, an application may want to send all log messages to a log file, all
517 log messages of error or higher to stdout, and all messages of critical to an
518 email address. This scenario requires three individual handlers where each
519 handler is responsible for sending messages of a specific severity to a specific
520 location.
521
522 The standard library includes quite a few handler types (see
523 :ref:`useful-handlers`); the tutorials use mainly :class:`StreamHandler` and
524 :class:`FileHandler` in its examples.
525
526 There are very few methods in a handler for application developers to concern
527 themselves with. The only handler methods that seem relevant for application
528 developers who are using the built-in handler objects (that is, not creating
529 custom handlers) are the following configuration methods:
530
531 * The :meth:`~Handler.setLevel` method, just as in logger objects, specifies the
532 lowest severity that will be dispatched to the appropriate destination. Why
533 are there two :meth:`~Handler.setLevel` methods? The level set in the logger
534 determines which severity of messages it will pass to its handlers. The level
535 set in each handler determines which messages that handler will send on.
536
537 * :meth:`~Handler.setFormatter` selects a Formatter object for this handler to
538 use.
539
540 * :meth:`~Handler.addFilter` and :meth:`~Handler.removeFilter` respectively
541 configure and deconfigure filter objects on handlers.
542
543 Application code should not directly instantiate and use instances of
544 :class:`Handler`. Instead, the :class:`Handler` class is a base class that
545 defines the interface that all handlers should have and establishes some
546 default behavior that child classes can use (or override).
547
548
549 Formatters
550 ^^^^^^^^^^
551
552 Formatter objects configure the final order, structure, and contents of the log
553 message. Unlike the base :class:`logging.Handler` class, application code may
554 instantiate formatter classes, although you could likely subclass the formatter
555 if your application needs special behavior. The constructor takes three
556 optional arguments -- a message format string, a date format string and a style
557 indicator.
558
559 .. method:: logging.Formatter.__init__(fmt=None, datefmt=None, style='%')
560
561 If there is no message format string, the default is to use the
562 raw message. If there is no date format string, the default date format is:
563
564 .. code-block:: none
565
566 %Y-%m-%d %H:%M:%S
567
568 with the milliseconds tacked on at the end. The ``style`` is one of ``'%'``,
569 ``'{'``, or ``'$'``. If one of these is not specified, then ``'%'`` will be used.
570
571 If the ``style`` is ``'%'``, the message format string uses
572 ``%(<dictionary key>)s`` styled string substitution; the possible keys are
573 documented in :ref:`logrecord-attributes`. If the style is ``'{'``, the message
574 format string is assumed to be compatible with :meth:`str.format` (using
575 keyword arguments), while if the style is ``'$'`` then the message format string
576 should conform to what is expected by :meth:`string.Template.substitute`.
577
578 .. versionchanged:: 3.2
579 Added the ``style`` parameter.
580
581 The following message format string will log the time in a human-readable
582 format, the severity of the message, and the contents of the message, in that
583 order::
584
585 '%(asctime)s - %(levelname)s - %(message)s'
586
587 Formatters use a user-configurable function to convert the creation time of a
588 record to a tuple. By default, :func:`time.localtime` is used; to change this
589 for a particular formatter instance, set the ``converter`` attribute of the
590 instance to a function with the same signature as :func:`time.localtime` or
591 :func:`time.gmtime`. To change it for all formatters, for example if you want
592 all logging times to be shown in GMT, set the ``converter`` attribute in the
593 Formatter class (to ``time.gmtime`` for GMT display).
594
595
596 Configuring Logging
597 ^^^^^^^^^^^^^^^^^^^
598
599 .. currentmodule:: logging.config
600
601 Programmers can configure logging in three ways:
602
603 1. Creating loggers, handlers, and formatters explicitly using Python
604 code that calls the configuration methods listed above.
605 2. Creating a logging config file and reading it using the :func:`fileConfig`
606 function.
607 3. Creating a dictionary of configuration information and passing it
608 to the :func:`dictConfig` function.
609
610 For the reference documentation on the last two options, see
611 :ref:`logging-config-api`. The following example configures a very simple
612 logger, a console handler, and a simple formatter using Python code::
613
614 import logging
615
616 # create logger
617 logger = logging.getLogger('simple_example')
618 logger.setLevel(logging.DEBUG)
619
620 # create console handler and set level to debug
621 ch = logging.StreamHandler()
622 ch.setLevel(logging.DEBUG)
623
624 # create formatter
625 formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
626
627 # add formatter to ch
628 ch.setFormatter(formatter)
629
630 # add ch to logger
631 logger.addHandler(ch)
632
633 # 'application' code
634 logger.debug('debug message')
635 logger.info('info message')
636 logger.warning('warn message')
637 logger.error('error message')
638 logger.critical('critical message')
639
640 Running this module from the command line produces the following output:
641
642 .. code-block:: shell-session
643
644 $ python simple_logging_module.py
645 2005-03-19 15:10:26,618 - simple_example - DEBUG - debug message
646 2005-03-19 15:10:26,620 - simple_example - INFO - info message
647 2005-03-19 15:10:26,695 - simple_example - WARNING - warn message
648 2005-03-19 15:10:26,697 - simple_example - ERROR - error message
649 2005-03-19 15:10:26,773 - simple_example - CRITICAL - critical message
650
651 The following Python module creates a logger, handler, and formatter nearly
652 identical to those in the example listed above, with the only difference being
653 the names of the objects::
654
655 import logging
656 import logging.config
657
658 logging.config.fileConfig('logging.conf')
659
660 # create logger
661 logger = logging.getLogger('simpleExample')
662
663 # 'application' code
664 logger.debug('debug message')
665 logger.info('info message')
666 logger.warning('warn message')
667 logger.error('error message')
668 logger.critical('critical message')
669
670 Here is the logging.conf file:
671
672 .. code-block:: ini
673
674 [loggers]
675 keys=root,simpleExample
676
677 [handlers]
678 keys=consoleHandler
679
680 [formatters]
681 keys=simpleFormatter
682
683 [logger_root]
684 level=DEBUG
685 handlers=consoleHandler
686
687 [logger_simpleExample]
688 level=DEBUG
689 handlers=consoleHandler
690 qualname=simpleExample
691 propagate=0
692
693 [handler_consoleHandler]
694 class=StreamHandler
695 level=DEBUG
696 formatter=simpleFormatter
697 args=(sys.stdout,)
698
699 [formatter_simpleFormatter]
700 format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
701
702 The output is nearly identical to that of the non-config-file-based example:
703
704 .. code-block:: shell-session
705
706 $ python simple_logging_config.py
707 2005-03-19 15:38:55,977 - simpleExample - DEBUG - debug message
708 2005-03-19 15:38:55,979 - simpleExample - INFO - info message
709 2005-03-19 15:38:56,054 - simpleExample - WARNING - warn message
710 2005-03-19 15:38:56,055 - simpleExample - ERROR - error message
711 2005-03-19 15:38:56,130 - simpleExample - CRITICAL - critical message
712
713 You can see that the config file approach has a few advantages over the Python
714 code approach, mainly separation of configuration and code and the ability of
715 noncoders to easily modify the logging properties.
716
717 .. warning:: The :func:`fileConfig` function takes a default parameter,
718 ``disable_existing_loggers``, which defaults to ``True`` for reasons of
719 backward compatibility. This may or may not be what you want, since it
720 will cause any non-root loggers existing before the :func:`fileConfig`
721 call to be disabled unless they (or an ancestor) are explicitly named in
722 the configuration. Please refer to the reference documentation for more
723 information, and specify ``False`` for this parameter if you wish.
724
725 The dictionary passed to :func:`dictConfig` can also specify a Boolean
726 value with key ``disable_existing_loggers``, which if not specified
727 explicitly in the dictionary also defaults to being interpreted as
728 ``True``. This leads to the logger-disabling behaviour described above,
729 which may not be what you want - in which case, provide the key
730 explicitly with a value of ``False``.
731
732
733 .. currentmodule:: logging
734
735 Note that the class names referenced in config files need to be either relative
736 to the logging module, or absolute values which can be resolved using normal
737 import mechanisms. Thus, you could use either
738 :class:`~logging.handlers.WatchedFileHandler` (relative to the logging module) or
739 ``mypackage.mymodule.MyHandler`` (for a class defined in package ``mypackage``
740 and module ``mymodule``, where ``mypackage`` is available on the Python import
741 path).
742
743 In Python 3.2, a new means of configuring logging has been introduced, using
744 dictionaries to hold configuration information. This provides a superset of the
745 functionality of the config-file-based approach outlined above, and is the
746 recommended configuration method for new applications and deployments. Because
747 a Python dictionary is used to hold configuration information, and since you
748 can populate that dictionary using different means, you have more options for
749 configuration. For example, you can use a configuration file in JSON format,
750 or, if you have access to YAML processing functionality, a file in YAML
751 format, to populate the configuration dictionary. Or, of course, you can
752 construct the dictionary in Python code, receive it in pickled form over a
753 socket, or use whatever approach makes sense for your application.
754
755 Here's an example of the same configuration as above, in YAML format for
756 the new dictionary-based approach:
757
758 .. code-block:: yaml
759
760 version: 1
761 formatters:
762 simple:
763 format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
764 handlers:
765 console:
766 class: logging.StreamHandler
767 level: DEBUG
768 formatter: simple
769 stream: ext://sys.stdout
770 loggers:
771 simpleExample:
772 level: DEBUG
773 handlers: [console]
774 propagate: no
775 root:
776 level: DEBUG
777 handlers: [console]
778
779 For more information about logging using a dictionary, see
780 :ref:`logging-config-api`.
781
782 What happens if no configuration is provided
783 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
784
785 If no logging configuration is provided, it is possible to have a situation
786 where a logging event needs to be output, but no handlers can be found to
787 output the event.
788
789 The event is output using a 'handler of last resort', stored in
790 :data:`lastResort`. This internal handler is not associated with any
791 logger, and acts like a :class:`~logging.StreamHandler` which writes the
792 event description message to the current value of ``sys.stderr`` (therefore
793 respecting any redirections which may be in effect). No formatting is
794 done on the message - just the bare event description message is printed.
795 The handler's level is set to ``WARNING``, so all events at this and
796 greater severities will be output.
797
798 .. versionchanged:: 3.2
799
800 For versions of Python prior to 3.2, the behaviour is as follows:
801
802 * If :data:`raiseExceptions` is ``False`` (production mode), the event is
803 silently dropped.
804
805 * If :data:`raiseExceptions` is ``True`` (development mode), a message
806 'No handlers could be found for logger X.Y.Z' is printed once.
807
808 To obtain the pre-3.2 behaviour,
809 :data:`lastResort` can be set to ``None``.
810
811 .. _library-config:
812
813 Configuring Logging for a Library
814 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
815
816 When developing a library which uses logging, you should take care to
817 document how the library uses logging - for example, the names of loggers
818 used. Some consideration also needs to be given to its logging configuration.
819 If the using application does not use logging, and library code makes logging
820 calls, then (as described in the previous section) events of severity
821 ``WARNING`` and greater will be printed to ``sys.stderr``. This is regarded as
822 the best default behaviour.
823
824 If for some reason you *don't* want these messages printed in the absence of
825 any logging configuration, you can attach a do-nothing handler to the top-level
826 logger for your library. This avoids the message being printed, since a handler
827 will always be found for the library's events: it just doesn't produce any
828 output. If the library user configures logging for application use, presumably
829 that configuration will add some handlers, and if levels are suitably
830 configured then logging calls made in library code will send output to those
831 handlers, as normal.
832
833 A do-nothing handler is included in the logging package:
834 :class:`~logging.NullHandler` (since Python 3.1). An instance of this handler
835 could be added to the top-level logger of the logging namespace used by the
836 library (*if* you want to prevent your library's logged events being output to
837 ``sys.stderr`` in the absence of logging configuration). If all logging by a
838 library *foo* is done using loggers with names matching 'foo.x', 'foo.x.y',
839 etc. then the code::
840
841 import logging
842 logging.getLogger('foo').addHandler(logging.NullHandler())
843
844 should have the desired effect. If an organisation produces a number of
845 libraries, then the logger name specified can be 'orgname.foo' rather than
846 just 'foo'.
847
848 .. note:: It is strongly advised that you *do not log to the root logger*
849 in your library. Instead, use a logger with a unique and easily
850 identifiable name, such as the ``__name__`` for your library's top-level package
851 or module. Logging to the root logger will make it difficult or impossible for
852 the application developer to configure the logging verbosity or handlers of
853 your library as they wish.
854
855 .. note:: It is strongly advised that you *do not add any handlers other
856 than* :class:`~logging.NullHandler` *to your library's loggers*. This is
857 because the configuration of handlers is the prerogative of the application
858 developer who uses your library. The application developer knows their
859 target audience and what handlers are most appropriate for their
860 application: if you add handlers 'under the hood', you might well interfere
861 with their ability to carry out unit tests and deliver logs which suit their
862 requirements.
863
864
865 Logging Levels
866 --------------
867
868 The numeric values of logging levels are given in the following table. These are
869 primarily of interest if you want to define your own levels, and need them to
870 have specific values relative to the predefined levels. If you define a level
871 with the same numeric value, it overwrites the predefined value; the predefined
872 name is lost.
873
874 +--------------+---------------+
875 | Level | Numeric value |
876 +==============+===============+
877 | ``CRITICAL`` | 50 |
878 +--------------+---------------+
879 | ``ERROR`` | 40 |
880 +--------------+---------------+
881 | ``WARNING`` | 30 |
882 +--------------+---------------+
883 | ``INFO`` | 20 |
884 +--------------+---------------+
885 | ``DEBUG`` | 10 |
886 +--------------+---------------+
887 | ``NOTSET`` | 0 |
888 +--------------+---------------+
889
890 Levels can also be associated with loggers, being set either by the developer or
891 through loading a saved logging configuration. When a logging method is called
892 on a logger, the logger compares its own level with the level associated with
893 the method call. If the logger's level is higher than the method call's, no
894 logging message is actually generated. This is the basic mechanism controlling
895 the verbosity of logging output.
896
897 Logging messages are encoded as instances of the :class:`~logging.LogRecord`
898 class. When a logger decides to actually log an event, a
899 :class:`~logging.LogRecord` instance is created from the logging message.
900
901 Logging messages are subjected to a dispatch mechanism through the use of
902 :dfn:`handlers`, which are instances of subclasses of the :class:`Handler`
903 class. Handlers are responsible for ensuring that a logged message (in the form
904 of a :class:`LogRecord`) ends up in a particular location (or set of locations)
905 which is useful for the target audience for that message (such as end users,
906 support desk staff, system administrators, developers). Handlers are passed
907 :class:`LogRecord` instances intended for particular destinations. Each logger
908 can have zero, one or more handlers associated with it (via the
909 :meth:`~Logger.addHandler` method of :class:`Logger`). In addition to any
910 handlers directly associated with a logger, *all handlers associated with all
911 ancestors of the logger* are called to dispatch the message (unless the
912 *propagate* flag for a logger is set to a false value, at which point the
913 passing to ancestor handlers stops).
914
915 Just as for loggers, handlers can have levels associated with them. A handler's
916 level acts as a filter in the same way as a logger's level does. If a handler
917 decides to actually dispatch an event, the :meth:`~Handler.emit` method is used
918 to send the message to its destination. Most user-defined subclasses of
919 :class:`Handler` will need to override this :meth:`~Handler.emit`.
920
921 .. _custom-levels:
922
923 Custom Levels
924 ^^^^^^^^^^^^^
925
926 Defining your own levels is possible, but should not be necessary, as the
927 existing levels have been chosen on the basis of practical experience.
928 However, if you are convinced that you need custom levels, great care should
929 be exercised when doing this, and it is possibly *a very bad idea to define
930 custom levels if you are developing a library*. That's because if multiple
931 library authors all define their own custom levels, there is a chance that
932 the logging output from such multiple libraries used together will be
933 difficult for the using developer to control and/or interpret, because a
934 given numeric value might mean different things for different libraries.
935
936 .. _useful-handlers:
937
938 Useful Handlers
939 ---------------
940
941 In addition to the base :class:`Handler` class, many useful subclasses are
942 provided:
943
944 #. :class:`StreamHandler` instances send messages to streams (file-like
945 objects).
946
947 #. :class:`FileHandler` instances send messages to disk files.
948
949 #. :class:`~handlers.BaseRotatingHandler` is the base class for handlers that
950 rotate log files at a certain point. It is not meant to be instantiated
951 directly. Instead, use :class:`~handlers.RotatingFileHandler` or
952 :class:`~handlers.TimedRotatingFileHandler`.
953
954 #. :class:`~handlers.RotatingFileHandler` instances send messages to disk
955 files, with support for maximum log file sizes and log file rotation.
956
957 #. :class:`~handlers.TimedRotatingFileHandler` instances send messages to
958 disk files, rotating the log file at certain timed intervals.
959
960 #. :class:`~handlers.SocketHandler` instances send messages to TCP/IP
961 sockets. Since 3.4, Unix domain sockets are also supported.
962
963 #. :class:`~handlers.DatagramHandler` instances send messages to UDP
964 sockets. Since 3.4, Unix domain sockets are also supported.
965
966 #. :class:`~handlers.SMTPHandler` instances send messages to a designated
967 email address.
968
969 #. :class:`~handlers.SysLogHandler` instances send messages to a Unix
970 syslog daemon, possibly on a remote machine.
971
972 #. :class:`~handlers.NTEventLogHandler` instances send messages to a
973 Windows NT/2000/XP event log.
974
975 #. :class:`~handlers.MemoryHandler` instances send messages to a buffer
976 in memory, which is flushed whenever specific criteria are met.
977
978 #. :class:`~handlers.HTTPHandler` instances send messages to an HTTP
979 server using either ``GET`` or ``POST`` semantics.
980
981 #. :class:`~handlers.WatchedFileHandler` instances watch the file they are
982 logging to. If the file changes, it is closed and reopened using the file
983 name. This handler is only useful on Unix-like systems; Windows does not
984 support the underlying mechanism used.
985
986 #. :class:`~handlers.QueueHandler` instances send messages to a queue, such as
987 those implemented in the :mod:`queue` or :mod:`multiprocessing` modules.
988
989 #. :class:`NullHandler` instances do nothing with error messages. They are used
990 by library developers who want to use logging, but want to avoid the 'No
991 handlers could be found for logger *XXX*' message which can be displayed if
992 the library user has not configured logging. See :ref:`library-config` for
993 more information.
994
995 .. versionadded:: 3.1
996 The :class:`NullHandler` class.
997
998 .. versionadded:: 3.2
999 The :class:`~handlers.QueueHandler` class.
1000
1001 The :class:`NullHandler`, :class:`StreamHandler` and :class:`FileHandler`
1002 classes are defined in the core logging package. The other handlers are
1003 defined in a sub-module, :mod:`logging.handlers`. (There is also another
1004 sub-module, :mod:`logging.config`, for configuration functionality.)
1005
1006 Logged messages are formatted for presentation through instances of the
1007 :class:`Formatter` class. They are initialized with a format string suitable for
1008 use with the % operator and a dictionary.
1009
1010 For formatting multiple messages in a batch, instances of
1011 :class:`BufferingFormatter` can be used. In addition to the format
1012 string (which is applied to each message in the batch), there is provision for
1013 header and trailer format strings.
1014
1015 When filtering based on logger level and/or handler level is not enough,
1016 instances of :class:`Filter` can be added to both :class:`Logger` and
1017 :class:`Handler` instances (through their :meth:`~Handler.addFilter` method).
1018 Before deciding to process a message further, both loggers and handlers consult
1019 all their filters for permission. If any filter returns a false value, the
1020 message is not processed further.
1021
1022 The basic :class:`Filter` functionality allows filtering by specific logger
1023 name. If this feature is used, messages sent to the named logger and its
1024 children are allowed through the filter, and all others dropped.
1025
1026
1027 .. _logging-exceptions:
1028
1029 Exceptions raised during logging
1030 --------------------------------
1031
1032 The logging package is designed to swallow exceptions which occur while logging
1033 in production. This is so that errors which occur while handling logging events
1034 - such as logging misconfiguration, network or other similar errors - do not
1035 cause the application using logging to terminate prematurely.
1036
1037 :class:`SystemExit` and :class:`KeyboardInterrupt` exceptions are never
1038 swallowed. Other exceptions which occur during the :meth:`~Handler.emit` method
1039 of a :class:`Handler` subclass are passed to its :meth:`~Handler.handleError`
1040 method.
1041
1042 The default implementation of :meth:`~Handler.handleError` in :class:`Handler`
1043 checks to see if a module-level variable, :data:`raiseExceptions`, is set. If
1044 set, a traceback is printed to :data:`sys.stderr`. If not set, the exception is
1045 swallowed.
1046
1047 .. note::
1048 The default value of :data:`raiseExceptions` is ``True``. This is
1049 because during development, you typically want to be notified of any
1050 exceptions that occur. It's advised that you set :data:`raiseExceptions` to
1051 ``False`` for production usage.
1052
1053 .. currentmodule:: logging
1054
1055 .. _arbitrary-object-messages:
1056
1057 Using arbitrary objects as messages
1058 -----------------------------------
1059
1060 In the preceding sections and examples, it has been assumed that the message
1061 passed when logging the event is a string. However, this is not the only
1062 possibility. You can pass an arbitrary object as a message, and its
1063 :meth:`~object.__str__` method will be called when the logging system needs to
1064 convert it to a string representation. In fact, if you want to, you can avoid
1065 computing a string representation altogether - for example, the
1066 :class:`~handlers.SocketHandler` emits an event by pickling it and sending it
1067 over the wire.
1068
1069
1070 Optimization
1071 ------------
1072
1073 Formatting of message arguments is deferred until it cannot be avoided.
1074 However, computing the arguments passed to the logging method can also be
1075 expensive, and you may want to avoid doing it if the logger will just throw
1076 away your event. To decide what to do, you can call the
1077 :meth:`~Logger.isEnabledFor` method which takes a level argument and returns
1078 true if the event would be created by the Logger for that level of call.
1079 You can write code like this::
1080
1081 if logger.isEnabledFor(logging.DEBUG):
1082 logger.debug('Message with %s, %s', expensive_func1(),
1083 expensive_func2())
1084
1085 so that if the logger's threshold is set above ``DEBUG``, the calls to
1086 ``expensive_func1`` and ``expensive_func2`` are never made.
1087
1088 .. note:: In some cases, :meth:`~Logger.isEnabledFor` can itself be more
1089 expensive than you'd like (e.g. for deeply nested loggers where an explicit
1090 level is only set high up in the logger hierarchy). In such cases (or if you
1091 want to avoid calling a method in tight loops), you can cache the result of a
1092 call to :meth:`~Logger.isEnabledFor` in a local or instance variable, and use
1093 that instead of calling the method each time. Such a cached value would only
1094 need to be recomputed when the logging configuration changes dynamically
1095 while the application is running (which is not all that common).
1096
1097 There are other optimizations which can be made for specific applications which
1098 need more precise control over what logging information is collected. Here's a
1099 list of things you can do to avoid processing during logging which you don't
1100 need:
1101
1102 +-----------------------------------------------------+---------------------------------------------------+
1103 | What you don't want to collect | How to avoid collecting it |
1104 +=====================================================+===================================================+
1105 | Information about where calls were made from. | Set ``logging._srcfile`` to ``None``. |
1106 | | This avoids calling :func:`sys._getframe`, which |
1107 | | may help to speed up your code in environments |
1108 | | like PyPy (which can't speed up code that uses |
1109 | | :func:`sys._getframe`). |
1110 +-----------------------------------------------------+---------------------------------------------------+
1111 | Threading information. | Set ``logging.logThreads`` to ``False``. |
1112 +-----------------------------------------------------+---------------------------------------------------+
1113 | Current process ID (:func:`os.getpid`) | Set ``logging.logProcesses`` to ``False``. |
1114 +-----------------------------------------------------+---------------------------------------------------+
1115 | Current process name when using ``multiprocessing`` | Set ``logging.logMultiprocessing`` to ``False``. |
1116 | to manage multiple processes. | |
1117 +-----------------------------------------------------+---------------------------------------------------+
1118 | Current :class:`asyncio.Task` name when using | Set ``logging.logAsyncioTasks`` to ``False``. |
1119 | ``asyncio``. | |
1120 +-----------------------------------------------------+---------------------------------------------------+
1121
1122 Also note that the core logging module only includes the basic handlers. If
1123 you don't import :mod:`logging.handlers` and :mod:`logging.config`, they won't
1124 take up any memory.
1125
1126 .. _tutorial-ref-links:
1127
1128 Other resources
1129 ---------------
1130
1131 .. seealso::
1132
1133 Module :mod:`logging`
1134 API reference for the logging module.
1135
1136 Module :mod:`logging.config`
1137 Configuration API for the logging module.
1138
1139 Module :mod:`logging.handlers`
1140 Useful handlers included with the logging module.
1141
1142 :ref:`A logging cookbook <logging-cookbook>`