| 1 |
.. _logging-cookbook: |
| 2 |
|
| 3 |
================ |
| 4 |
Logging Cookbook |
| 5 |
================ |
| 6 |
|
| 7 |
:Author: Vinay Sajip <vinay_sajip at red-dove dot com> |
| 8 |
|
| 9 |
This page contains a number of recipes related to logging, which have been found |
| 10 |
useful in the past. For links to tutorial and reference information, please see |
| 11 |
:ref:`cookbook-ref-links`. |
| 12 |
|
| 13 |
.. currentmodule:: logging |
| 14 |
|
| 15 |
Using logging in multiple modules |
| 16 |
--------------------------------- |
| 17 |
|
| 18 |
Multiple calls to ``logging.getLogger('someLogger')`` return a reference to the |
| 19 |
same logger object. This is true not only within the same module, but also |
| 20 |
across modules as long as it is in the same Python interpreter process. It is |
| 21 |
true for references to the same object; additionally, application code can |
| 22 |
define and configure a parent logger in one module and create (but not |
| 23 |
configure) a child logger in a separate module, and all logger calls to the |
| 24 |
child will pass up to the parent. Here is a main module:: |
| 25 |
|
| 26 |
import logging |
| 27 |
import auxiliary_module |
| 28 |
|
| 29 |
# create logger with 'spam_application' |
| 30 |
logger = logging.getLogger('spam_application') |
| 31 |
logger.setLevel(logging.DEBUG) |
| 32 |
# create file handler which logs even debug messages |
| 33 |
fh = logging.FileHandler('spam.log') |
| 34 |
fh.setLevel(logging.DEBUG) |
| 35 |
# create console handler with a higher log level |
| 36 |
ch = logging.StreamHandler() |
| 37 |
ch.setLevel(logging.ERROR) |
| 38 |
# create formatter and add it to the handlers |
| 39 |
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') |
| 40 |
fh.setFormatter(formatter) |
| 41 |
ch.setFormatter(formatter) |
| 42 |
# add the handlers to the logger |
| 43 |
logger.addHandler(fh) |
| 44 |
logger.addHandler(ch) |
| 45 |
|
| 46 |
logger.info('creating an instance of auxiliary_module.Auxiliary') |
| 47 |
a = auxiliary_module.Auxiliary() |
| 48 |
logger.info('created an instance of auxiliary_module.Auxiliary') |
| 49 |
logger.info('calling auxiliary_module.Auxiliary.do_something') |
| 50 |
a.do_something() |
| 51 |
logger.info('finished auxiliary_module.Auxiliary.do_something') |
| 52 |
logger.info('calling auxiliary_module.some_function()') |
| 53 |
auxiliary_module.some_function() |
| 54 |
logger.info('done with auxiliary_module.some_function()') |
| 55 |
|
| 56 |
Here is the auxiliary module:: |
| 57 |
|
| 58 |
import logging |
| 59 |
|
| 60 |
# create logger |
| 61 |
module_logger = logging.getLogger('spam_application.auxiliary') |
| 62 |
|
| 63 |
class Auxiliary: |
| 64 |
def __init__(self): |
| 65 |
self.logger = logging.getLogger('spam_application.auxiliary.Auxiliary') |
| 66 |
self.logger.info('creating an instance of Auxiliary') |
| 67 |
|
| 68 |
def do_something(self): |
| 69 |
self.logger.info('doing something') |
| 70 |
a = 1 + 1 |
| 71 |
self.logger.info('done doing something') |
| 72 |
|
| 73 |
def some_function(): |
| 74 |
module_logger.info('received a call to "some_function"') |
| 75 |
|
| 76 |
The output looks like this: |
| 77 |
|
| 78 |
.. code-block:: none |
| 79 |
|
| 80 |
2005-03-23 23:47:11,663 - spam_application - INFO - |
| 81 |
creating an instance of auxiliary_module.Auxiliary |
| 82 |
2005-03-23 23:47:11,665 - spam_application.auxiliary.Auxiliary - INFO - |
| 83 |
creating an instance of Auxiliary |
| 84 |
2005-03-23 23:47:11,665 - spam_application - INFO - |
| 85 |
created an instance of auxiliary_module.Auxiliary |
| 86 |
2005-03-23 23:47:11,668 - spam_application - INFO - |
| 87 |
calling auxiliary_module.Auxiliary.do_something |
| 88 |
2005-03-23 23:47:11,668 - spam_application.auxiliary.Auxiliary - INFO - |
| 89 |
doing something |
| 90 |
2005-03-23 23:47:11,669 - spam_application.auxiliary.Auxiliary - INFO - |
| 91 |
done doing something |
| 92 |
2005-03-23 23:47:11,670 - spam_application - INFO - |
| 93 |
finished auxiliary_module.Auxiliary.do_something |
| 94 |
2005-03-23 23:47:11,671 - spam_application - INFO - |
| 95 |
calling auxiliary_module.some_function() |
| 96 |
2005-03-23 23:47:11,672 - spam_application.auxiliary - INFO - |
| 97 |
received a call to 'some_function' |
| 98 |
2005-03-23 23:47:11,673 - spam_application - INFO - |
| 99 |
done with auxiliary_module.some_function() |
| 100 |
|
| 101 |
Logging from multiple threads |
| 102 |
----------------------------- |
| 103 |
|
| 104 |
Logging from multiple threads requires no special effort. The following example |
| 105 |
shows logging from the main (initial) thread and another thread:: |
| 106 |
|
| 107 |
import logging |
| 108 |
import threading |
| 109 |
import time |
| 110 |
|
| 111 |
def worker(arg): |
| 112 |
while not arg['stop']: |
| 113 |
logging.debug('Hi from myfunc') |
| 114 |
time.sleep(0.5) |
| 115 |
|
| 116 |
def main(): |
| 117 |
logging.basicConfig(level=logging.DEBUG, format='%(relativeCreated)6d %(threadName)s %(message)s') |
| 118 |
info = {'stop': False} |
| 119 |
thread = threading.Thread(target=worker, args=(info,)) |
| 120 |
thread.start() |
| 121 |
while True: |
| 122 |
try: |
| 123 |
logging.debug('Hello from main') |
| 124 |
time.sleep(0.75) |
| 125 |
except KeyboardInterrupt: |
| 126 |
info['stop'] = True |
| 127 |
break |
| 128 |
thread.join() |
| 129 |
|
| 130 |
if __name__ == '__main__': |
| 131 |
main() |
| 132 |
|
| 133 |
When run, the script should print something like the following: |
| 134 |
|
| 135 |
.. code-block:: none |
| 136 |
|
| 137 |
0 Thread-1 Hi from myfunc |
| 138 |
3 MainThread Hello from main |
| 139 |
505 Thread-1 Hi from myfunc |
| 140 |
755 MainThread Hello from main |
| 141 |
1007 Thread-1 Hi from myfunc |
| 142 |
1507 MainThread Hello from main |
| 143 |
1508 Thread-1 Hi from myfunc |
| 144 |
2010 Thread-1 Hi from myfunc |
| 145 |
2258 MainThread Hello from main |
| 146 |
2512 Thread-1 Hi from myfunc |
| 147 |
3009 MainThread Hello from main |
| 148 |
3013 Thread-1 Hi from myfunc |
| 149 |
3515 Thread-1 Hi from myfunc |
| 150 |
3761 MainThread Hello from main |
| 151 |
4017 Thread-1 Hi from myfunc |
| 152 |
4513 MainThread Hello from main |
| 153 |
4518 Thread-1 Hi from myfunc |
| 154 |
|
| 155 |
This shows the logging output interspersed as one might expect. This approach |
| 156 |
works for more threads than shown here, of course. |
| 157 |
|
| 158 |
Multiple handlers and formatters |
| 159 |
-------------------------------- |
| 160 |
|
| 161 |
Loggers are plain Python objects. The :meth:`~Logger.addHandler` method has no |
| 162 |
minimum or maximum quota for the number of handlers you may add. Sometimes it |
| 163 |
will be beneficial for an application to log all messages of all severities to a |
| 164 |
text file while simultaneously logging errors or above to the console. To set |
| 165 |
this up, simply configure the appropriate handlers. The logging calls in the |
| 166 |
application code will remain unchanged. Here is a slight modification to the |
| 167 |
previous simple module-based configuration example:: |
| 168 |
|
| 169 |
import logging |
| 170 |
|
| 171 |
logger = logging.getLogger('simple_example') |
| 172 |
logger.setLevel(logging.DEBUG) |
| 173 |
# create file handler which logs even debug messages |
| 174 |
fh = logging.FileHandler('spam.log') |
| 175 |
fh.setLevel(logging.DEBUG) |
| 176 |
# create console handler with a higher log level |
| 177 |
ch = logging.StreamHandler() |
| 178 |
ch.setLevel(logging.ERROR) |
| 179 |
# create formatter and add it to the handlers |
| 180 |
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') |
| 181 |
ch.setFormatter(formatter) |
| 182 |
fh.setFormatter(formatter) |
| 183 |
# add the handlers to logger |
| 184 |
logger.addHandler(ch) |
| 185 |
logger.addHandler(fh) |
| 186 |
|
| 187 |
# 'application' code |
| 188 |
logger.debug('debug message') |
| 189 |
logger.info('info message') |
| 190 |
logger.warning('warn message') |
| 191 |
logger.error('error message') |
| 192 |
logger.critical('critical message') |
| 193 |
|
| 194 |
Notice that the 'application' code does not care about multiple handlers. All |
| 195 |
that changed was the addition and configuration of a new handler named *fh*. |
| 196 |
|
| 197 |
The ability to create new handlers with higher- or lower-severity filters can be |
| 198 |
very helpful when writing and testing an application. Instead of using many |
| 199 |
``print`` statements for debugging, use ``logger.debug``: Unlike the print |
| 200 |
statements, which you will have to delete or comment out later, the logger.debug |
| 201 |
statements can remain intact in the source code and remain dormant until you |
| 202 |
need them again. At that time, the only change that needs to happen is to |
| 203 |
modify the severity level of the logger and/or handler to debug. |
| 204 |
|
| 205 |
.. _multiple-destinations: |
| 206 |
|
| 207 |
Logging to multiple destinations |
| 208 |
-------------------------------- |
| 209 |
|
| 210 |
Let's say you want to log to console and file with different message formats and |
| 211 |
in differing circumstances. Say you want to log messages with levels of DEBUG |
| 212 |
and higher to file, and those messages at level INFO and higher to the console. |
| 213 |
Let's also assume that the file should contain timestamps, but the console |
| 214 |
messages should not. Here's how you can achieve this:: |
| 215 |
|
| 216 |
import logging |
| 217 |
|
| 218 |
# set up logging to file - see previous section for more details |
| 219 |
logging.basicConfig(level=logging.DEBUG, |
| 220 |
format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s', |
| 221 |
datefmt='%m-%d %H:%M', |
| 222 |
filename='/tmp/myapp.log', |
| 223 |
filemode='w') |
| 224 |
# define a Handler which writes INFO messages or higher to the sys.stderr |
| 225 |
console = logging.StreamHandler() |
| 226 |
console.setLevel(logging.INFO) |
| 227 |
# set a format which is simpler for console use |
| 228 |
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s') |
| 229 |
# tell the handler to use this format |
| 230 |
console.setFormatter(formatter) |
| 231 |
# add the handler to the root logger |
| 232 |
logging.getLogger().addHandler(console) |
| 233 |
|
| 234 |
# Now, we can log to the root logger, or any other logger. First the root... |
| 235 |
logging.info('Jackdaws love my big sphinx of quartz.') |
| 236 |
|
| 237 |
# Now, define a couple of other loggers which might represent areas in your |
| 238 |
# application: |
| 239 |
|
| 240 |
logger1 = logging.getLogger('myapp.area1') |
| 241 |
logger2 = logging.getLogger('myapp.area2') |
| 242 |
|
| 243 |
logger1.debug('Quick zephyrs blow, vexing daft Jim.') |
| 244 |
logger1.info('How quickly daft jumping zebras vex.') |
| 245 |
logger2.warning('Jail zesty vixen who grabbed pay from quack.') |
| 246 |
logger2.error('The five boxing wizards jump quickly.') |
| 247 |
|
| 248 |
When you run this, on the console you will see |
| 249 |
|
| 250 |
.. code-block:: none |
| 251 |
|
| 252 |
root : INFO Jackdaws love my big sphinx of quartz. |
| 253 |
myapp.area1 : INFO How quickly daft jumping zebras vex. |
| 254 |
myapp.area2 : WARNING Jail zesty vixen who grabbed pay from quack. |
| 255 |
myapp.area2 : ERROR The five boxing wizards jump quickly. |
| 256 |
|
| 257 |
and in the file you will see something like |
| 258 |
|
| 259 |
.. code-block:: none |
| 260 |
|
| 261 |
10-22 22:19 root INFO Jackdaws love my big sphinx of quartz. |
| 262 |
10-22 22:19 myapp.area1 DEBUG Quick zephyrs blow, vexing daft Jim. |
| 263 |
10-22 22:19 myapp.area1 INFO How quickly daft jumping zebras vex. |
| 264 |
10-22 22:19 myapp.area2 WARNING Jail zesty vixen who grabbed pay from quack. |
| 265 |
10-22 22:19 myapp.area2 ERROR The five boxing wizards jump quickly. |
| 266 |
|
| 267 |
As you can see, the DEBUG message only shows up in the file. The other messages |
| 268 |
are sent to both destinations. |
| 269 |
|
| 270 |
This example uses console and file handlers, but you can use any number and |
| 271 |
combination of handlers you choose. |
| 272 |
|
| 273 |
Note that the above choice of log filename ``/tmp/myapp.log`` implies use of a |
| 274 |
standard location for temporary files on POSIX systems. On Windows, you may need to |
| 275 |
choose a different directory name for the log - just ensure that the directory exists |
| 276 |
and that you have the permissions to create and update files in it. |
| 277 |
|
| 278 |
|
| 279 |
.. _custom-level-handling: |
| 280 |
|
| 281 |
Custom handling of levels |
| 282 |
------------------------- |
| 283 |
|
| 284 |
Sometimes, you might want to do something slightly different from the standard |
| 285 |
handling of levels in handlers, where all levels above a threshold get |
| 286 |
processed by a handler. To do this, you need to use filters. Let's look at a |
| 287 |
scenario where you want to arrange things as follows: |
| 288 |
|
| 289 |
* Send messages of severity ``INFO`` and ``WARNING`` to ``sys.stdout`` |
| 290 |
* Send messages of severity ``ERROR`` and above to ``sys.stderr`` |
| 291 |
* Send messages of severity ``DEBUG`` and above to file ``app.log`` |
| 292 |
|
| 293 |
Suppose you configure logging with the following JSON: |
| 294 |
|
| 295 |
.. code-block:: json |
| 296 |
|
| 297 |
{ |
| 298 |
"version": 1, |
| 299 |
"disable_existing_loggers": false, |
| 300 |
"formatters": { |
| 301 |
"simple": { |
| 302 |
"format": "%(levelname)-8s - %(message)s" |
| 303 |
} |
| 304 |
}, |
| 305 |
"handlers": { |
| 306 |
"stdout": { |
| 307 |
"class": "logging.StreamHandler", |
| 308 |
"level": "INFO", |
| 309 |
"formatter": "simple", |
| 310 |
"stream": "ext://sys.stdout" |
| 311 |
}, |
| 312 |
"stderr": { |
| 313 |
"class": "logging.StreamHandler", |
| 314 |
"level": "ERROR", |
| 315 |
"formatter": "simple", |
| 316 |
"stream": "ext://sys.stderr" |
| 317 |
}, |
| 318 |
"file": { |
| 319 |
"class": "logging.FileHandler", |
| 320 |
"formatter": "simple", |
| 321 |
"filename": "app.log", |
| 322 |
"mode": "w" |
| 323 |
} |
| 324 |
}, |
| 325 |
"root": { |
| 326 |
"level": "DEBUG", |
| 327 |
"handlers": [ |
| 328 |
"stderr", |
| 329 |
"stdout", |
| 330 |
"file" |
| 331 |
] |
| 332 |
} |
| 333 |
} |
| 334 |
|
| 335 |
This configuration does *almost* what we want, except that ``sys.stdout`` would show messages |
| 336 |
of severity ``ERROR`` and only events of this severity and higher will be tracked |
| 337 |
as well as ``INFO`` and ``WARNING`` messages. To prevent this, we can set up a filter which |
| 338 |
excludes those messages and add it to the relevant handler. This can be configured by |
| 339 |
adding a ``filters`` section parallel to ``formatters`` and ``handlers``: |
| 340 |
|
| 341 |
.. code-block:: json |
| 342 |
|
| 343 |
{ |
| 344 |
"filters": { |
| 345 |
"warnings_and_below": { |
| 346 |
"()" : "__main__.filter_maker", |
| 347 |
"level": "WARNING" |
| 348 |
} |
| 349 |
} |
| 350 |
} |
| 351 |
|
| 352 |
and changing the section on the ``stdout`` handler to add it: |
| 353 |
|
| 354 |
.. code-block:: json |
| 355 |
|
| 356 |
{ |
| 357 |
"stdout": { |
| 358 |
"class": "logging.StreamHandler", |
| 359 |
"level": "INFO", |
| 360 |
"formatter": "simple", |
| 361 |
"stream": "ext://sys.stdout", |
| 362 |
"filters": ["warnings_and_below"] |
| 363 |
} |
| 364 |
} |
| 365 |
|
| 366 |
A filter is just a function, so we can define the ``filter_maker`` (a factory |
| 367 |
function) as follows: |
| 368 |
|
| 369 |
.. code-block:: python |
| 370 |
|
| 371 |
def filter_maker(level): |
| 372 |
level = getattr(logging, level) |
| 373 |
|
| 374 |
def filter(record): |
| 375 |
return record.levelno <= level |
| 376 |
|
| 377 |
return filter |
| 378 |
|
| 379 |
This converts the string argument passed in to a numeric level, and returns a |
| 380 |
function which only returns ``True`` if the level of the passed in record is |
| 381 |
at or below the specified level. Note that in this example I have defined the |
| 382 |
``filter_maker`` in a test script ``main.py`` that I run from the command line, |
| 383 |
so its module will be ``__main__`` - hence the ``__main__.filter_maker`` in the |
| 384 |
filter configuration. You will need to change that if you define it in a |
| 385 |
different module. |
| 386 |
|
| 387 |
With the filter added, we can run ``main.py``, which in full is: |
| 388 |
|
| 389 |
.. code-block:: python |
| 390 |
|
| 391 |
import json |
| 392 |
import logging |
| 393 |
import logging.config |
| 394 |
|
| 395 |
CONFIG = ''' |
| 396 |
{ |
| 397 |
"version": 1, |
| 398 |
"disable_existing_loggers": false, |
| 399 |
"formatters": { |
| 400 |
"simple": { |
| 401 |
"format": "%(levelname)-8s - %(message)s" |
| 402 |
} |
| 403 |
}, |
| 404 |
"filters": { |
| 405 |
"warnings_and_below": { |
| 406 |
"()" : "__main__.filter_maker", |
| 407 |
"level": "WARNING" |
| 408 |
} |
| 409 |
}, |
| 410 |
"handlers": { |
| 411 |
"stdout": { |
| 412 |
"class": "logging.StreamHandler", |
| 413 |
"level": "INFO", |
| 414 |
"formatter": "simple", |
| 415 |
"stream": "ext://sys.stdout", |
| 416 |
"filters": ["warnings_and_below"] |
| 417 |
}, |
| 418 |
"stderr": { |
| 419 |
"class": "logging.StreamHandler", |
| 420 |
"level": "ERROR", |
| 421 |
"formatter": "simple", |
| 422 |
"stream": "ext://sys.stderr" |
| 423 |
}, |
| 424 |
"file": { |
| 425 |
"class": "logging.FileHandler", |
| 426 |
"formatter": "simple", |
| 427 |
"filename": "app.log", |
| 428 |
"mode": "w" |
| 429 |
} |
| 430 |
}, |
| 431 |
"root": { |
| 432 |
"level": "DEBUG", |
| 433 |
"handlers": [ |
| 434 |
"stderr", |
| 435 |
"stdout", |
| 436 |
"file" |
| 437 |
] |
| 438 |
} |
| 439 |
} |
| 440 |
''' |
| 441 |
|
| 442 |
def filter_maker(level): |
| 443 |
level = getattr(logging, level) |
| 444 |
|
| 445 |
def filter(record): |
| 446 |
return record.levelno <= level |
| 447 |
|
| 448 |
return filter |
| 449 |
|
| 450 |
logging.config.dictConfig(json.loads(CONFIG)) |
| 451 |
logging.debug('A DEBUG message') |
| 452 |
logging.info('An INFO message') |
| 453 |
logging.warning('A WARNING message') |
| 454 |
logging.error('An ERROR message') |
| 455 |
logging.critical('A CRITICAL message') |
| 456 |
|
| 457 |
And after running it like this: |
| 458 |
|
| 459 |
.. code-block:: shell |
| 460 |
|
| 461 |
python main.py 2>stderr.log >stdout.log |
| 462 |
|
| 463 |
We can see the results are as expected: |
| 464 |
|
| 465 |
.. code-block:: shell |
| 466 |
|
| 467 |
$ more *.log |
| 468 |
:::::::::::::: |
| 469 |
app.log |
| 470 |
:::::::::::::: |
| 471 |
DEBUG - A DEBUG message |
| 472 |
INFO - An INFO message |
| 473 |
WARNING - A WARNING message |
| 474 |
ERROR - An ERROR message |
| 475 |
CRITICAL - A CRITICAL message |
| 476 |
:::::::::::::: |
| 477 |
stderr.log |
| 478 |
:::::::::::::: |
| 479 |
ERROR - An ERROR message |
| 480 |
CRITICAL - A CRITICAL message |
| 481 |
:::::::::::::: |
| 482 |
stdout.log |
| 483 |
:::::::::::::: |
| 484 |
INFO - An INFO message |
| 485 |
WARNING - A WARNING message |
| 486 |
|
| 487 |
|
| 488 |
Configuration server example |
| 489 |
---------------------------- |
| 490 |
|
| 491 |
Here is an example of a module using the logging configuration server:: |
| 492 |
|
| 493 |
import logging |
| 494 |
import logging.config |
| 495 |
import time |
| 496 |
import os |
| 497 |
|
| 498 |
# read initial config file |
| 499 |
logging.config.fileConfig('logging.conf') |
| 500 |
|
| 501 |
# create and start listener on port 9999 |
| 502 |
t = logging.config.listen(9999) |
| 503 |
t.start() |
| 504 |
|
| 505 |
logger = logging.getLogger('simpleExample') |
| 506 |
|
| 507 |
try: |
| 508 |
# loop through logging calls to see the difference |
| 509 |
# new configurations make, until Ctrl+C is pressed |
| 510 |
while True: |
| 511 |
logger.debug('debug message') |
| 512 |
logger.info('info message') |
| 513 |
logger.warning('warn message') |
| 514 |
logger.error('error message') |
| 515 |
logger.critical('critical message') |
| 516 |
time.sleep(5) |
| 517 |
except KeyboardInterrupt: |
| 518 |
# cleanup |
| 519 |
logging.config.stopListening() |
| 520 |
t.join() |
| 521 |
|
| 522 |
And here is a script that takes a filename and sends that file to the server, |
| 523 |
properly preceded with the binary-encoded length, as the new logging |
| 524 |
configuration:: |
| 525 |
|
| 526 |
#!/usr/bin/env python |
| 527 |
import socket, sys, struct |
| 528 |
|
| 529 |
with open(sys.argv[1], 'rb') as f: |
| 530 |
data_to_send = f.read() |
| 531 |
|
| 532 |
HOST = 'localhost' |
| 533 |
PORT = 9999 |
| 534 |
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) |
| 535 |
print('connecting...') |
| 536 |
s.connect((HOST, PORT)) |
| 537 |
print('sending config...') |
| 538 |
s.send(struct.pack('>L', len(data_to_send))) |
| 539 |
s.send(data_to_send) |
| 540 |
s.close() |
| 541 |
print('complete') |
| 542 |
|
| 543 |
|
| 544 |
.. _blocking-handlers: |
| 545 |
|
| 546 |
Dealing with handlers that block |
| 547 |
-------------------------------- |
| 548 |
|
| 549 |
.. currentmodule:: logging.handlers |
| 550 |
|
| 551 |
Sometimes you have to get your logging handlers to do their work without |
| 552 |
blocking the thread you're logging from. This is common in web applications, |
| 553 |
though of course it also occurs in other scenarios. |
| 554 |
|
| 555 |
A common culprit which demonstrates sluggish behaviour is the |
| 556 |
:class:`SMTPHandler`: sending emails can take a long time, for a |
| 557 |
number of reasons outside the developer's control (for example, a poorly |
| 558 |
performing mail or network infrastructure). But almost any network-based |
| 559 |
handler can block: Even a :class:`SocketHandler` operation may do a |
| 560 |
DNS query under the hood which is too slow (and this query can be deep in the |
| 561 |
socket library code, below the Python layer, and outside your control). |
| 562 |
|
| 563 |
One solution is to use a two-part approach. For the first part, attach only a |
| 564 |
:class:`QueueHandler` to those loggers which are accessed from |
| 565 |
performance-critical threads. They simply write to their queue, which can be |
| 566 |
sized to a large enough capacity or initialized with no upper bound to their |
| 567 |
size. The write to the queue will typically be accepted quickly, though you |
| 568 |
will probably need to catch the :exc:`queue.Full` exception as a precaution |
| 569 |
in your code. If you are a library developer who has performance-critical |
| 570 |
threads in their code, be sure to document this (together with a suggestion to |
| 571 |
attach only ``QueueHandlers`` to your loggers) for the benefit of other |
| 572 |
developers who will use your code. |
| 573 |
|
| 574 |
The second part of the solution is :class:`QueueListener`, which has been |
| 575 |
designed as the counterpart to :class:`QueueHandler`. A |
| 576 |
:class:`QueueListener` is very simple: it's passed a queue and some handlers, |
| 577 |
and it fires up an internal thread which listens to its queue for LogRecords |
| 578 |
sent from ``QueueHandlers`` (or any other source of ``LogRecords``, for that |
| 579 |
matter). The ``LogRecords`` are removed from the queue and passed to the |
| 580 |
handlers for processing. |
| 581 |
|
| 582 |
The advantage of having a separate :class:`QueueListener` class is that you |
| 583 |
can use the same instance to service multiple ``QueueHandlers``. This is more |
| 584 |
resource-friendly than, say, having threaded versions of the existing handler |
| 585 |
classes, which would eat up one thread per handler for no particular benefit. |
| 586 |
|
| 587 |
An example of using these two classes follows (imports omitted):: |
| 588 |
|
| 589 |
que = queue.Queue(-1) # no limit on size |
| 590 |
queue_handler = QueueHandler(que) |
| 591 |
handler = logging.StreamHandler() |
| 592 |
listener = QueueListener(que, handler) |
| 593 |
root = logging.getLogger() |
| 594 |
root.addHandler(queue_handler) |
| 595 |
formatter = logging.Formatter('%(threadName)s: %(message)s') |
| 596 |
handler.setFormatter(formatter) |
| 597 |
listener.start() |
| 598 |
# The log output will display the thread which generated |
| 599 |
# the event (the main thread) rather than the internal |
| 600 |
# thread which monitors the internal queue. This is what |
| 601 |
# you want to happen. |
| 602 |
root.warning('Look out!') |
| 603 |
listener.stop() |
| 604 |
|
| 605 |
which, when run, will produce: |
| 606 |
|
| 607 |
.. code-block:: none |
| 608 |
|
| 609 |
MainThread: Look out! |
| 610 |
|
| 611 |
.. note:: Although the earlier discussion wasn't specifically talking about |
| 612 |
async code, but rather about slow logging handlers, it should be noted that |
| 613 |
when logging from async code, network and even file handlers could lead to |
| 614 |
problems (blocking the event loop) because some logging is done from |
| 615 |
:mod:`asyncio` internals. It might be best, if any async code is used in an |
| 616 |
application, to use the above approach for logging, so that any blocking code |
| 617 |
runs only in the ``QueueListener`` thread. |
| 618 |
|
| 619 |
.. versionchanged:: 3.5 |
| 620 |
Prior to Python 3.5, the :class:`QueueListener` always passed every message |
| 621 |
received from the queue to every handler it was initialized with. (This was |
| 622 |
because it was assumed that level filtering was all done on the other side, |
| 623 |
where the queue is filled.) From 3.5 onwards, this behaviour can be changed |
| 624 |
by passing a keyword argument ``respect_handler_level=True`` to the |
| 625 |
listener's constructor. When this is done, the listener compares the level |
| 626 |
of each message with the handler's level, and only passes a message to a |
| 627 |
handler if it's appropriate to do so. |
| 628 |
|
| 629 |
.. versionchanged:: 3.14 |
| 630 |
The :class:`QueueListener` can be started (and stopped) via the |
| 631 |
:keyword:`with` statement. For example: |
| 632 |
|
| 633 |
.. code-block:: python |
| 634 |
|
| 635 |
with QueueListener(que, handler) as listener: |
| 636 |
# The queue listener automatically starts |
| 637 |
# when the 'with' block is entered. |
| 638 |
pass |
| 639 |
# The queue listener automatically stops once |
| 640 |
# the 'with' block is exited. |
| 641 |
|
| 642 |
.. _network-logging: |
| 643 |
|
| 644 |
Sending and receiving logging events across a network |
| 645 |
----------------------------------------------------- |
| 646 |
|
| 647 |
Let's say you want to send logging events across a network, and handle them at |
| 648 |
the receiving end. A simple way of doing this is attaching a |
| 649 |
:class:`SocketHandler` instance to the root logger at the sending end:: |
| 650 |
|
| 651 |
import logging, logging.handlers |
| 652 |
|
| 653 |
rootLogger = logging.getLogger() |
| 654 |
rootLogger.setLevel(logging.DEBUG) |
| 655 |
socketHandler = logging.handlers.SocketHandler('localhost', |
| 656 |
logging.handlers.DEFAULT_TCP_LOGGING_PORT) |
| 657 |
# don't bother with a formatter, since a socket handler sends the event as |
| 658 |
# an unformatted pickle |
| 659 |
rootLogger.addHandler(socketHandler) |
| 660 |
|
| 661 |
# Now, we can log to the root logger, or any other logger. First the root... |
| 662 |
logging.info('Jackdaws love my big sphinx of quartz.') |
| 663 |
|
| 664 |
# Now, define a couple of other loggers which might represent areas in your |
| 665 |
# application: |
| 666 |
|
| 667 |
logger1 = logging.getLogger('myapp.area1') |
| 668 |
logger2 = logging.getLogger('myapp.area2') |
| 669 |
|
| 670 |
logger1.debug('Quick zephyrs blow, vexing daft Jim.') |
| 671 |
logger1.info('How quickly daft jumping zebras vex.') |
| 672 |
logger2.warning('Jail zesty vixen who grabbed pay from quack.') |
| 673 |
logger2.error('The five boxing wizards jump quickly.') |
| 674 |
|
| 675 |
At the receiving end, you can set up a receiver using the :mod:`socketserver` |
| 676 |
module. Here is a basic working example:: |
| 677 |
|
| 678 |
import pickle |
| 679 |
import logging |
| 680 |
import logging.handlers |
| 681 |
import socketserver |
| 682 |
import struct |
| 683 |
|
| 684 |
|
| 685 |
class LogRecordStreamHandler(socketserver.StreamRequestHandler): |
| 686 |
"""Handler for a streaming logging request. |
| 687 |
|
| 688 |
This basically logs the record using whatever logging policy is |
| 689 |
configured locally. |
| 690 |
""" |
| 691 |
|
| 692 |
def handle(self): |
| 693 |
""" |
| 694 |
Handle multiple requests - each expected to be a 4-byte length, |
| 695 |
followed by the LogRecord in pickle format. Logs the record |
| 696 |
according to whatever policy is configured locally. |
| 697 |
""" |
| 698 |
while True: |
| 699 |
chunk = self.connection.recv(4) |
| 700 |
if len(chunk) < 4: |
| 701 |
break |
| 702 |
slen = struct.unpack('>L', chunk)[0] |
| 703 |
chunk = self.connection.recv(slen) |
| 704 |
while len(chunk) < slen: |
| 705 |
chunk = chunk + self.connection.recv(slen - len(chunk)) |
| 706 |
obj = self.unPickle(chunk) |
| 707 |
record = logging.makeLogRecord(obj) |
| 708 |
self.handleLogRecord(record) |
| 709 |
|
| 710 |
def unPickle(self, data): |
| 711 |
return pickle.loads(data) |
| 712 |
|
| 713 |
def handleLogRecord(self, record): |
| 714 |
# if a name is specified, we use the named logger rather than the one |
| 715 |
# implied by the record. |
| 716 |
if self.server.logname is not None: |
| 717 |
name = self.server.logname |
| 718 |
else: |
| 719 |
name = record.name |
| 720 |
logger = logging.getLogger(name) |
| 721 |
# N.B. EVERY record gets logged. This is because Logger.handle |
| 722 |
# is normally called AFTER logger-level filtering. If you want |
| 723 |
# to do filtering, do it at the client end to save wasting |
| 724 |
# cycles and network bandwidth! |
| 725 |
logger.handle(record) |
| 726 |
|
| 727 |
class LogRecordSocketReceiver(socketserver.ThreadingTCPServer): |
| 728 |
""" |
| 729 |
Simple TCP socket-based logging receiver suitable for testing. |
| 730 |
""" |
| 731 |
|
| 732 |
allow_reuse_address = True |
| 733 |
|
| 734 |
def __init__(self, host='localhost', |
| 735 |
port=logging.handlers.DEFAULT_TCP_LOGGING_PORT, |
| 736 |
handler=LogRecordStreamHandler): |
| 737 |
socketserver.ThreadingTCPServer.__init__(self, (host, port), handler) |
| 738 |
self.abort = 0 |
| 739 |
self.timeout = 1 |
| 740 |
self.logname = None |
| 741 |
|
| 742 |
def serve_until_stopped(self): |
| 743 |
import select |
| 744 |
abort = 0 |
| 745 |
while not abort: |
| 746 |
rd, wr, ex = select.select([self.socket.fileno()], |
| 747 |
[], [], |
| 748 |
self.timeout) |
| 749 |
if rd: |
| 750 |
self.handle_request() |
| 751 |
abort = self.abort |
| 752 |
|
| 753 |
def main(): |
| 754 |
logging.basicConfig( |
| 755 |
format='%(relativeCreated)5d %(name)-15s %(levelname)-8s %(message)s') |
| 756 |
tcpserver = LogRecordSocketReceiver() |
| 757 |
print('About to start TCP server...') |
| 758 |
tcpserver.serve_until_stopped() |
| 759 |
|
| 760 |
if __name__ == '__main__': |
| 761 |
main() |
| 762 |
|
| 763 |
First run the server, and then the client. On the client side, nothing is |
| 764 |
printed on the console; on the server side, you should see something like: |
| 765 |
|
| 766 |
.. code-block:: none |
| 767 |
|
| 768 |
About to start TCP server... |
| 769 |
59 root INFO Jackdaws love my big sphinx of quartz. |
| 770 |
59 myapp.area1 DEBUG Quick zephyrs blow, vexing daft Jim. |
| 771 |
69 myapp.area1 INFO How quickly daft jumping zebras vex. |
| 772 |
69 myapp.area2 WARNING Jail zesty vixen who grabbed pay from quack. |
| 773 |
69 myapp.area2 ERROR The five boxing wizards jump quickly. |
| 774 |
|
| 775 |
Note that there are some security issues with pickle in some scenarios. If |
| 776 |
these affect you, you can use an alternative serialization scheme by overriding |
| 777 |
the :meth:`~SocketHandler.makePickle` method and implementing your |
| 778 |
alternative there, as well as adapting the above script to use your alternative |
| 779 |
serialization. |
| 780 |
|
| 781 |
|
| 782 |
Running a logging socket listener in production |
| 783 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 784 |
|
| 785 |
.. _socket-listener-gist: https://gist.github.com/vsajip/4b227eeec43817465ca835ca66f75e2b |
| 786 |
|
| 787 |
To run a logging listener in production, you may need to use a |
| 788 |
process-management tool such as `Supervisor <http://supervisord.org/>`_. |
| 789 |
`Here is a Gist <socket-listener-gist_>`__ |
| 790 |
which provides the bare-bones files to run the above functionality using |
| 791 |
Supervisor. It consists of the following files: |
| 792 |
|
| 793 |
+-------------------------+----------------------------------------------------+ |
| 794 |
| File | Purpose | |
| 795 |
+=========================+====================================================+ |
| 796 |
| :file:`prepare.sh` | A Bash script to prepare the environment for | |
| 797 |
| | testing | |
| 798 |
+-------------------------+----------------------------------------------------+ |
| 799 |
| :file:`supervisor.conf` | The Supervisor configuration file, which has | |
| 800 |
| | entries for the listener and a multi-process web | |
| 801 |
| | application | |
| 802 |
+-------------------------+----------------------------------------------------+ |
| 803 |
| :file:`ensure_app.sh` | A Bash script to ensure that Supervisor is running | |
| 804 |
| | with the above configuration | |
| 805 |
+-------------------------+----------------------------------------------------+ |
| 806 |
| :file:`log_listener.py` | The socket listener program which receives log | |
| 807 |
| | events and records them to a file | |
| 808 |
+-------------------------+----------------------------------------------------+ |
| 809 |
| :file:`main.py` | A simple web application which performs logging | |
| 810 |
| | via a socket connected to the listener | |
| 811 |
+-------------------------+----------------------------------------------------+ |
| 812 |
| :file:`webapp.json` | A JSON configuration file for the web application | |
| 813 |
+-------------------------+----------------------------------------------------+ |
| 814 |
| :file:`client.py` | A Python script to exercise the web application | |
| 815 |
+-------------------------+----------------------------------------------------+ |
| 816 |
|
| 817 |
The web application uses `Gunicorn <https://gunicorn.org/>`_, which is a |
| 818 |
popular web application server that starts multiple worker processes to handle |
| 819 |
requests. This example setup shows how the workers can write to the same log file |
| 820 |
without conflicting with one another --- they all go through the socket listener. |
| 821 |
|
| 822 |
To test these files, do the following in a POSIX environment: |
| 823 |
|
| 824 |
#. Download `the Gist <socket-listener-gist_>`__ |
| 825 |
as a ZIP archive using the :guilabel:`Download ZIP` button. |
| 826 |
|
| 827 |
#. Unzip the above files from the archive into a scratch directory. |
| 828 |
|
| 829 |
#. In the scratch directory, run ``bash prepare.sh`` to get things ready. |
| 830 |
This creates a :file:`run` subdirectory to contain Supervisor-related and |
| 831 |
log files, and a :file:`venv` subdirectory to contain a virtual environment |
| 832 |
into which ``bottle``, ``gunicorn`` and ``supervisor`` are installed. |
| 833 |
|
| 834 |
#. Run ``bash ensure_app.sh`` to ensure that Supervisor is running with |
| 835 |
the above configuration. |
| 836 |
|
| 837 |
#. Run ``venv/bin/python client.py`` to exercise the web application, |
| 838 |
which will lead to records being written to the log. |
| 839 |
|
| 840 |
#. Inspect the log files in the :file:`run` subdirectory. You should see the |
| 841 |
most recent log lines in files matching the pattern :file:`app.log*`. They |
| 842 |
won't be in any particular order, since they have been handled concurrently |
| 843 |
by different worker processes in a non-deterministic way. |
| 844 |
|
| 845 |
#. You can shut down the listener and the web application by running |
| 846 |
``venv/bin/supervisorctl -c supervisor.conf shutdown``. |
| 847 |
|
| 848 |
You may need to tweak the configuration files in the unlikely event that the |
| 849 |
configured ports clash with something else in your test environment. |
| 850 |
|
| 851 |
The default configuration uses a TCP socket on port 9020. You can use a Unix |
| 852 |
Domain socket instead of a TCP socket by doing the following: |
| 853 |
|
| 854 |
#. In :file:`listener.json`, add a ``socket`` key with the path to the domain |
| 855 |
socket you want to use. If this key is present, the listener listens on the |
| 856 |
corresponding domain socket and not on a TCP socket (the ``port`` key is |
| 857 |
ignored). |
| 858 |
|
| 859 |
#. In :file:`webapp.json`, change the socket handler configuration dictionary |
| 860 |
so that the ``host`` value is the path to the domain socket, and set the |
| 861 |
``port`` value to ``null``. |
| 862 |
|
| 863 |
|
| 864 |
.. currentmodule:: logging |
| 865 |
|
| 866 |
.. _context-info: |
| 867 |
|
| 868 |
Adding contextual information to your logging output |
| 869 |
---------------------------------------------------- |
| 870 |
|
| 871 |
Sometimes you want logging output to contain contextual information in |
| 872 |
addition to the parameters passed to the logging call. For example, in a |
| 873 |
networked application, it may be desirable to log client-specific information |
| 874 |
in the log (e.g. remote client's username, or IP address). Although you could |
| 875 |
use the *extra* parameter to achieve this, it's not always convenient to pass |
| 876 |
the information in this way. While it might be tempting to create |
| 877 |
:class:`Logger` instances on a per-connection basis, this is not a good idea |
| 878 |
because these instances are not garbage collected. While this is not a problem |
| 879 |
in practice, when the number of :class:`Logger` instances is dependent on the |
| 880 |
level of granularity you want to use in logging an application, it could |
| 881 |
be hard to manage if the number of :class:`Logger` instances becomes |
| 882 |
effectively unbounded. |
| 883 |
|
| 884 |
|
| 885 |
Using LoggerAdapters to impart contextual information |
| 886 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 887 |
|
| 888 |
An easy way in which you can pass contextual information to be output along |
| 889 |
with logging event information is to use the :class:`LoggerAdapter` class. |
| 890 |
This class is designed to look like a :class:`Logger`, so that you can call |
| 891 |
:meth:`debug`, :meth:`info`, :meth:`warning`, :meth:`error`, |
| 892 |
:meth:`exception`, :meth:`critical` and :meth:`log`. These methods have the |
| 893 |
same signatures as their counterparts in :class:`Logger`, so you can use the |
| 894 |
two types of instances interchangeably. |
| 895 |
|
| 896 |
When you create an instance of :class:`LoggerAdapter`, you pass it a |
| 897 |
:class:`Logger` instance and a dict-like object which contains your contextual |
| 898 |
information. When you call one of the logging methods on an instance of |
| 899 |
:class:`LoggerAdapter`, it delegates the call to the underlying instance of |
| 900 |
:class:`Logger` passed to its constructor, and arranges to pass the contextual |
| 901 |
information in the delegated call. Here's a snippet from the code of |
| 902 |
:class:`LoggerAdapter`:: |
| 903 |
|
| 904 |
def debug(self, msg, /, *args, **kwargs): |
| 905 |
""" |
| 906 |
Delegate a debug call to the underlying logger, after adding |
| 907 |
contextual information from this adapter instance. |
| 908 |
""" |
| 909 |
msg, kwargs = self.process(msg, kwargs) |
| 910 |
self.logger.debug(msg, *args, **kwargs) |
| 911 |
|
| 912 |
The :meth:`~LoggerAdapter.process` method of :class:`LoggerAdapter` is where the |
| 913 |
contextual information is added to the logging output. It's passed the message |
| 914 |
and keyword arguments of the logging call, and it passes back (potentially) |
| 915 |
modified versions of these to use in the call to the underlying logger. The |
| 916 |
default implementation of this method leaves the message alone, but inserts |
| 917 |
an 'extra' key in the keyword argument whose value is the dict-like object |
| 918 |
passed to the constructor. Of course, if you had passed an 'extra' keyword |
| 919 |
argument in the call to the adapter, it will be silently overwritten. |
| 920 |
|
| 921 |
The advantage of using 'extra' is that the values in the dict-like object are |
| 922 |
merged into the :class:`LogRecord` instance's __dict__, allowing you to use |
| 923 |
customized strings with your :class:`Formatter` instances which know about |
| 924 |
the keys of the dict-like object. If you need a different method, e.g. if you |
| 925 |
want to prepend or append the contextual information to the message string, |
| 926 |
you just need to subclass :class:`LoggerAdapter` and override |
| 927 |
:meth:`~LoggerAdapter.process` to do what you need. Here is a simple example:: |
| 928 |
|
| 929 |
class CustomAdapter(logging.LoggerAdapter): |
| 930 |
""" |
| 931 |
This example adapter expects the passed in dict-like object to have a |
| 932 |
'connid' key, whose value in brackets is prepended to the log message. |
| 933 |
""" |
| 934 |
def process(self, msg, kwargs): |
| 935 |
return '[%s] %s' % (self.extra['connid'], msg), kwargs |
| 936 |
|
| 937 |
which you can use like this:: |
| 938 |
|
| 939 |
logger = logging.getLogger(__name__) |
| 940 |
adapter = CustomAdapter(logger, {'connid': some_conn_id}) |
| 941 |
|
| 942 |
Then any events that you log to the adapter will have the value of |
| 943 |
``some_conn_id`` prepended to the log messages. |
| 944 |
|
| 945 |
Using objects other than dicts to pass contextual information |
| 946 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 947 |
|
| 948 |
You don't need to pass an actual dict to a :class:`LoggerAdapter` - you could |
| 949 |
pass an instance of a class which implements ``__getitem__`` and ``__iter__`` so |
| 950 |
that it looks like a dict to logging. This would be useful if you want to |
| 951 |
generate values dynamically (whereas the values in a dict would be constant). |
| 952 |
|
| 953 |
|
| 954 |
.. _filters-contextual: |
| 955 |
|
| 956 |
Using Filters to impart contextual information |
| 957 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 958 |
|
| 959 |
You can also add contextual information to log output using a user-defined |
| 960 |
:class:`Filter`. ``Filter`` instances are allowed to modify the ``LogRecords`` |
| 961 |
passed to them, including adding additional attributes which can then be output |
| 962 |
using a suitable format string, or if needed a custom :class:`Formatter`. |
| 963 |
|
| 964 |
For example in a web application, the request being processed (or at least, |
| 965 |
the interesting parts of it) can be stored in a threadlocal |
| 966 |
(:class:`threading.local`) variable, and then accessed from a ``Filter`` to |
| 967 |
add, say, information from the request - say, the remote IP address and remote |
| 968 |
user's username - to the ``LogRecord``, using the attribute names 'ip' and |
| 969 |
'user' as in the ``LoggerAdapter`` example above. In that case, the same format |
| 970 |
string can be used to get similar output to that shown above. Here's an example |
| 971 |
script:: |
| 972 |
|
| 973 |
import logging |
| 974 |
from random import choice |
| 975 |
|
| 976 |
class ContextFilter(logging.Filter): |
| 977 |
""" |
| 978 |
This is a filter which injects contextual information into the log. |
| 979 |
|
| 980 |
Rather than use actual contextual information, we just use random |
| 981 |
data in this demo. |
| 982 |
""" |
| 983 |
|
| 984 |
USERS = ['jim', 'fred', 'sheila'] |
| 985 |
IPS = ['123.231.231.123', '127.0.0.1', '192.168.0.1'] |
| 986 |
|
| 987 |
def filter(self, record): |
| 988 |
|
| 989 |
record.ip = choice(ContextFilter.IPS) |
| 990 |
record.user = choice(ContextFilter.USERS) |
| 991 |
return True |
| 992 |
|
| 993 |
if __name__ == '__main__': |
| 994 |
levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL) |
| 995 |
logging.basicConfig(level=logging.DEBUG, |
| 996 |
format='%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s') |
| 997 |
a1 = logging.getLogger('a.b.c') |
| 998 |
a2 = logging.getLogger('d.e.f') |
| 999 |
|
| 1000 |
f = ContextFilter() |
| 1001 |
a1.addFilter(f) |
| 1002 |
a2.addFilter(f) |
| 1003 |
a1.debug('A debug message') |
| 1004 |
a1.info('An info message with %s', 'some parameters') |
| 1005 |
for x in range(10): |
| 1006 |
lvl = choice(levels) |
| 1007 |
lvlname = logging.getLevelName(lvl) |
| 1008 |
a2.log(lvl, 'A message at %s level with %d %s', lvlname, 2, 'parameters') |
| 1009 |
|
| 1010 |
which, when run, produces something like: |
| 1011 |
|
| 1012 |
.. code-block:: none |
| 1013 |
|
| 1014 |
2010-09-06 22:38:15,292 a.b.c DEBUG IP: 123.231.231.123 User: fred A debug message |
| 1015 |
2010-09-06 22:38:15,300 a.b.c INFO IP: 192.168.0.1 User: sheila An info message with some parameters |
| 1016 |
2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1 User: sheila A message at CRITICAL level with 2 parameters |
| 1017 |
2010-09-06 22:38:15,300 d.e.f ERROR IP: 127.0.0.1 User: jim A message at ERROR level with 2 parameters |
| 1018 |
2010-09-06 22:38:15,300 d.e.f DEBUG IP: 127.0.0.1 User: sheila A message at DEBUG level with 2 parameters |
| 1019 |
2010-09-06 22:38:15,300 d.e.f ERROR IP: 123.231.231.123 User: fred A message at ERROR level with 2 parameters |
| 1020 |
2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 192.168.0.1 User: jim A message at CRITICAL level with 2 parameters |
| 1021 |
2010-09-06 22:38:15,300 d.e.f CRITICAL IP: 127.0.0.1 User: sheila A message at CRITICAL level with 2 parameters |
| 1022 |
2010-09-06 22:38:15,300 d.e.f DEBUG IP: 192.168.0.1 User: jim A message at DEBUG level with 2 parameters |
| 1023 |
2010-09-06 22:38:15,301 d.e.f ERROR IP: 127.0.0.1 User: sheila A message at ERROR level with 2 parameters |
| 1024 |
2010-09-06 22:38:15,301 d.e.f DEBUG IP: 123.231.231.123 User: fred A message at DEBUG level with 2 parameters |
| 1025 |
2010-09-06 22:38:15,301 d.e.f INFO IP: 123.231.231.123 User: fred A message at INFO level with 2 parameters |
| 1026 |
|
| 1027 |
Use of ``contextvars`` |
| 1028 |
---------------------- |
| 1029 |
|
| 1030 |
Since Python 3.7, the :mod:`contextvars` module has provided context-local storage |
| 1031 |
which works for both :mod:`threading` and :mod:`asyncio` processing needs. This type |
| 1032 |
of storage may thus be generally preferable to thread-locals. The following example |
| 1033 |
shows how, in a multi-threaded environment, logs can populated with contextual |
| 1034 |
information such as, for example, request attributes handled by web applications. |
| 1035 |
|
| 1036 |
For the purposes of illustration, say that you have different web applications, each |
| 1037 |
independent of the other but running in the same Python process and using a library |
| 1038 |
common to them. How can each of these applications have their own log, where all |
| 1039 |
logging messages from the library (and other request processing code) are directed to |
| 1040 |
the appropriate application's log file, while including in the log additional |
| 1041 |
contextual information such as client IP, HTTP request method and client username? |
| 1042 |
|
| 1043 |
Let's assume that the library can be simulated by the following code: |
| 1044 |
|
| 1045 |
.. code-block:: python |
| 1046 |
|
| 1047 |
# webapplib.py |
| 1048 |
import logging |
| 1049 |
import time |
| 1050 |
|
| 1051 |
logger = logging.getLogger(__name__) |
| 1052 |
|
| 1053 |
def useful(): |
| 1054 |
# Just a representative event logged from the library |
| 1055 |
logger.debug('Hello from webapplib!') |
| 1056 |
# Just sleep for a bit so other threads get to run |
| 1057 |
time.sleep(0.01) |
| 1058 |
|
| 1059 |
We can simulate the multiple web applications by means of two simple classes, |
| 1060 |
``Request`` and ``WebApp``. These simulate how real threaded web applications work - |
| 1061 |
each request is handled by a thread: |
| 1062 |
|
| 1063 |
.. code-block:: python |
| 1064 |
|
| 1065 |
# main.py |
| 1066 |
import argparse |
| 1067 |
from contextvars import ContextVar |
| 1068 |
import logging |
| 1069 |
import os |
| 1070 |
from random import choice |
| 1071 |
import threading |
| 1072 |
import webapplib |
| 1073 |
|
| 1074 |
logger = logging.getLogger(__name__) |
| 1075 |
root = logging.getLogger() |
| 1076 |
root.setLevel(logging.DEBUG) |
| 1077 |
|
| 1078 |
class Request: |
| 1079 |
""" |
| 1080 |
A simple dummy request class which just holds dummy HTTP request method, |
| 1081 |
client IP address and client username |
| 1082 |
""" |
| 1083 |
def __init__(self, method, ip, user): |
| 1084 |
self.method = method |
| 1085 |
self.ip = ip |
| 1086 |
self.user = user |
| 1087 |
|
| 1088 |
# A dummy set of requests which will be used in the simulation - we'll just pick |
| 1089 |
# from this list randomly. Note that all GET requests are from 192.168.2.XXX |
| 1090 |
# addresses, whereas POST requests are from 192.16.3.XXX addresses. Three users |
| 1091 |
# are represented in the sample requests. |
| 1092 |
|
| 1093 |
REQUESTS = [ |
| 1094 |
Request('GET', '192.168.2.20', 'jim'), |
| 1095 |
Request('POST', '192.168.3.20', 'fred'), |
| 1096 |
Request('GET', '192.168.2.21', 'sheila'), |
| 1097 |
Request('POST', '192.168.3.21', 'jim'), |
| 1098 |
Request('GET', '192.168.2.22', 'fred'), |
| 1099 |
Request('POST', '192.168.3.22', 'sheila'), |
| 1100 |
] |
| 1101 |
|
| 1102 |
# Note that the format string includes references to request context information |
| 1103 |
# such as HTTP method, client IP and username |
| 1104 |
|
| 1105 |
formatter = logging.Formatter('%(threadName)-11s %(appName)s %(name)-9s %(user)-6s %(ip)s %(method)-4s %(message)s') |
| 1106 |
|
| 1107 |
# Create our context variables. These will be filled at the start of request |
| 1108 |
# processing, and used in the logging that happens during that processing |
| 1109 |
|
| 1110 |
ctx_request = ContextVar('request') |
| 1111 |
ctx_appname = ContextVar('appname') |
| 1112 |
|
| 1113 |
class InjectingFilter(logging.Filter): |
| 1114 |
""" |
| 1115 |
A filter which injects context-specific information into logs and ensures |
| 1116 |
that only information for a specific webapp is included in its log |
| 1117 |
""" |
| 1118 |
def __init__(self, app): |
| 1119 |
self.app = app |
| 1120 |
|
| 1121 |
def filter(self, record): |
| 1122 |
request = ctx_request.get() |
| 1123 |
record.method = request.method |
| 1124 |
record.ip = request.ip |
| 1125 |
record.user = request.user |
| 1126 |
record.appName = appName = ctx_appname.get() |
| 1127 |
return appName == self.app.name |
| 1128 |
|
| 1129 |
class WebApp: |
| 1130 |
""" |
| 1131 |
A dummy web application class which has its own handler and filter for a |
| 1132 |
webapp-specific log. |
| 1133 |
""" |
| 1134 |
def __init__(self, name): |
| 1135 |
self.name = name |
| 1136 |
handler = logging.FileHandler(name + '.log', 'w') |
| 1137 |
f = InjectingFilter(self) |
| 1138 |
handler.setFormatter(formatter) |
| 1139 |
handler.addFilter(f) |
| 1140 |
root.addHandler(handler) |
| 1141 |
self.num_requests = 0 |
| 1142 |
|
| 1143 |
def process_request(self, request): |
| 1144 |
""" |
| 1145 |
This is the dummy method for processing a request. It's called on a |
| 1146 |
different thread for every request. We store the context information into |
| 1147 |
the context vars before doing anything else. |
| 1148 |
""" |
| 1149 |
ctx_request.set(request) |
| 1150 |
ctx_appname.set(self.name) |
| 1151 |
self.num_requests += 1 |
| 1152 |
logger.debug('Request processing started') |
| 1153 |
webapplib.useful() |
| 1154 |
logger.debug('Request processing finished') |
| 1155 |
|
| 1156 |
def main(): |
| 1157 |
fn = os.path.splitext(os.path.basename(__file__))[0] |
| 1158 |
adhf = argparse.ArgumentDefaultsHelpFormatter |
| 1159 |
ap = argparse.ArgumentParser(formatter_class=adhf, prog=fn, |
| 1160 |
description='Simulate a couple of web ' |
| 1161 |
'applications handling some ' |
| 1162 |
'requests, showing how request ' |
| 1163 |
'context can be used to ' |
| 1164 |
'populate logs') |
| 1165 |
aa = ap.add_argument |
| 1166 |
aa('--count', '-c', type=int, default=100, help='How many requests to simulate') |
| 1167 |
options = ap.parse_args() |
| 1168 |
|
| 1169 |
# Create the dummy webapps and put them in a list which we can use to select |
| 1170 |
# from randomly |
| 1171 |
app1 = WebApp('app1') |
| 1172 |
app2 = WebApp('app2') |
| 1173 |
apps = [app1, app2] |
| 1174 |
threads = [] |
| 1175 |
# Add a common handler which will capture all events |
| 1176 |
handler = logging.FileHandler('app.log', 'w') |
| 1177 |
handler.setFormatter(formatter) |
| 1178 |
root.addHandler(handler) |
| 1179 |
|
| 1180 |
# Generate calls to process requests |
| 1181 |
for i in range(options.count): |
| 1182 |
try: |
| 1183 |
# Pick an app at random and a request for it to process |
| 1184 |
app = choice(apps) |
| 1185 |
request = choice(REQUESTS) |
| 1186 |
# Process the request in its own thread |
| 1187 |
t = threading.Thread(target=app.process_request, args=(request,)) |
| 1188 |
threads.append(t) |
| 1189 |
t.start() |
| 1190 |
except KeyboardInterrupt: |
| 1191 |
break |
| 1192 |
|
| 1193 |
# Wait for the threads to terminate |
| 1194 |
for t in threads: |
| 1195 |
t.join() |
| 1196 |
|
| 1197 |
for app in apps: |
| 1198 |
print('%s processed %s requests' % (app.name, app.num_requests)) |
| 1199 |
|
| 1200 |
if __name__ == '__main__': |
| 1201 |
main() |
| 1202 |
|
| 1203 |
If you run the above, you should find that roughly half the requests go |
| 1204 |
into :file:`app1.log` and the rest into :file:`app2.log`, and the all the requests are |
| 1205 |
logged to :file:`app.log`. Each webapp-specific log will contain only log entries for |
| 1206 |
only that webapp, and the request information will be displayed consistently in the |
| 1207 |
log (i.e. the information in each dummy request will always appear together in a log |
| 1208 |
line). This is illustrated by the following shell output: |
| 1209 |
|
| 1210 |
.. code-block:: shell |
| 1211 |
|
| 1212 |
~/logging-contextual-webapp$ python main.py |
| 1213 |
app1 processed 51 requests |
| 1214 |
app2 processed 49 requests |
| 1215 |
~/logging-contextual-webapp$ wc -l *.log |
| 1216 |
153 app1.log |
| 1217 |
147 app2.log |
| 1218 |
300 app.log |
| 1219 |
600 total |
| 1220 |
~/logging-contextual-webapp$ head -3 app1.log |
| 1221 |
Thread-3 (process_request) app1 __main__ jim 192.168.3.21 POST Request processing started |
| 1222 |
Thread-3 (process_request) app1 webapplib jim 192.168.3.21 POST Hello from webapplib! |
| 1223 |
Thread-5 (process_request) app1 __main__ jim 192.168.3.21 POST Request processing started |
| 1224 |
~/logging-contextual-webapp$ head -3 app2.log |
| 1225 |
Thread-1 (process_request) app2 __main__ sheila 192.168.2.21 GET Request processing started |
| 1226 |
Thread-1 (process_request) app2 webapplib sheila 192.168.2.21 GET Hello from webapplib! |
| 1227 |
Thread-2 (process_request) app2 __main__ jim 192.168.2.20 GET Request processing started |
| 1228 |
~/logging-contextual-webapp$ head app.log |
| 1229 |
Thread-1 (process_request) app2 __main__ sheila 192.168.2.21 GET Request processing started |
| 1230 |
Thread-1 (process_request) app2 webapplib sheila 192.168.2.21 GET Hello from webapplib! |
| 1231 |
Thread-2 (process_request) app2 __main__ jim 192.168.2.20 GET Request processing started |
| 1232 |
Thread-3 (process_request) app1 __main__ jim 192.168.3.21 POST Request processing started |
| 1233 |
Thread-2 (process_request) app2 webapplib jim 192.168.2.20 GET Hello from webapplib! |
| 1234 |
Thread-3 (process_request) app1 webapplib jim 192.168.3.21 POST Hello from webapplib! |
| 1235 |
Thread-4 (process_request) app2 __main__ fred 192.168.2.22 GET Request processing started |
| 1236 |
Thread-5 (process_request) app1 __main__ jim 192.168.3.21 POST Request processing started |
| 1237 |
Thread-4 (process_request) app2 webapplib fred 192.168.2.22 GET Hello from webapplib! |
| 1238 |
Thread-6 (process_request) app1 __main__ jim 192.168.3.21 POST Request processing started |
| 1239 |
~/logging-contextual-webapp$ grep app1 app1.log | wc -l |
| 1240 |
153 |
| 1241 |
~/logging-contextual-webapp$ grep app2 app2.log | wc -l |
| 1242 |
147 |
| 1243 |
~/logging-contextual-webapp$ grep app1 app.log | wc -l |
| 1244 |
153 |
| 1245 |
~/logging-contextual-webapp$ grep app2 app.log | wc -l |
| 1246 |
147 |
| 1247 |
|
| 1248 |
|
| 1249 |
Imparting contextual information in handlers |
| 1250 |
-------------------------------------------- |
| 1251 |
|
| 1252 |
Each :class:`~Handler` has its own chain of filters. |
| 1253 |
If you want to add contextual information to a :class:`LogRecord` without leaking |
| 1254 |
it to other handlers, you can use a filter that returns |
| 1255 |
a new :class:`~LogRecord` instead of modifying it in-place, as shown in the following script:: |
| 1256 |
|
| 1257 |
import copy |
| 1258 |
import logging |
| 1259 |
|
| 1260 |
def filter(record: logging.LogRecord): |
| 1261 |
record = copy.copy(record) |
| 1262 |
record.user = 'jim' |
| 1263 |
return record |
| 1264 |
|
| 1265 |
if __name__ == '__main__': |
| 1266 |
logger = logging.getLogger() |
| 1267 |
logger.setLevel(logging.INFO) |
| 1268 |
handler = logging.StreamHandler() |
| 1269 |
formatter = logging.Formatter('%(message)s from %(user)-8s') |
| 1270 |
handler.setFormatter(formatter) |
| 1271 |
handler.addFilter(filter) |
| 1272 |
logger.addHandler(handler) |
| 1273 |
|
| 1274 |
logger.info('A log message') |
| 1275 |
|
| 1276 |
.. _multiple-processes: |
| 1277 |
|
| 1278 |
Logging to a single file from multiple processes |
| 1279 |
------------------------------------------------ |
| 1280 |
|
| 1281 |
Although logging is thread-safe, and logging to a single file from multiple |
| 1282 |
threads in a single process *is* supported, logging to a single file from |
| 1283 |
*multiple processes* is *not* supported, because there is no standard way to |
| 1284 |
serialize access to a single file across multiple processes in Python. If you |
| 1285 |
need to log to a single file from multiple processes, one way of doing this is |
| 1286 |
to have all the processes log to a :class:`~handlers.SocketHandler`, and have a |
| 1287 |
separate process which implements a socket server which reads from the socket |
| 1288 |
and logs to file. (If you prefer, you can dedicate one thread in one of the |
| 1289 |
existing processes to perform this function.) |
| 1290 |
:ref:`This section <network-logging>` documents this approach in more detail and |
| 1291 |
includes a working socket receiver which can be used as a starting point for you |
| 1292 |
to adapt in your own applications. |
| 1293 |
|
| 1294 |
You could also write your own handler which uses the :class:`~multiprocessing.Lock` |
| 1295 |
class from the :mod:`multiprocessing` module to serialize access to the |
| 1296 |
file from your processes. The stdlib :class:`FileHandler` and subclasses do |
| 1297 |
not make use of :mod:`multiprocessing`. |
| 1298 |
|
| 1299 |
.. currentmodule:: logging.handlers |
| 1300 |
|
| 1301 |
Alternatively, you can use a ``Queue`` and a :class:`QueueHandler` to send |
| 1302 |
all logging events to one of the processes in your multi-process application. |
| 1303 |
The following example script demonstrates how you can do this; in the example |
| 1304 |
a separate listener process listens for events sent by other processes and logs |
| 1305 |
them according to its own logging configuration. Although the example only |
| 1306 |
demonstrates one way of doing it (for example, you may want to use a listener |
| 1307 |
thread rather than a separate listener process -- the implementation would be |
| 1308 |
analogous) it does allow for completely different logging configurations for |
| 1309 |
the listener and the other processes in your application, and can be used as |
| 1310 |
the basis for code meeting your own specific requirements:: |
| 1311 |
|
| 1312 |
# You'll need these imports in your own code |
| 1313 |
import logging |
| 1314 |
import logging.handlers |
| 1315 |
import multiprocessing |
| 1316 |
|
| 1317 |
# Next two import lines for this demo only |
| 1318 |
from random import choice, random |
| 1319 |
import time |
| 1320 |
|
| 1321 |
# |
| 1322 |
# Because you'll want to define the logging configurations for listener and workers, the |
| 1323 |
# listener and worker process functions take a configurer parameter which is a callable |
| 1324 |
# for configuring logging for that process. These functions are also passed the queue, |
| 1325 |
# which they use for communication. |
| 1326 |
# |
| 1327 |
# In practice, you can configure the listener however you want, but note that in this |
| 1328 |
# simple example, the listener does not apply level or filter logic to received records. |
| 1329 |
# In practice, you would probably want to do this logic in the worker processes, to avoid |
| 1330 |
# sending events which would be filtered out between processes. |
| 1331 |
# |
| 1332 |
# The size of the rotated files is made small so you can see the results easily. |
| 1333 |
def listener_configurer(): |
| 1334 |
root = logging.getLogger() |
| 1335 |
h = logging.handlers.RotatingFileHandler('mptest.log', 'a', 300, 10) |
| 1336 |
f = logging.Formatter('%(asctime)s %(processName)-10s %(name)s %(levelname)-8s %(message)s') |
| 1337 |
h.setFormatter(f) |
| 1338 |
root.addHandler(h) |
| 1339 |
|
| 1340 |
# This is the listener process top-level loop: wait for logging events |
| 1341 |
# (LogRecords)on the queue and handle them, quit when you get a None for a |
| 1342 |
# LogRecord. |
| 1343 |
def listener_process(queue, configurer): |
| 1344 |
configurer() |
| 1345 |
while True: |
| 1346 |
try: |
| 1347 |
record = queue.get() |
| 1348 |
if record is None: # We send this as a sentinel to tell the listener to quit. |
| 1349 |
break |
| 1350 |
logger = logging.getLogger(record.name) |
| 1351 |
logger.handle(record) # No level or filter logic applied - just do it! |
| 1352 |
except Exception: |
| 1353 |
import sys, traceback |
| 1354 |
print('Whoops! Problem:', file=sys.stderr) |
| 1355 |
traceback.print_exc(file=sys.stderr) |
| 1356 |
|
| 1357 |
# Arrays used for random selections in this demo |
| 1358 |
|
| 1359 |
LEVELS = [logging.DEBUG, logging.INFO, logging.WARNING, |
| 1360 |
logging.ERROR, logging.CRITICAL] |
| 1361 |
|
| 1362 |
LOGGERS = ['a.b.c', 'd.e.f'] |
| 1363 |
|
| 1364 |
MESSAGES = [ |
| 1365 |
'Random message #1', |
| 1366 |
'Random message #2', |
| 1367 |
'Random message #3', |
| 1368 |
] |
| 1369 |
|
| 1370 |
# The worker configuration is done at the start of the worker process run. |
| 1371 |
# Note that on Windows you can't rely on fork semantics, so each process |
| 1372 |
# will run the logging configuration code when it starts. |
| 1373 |
def worker_configurer(queue): |
| 1374 |
h = logging.handlers.QueueHandler(queue) # Just the one handler needed |
| 1375 |
root = logging.getLogger() |
| 1376 |
root.addHandler(h) |
| 1377 |
# send all messages, for demo; no other level or filter logic applied. |
| 1378 |
root.setLevel(logging.DEBUG) |
| 1379 |
|
| 1380 |
# This is the worker process top-level loop, which just logs ten events with |
| 1381 |
# random intervening delays before terminating. |
| 1382 |
# The print messages are just so you know it's doing something! |
| 1383 |
def worker_process(queue, configurer): |
| 1384 |
configurer(queue) |
| 1385 |
name = multiprocessing.current_process().name |
| 1386 |
print('Worker started: %s' % name) |
| 1387 |
for i in range(10): |
| 1388 |
time.sleep(random()) |
| 1389 |
logger = logging.getLogger(choice(LOGGERS)) |
| 1390 |
level = choice(LEVELS) |
| 1391 |
message = choice(MESSAGES) |
| 1392 |
logger.log(level, message) |
| 1393 |
print('Worker finished: %s' % name) |
| 1394 |
|
| 1395 |
# Here's where the demo gets orchestrated. Create the queue, create and start |
| 1396 |
# the listener, create ten workers and start them, wait for them to finish, |
| 1397 |
# then send a None to the queue to tell the listener to finish. |
| 1398 |
def main(): |
| 1399 |
queue = multiprocessing.Queue(-1) |
| 1400 |
listener = multiprocessing.Process(target=listener_process, |
| 1401 |
args=(queue, listener_configurer)) |
| 1402 |
listener.start() |
| 1403 |
workers = [] |
| 1404 |
for i in range(10): |
| 1405 |
worker = multiprocessing.Process(target=worker_process, |
| 1406 |
args=(queue, worker_configurer)) |
| 1407 |
workers.append(worker) |
| 1408 |
worker.start() |
| 1409 |
for w in workers: |
| 1410 |
w.join() |
| 1411 |
queue.put_nowait(None) |
| 1412 |
listener.join() |
| 1413 |
|
| 1414 |
if __name__ == '__main__': |
| 1415 |
main() |
| 1416 |
|
| 1417 |
A variant of the above script keeps the logging in the main process, in a |
| 1418 |
separate thread:: |
| 1419 |
|
| 1420 |
import logging |
| 1421 |
import logging.config |
| 1422 |
import logging.handlers |
| 1423 |
from multiprocessing import Process, Queue |
| 1424 |
import random |
| 1425 |
import threading |
| 1426 |
import time |
| 1427 |
|
| 1428 |
def logger_thread(q): |
| 1429 |
while True: |
| 1430 |
record = q.get() |
| 1431 |
if record is None: |
| 1432 |
break |
| 1433 |
logger = logging.getLogger(record.name) |
| 1434 |
logger.handle(record) |
| 1435 |
|
| 1436 |
|
| 1437 |
def worker_process(q): |
| 1438 |
qh = logging.handlers.QueueHandler(q) |
| 1439 |
root = logging.getLogger() |
| 1440 |
root.setLevel(logging.DEBUG) |
| 1441 |
root.addHandler(qh) |
| 1442 |
levels = [logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, |
| 1443 |
logging.CRITICAL] |
| 1444 |
loggers = ['foo', 'foo.bar', 'foo.bar.baz', |
| 1445 |
'spam', 'spam.ham', 'spam.ham.eggs'] |
| 1446 |
for i in range(100): |
| 1447 |
lvl = random.choice(levels) |
| 1448 |
logger = logging.getLogger(random.choice(loggers)) |
| 1449 |
logger.log(lvl, 'Message no. %d', i) |
| 1450 |
|
| 1451 |
if __name__ == '__main__': |
| 1452 |
q = Queue() |
| 1453 |
d = { |
| 1454 |
'version': 1, |
| 1455 |
'formatters': { |
| 1456 |
'detailed': { |
| 1457 |
'class': 'logging.Formatter', |
| 1458 |
'format': '%(asctime)s %(name)-15s %(levelname)-8s %(processName)-10s %(message)s' |
| 1459 |
} |
| 1460 |
}, |
| 1461 |
'handlers': { |
| 1462 |
'console': { |
| 1463 |
'class': 'logging.StreamHandler', |
| 1464 |
'level': 'INFO', |
| 1465 |
}, |
| 1466 |
'file': { |
| 1467 |
'class': 'logging.FileHandler', |
| 1468 |
'filename': 'mplog.log', |
| 1469 |
'mode': 'w', |
| 1470 |
'formatter': 'detailed', |
| 1471 |
}, |
| 1472 |
'foofile': { |
| 1473 |
'class': 'logging.FileHandler', |
| 1474 |
'filename': 'mplog-foo.log', |
| 1475 |
'mode': 'w', |
| 1476 |
'formatter': 'detailed', |
| 1477 |
}, |
| 1478 |
'errors': { |
| 1479 |
'class': 'logging.FileHandler', |
| 1480 |
'filename': 'mplog-errors.log', |
| 1481 |
'mode': 'w', |
| 1482 |
'level': 'ERROR', |
| 1483 |
'formatter': 'detailed', |
| 1484 |
}, |
| 1485 |
}, |
| 1486 |
'loggers': { |
| 1487 |
'foo': { |
| 1488 |
'handlers': ['foofile'] |
| 1489 |
} |
| 1490 |
}, |
| 1491 |
'root': { |
| 1492 |
'level': 'DEBUG', |
| 1493 |
'handlers': ['console', 'file', 'errors'] |
| 1494 |
}, |
| 1495 |
} |
| 1496 |
workers = [] |
| 1497 |
for i in range(5): |
| 1498 |
wp = Process(target=worker_process, name='worker %d' % (i + 1), args=(q,)) |
| 1499 |
workers.append(wp) |
| 1500 |
wp.start() |
| 1501 |
logging.config.dictConfig(d) |
| 1502 |
lp = threading.Thread(target=logger_thread, args=(q,)) |
| 1503 |
lp.start() |
| 1504 |
# At this point, the main process could do some useful work of its own |
| 1505 |
# Once it's done that, it can wait for the workers to terminate... |
| 1506 |
for wp in workers: |
| 1507 |
wp.join() |
| 1508 |
# And now tell the logging thread to finish up, too |
| 1509 |
q.put(None) |
| 1510 |
lp.join() |
| 1511 |
|
| 1512 |
This variant shows how you can e.g. apply configuration for particular loggers |
| 1513 |
- e.g. the ``foo`` logger has a special handler which stores all events in the |
| 1514 |
``foo`` subsystem in a file ``mplog-foo.log``. This will be used by the logging |
| 1515 |
machinery in the main process (even though the logging events are generated in |
| 1516 |
the worker processes) to direct the messages to the appropriate destinations. |
| 1517 |
|
| 1518 |
Using concurrent.futures.ProcessPoolExecutor |
| 1519 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1520 |
|
| 1521 |
If you want to use :class:`concurrent.futures.ProcessPoolExecutor` to start |
| 1522 |
your worker processes, you need to create the queue slightly differently. |
| 1523 |
Instead of |
| 1524 |
|
| 1525 |
.. code-block:: python |
| 1526 |
|
| 1527 |
queue = multiprocessing.Queue(-1) |
| 1528 |
|
| 1529 |
you should use |
| 1530 |
|
| 1531 |
.. code-block:: python |
| 1532 |
|
| 1533 |
queue = multiprocessing.Manager().Queue(-1) # also works with the examples above |
| 1534 |
|
| 1535 |
and you can then replace the worker creation from this:: |
| 1536 |
|
| 1537 |
workers = [] |
| 1538 |
for i in range(10): |
| 1539 |
worker = multiprocessing.Process(target=worker_process, |
| 1540 |
args=(queue, worker_configurer)) |
| 1541 |
workers.append(worker) |
| 1542 |
worker.start() |
| 1543 |
for w in workers: |
| 1544 |
w.join() |
| 1545 |
|
| 1546 |
to this (remembering to first import :mod:`concurrent.futures`):: |
| 1547 |
|
| 1548 |
with concurrent.futures.ProcessPoolExecutor(max_workers=10) as executor: |
| 1549 |
for i in range(10): |
| 1550 |
executor.submit(worker_process, queue, worker_configurer) |
| 1551 |
|
| 1552 |
Deploying web applications using Gunicorn and uWSGI |
| 1553 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1554 |
|
| 1555 |
When deploying web applications using `Gunicorn <https://gunicorn.org/>`_ or `uWSGI |
| 1556 |
<https://uwsgi-docs.readthedocs.io/en/latest/>`_ (or similar), multiple worker |
| 1557 |
processes are created to handle client requests. In such environments, avoid creating |
| 1558 |
file-based handlers directly in your web application. Instead, use a |
| 1559 |
:class:`SocketHandler` to log from the web application to a listener in a separate |
| 1560 |
process. This can be set up using a process management tool such as Supervisor - see |
| 1561 |
`Running a logging socket listener in production`_ for more details. |
| 1562 |
|
| 1563 |
|
| 1564 |
Using file rotation |
| 1565 |
------------------- |
| 1566 |
|
| 1567 |
.. sectionauthor:: Doug Hellmann, Vinay Sajip (changes) |
| 1568 |
.. (see <https://pymotw.com/3/logging/>) |
| 1569 |
|
| 1570 |
Sometimes you want to let a log file grow to a certain size, then open a new |
| 1571 |
file and log to that. You may want to keep a certain number of these files, and |
| 1572 |
when that many files have been created, rotate the files so that the number of |
| 1573 |
files and the size of the files both remain bounded. For this usage pattern, the |
| 1574 |
logging package provides a :class:`RotatingFileHandler`:: |
| 1575 |
|
| 1576 |
import glob |
| 1577 |
import logging |
| 1578 |
import logging.handlers |
| 1579 |
|
| 1580 |
LOG_FILENAME = 'logging_rotatingfile_example.out' |
| 1581 |
|
| 1582 |
# Set up a specific logger with our desired output level |
| 1583 |
my_logger = logging.getLogger('MyLogger') |
| 1584 |
my_logger.setLevel(logging.DEBUG) |
| 1585 |
|
| 1586 |
# Add the log message handler to the logger |
| 1587 |
handler = logging.handlers.RotatingFileHandler( |
| 1588 |
LOG_FILENAME, maxBytes=20, backupCount=5) |
| 1589 |
|
| 1590 |
my_logger.addHandler(handler) |
| 1591 |
|
| 1592 |
# Log some messages |
| 1593 |
for i in range(20): |
| 1594 |
my_logger.debug('i = %d' % i) |
| 1595 |
|
| 1596 |
# See what files are created |
| 1597 |
logfiles = glob.glob('%s*' % LOG_FILENAME) |
| 1598 |
|
| 1599 |
for filename in logfiles: |
| 1600 |
print(filename) |
| 1601 |
|
| 1602 |
The result should be 6 separate files, each with part of the log history for the |
| 1603 |
application: |
| 1604 |
|
| 1605 |
.. code-block:: none |
| 1606 |
|
| 1607 |
logging_rotatingfile_example.out |
| 1608 |
logging_rotatingfile_example.out.1 |
| 1609 |
logging_rotatingfile_example.out.2 |
| 1610 |
logging_rotatingfile_example.out.3 |
| 1611 |
logging_rotatingfile_example.out.4 |
| 1612 |
logging_rotatingfile_example.out.5 |
| 1613 |
|
| 1614 |
The most current file is always :file:`logging_rotatingfile_example.out`, |
| 1615 |
and each time it reaches the size limit it is renamed with the suffix |
| 1616 |
``.1``. Each of the existing backup files is renamed to increment the suffix |
| 1617 |
(``.1`` becomes ``.2``, etc.) and the ``.6`` file is erased. |
| 1618 |
|
| 1619 |
Obviously this example sets the log length much too small as an extreme |
| 1620 |
example. You would want to set *maxBytes* to an appropriate value. |
| 1621 |
|
| 1622 |
.. currentmodule:: logging |
| 1623 |
|
| 1624 |
.. _format-styles: |
| 1625 |
|
| 1626 |
Use of alternative formatting styles |
| 1627 |
------------------------------------ |
| 1628 |
|
| 1629 |
When logging was added to the Python standard library, the only way of |
| 1630 |
formatting messages with variable content was to use the %-formatting |
| 1631 |
method. Since then, Python has gained two new formatting approaches: |
| 1632 |
:class:`string.Template` (added in Python 2.4) and :meth:`str.format` |
| 1633 |
(added in Python 2.6). |
| 1634 |
|
| 1635 |
Logging (as of 3.2) provides improved support for these two additional |
| 1636 |
formatting styles. The :class:`Formatter` class been enhanced to take an |
| 1637 |
additional, optional keyword parameter named ``style``. This defaults to |
| 1638 |
``'%'``, but other possible values are ``'{'`` and ``'$'``, which correspond |
| 1639 |
to the other two formatting styles. Backwards compatibility is maintained by |
| 1640 |
default (as you would expect), but by explicitly specifying a style parameter, |
| 1641 |
you get the ability to specify format strings which work with |
| 1642 |
:meth:`str.format` or :class:`string.Template`. Here's an example console |
| 1643 |
session to show the possibilities: |
| 1644 |
|
| 1645 |
.. code-block:: pycon |
| 1646 |
|
| 1647 |
>>> import logging |
| 1648 |
>>> root = logging.getLogger() |
| 1649 |
>>> root.setLevel(logging.DEBUG) |
| 1650 |
>>> handler = logging.StreamHandler() |
| 1651 |
>>> bf = logging.Formatter('{asctime} {name} {levelname:8s} {message}', |
| 1652 |
... style='{') |
| 1653 |
>>> handler.setFormatter(bf) |
| 1654 |
>>> root.addHandler(handler) |
| 1655 |
>>> logger = logging.getLogger('foo.bar') |
| 1656 |
>>> logger.debug('This is a DEBUG message') |
| 1657 |
2010-10-28 15:11:55,341 foo.bar DEBUG This is a DEBUG message |
| 1658 |
>>> logger.critical('This is a CRITICAL message') |
| 1659 |
2010-10-28 15:12:11,526 foo.bar CRITICAL This is a CRITICAL message |
| 1660 |
>>> df = logging.Formatter('$asctime $name ${levelname} $message', |
| 1661 |
... style='$') |
| 1662 |
>>> handler.setFormatter(df) |
| 1663 |
>>> logger.debug('This is a DEBUG message') |
| 1664 |
2010-10-28 15:13:06,924 foo.bar DEBUG This is a DEBUG message |
| 1665 |
>>> logger.critical('This is a CRITICAL message') |
| 1666 |
2010-10-28 15:13:11,494 foo.bar CRITICAL This is a CRITICAL message |
| 1667 |
>>> |
| 1668 |
|
| 1669 |
Note that the formatting of logging messages for final output to logs is |
| 1670 |
completely independent of how an individual logging message is constructed. |
| 1671 |
That can still use %-formatting, as shown here:: |
| 1672 |
|
| 1673 |
>>> logger.error('This is an%s %s %s', 'other,', 'ERROR,', 'message') |
| 1674 |
2010-10-28 15:19:29,833 foo.bar ERROR This is another, ERROR, message |
| 1675 |
>>> |
| 1676 |
|
| 1677 |
Logging calls (``logger.debug()``, ``logger.info()`` etc.) only take |
| 1678 |
positional parameters for the actual logging message itself, with keyword |
| 1679 |
parameters used only for determining options for how to handle the actual |
| 1680 |
logging call (e.g. the ``exc_info`` keyword parameter to indicate that |
| 1681 |
traceback information should be logged, or the ``extra`` keyword parameter |
| 1682 |
to indicate additional contextual information to be added to the log). So |
| 1683 |
you cannot directly make logging calls using :meth:`str.format` or |
| 1684 |
:class:`string.Template` syntax, because internally the logging package |
| 1685 |
uses %-formatting to merge the format string and the variable arguments. |
| 1686 |
There would be no changing this while preserving backward compatibility, since |
| 1687 |
all logging calls which are out there in existing code will be using %-format |
| 1688 |
strings. |
| 1689 |
|
| 1690 |
There is, however, a way that you can use {}- and $- formatting to construct |
| 1691 |
your individual log messages. Recall that for a message you can use an |
| 1692 |
arbitrary object as a message format string, and that the logging package will |
| 1693 |
call ``str()`` on that object to get the actual format string. Consider the |
| 1694 |
following two classes:: |
| 1695 |
|
| 1696 |
class BraceMessage: |
| 1697 |
def __init__(self, fmt, /, *args, **kwargs): |
| 1698 |
self.fmt = fmt |
| 1699 |
self.args = args |
| 1700 |
self.kwargs = kwargs |
| 1701 |
|
| 1702 |
def __str__(self): |
| 1703 |
return self.fmt.format(*self.args, **self.kwargs) |
| 1704 |
|
| 1705 |
class DollarMessage: |
| 1706 |
def __init__(self, fmt, /, **kwargs): |
| 1707 |
self.fmt = fmt |
| 1708 |
self.kwargs = kwargs |
| 1709 |
|
| 1710 |
def __str__(self): |
| 1711 |
from string import Template |
| 1712 |
return Template(self.fmt).substitute(**self.kwargs) |
| 1713 |
|
| 1714 |
Either of these can be used in place of a format string, to allow {}- or |
| 1715 |
$-formatting to be used to build the actual "message" part which appears in the |
| 1716 |
formatted log output in place of "%(message)s" or "{message}" or "$message". |
| 1717 |
It's a little unwieldy to use the class names whenever you want to log |
| 1718 |
something, but it's quite palatable if you use an alias such as __ (double |
| 1719 |
underscore --- not to be confused with _, the single underscore used as a |
| 1720 |
synonym/alias for :func:`gettext.gettext` or its brethren). |
| 1721 |
|
| 1722 |
The above classes are not included in Python, though they're easy enough to |
| 1723 |
copy and paste into your own code. They can be used as follows (assuming that |
| 1724 |
they're declared in a module called ``wherever``): |
| 1725 |
|
| 1726 |
.. code-block:: pycon |
| 1727 |
|
| 1728 |
>>> from wherever import BraceMessage as __ |
| 1729 |
>>> print(__('Message with {0} {name}', 2, name='placeholders')) |
| 1730 |
Message with 2 placeholders |
| 1731 |
>>> class Point: pass |
| 1732 |
... |
| 1733 |
>>> p = Point() |
| 1734 |
>>> p.x = 0.5 |
| 1735 |
>>> p.y = 0.5 |
| 1736 |
>>> print(__('Message with coordinates: ({point.x:.2f}, {point.y:.2f})', |
| 1737 |
... point=p)) |
| 1738 |
Message with coordinates: (0.50, 0.50) |
| 1739 |
>>> from wherever import DollarMessage as __ |
| 1740 |
>>> print(__('Message with $num $what', num=2, what='placeholders')) |
| 1741 |
Message with 2 placeholders |
| 1742 |
>>> |
| 1743 |
|
| 1744 |
While the above examples use ``print()`` to show how the formatting works, you |
| 1745 |
would of course use ``logger.debug()`` or similar to actually log using this |
| 1746 |
approach. |
| 1747 |
|
| 1748 |
One thing to note is that you pay no significant performance penalty with this |
| 1749 |
approach: the actual formatting happens not when you make the logging call, but |
| 1750 |
when (and if) the logged message is actually about to be output to a log by a |
| 1751 |
handler. So the only slightly unusual thing which might trip you up is that the |
| 1752 |
parentheses go around the format string and the arguments, not just the format |
| 1753 |
string. That's because the __ notation is just syntax sugar for a constructor |
| 1754 |
call to one of the :samp:`{XXX}Message` classes. |
| 1755 |
|
| 1756 |
If you prefer, you can use a :class:`LoggerAdapter` to achieve a similar effect |
| 1757 |
to the above, as in the following example:: |
| 1758 |
|
| 1759 |
import logging |
| 1760 |
|
| 1761 |
class Message: |
| 1762 |
def __init__(self, fmt, args): |
| 1763 |
self.fmt = fmt |
| 1764 |
self.args = args |
| 1765 |
|
| 1766 |
def __str__(self): |
| 1767 |
return self.fmt.format(*self.args) |
| 1768 |
|
| 1769 |
class StyleAdapter(logging.LoggerAdapter): |
| 1770 |
def log(self, level, msg, /, *args, stacklevel=1, **kwargs): |
| 1771 |
if self.isEnabledFor(level): |
| 1772 |
msg, kwargs = self.process(msg, kwargs) |
| 1773 |
self.logger.log(level, Message(msg, args), **kwargs, |
| 1774 |
stacklevel=stacklevel+1) |
| 1775 |
|
| 1776 |
logger = StyleAdapter(logging.getLogger(__name__)) |
| 1777 |
|
| 1778 |
def main(): |
| 1779 |
logger.debug('Hello, {}', 'world!') |
| 1780 |
|
| 1781 |
if __name__ == '__main__': |
| 1782 |
logging.basicConfig(level=logging.DEBUG) |
| 1783 |
main() |
| 1784 |
|
| 1785 |
The above script should log the message ``Hello, world!`` when run with |
| 1786 |
Python 3.8 or later. |
| 1787 |
|
| 1788 |
|
| 1789 |
.. currentmodule:: logging |
| 1790 |
|
| 1791 |
.. _custom-logrecord: |
| 1792 |
|
| 1793 |
Customizing ``LogRecord`` |
| 1794 |
------------------------- |
| 1795 |
|
| 1796 |
Every logging event is represented by a :class:`LogRecord` instance. |
| 1797 |
When an event is logged and not filtered out by a logger's level, a |
| 1798 |
:class:`LogRecord` is created, populated with information about the event and |
| 1799 |
then passed to the handlers for that logger (and its ancestors, up to and |
| 1800 |
including the logger where further propagation up the hierarchy is disabled). |
| 1801 |
Before Python 3.2, there were only two places where this creation was done: |
| 1802 |
|
| 1803 |
* :meth:`Logger.makeRecord`, which is called in the normal process of |
| 1804 |
logging an event. This invoked :class:`LogRecord` directly to create an |
| 1805 |
instance. |
| 1806 |
* :func:`makeLogRecord`, which is called with a dictionary containing |
| 1807 |
attributes to be added to the LogRecord. This is typically invoked when a |
| 1808 |
suitable dictionary has been received over the network (e.g. in pickle form |
| 1809 |
via a :class:`~handlers.SocketHandler`, or in JSON form via an |
| 1810 |
:class:`~handlers.HTTPHandler`). |
| 1811 |
|
| 1812 |
This has usually meant that if you need to do anything special with a |
| 1813 |
:class:`LogRecord`, you've had to do one of the following. |
| 1814 |
|
| 1815 |
* Create your own :class:`Logger` subclass, which overrides |
| 1816 |
:meth:`Logger.makeRecord`, and set it using :func:`~logging.setLoggerClass` |
| 1817 |
before any loggers that you care about are instantiated. |
| 1818 |
* Add a :class:`Filter` to a logger or handler, which does the |
| 1819 |
necessary special manipulation you need when its |
| 1820 |
:meth:`~Filter.filter` method is called. |
| 1821 |
|
| 1822 |
The first approach would be a little unwieldy in the scenario where (say) |
| 1823 |
several different libraries wanted to do different things. Each would attempt |
| 1824 |
to set its own :class:`Logger` subclass, and the one which did this last would |
| 1825 |
win. |
| 1826 |
|
| 1827 |
The second approach works reasonably well for many cases, but does not allow |
| 1828 |
you to e.g. use a specialized subclass of :class:`LogRecord`. Library |
| 1829 |
developers can set a suitable filter on their loggers, but they would have to |
| 1830 |
remember to do this every time they introduced a new logger (which they would |
| 1831 |
do simply by adding new packages or modules and doing :: |
| 1832 |
|
| 1833 |
logger = logging.getLogger(__name__) |
| 1834 |
|
| 1835 |
at module level). It's probably one too many things to think about. Developers |
| 1836 |
could also add the filter to a :class:`~logging.NullHandler` attached to their |
| 1837 |
top-level logger, but this would not be invoked if an application developer |
| 1838 |
attached a handler to a lower-level library logger --- so output from that |
| 1839 |
handler would not reflect the intentions of the library developer. |
| 1840 |
|
| 1841 |
In Python 3.2 and later, :class:`~logging.LogRecord` creation is done through a |
| 1842 |
factory, which you can specify. The factory is just a callable you can set with |
| 1843 |
:func:`~logging.setLogRecordFactory`, and interrogate with |
| 1844 |
:func:`~logging.getLogRecordFactory`. The factory is invoked with the same |
| 1845 |
signature as the :class:`~logging.LogRecord` constructor, as :class:`LogRecord` |
| 1846 |
is the default setting for the factory. |
| 1847 |
|
| 1848 |
This approach allows a custom factory to control all aspects of LogRecord |
| 1849 |
creation. For example, you could return a subclass, or just add some additional |
| 1850 |
attributes to the record once created, using a pattern similar to this:: |
| 1851 |
|
| 1852 |
old_factory = logging.getLogRecordFactory() |
| 1853 |
|
| 1854 |
def record_factory(*args, **kwargs): |
| 1855 |
record = old_factory(*args, **kwargs) |
| 1856 |
record.custom_attribute = 0xdecafbad |
| 1857 |
return record |
| 1858 |
|
| 1859 |
logging.setLogRecordFactory(record_factory) |
| 1860 |
|
| 1861 |
This pattern allows different libraries to chain factories together, and as |
| 1862 |
long as they don't overwrite each other's attributes or unintentionally |
| 1863 |
overwrite the attributes provided as standard, there should be no surprises. |
| 1864 |
However, it should be borne in mind that each link in the chain adds run-time |
| 1865 |
overhead to all logging operations, and the technique should only be used when |
| 1866 |
the use of a :class:`Filter` does not provide the desired result. |
| 1867 |
|
| 1868 |
.. currentmodule:: logging.handlers |
| 1869 |
|
| 1870 |
.. _zeromq-handlers: |
| 1871 |
|
| 1872 |
Subclassing QueueHandler and QueueListener- a ZeroMQ example |
| 1873 |
------------------------------------------------------------ |
| 1874 |
|
| 1875 |
Subclass ``QueueHandler`` |
| 1876 |
^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1877 |
|
| 1878 |
You can use a :class:`QueueHandler` subclass to send messages to other kinds |
| 1879 |
of queues, for example a ZeroMQ 'publish' socket. In the example below,the |
| 1880 |
socket is created separately and passed to the handler (as its 'queue'):: |
| 1881 |
|
| 1882 |
import zmq # using pyzmq, the Python binding for ZeroMQ |
| 1883 |
import json # for serializing records portably |
| 1884 |
|
| 1885 |
ctx = zmq.Context() |
| 1886 |
sock = zmq.Socket(ctx, zmq.PUB) # or zmq.PUSH, or other suitable value |
| 1887 |
sock.bind('tcp://*:5556') # or wherever |
| 1888 |
|
| 1889 |
class ZeroMQSocketHandler(QueueHandler): |
| 1890 |
def enqueue(self, record): |
| 1891 |
self.queue.send_json(record.__dict__) |
| 1892 |
|
| 1893 |
|
| 1894 |
handler = ZeroMQSocketHandler(sock) |
| 1895 |
|
| 1896 |
|
| 1897 |
Of course there are other ways of organizing this, for example passing in the |
| 1898 |
data needed by the handler to create the socket:: |
| 1899 |
|
| 1900 |
class ZeroMQSocketHandler(QueueHandler): |
| 1901 |
def __init__(self, uri, socktype=zmq.PUB, ctx=None): |
| 1902 |
self.ctx = ctx or zmq.Context() |
| 1903 |
socket = zmq.Socket(self.ctx, socktype) |
| 1904 |
socket.bind(uri) |
| 1905 |
super().__init__(socket) |
| 1906 |
|
| 1907 |
def enqueue(self, record): |
| 1908 |
self.queue.send_json(record.__dict__) |
| 1909 |
|
| 1910 |
def close(self): |
| 1911 |
self.queue.close() |
| 1912 |
|
| 1913 |
|
| 1914 |
Subclass ``QueueListener`` |
| 1915 |
^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1916 |
|
| 1917 |
You can also subclass :class:`QueueListener` to get messages from other kinds |
| 1918 |
of queues, for example a ZeroMQ 'subscribe' socket. Here's an example:: |
| 1919 |
|
| 1920 |
class ZeroMQSocketListener(QueueListener): |
| 1921 |
def __init__(self, uri, /, *handlers, **kwargs): |
| 1922 |
self.ctx = kwargs.get('ctx') or zmq.Context() |
| 1923 |
socket = zmq.Socket(self.ctx, zmq.SUB) |
| 1924 |
socket.setsockopt_string(zmq.SUBSCRIBE, '') # subscribe to everything |
| 1925 |
socket.connect(uri) |
| 1926 |
super().__init__(socket, *handlers, **kwargs) |
| 1927 |
|
| 1928 |
def dequeue(self): |
| 1929 |
msg = self.queue.recv_json() |
| 1930 |
return logging.makeLogRecord(msg) |
| 1931 |
|
| 1932 |
.. _pynng-handlers: |
| 1933 |
|
| 1934 |
Subclassing QueueHandler and QueueListener- a ``pynng`` example |
| 1935 |
--------------------------------------------------------------- |
| 1936 |
|
| 1937 |
In a similar way to the above section, we can implement a listener and handler |
| 1938 |
using :pypi:`pynng`, which is a Python binding to |
| 1939 |
`NNG <https://nng.nanomsg.org/>`_, billed as a spiritual successor to ZeroMQ. |
| 1940 |
The following snippets illustrate -- you can test them in an environment which has |
| 1941 |
``pynng`` installed. Just for variety, we present the listener first. |
| 1942 |
|
| 1943 |
|
| 1944 |
Subclass ``QueueListener`` |
| 1945 |
^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 1946 |
|
| 1947 |
.. code-block:: python |
| 1948 |
|
| 1949 |
# listener.py |
| 1950 |
import json |
| 1951 |
import logging |
| 1952 |
import logging.handlers |
| 1953 |
|
| 1954 |
import pynng |
| 1955 |
|
| 1956 |
DEFAULT_ADDR = "tcp://localhost:13232" |
| 1957 |
|
| 1958 |
interrupted = False |
| 1959 |
|
| 1960 |
class NNGSocketListener(logging.handlers.QueueListener): |
| 1961 |
|
| 1962 |
def __init__(self, uri, /, *handlers, **kwargs): |
| 1963 |
# Have a timeout for interruptability, and open a |
| 1964 |
# subscriber socket |
| 1965 |
socket = pynng.Sub0(listen=uri, recv_timeout=500) |
| 1966 |
# The b'' subscription matches all topics |
| 1967 |
topics = kwargs.pop('topics', None) or b'' |
| 1968 |
socket.subscribe(topics) |
| 1969 |
# We treat the socket as a queue |
| 1970 |
super().__init__(socket, *handlers, **kwargs) |
| 1971 |
|
| 1972 |
def dequeue(self, block): |
| 1973 |
data = None |
| 1974 |
# Keep looping while not interrupted and no data received over the |
| 1975 |
# socket |
| 1976 |
while not interrupted: |
| 1977 |
try: |
| 1978 |
data = self.queue.recv(block=block) |
| 1979 |
break |
| 1980 |
except pynng.Timeout: |
| 1981 |
pass |
| 1982 |
except pynng.Closed: # sometimes happens when you hit Ctrl-C |
| 1983 |
break |
| 1984 |
if data is None: |
| 1985 |
return None |
| 1986 |
# Get the logging event sent from a publisher |
| 1987 |
event = json.loads(data.decode('utf-8')) |
| 1988 |
return logging.makeLogRecord(event) |
| 1989 |
|
| 1990 |
def enqueue_sentinel(self): |
| 1991 |
# Not used in this implementation, as the socket isn't really a |
| 1992 |
# queue |
| 1993 |
pass |
| 1994 |
|
| 1995 |
logging.getLogger('pynng').propagate = False |
| 1996 |
listener = NNGSocketListener(DEFAULT_ADDR, logging.StreamHandler(), topics=b'') |
| 1997 |
listener.start() |
| 1998 |
print('Press Ctrl-C to stop.') |
| 1999 |
try: |
| 2000 |
while True: |
| 2001 |
pass |
| 2002 |
except KeyboardInterrupt: |
| 2003 |
interrupted = True |
| 2004 |
finally: |
| 2005 |
listener.stop() |
| 2006 |
|
| 2007 |
|
| 2008 |
Subclass ``QueueHandler`` |
| 2009 |
^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 2010 |
|
| 2011 |
.. currentmodule:: logging |
| 2012 |
|
| 2013 |
.. code-block:: python |
| 2014 |
|
| 2015 |
# sender.py |
| 2016 |
import json |
| 2017 |
import logging |
| 2018 |
import logging.handlers |
| 2019 |
import time |
| 2020 |
import random |
| 2021 |
|
| 2022 |
import pynng |
| 2023 |
|
| 2024 |
DEFAULT_ADDR = "tcp://localhost:13232" |
| 2025 |
|
| 2026 |
class NNGSocketHandler(logging.handlers.QueueHandler): |
| 2027 |
|
| 2028 |
def __init__(self, uri): |
| 2029 |
socket = pynng.Pub0(dial=uri, send_timeout=500) |
| 2030 |
super().__init__(socket) |
| 2031 |
|
| 2032 |
def enqueue(self, record): |
| 2033 |
# Send the record as UTF-8 encoded JSON |
| 2034 |
d = dict(record.__dict__) |
| 2035 |
data = json.dumps(d) |
| 2036 |
self.queue.send(data.encode('utf-8')) |
| 2037 |
|
| 2038 |
def close(self): |
| 2039 |
self.queue.close() |
| 2040 |
|
| 2041 |
logging.getLogger('pynng').propagate = False |
| 2042 |
handler = NNGSocketHandler(DEFAULT_ADDR) |
| 2043 |
# Make sure the process ID is in the output |
| 2044 |
logging.basicConfig(level=logging.DEBUG, |
| 2045 |
handlers=[logging.StreamHandler(), handler], |
| 2046 |
format='%(levelname)-8s %(name)10s %(process)6s %(message)s') |
| 2047 |
levels = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, |
| 2048 |
logging.CRITICAL) |
| 2049 |
logger_names = ('myapp', 'myapp.lib1', 'myapp.lib2') |
| 2050 |
msgno = 1 |
| 2051 |
while True: |
| 2052 |
# Just randomly select some loggers and levels and log away |
| 2053 |
level = random.choice(levels) |
| 2054 |
logger = logging.getLogger(random.choice(logger_names)) |
| 2055 |
logger.log(level, 'Message no. %5d' % msgno) |
| 2056 |
msgno += 1 |
| 2057 |
delay = random.random() * 2 + 0.5 |
| 2058 |
time.sleep(delay) |
| 2059 |
|
| 2060 |
You can run the above two snippets in separate command shells. If we run the |
| 2061 |
listener in one shell and run the sender in two separate shells, we should see |
| 2062 |
something like the following. In the first sender shell: |
| 2063 |
|
| 2064 |
.. code-block:: console |
| 2065 |
|
| 2066 |
$ python sender.py |
| 2067 |
DEBUG myapp 613 Message no. 1 |
| 2068 |
WARNING myapp.lib2 613 Message no. 2 |
| 2069 |
CRITICAL myapp.lib2 613 Message no. 3 |
| 2070 |
WARNING myapp.lib2 613 Message no. 4 |
| 2071 |
CRITICAL myapp.lib1 613 Message no. 5 |
| 2072 |
DEBUG myapp 613 Message no. 6 |
| 2073 |
CRITICAL myapp.lib1 613 Message no. 7 |
| 2074 |
INFO myapp.lib1 613 Message no. 8 |
| 2075 |
(and so on) |
| 2076 |
|
| 2077 |
In the second sender shell: |
| 2078 |
|
| 2079 |
.. code-block:: console |
| 2080 |
|
| 2081 |
$ python sender.py |
| 2082 |
INFO myapp.lib2 657 Message no. 1 |
| 2083 |
CRITICAL myapp.lib2 657 Message no. 2 |
| 2084 |
CRITICAL myapp 657 Message no. 3 |
| 2085 |
CRITICAL myapp.lib1 657 Message no. 4 |
| 2086 |
INFO myapp.lib1 657 Message no. 5 |
| 2087 |
WARNING myapp.lib2 657 Message no. 6 |
| 2088 |
CRITICAL myapp 657 Message no. 7 |
| 2089 |
DEBUG myapp.lib1 657 Message no. 8 |
| 2090 |
(and so on) |
| 2091 |
|
| 2092 |
In the listener shell: |
| 2093 |
|
| 2094 |
.. code-block:: console |
| 2095 |
|
| 2096 |
$ python listener.py |
| 2097 |
Press Ctrl-C to stop. |
| 2098 |
DEBUG myapp 613 Message no. 1 |
| 2099 |
WARNING myapp.lib2 613 Message no. 2 |
| 2100 |
INFO myapp.lib2 657 Message no. 1 |
| 2101 |
CRITICAL myapp.lib2 613 Message no. 3 |
| 2102 |
CRITICAL myapp.lib2 657 Message no. 2 |
| 2103 |
CRITICAL myapp 657 Message no. 3 |
| 2104 |
WARNING myapp.lib2 613 Message no. 4 |
| 2105 |
CRITICAL myapp.lib1 613 Message no. 5 |
| 2106 |
CRITICAL myapp.lib1 657 Message no. 4 |
| 2107 |
INFO myapp.lib1 657 Message no. 5 |
| 2108 |
DEBUG myapp 613 Message no. 6 |
| 2109 |
WARNING myapp.lib2 657 Message no. 6 |
| 2110 |
CRITICAL myapp 657 Message no. 7 |
| 2111 |
CRITICAL myapp.lib1 613 Message no. 7 |
| 2112 |
INFO myapp.lib1 613 Message no. 8 |
| 2113 |
DEBUG myapp.lib1 657 Message no. 8 |
| 2114 |
(and so on) |
| 2115 |
|
| 2116 |
As you can see, the logging from the two sender processes is interleaved in the |
| 2117 |
listener's output. |
| 2118 |
|
| 2119 |
|
| 2120 |
An example dictionary-based configuration |
| 2121 |
----------------------------------------- |
| 2122 |
|
| 2123 |
Below is an example of a logging configuration dictionary - it's taken from |
| 2124 |
the `documentation on the Django project <https://docs.djangoproject.com/en/stable/topics/logging/#configuring-logging>`_. |
| 2125 |
This dictionary is passed to :func:`~config.dictConfig` to put the configuration into effect:: |
| 2126 |
|
| 2127 |
LOGGING = { |
| 2128 |
'version': 1, |
| 2129 |
'disable_existing_loggers': False, |
| 2130 |
'formatters': { |
| 2131 |
'verbose': { |
| 2132 |
'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}', |
| 2133 |
'style': '{', |
| 2134 |
}, |
| 2135 |
'simple': { |
| 2136 |
'format': '{levelname} {message}', |
| 2137 |
'style': '{', |
| 2138 |
}, |
| 2139 |
}, |
| 2140 |
'filters': { |
| 2141 |
'special': { |
| 2142 |
'()': 'project.logging.SpecialFilter', |
| 2143 |
'foo': 'bar', |
| 2144 |
}, |
| 2145 |
}, |
| 2146 |
'handlers': { |
| 2147 |
'console': { |
| 2148 |
'level': 'INFO', |
| 2149 |
'class': 'logging.StreamHandler', |
| 2150 |
'formatter': 'simple', |
| 2151 |
}, |
| 2152 |
'mail_admins': { |
| 2153 |
'level': 'ERROR', |
| 2154 |
'class': 'django.utils.log.AdminEmailHandler', |
| 2155 |
'filters': ['special'] |
| 2156 |
} |
| 2157 |
}, |
| 2158 |
'loggers': { |
| 2159 |
'django': { |
| 2160 |
'handlers': ['console'], |
| 2161 |
'propagate': True, |
| 2162 |
}, |
| 2163 |
'django.request': { |
| 2164 |
'handlers': ['mail_admins'], |
| 2165 |
'level': 'ERROR', |
| 2166 |
'propagate': False, |
| 2167 |
}, |
| 2168 |
'myproject.custom': { |
| 2169 |
'handlers': ['console', 'mail_admins'], |
| 2170 |
'level': 'INFO', |
| 2171 |
'filters': ['special'] |
| 2172 |
} |
| 2173 |
} |
| 2174 |
} |
| 2175 |
|
| 2176 |
For more information about this configuration, you can see the `relevant |
| 2177 |
section <https://docs.djangoproject.com/en/stable/topics/logging/#configuring-logging>`_ |
| 2178 |
of the Django documentation. |
| 2179 |
|
| 2180 |
.. _cookbook-rotator-namer: |
| 2181 |
|
| 2182 |
Using a rotator and namer to customize log rotation processing |
| 2183 |
-------------------------------------------------------------- |
| 2184 |
|
| 2185 |
An example of how you can define a namer and rotator is given in the following |
| 2186 |
runnable script, which shows gzip compression of the log file:: |
| 2187 |
|
| 2188 |
import gzip |
| 2189 |
import logging |
| 2190 |
import logging.handlers |
| 2191 |
import os |
| 2192 |
import shutil |
| 2193 |
|
| 2194 |
def namer(name): |
| 2195 |
return name + ".gz" |
| 2196 |
|
| 2197 |
def rotator(source, dest): |
| 2198 |
with open(source, 'rb') as f_in: |
| 2199 |
with gzip.open(dest, 'wb') as f_out: |
| 2200 |
shutil.copyfileobj(f_in, f_out) |
| 2201 |
os.remove(source) |
| 2202 |
|
| 2203 |
|
| 2204 |
rh = logging.handlers.RotatingFileHandler('rotated.log', maxBytes=128, backupCount=5) |
| 2205 |
rh.rotator = rotator |
| 2206 |
rh.namer = namer |
| 2207 |
|
| 2208 |
root = logging.getLogger() |
| 2209 |
root.setLevel(logging.INFO) |
| 2210 |
root.addHandler(rh) |
| 2211 |
f = logging.Formatter('%(asctime)s %(message)s') |
| 2212 |
rh.setFormatter(f) |
| 2213 |
for i in range(1000): |
| 2214 |
root.info(f'Message no. {i + 1}') |
| 2215 |
|
| 2216 |
After running this, you will see six new files, five of which are compressed: |
| 2217 |
|
| 2218 |
.. code-block:: shell-session |
| 2219 |
|
| 2220 |
$ ls rotated.log* |
| 2221 |
rotated.log rotated.log.2.gz rotated.log.4.gz |
| 2222 |
rotated.log.1.gz rotated.log.3.gz rotated.log.5.gz |
| 2223 |
$ zcat rotated.log.1.gz |
| 2224 |
2023-01-20 02:28:17,767 Message no. 996 |
| 2225 |
2023-01-20 02:28:17,767 Message no. 997 |
| 2226 |
2023-01-20 02:28:17,767 Message no. 998 |
| 2227 |
|
| 2228 |
A more elaborate multiprocessing example |
| 2229 |
---------------------------------------- |
| 2230 |
|
| 2231 |
The following working example shows how logging can be used with multiprocessing |
| 2232 |
using configuration files. The configurations are fairly simple, but serve to |
| 2233 |
illustrate how more complex ones could be implemented in a real multiprocessing |
| 2234 |
scenario. |
| 2235 |
|
| 2236 |
In the example, the main process spawns a listener process and some worker |
| 2237 |
processes. Each of the main process, the listener and the workers have three |
| 2238 |
separate configurations (the workers all share the same configuration). We can |
| 2239 |
see logging in the main process, how the workers log to a QueueHandler and how |
| 2240 |
the listener implements a QueueListener and a more complex logging |
| 2241 |
configuration, and arranges to dispatch events received via the queue to the |
| 2242 |
handlers specified in the configuration. Note that these configurations are |
| 2243 |
purely illustrative, but you should be able to adapt this example to your own |
| 2244 |
scenario. |
| 2245 |
|
| 2246 |
Here's the script - the docstrings and the comments hopefully explain how it |
| 2247 |
works:: |
| 2248 |
|
| 2249 |
import logging |
| 2250 |
import logging.config |
| 2251 |
import logging.handlers |
| 2252 |
from multiprocessing import Process, Queue, Event, current_process |
| 2253 |
import os |
| 2254 |
import random |
| 2255 |
import time |
| 2256 |
|
| 2257 |
class MyHandler: |
| 2258 |
""" |
| 2259 |
A simple handler for logging events. It runs in the listener process and |
| 2260 |
dispatches events to loggers based on the name in the received record, |
| 2261 |
which then get dispatched, by the logging system, to the handlers |
| 2262 |
configured for those loggers. |
| 2263 |
""" |
| 2264 |
|
| 2265 |
def handle(self, record): |
| 2266 |
if record.name == "root": |
| 2267 |
logger = logging.getLogger() |
| 2268 |
else: |
| 2269 |
logger = logging.getLogger(record.name) |
| 2270 |
|
| 2271 |
if logger.isEnabledFor(record.levelno): |
| 2272 |
# The process name is transformed just to show that it's the listener |
| 2273 |
# doing the logging to files and console |
| 2274 |
record.processName = '%s (for %s)' % (current_process().name, record.processName) |
| 2275 |
logger.handle(record) |
| 2276 |
|
| 2277 |
def listener_process(q, stop_event, config): |
| 2278 |
""" |
| 2279 |
This could be done in the main process, but is just done in a separate |
| 2280 |
process for illustrative purposes. |
| 2281 |
|
| 2282 |
This initialises logging according to the specified configuration, |
| 2283 |
starts the listener and waits for the main process to signal completion |
| 2284 |
via the event. The listener is then stopped, and the process exits. |
| 2285 |
""" |
| 2286 |
logging.config.dictConfig(config) |
| 2287 |
listener = logging.handlers.QueueListener(q, MyHandler()) |
| 2288 |
listener.start() |
| 2289 |
if os.name == 'posix': |
| 2290 |
# On POSIX, the setup logger will have been configured in the |
| 2291 |
# parent process, but should have been disabled following the |
| 2292 |
# dictConfig call. |
| 2293 |
# On Windows, since fork isn't used, the setup logger won't |
| 2294 |
# exist in the child, so it would be created and the message |
| 2295 |
# would appear - hence the "if posix" clause. |
| 2296 |
logger = logging.getLogger('setup') |
| 2297 |
logger.critical('Should not appear, because of disabled logger ...') |
| 2298 |
stop_event.wait() |
| 2299 |
listener.stop() |
| 2300 |
|
| 2301 |
def worker_process(config): |
| 2302 |
""" |
| 2303 |
A number of these are spawned for the purpose of illustration. In |
| 2304 |
practice, they could be a heterogeneous bunch of processes rather than |
| 2305 |
ones which are identical to each other. |
| 2306 |
|
| 2307 |
This initialises logging according to the specified configuration, |
| 2308 |
and logs a hundred messages with random levels to randomly selected |
| 2309 |
loggers. |
| 2310 |
|
| 2311 |
A small sleep is added to allow other processes a chance to run. This |
| 2312 |
is not strictly needed, but it mixes the output from the different |
| 2313 |
processes a bit more than if it's left out. |
| 2314 |
""" |
| 2315 |
logging.config.dictConfig(config) |
| 2316 |
levels = [logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, |
| 2317 |
logging.CRITICAL] |
| 2318 |
loggers = ['foo', 'foo.bar', 'foo.bar.baz', |
| 2319 |
'spam', 'spam.ham', 'spam.ham.eggs'] |
| 2320 |
if os.name == 'posix': |
| 2321 |
# On POSIX, the setup logger will have been configured in the |
| 2322 |
# parent process, but should have been disabled following the |
| 2323 |
# dictConfig call. |
| 2324 |
# On Windows, since fork isn't used, the setup logger won't |
| 2325 |
# exist in the child, so it would be created and the message |
| 2326 |
# would appear - hence the "if posix" clause. |
| 2327 |
logger = logging.getLogger('setup') |
| 2328 |
logger.critical('Should not appear, because of disabled logger ...') |
| 2329 |
for i in range(100): |
| 2330 |
lvl = random.choice(levels) |
| 2331 |
logger = logging.getLogger(random.choice(loggers)) |
| 2332 |
logger.log(lvl, 'Message no. %d', i) |
| 2333 |
time.sleep(0.01) |
| 2334 |
|
| 2335 |
def main(): |
| 2336 |
q = Queue() |
| 2337 |
# The main process gets a simple configuration which prints to the console. |
| 2338 |
config_initial = { |
| 2339 |
'version': 1, |
| 2340 |
'handlers': { |
| 2341 |
'console': { |
| 2342 |
'class': 'logging.StreamHandler', |
| 2343 |
'level': 'INFO' |
| 2344 |
} |
| 2345 |
}, |
| 2346 |
'root': { |
| 2347 |
'handlers': ['console'], |
| 2348 |
'level': 'DEBUG' |
| 2349 |
} |
| 2350 |
} |
| 2351 |
# The worker process configuration is just a QueueHandler attached to the |
| 2352 |
# root logger, which allows all messages to be sent to the queue. |
| 2353 |
# We disable existing loggers to disable the "setup" logger used in the |
| 2354 |
# parent process. This is needed on POSIX because the logger will |
| 2355 |
# be there in the child following a fork(). |
| 2356 |
config_worker = { |
| 2357 |
'version': 1, |
| 2358 |
'disable_existing_loggers': True, |
| 2359 |
'handlers': { |
| 2360 |
'queue': { |
| 2361 |
'class': 'logging.handlers.QueueHandler', |
| 2362 |
'queue': q |
| 2363 |
} |
| 2364 |
}, |
| 2365 |
'root': { |
| 2366 |
'handlers': ['queue'], |
| 2367 |
'level': 'DEBUG' |
| 2368 |
} |
| 2369 |
} |
| 2370 |
# The listener process configuration shows that the full flexibility of |
| 2371 |
# logging configuration is available to dispatch events to handlers however |
| 2372 |
# you want. |
| 2373 |
# We disable existing loggers to disable the "setup" logger used in the |
| 2374 |
# parent process. This is needed on POSIX because the logger will |
| 2375 |
# be there in the child following a fork(). |
| 2376 |
config_listener = { |
| 2377 |
'version': 1, |
| 2378 |
'disable_existing_loggers': True, |
| 2379 |
'formatters': { |
| 2380 |
'detailed': { |
| 2381 |
'class': 'logging.Formatter', |
| 2382 |
'format': '%(asctime)s %(name)-15s %(levelname)-8s %(processName)-10s %(message)s' |
| 2383 |
}, |
| 2384 |
'simple': { |
| 2385 |
'class': 'logging.Formatter', |
| 2386 |
'format': '%(name)-15s %(levelname)-8s %(processName)-10s %(message)s' |
| 2387 |
} |
| 2388 |
}, |
| 2389 |
'handlers': { |
| 2390 |
'console': { |
| 2391 |
'class': 'logging.StreamHandler', |
| 2392 |
'formatter': 'simple', |
| 2393 |
'level': 'INFO' |
| 2394 |
}, |
| 2395 |
'file': { |
| 2396 |
'class': 'logging.FileHandler', |
| 2397 |
'filename': 'mplog.log', |
| 2398 |
'mode': 'w', |
| 2399 |
'formatter': 'detailed' |
| 2400 |
}, |
| 2401 |
'foofile': { |
| 2402 |
'class': 'logging.FileHandler', |
| 2403 |
'filename': 'mplog-foo.log', |
| 2404 |
'mode': 'w', |
| 2405 |
'formatter': 'detailed' |
| 2406 |
}, |
| 2407 |
'errors': { |
| 2408 |
'class': 'logging.FileHandler', |
| 2409 |
'filename': 'mplog-errors.log', |
| 2410 |
'mode': 'w', |
| 2411 |
'formatter': 'detailed', |
| 2412 |
'level': 'ERROR' |
| 2413 |
} |
| 2414 |
}, |
| 2415 |
'loggers': { |
| 2416 |
'foo': { |
| 2417 |
'handlers': ['foofile'] |
| 2418 |
} |
| 2419 |
}, |
| 2420 |
'root': { |
| 2421 |
'handlers': ['console', 'file', 'errors'], |
| 2422 |
'level': 'DEBUG' |
| 2423 |
} |
| 2424 |
} |
| 2425 |
# Log some initial events, just to show that logging in the parent works |
| 2426 |
# normally. |
| 2427 |
logging.config.dictConfig(config_initial) |
| 2428 |
logger = logging.getLogger('setup') |
| 2429 |
logger.info('About to create workers ...') |
| 2430 |
workers = [] |
| 2431 |
for i in range(5): |
| 2432 |
wp = Process(target=worker_process, name='worker %d' % (i + 1), |
| 2433 |
args=(config_worker,)) |
| 2434 |
workers.append(wp) |
| 2435 |
wp.start() |
| 2436 |
logger.info('Started worker: %s', wp.name) |
| 2437 |
logger.info('About to create listener ...') |
| 2438 |
stop_event = Event() |
| 2439 |
lp = Process(target=listener_process, name='listener', |
| 2440 |
args=(q, stop_event, config_listener)) |
| 2441 |
lp.start() |
| 2442 |
logger.info('Started listener') |
| 2443 |
# We now hang around for the workers to finish their work. |
| 2444 |
for wp in workers: |
| 2445 |
wp.join() |
| 2446 |
# Workers all done, listening can now stop. |
| 2447 |
# Logging in the parent still works normally. |
| 2448 |
logger.info('Telling listener to stop ...') |
| 2449 |
stop_event.set() |
| 2450 |
lp.join() |
| 2451 |
logger.info('All done.') |
| 2452 |
|
| 2453 |
if __name__ == '__main__': |
| 2454 |
main() |
| 2455 |
|
| 2456 |
|
| 2457 |
Inserting a BOM into messages sent to a SysLogHandler |
| 2458 |
----------------------------------------------------- |
| 2459 |
|
| 2460 |
:rfc:`5424` requires that a |
| 2461 |
Unicode message be sent to a syslog daemon as a set of bytes which have the |
| 2462 |
following structure: an optional pure-ASCII component, followed by a UTF-8 Byte |
| 2463 |
Order Mark (BOM), followed by Unicode encoded using UTF-8. (See the |
| 2464 |
:rfc:`relevant section of the specification <5424#section-6>`.) |
| 2465 |
|
| 2466 |
In Python 3.1, code was added to |
| 2467 |
:class:`~logging.handlers.SysLogHandler` to insert a BOM into the message, but |
| 2468 |
unfortunately, it was implemented incorrectly, with the BOM appearing at the |
| 2469 |
beginning of the message and hence not allowing any pure-ASCII component to |
| 2470 |
appear before it. |
| 2471 |
|
| 2472 |
As this behaviour is broken, the incorrect BOM insertion code is being removed |
| 2473 |
from Python 3.2.4 and later. However, it is not being replaced, and if you |
| 2474 |
want to produce :rfc:`5424`-compliant messages which include a BOM, an optional |
| 2475 |
pure-ASCII sequence before it and arbitrary Unicode after it, encoded using |
| 2476 |
UTF-8, then you need to do the following: |
| 2477 |
|
| 2478 |
#. Attach a :class:`~logging.Formatter` instance to your |
| 2479 |
:class:`~logging.handlers.SysLogHandler` instance, with a format string |
| 2480 |
such as:: |
| 2481 |
|
| 2482 |
'ASCII section\ufeffUnicode section' |
| 2483 |
|
| 2484 |
The Unicode code point U+FEFF, when encoded using UTF-8, will be |
| 2485 |
encoded as a UTF-8 BOM -- the byte-string ``b'\xef\xbb\xbf'``. |
| 2486 |
|
| 2487 |
#. Replace the ASCII section with whatever placeholders you like, but make sure |
| 2488 |
that the data that appears in there after substitution is always ASCII (that |
| 2489 |
way, it will remain unchanged after UTF-8 encoding). |
| 2490 |
|
| 2491 |
#. Replace the Unicode section with whatever placeholders you like; if the data |
| 2492 |
which appears there after substitution contains characters outside the ASCII |
| 2493 |
range, that's fine -- it will be encoded using UTF-8. |
| 2494 |
|
| 2495 |
The formatted message *will* be encoded using UTF-8 encoding by |
| 2496 |
``SysLogHandler``. If you follow the above rules, you should be able to produce |
| 2497 |
:rfc:`5424`-compliant messages. If you don't, logging may not complain, but your |
| 2498 |
messages will not be RFC 5424-compliant, and your syslog daemon may complain. |
| 2499 |
|
| 2500 |
|
| 2501 |
Implementing structured logging |
| 2502 |
------------------------------- |
| 2503 |
|
| 2504 |
Although most logging messages are intended for reading by humans, and thus not |
| 2505 |
readily machine-parseable, there might be circumstances where you want to output |
| 2506 |
messages in a structured format which *is* capable of being parsed by a program |
| 2507 |
(without needing complex regular expressions to parse the log message). This is |
| 2508 |
straightforward to achieve using the logging package. There are a number of |
| 2509 |
ways in which this could be achieved, but the following is a simple approach |
| 2510 |
which uses JSON to serialise the event in a machine-parseable manner:: |
| 2511 |
|
| 2512 |
import json |
| 2513 |
import logging |
| 2514 |
|
| 2515 |
class StructuredMessage: |
| 2516 |
def __init__(self, message, /, **kwargs): |
| 2517 |
self.message = message |
| 2518 |
self.kwargs = kwargs |
| 2519 |
|
| 2520 |
def __str__(self): |
| 2521 |
return '%s >>> %s' % (self.message, json.dumps(self.kwargs)) |
| 2522 |
|
| 2523 |
_ = StructuredMessage # optional, to improve readability |
| 2524 |
|
| 2525 |
logging.basicConfig(level=logging.INFO, format='%(message)s') |
| 2526 |
logging.info(_('message 1', foo='bar', bar='baz', num=123, fnum=123.456)) |
| 2527 |
|
| 2528 |
If the above script is run, it prints: |
| 2529 |
|
| 2530 |
.. code-block:: none |
| 2531 |
|
| 2532 |
message 1 >>> {"fnum": 123.456, "num": 123, "bar": "baz", "foo": "bar"} |
| 2533 |
|
| 2534 |
Note that the order of items might be different according to the version of |
| 2535 |
Python used. |
| 2536 |
|
| 2537 |
If you need more specialised processing, you can use a custom JSON encoder, |
| 2538 |
as in the following complete example:: |
| 2539 |
|
| 2540 |
import json |
| 2541 |
import logging |
| 2542 |
|
| 2543 |
|
| 2544 |
class Encoder(json.JSONEncoder): |
| 2545 |
def default(self, o): |
| 2546 |
if isinstance(o, set): |
| 2547 |
return tuple(o) |
| 2548 |
elif isinstance(o, str): |
| 2549 |
return o.encode('unicode_escape').decode('ascii') |
| 2550 |
return super().default(o) |
| 2551 |
|
| 2552 |
class StructuredMessage: |
| 2553 |
def __init__(self, message, /, **kwargs): |
| 2554 |
self.message = message |
| 2555 |
self.kwargs = kwargs |
| 2556 |
|
| 2557 |
def __str__(self): |
| 2558 |
s = Encoder().encode(self.kwargs) |
| 2559 |
return '%s >>> %s' % (self.message, s) |
| 2560 |
|
| 2561 |
_ = StructuredMessage # optional, to improve readability |
| 2562 |
|
| 2563 |
def main(): |
| 2564 |
logging.basicConfig(level=logging.INFO, format='%(message)s') |
| 2565 |
logging.info(_('message 1', set_value={1, 2, 3}, snowman='\u2603')) |
| 2566 |
|
| 2567 |
if __name__ == '__main__': |
| 2568 |
main() |
| 2569 |
|
| 2570 |
When the above script is run, it prints: |
| 2571 |
|
| 2572 |
.. code-block:: none |
| 2573 |
|
| 2574 |
message 1 >>> {"snowman": "\u2603", "set_value": [1, 2, 3]} |
| 2575 |
|
| 2576 |
Note that the order of items might be different according to the version of |
| 2577 |
Python used. |
| 2578 |
|
| 2579 |
|
| 2580 |
.. _custom-handlers: |
| 2581 |
|
| 2582 |
.. currentmodule:: logging.config |
| 2583 |
|
| 2584 |
Customizing handlers with :func:`dictConfig` |
| 2585 |
-------------------------------------------- |
| 2586 |
|
| 2587 |
There are times when you want to customize logging handlers in particular ways, |
| 2588 |
and if you use :func:`dictConfig` you may be able to do this without |
| 2589 |
subclassing. As an example, consider that you may want to set the ownership of a |
| 2590 |
log file. On POSIX, this is easily done using :func:`shutil.chown`, but the file |
| 2591 |
handlers in the stdlib don't offer built-in support. You can customize handler |
| 2592 |
creation using a plain function such as:: |
| 2593 |
|
| 2594 |
def owned_file_handler(filename, mode='a', encoding=None, owner=None): |
| 2595 |
if owner: |
| 2596 |
if not os.path.exists(filename): |
| 2597 |
open(filename, 'a').close() |
| 2598 |
shutil.chown(filename, *owner) |
| 2599 |
return logging.FileHandler(filename, mode, encoding) |
| 2600 |
|
| 2601 |
You can then specify, in a logging configuration passed to :func:`dictConfig`, |
| 2602 |
that a logging handler be created by calling this function:: |
| 2603 |
|
| 2604 |
LOGGING = { |
| 2605 |
'version': 1, |
| 2606 |
'disable_existing_loggers': False, |
| 2607 |
'formatters': { |
| 2608 |
'default': { |
| 2609 |
'format': '%(asctime)s %(levelname)s %(name)s %(message)s' |
| 2610 |
}, |
| 2611 |
}, |
| 2612 |
'handlers': { |
| 2613 |
'file':{ |
| 2614 |
# The values below are popped from this dictionary and |
| 2615 |
# used to create the handler, set the handler's level and |
| 2616 |
# its formatter. |
| 2617 |
'()': owned_file_handler, |
| 2618 |
'level':'DEBUG', |
| 2619 |
'formatter': 'default', |
| 2620 |
# The values below are passed to the handler creator callable |
| 2621 |
# as keyword arguments. |
| 2622 |
'owner': ['pulse', 'pulse'], |
| 2623 |
'filename': 'chowntest.log', |
| 2624 |
'mode': 'w', |
| 2625 |
'encoding': 'utf-8', |
| 2626 |
}, |
| 2627 |
}, |
| 2628 |
'root': { |
| 2629 |
'handlers': ['file'], |
| 2630 |
'level': 'DEBUG', |
| 2631 |
}, |
| 2632 |
} |
| 2633 |
|
| 2634 |
In this example I am setting the ownership using the ``pulse`` user and group, |
| 2635 |
just for the purposes of illustration. Putting it together into a working |
| 2636 |
script, ``chowntest.py``:: |
| 2637 |
|
| 2638 |
import logging, logging.config, os, shutil |
| 2639 |
|
| 2640 |
def owned_file_handler(filename, mode='a', encoding=None, owner=None): |
| 2641 |
if owner: |
| 2642 |
if not os.path.exists(filename): |
| 2643 |
open(filename, 'a').close() |
| 2644 |
shutil.chown(filename, *owner) |
| 2645 |
return logging.FileHandler(filename, mode, encoding) |
| 2646 |
|
| 2647 |
LOGGING = { |
| 2648 |
'version': 1, |
| 2649 |
'disable_existing_loggers': False, |
| 2650 |
'formatters': { |
| 2651 |
'default': { |
| 2652 |
'format': '%(asctime)s %(levelname)s %(name)s %(message)s' |
| 2653 |
}, |
| 2654 |
}, |
| 2655 |
'handlers': { |
| 2656 |
'file':{ |
| 2657 |
# The values below are popped from this dictionary and |
| 2658 |
# used to create the handler, set the handler's level and |
| 2659 |
# its formatter. |
| 2660 |
'()': owned_file_handler, |
| 2661 |
'level':'DEBUG', |
| 2662 |
'formatter': 'default', |
| 2663 |
# The values below are passed to the handler creator callable |
| 2664 |
# as keyword arguments. |
| 2665 |
'owner': ['pulse', 'pulse'], |
| 2666 |
'filename': 'chowntest.log', |
| 2667 |
'mode': 'w', |
| 2668 |
'encoding': 'utf-8', |
| 2669 |
}, |
| 2670 |
}, |
| 2671 |
'root': { |
| 2672 |
'handlers': ['file'], |
| 2673 |
'level': 'DEBUG', |
| 2674 |
}, |
| 2675 |
} |
| 2676 |
|
| 2677 |
logging.config.dictConfig(LOGGING) |
| 2678 |
logger = logging.getLogger('mylogger') |
| 2679 |
logger.debug('A debug message') |
| 2680 |
|
| 2681 |
To run this, you will probably need to run as ``root``: |
| 2682 |
|
| 2683 |
.. code-block:: shell-session |
| 2684 |
|
| 2685 |
$ sudo python3.3 chowntest.py |
| 2686 |
$ cat chowntest.log |
| 2687 |
2013-11-05 09:34:51,128 DEBUG mylogger A debug message |
| 2688 |
$ ls -l chowntest.log |
| 2689 |
-rw-r--r-- 1 pulse pulse 55 2013-11-05 09:34 chowntest.log |
| 2690 |
|
| 2691 |
Note that this example uses Python 3.3 because that's where :func:`shutil.chown` |
| 2692 |
makes an appearance. This approach should work with any Python version that |
| 2693 |
supports :func:`dictConfig` - namely, Python 2.7, 3.2 or later. With pre-3.3 |
| 2694 |
versions, you would need to implement the actual ownership change using e.g. |
| 2695 |
:func:`os.chown`. |
| 2696 |
|
| 2697 |
In practice, the handler-creating function may be in a utility module somewhere |
| 2698 |
in your project. Instead of the line in the configuration:: |
| 2699 |
|
| 2700 |
'()': owned_file_handler, |
| 2701 |
|
| 2702 |
you could use e.g.:: |
| 2703 |
|
| 2704 |
'()': 'ext://project.util.owned_file_handler', |
| 2705 |
|
| 2706 |
where ``project.util`` can be replaced with the actual name of the package |
| 2707 |
where the function resides. In the above working script, using |
| 2708 |
``'ext://__main__.owned_file_handler'`` should work. Here, the actual callable |
| 2709 |
is resolved by :func:`dictConfig` from the ``ext://`` specification. |
| 2710 |
|
| 2711 |
This example hopefully also points the way to how you could implement other |
| 2712 |
types of file change - e.g. setting specific POSIX permission bits - in the |
| 2713 |
same way, using :func:`os.chmod`. |
| 2714 |
|
| 2715 |
Of course, the approach could also be extended to types of handler other than a |
| 2716 |
:class:`~logging.FileHandler` - for example, one of the rotating file handlers, |
| 2717 |
or a different type of handler altogether. |
| 2718 |
|
| 2719 |
|
| 2720 |
.. currentmodule:: logging |
| 2721 |
|
| 2722 |
.. _formatting-styles: |
| 2723 |
|
| 2724 |
Using particular formatting styles throughout your application |
| 2725 |
-------------------------------------------------------------- |
| 2726 |
|
| 2727 |
In Python 3.2, the :class:`~logging.Formatter` gained a ``style`` keyword |
| 2728 |
parameter which, while defaulting to ``%`` for backward compatibility, allowed |
| 2729 |
the specification of ``{`` or ``$`` to support the formatting approaches |
| 2730 |
supported by :meth:`str.format` and :class:`string.Template`. Note that this |
| 2731 |
governs the formatting of logging messages for final output to logs, and is |
| 2732 |
completely orthogonal to how an individual logging message is constructed. |
| 2733 |
|
| 2734 |
Logging calls (:meth:`~Logger.debug`, :meth:`~Logger.info` etc.) only take |
| 2735 |
positional parameters for the actual logging message itself, with keyword |
| 2736 |
parameters used only for determining options for how to handle the logging call |
| 2737 |
(e.g. the ``exc_info`` keyword parameter to indicate that traceback information |
| 2738 |
should be logged, or the ``extra`` keyword parameter to indicate additional |
| 2739 |
contextual information to be added to the log). So you cannot directly make |
| 2740 |
logging calls using :meth:`str.format` or :class:`string.Template` syntax, |
| 2741 |
because internally the logging package uses %-formatting to merge the format |
| 2742 |
string and the variable arguments. There would be no changing this while preserving |
| 2743 |
backward compatibility, since all logging calls which are out there in existing |
| 2744 |
code will be using %-format strings. |
| 2745 |
|
| 2746 |
There have been suggestions to associate format styles with specific loggers, |
| 2747 |
but that approach also runs into backward compatibility problems because any |
| 2748 |
existing code could be using a given logger name and using %-formatting. |
| 2749 |
|
| 2750 |
For logging to work interoperably between any third-party libraries and your |
| 2751 |
code, decisions about formatting need to be made at the level of the |
| 2752 |
individual logging call. This opens up a couple of ways in which alternative |
| 2753 |
formatting styles can be accommodated. |
| 2754 |
|
| 2755 |
|
| 2756 |
Using LogRecord factories |
| 2757 |
^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 2758 |
|
| 2759 |
In Python 3.2, along with the :class:`~logging.Formatter` changes mentioned |
| 2760 |
above, the logging package gained the ability to allow users to set their own |
| 2761 |
:class:`LogRecord` subclasses, using the :func:`setLogRecordFactory` function. |
| 2762 |
You can use this to set your own subclass of :class:`LogRecord`, which does the |
| 2763 |
Right Thing by overriding the :meth:`~LogRecord.getMessage` method. The base |
| 2764 |
class implementation of this method is where the ``msg % args`` formatting |
| 2765 |
happens, and where you can substitute your alternate formatting; however, you |
| 2766 |
should be careful to support all formatting styles and allow %-formatting as |
| 2767 |
the default, to ensure interoperability with other code. Care should also be |
| 2768 |
taken to call ``str(self.msg)``, just as the base implementation does. |
| 2769 |
|
| 2770 |
Refer to the reference documentation on :func:`setLogRecordFactory` and |
| 2771 |
:class:`LogRecord` for more information. |
| 2772 |
|
| 2773 |
|
| 2774 |
Using custom message objects |
| 2775 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 2776 |
|
| 2777 |
There is another, perhaps simpler way that you can use {}- and $- formatting to |
| 2778 |
construct your individual log messages. You may recall (from |
| 2779 |
:ref:`arbitrary-object-messages`) that when logging you can use an arbitrary |
| 2780 |
object as a message format string, and that the logging package will call |
| 2781 |
:func:`str` on that object to get the actual format string. Consider the |
| 2782 |
following two classes:: |
| 2783 |
|
| 2784 |
class BraceMessage: |
| 2785 |
def __init__(self, fmt, /, *args, **kwargs): |
| 2786 |
self.fmt = fmt |
| 2787 |
self.args = args |
| 2788 |
self.kwargs = kwargs |
| 2789 |
|
| 2790 |
def __str__(self): |
| 2791 |
return self.fmt.format(*self.args, **self.kwargs) |
| 2792 |
|
| 2793 |
class DollarMessage: |
| 2794 |
def __init__(self, fmt, /, **kwargs): |
| 2795 |
self.fmt = fmt |
| 2796 |
self.kwargs = kwargs |
| 2797 |
|
| 2798 |
def __str__(self): |
| 2799 |
from string import Template |
| 2800 |
return Template(self.fmt).substitute(**self.kwargs) |
| 2801 |
|
| 2802 |
Either of these can be used in place of a format string, to allow {}- or |
| 2803 |
$-formatting to be used to build the actual "message" part which appears in the |
| 2804 |
formatted log output in place of “%(message)s” or “{message}” or “$message”. |
| 2805 |
If you find it a little unwieldy to use the class names whenever you want to log |
| 2806 |
something, you can make it more palatable if you use an alias such as ``M`` or |
| 2807 |
``_`` for the message (or perhaps ``__``, if you are using ``_`` for |
| 2808 |
localization). |
| 2809 |
|
| 2810 |
Examples of this approach are given below. Firstly, formatting with |
| 2811 |
:meth:`str.format`:: |
| 2812 |
|
| 2813 |
>>> __ = BraceMessage |
| 2814 |
>>> print(__('Message with {0} {1}', 2, 'placeholders')) |
| 2815 |
Message with 2 placeholders |
| 2816 |
>>> class Point: pass |
| 2817 |
... |
| 2818 |
>>> p = Point() |
| 2819 |
>>> p.x = 0.5 |
| 2820 |
>>> p.y = 0.5 |
| 2821 |
>>> print(__('Message with coordinates: ({point.x:.2f}, {point.y:.2f})', point=p)) |
| 2822 |
Message with coordinates: (0.50, 0.50) |
| 2823 |
|
| 2824 |
Secondly, formatting with :class:`string.Template`:: |
| 2825 |
|
| 2826 |
>>> __ = DollarMessage |
| 2827 |
>>> print(__('Message with $num $what', num=2, what='placeholders')) |
| 2828 |
Message with 2 placeholders |
| 2829 |
>>> |
| 2830 |
|
| 2831 |
One thing to note is that you pay no significant performance penalty with this |
| 2832 |
approach: the actual formatting happens not when you make the logging call, but |
| 2833 |
when (and if) the logged message is actually about to be output to a log by a |
| 2834 |
handler. So the only slightly unusual thing which might trip you up is that the |
| 2835 |
parentheses go around the format string and the arguments, not just the format |
| 2836 |
string. That’s because the __ notation is just syntax sugar for a constructor |
| 2837 |
call to one of the :samp:`{XXX}Message` classes shown above. |
| 2838 |
|
| 2839 |
|
| 2840 |
.. _filters-dictconfig: |
| 2841 |
|
| 2842 |
.. currentmodule:: logging.config |
| 2843 |
|
| 2844 |
Configuring filters with :func:`dictConfig` |
| 2845 |
------------------------------------------- |
| 2846 |
|
| 2847 |
You *can* configure filters using :func:`~logging.config.dictConfig`, though it |
| 2848 |
might not be obvious at first glance how to do it (hence this recipe). Since |
| 2849 |
:class:`~logging.Filter` is the only filter class included in the standard |
| 2850 |
library, and it is unlikely to cater to many requirements (it's only there as a |
| 2851 |
base class), you will typically need to define your own :class:`~logging.Filter` |
| 2852 |
subclass with an overridden :meth:`~logging.Filter.filter` method. To do this, |
| 2853 |
specify the ``()`` key in the configuration dictionary for the filter, |
| 2854 |
specifying a callable which will be used to create the filter (a class is the |
| 2855 |
most obvious, but you can provide any callable which returns a |
| 2856 |
:class:`~logging.Filter` instance). Here is a complete example:: |
| 2857 |
|
| 2858 |
import logging |
| 2859 |
import logging.config |
| 2860 |
import sys |
| 2861 |
|
| 2862 |
class MyFilter(logging.Filter): |
| 2863 |
def __init__(self, param=None): |
| 2864 |
self.param = param |
| 2865 |
|
| 2866 |
def filter(self, record): |
| 2867 |
if self.param is None: |
| 2868 |
allow = True |
| 2869 |
else: |
| 2870 |
allow = self.param not in record.msg |
| 2871 |
if allow: |
| 2872 |
record.msg = 'changed: ' + record.msg |
| 2873 |
return allow |
| 2874 |
|
| 2875 |
LOGGING = { |
| 2876 |
'version': 1, |
| 2877 |
'filters': { |
| 2878 |
'myfilter': { |
| 2879 |
'()': MyFilter, |
| 2880 |
'param': 'noshow', |
| 2881 |
} |
| 2882 |
}, |
| 2883 |
'handlers': { |
| 2884 |
'console': { |
| 2885 |
'class': 'logging.StreamHandler', |
| 2886 |
'filters': ['myfilter'] |
| 2887 |
} |
| 2888 |
}, |
| 2889 |
'root': { |
| 2890 |
'level': 'DEBUG', |
| 2891 |
'handlers': ['console'] |
| 2892 |
}, |
| 2893 |
} |
| 2894 |
|
| 2895 |
if __name__ == '__main__': |
| 2896 |
logging.config.dictConfig(LOGGING) |
| 2897 |
logging.debug('hello') |
| 2898 |
logging.debug('hello - noshow') |
| 2899 |
|
| 2900 |
This example shows how you can pass configuration data to the callable which |
| 2901 |
constructs the instance, in the form of keyword parameters. When run, the above |
| 2902 |
script will print: |
| 2903 |
|
| 2904 |
.. code-block:: none |
| 2905 |
|
| 2906 |
changed: hello |
| 2907 |
|
| 2908 |
which shows that the filter is working as configured. |
| 2909 |
|
| 2910 |
A couple of extra points to note: |
| 2911 |
|
| 2912 |
* If you can't refer to the callable directly in the configuration (e.g. if it |
| 2913 |
lives in a different module, and you can't import it directly where the |
| 2914 |
configuration dictionary is), you can use the form ``ext://...`` as described |
| 2915 |
in :ref:`logging-config-dict-externalobj`. For example, you could have used |
| 2916 |
the text ``'ext://__main__.MyFilter'`` instead of ``MyFilter`` in the above |
| 2917 |
example. |
| 2918 |
|
| 2919 |
* As well as for filters, this technique can also be used to configure custom |
| 2920 |
handlers and formatters. See :ref:`logging-config-dict-userdef` for more |
| 2921 |
information on how logging supports using user-defined objects in its |
| 2922 |
configuration, and see the other cookbook recipe :ref:`custom-handlers` above. |
| 2923 |
|
| 2924 |
|
| 2925 |
.. _custom-format-exception: |
| 2926 |
|
| 2927 |
Customized exception formatting |
| 2928 |
------------------------------- |
| 2929 |
|
| 2930 |
There might be times when you want to do customized exception formatting - for |
| 2931 |
argument's sake, let's say you want exactly one line per logged event, even |
| 2932 |
when exception information is present. You can do this with a custom formatter |
| 2933 |
class, as shown in the following example:: |
| 2934 |
|
| 2935 |
import logging |
| 2936 |
|
| 2937 |
class OneLineExceptionFormatter(logging.Formatter): |
| 2938 |
def formatException(self, exc_info): |
| 2939 |
""" |
| 2940 |
Format an exception so that it prints on a single line. |
| 2941 |
""" |
| 2942 |
result = super().formatException(exc_info) |
| 2943 |
return repr(result) # or format into one line however you want to |
| 2944 |
|
| 2945 |
def format(self, record): |
| 2946 |
s = super().format(record) |
| 2947 |
if record.exc_text: |
| 2948 |
s = s.replace('\n', '') + '|' |
| 2949 |
return s |
| 2950 |
|
| 2951 |
def configure_logging(): |
| 2952 |
fh = logging.FileHandler('output.txt', 'w') |
| 2953 |
f = OneLineExceptionFormatter('%(asctime)s|%(levelname)s|%(message)s|', |
| 2954 |
'%d/%m/%Y %H:%M:%S') |
| 2955 |
fh.setFormatter(f) |
| 2956 |
root = logging.getLogger() |
| 2957 |
root.setLevel(logging.DEBUG) |
| 2958 |
root.addHandler(fh) |
| 2959 |
|
| 2960 |
def main(): |
| 2961 |
configure_logging() |
| 2962 |
logging.info('Sample message') |
| 2963 |
try: |
| 2964 |
x = 1 / 0 |
| 2965 |
except ZeroDivisionError as e: |
| 2966 |
logging.exception('ZeroDivisionError: %s', e) |
| 2967 |
|
| 2968 |
if __name__ == '__main__': |
| 2969 |
main() |
| 2970 |
|
| 2971 |
When run, this produces a file with exactly two lines: |
| 2972 |
|
| 2973 |
.. code-block:: none |
| 2974 |
|
| 2975 |
28/01/2015 07:21:23|INFO|Sample message| |
| 2976 |
28/01/2015 07:21:23|ERROR|ZeroDivisionError: division by zero|'Traceback (most recent call last):\n File "logtest7.py", line 30, in main\n x = 1 / 0\nZeroDivisionError: division by zero'| |
| 2977 |
|
| 2978 |
While the above treatment is simplistic, it points the way to how exception |
| 2979 |
information can be formatted to your liking. The :mod:`traceback` module may be |
| 2980 |
helpful for more specialized needs. |
| 2981 |
|
| 2982 |
.. _spoken-messages: |
| 2983 |
|
| 2984 |
Speaking logging messages |
| 2985 |
------------------------- |
| 2986 |
|
| 2987 |
There might be situations when it is desirable to have logging messages rendered |
| 2988 |
in an audible rather than a visible format. This is easy to do if you have |
| 2989 |
text-to-speech (TTS) functionality available in your system, even if it doesn't have |
| 2990 |
a Python binding. Most TTS systems have a command line program you can run, and |
| 2991 |
this can be invoked from a handler using :mod:`subprocess`. It's assumed here |
| 2992 |
that TTS command line programs won't expect to interact with users or take a |
| 2993 |
long time to complete, and that the frequency of logged messages will be not so |
| 2994 |
high as to swamp the user with messages, and that it's acceptable to have the |
| 2995 |
messages spoken one at a time rather than concurrently, The example implementation |
| 2996 |
below waits for one message to be spoken before the next is processed, and this |
| 2997 |
might cause other handlers to be kept waiting. Here is a short example showing |
| 2998 |
the approach, which assumes that the ``espeak`` TTS package is available:: |
| 2999 |
|
| 3000 |
import logging |
| 3001 |
import subprocess |
| 3002 |
import sys |
| 3003 |
|
| 3004 |
class TTSHandler(logging.Handler): |
| 3005 |
def emit(self, record): |
| 3006 |
msg = self.format(record) |
| 3007 |
# Speak slowly in a female English voice |
| 3008 |
cmd = ['espeak', '-s150', '-ven+f3', msg] |
| 3009 |
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, |
| 3010 |
stderr=subprocess.STDOUT) |
| 3011 |
# wait for the program to finish |
| 3012 |
p.communicate() |
| 3013 |
|
| 3014 |
def configure_logging(): |
| 3015 |
h = TTSHandler() |
| 3016 |
root = logging.getLogger() |
| 3017 |
root.addHandler(h) |
| 3018 |
# the default formatter just returns the message |
| 3019 |
root.setLevel(logging.DEBUG) |
| 3020 |
|
| 3021 |
def main(): |
| 3022 |
logging.info('Hello') |
| 3023 |
logging.debug('Goodbye') |
| 3024 |
|
| 3025 |
if __name__ == '__main__': |
| 3026 |
configure_logging() |
| 3027 |
sys.exit(main()) |
| 3028 |
|
| 3029 |
When run, this script should say "Hello" and then "Goodbye" in a female voice. |
| 3030 |
|
| 3031 |
The above approach can, of course, be adapted to other TTS systems and even |
| 3032 |
other systems altogether which can process messages via external programs run |
| 3033 |
from a command line. |
| 3034 |
|
| 3035 |
|
| 3036 |
.. _buffered-logging: |
| 3037 |
|
| 3038 |
Buffering logging messages and outputting them conditionally |
| 3039 |
------------------------------------------------------------ |
| 3040 |
|
| 3041 |
There might be situations where you want to log messages in a temporary area |
| 3042 |
and only output them if a certain condition occurs. For example, you may want to |
| 3043 |
start logging debug events in a function, and if the function completes without |
| 3044 |
errors, you don't want to clutter the log with the collected debug information, |
| 3045 |
but if there is an error, you want all the debug information to be output as well |
| 3046 |
as the error. |
| 3047 |
|
| 3048 |
Here is an example which shows how you could do this using a decorator for your |
| 3049 |
functions where you want logging to behave this way. It makes use of the |
| 3050 |
:class:`logging.handlers.MemoryHandler`, which allows buffering of logged events |
| 3051 |
until some condition occurs, at which point the buffered events are ``flushed`` |
| 3052 |
- passed to another handler (the ``target`` handler) for processing. By default, |
| 3053 |
the ``MemoryHandler`` flushed when its buffer gets filled up or an event whose |
| 3054 |
level is greater than or equal to a specified threshold is seen. You can use this |
| 3055 |
recipe with a more specialised subclass of ``MemoryHandler`` if you want custom |
| 3056 |
flushing behavior. |
| 3057 |
|
| 3058 |
The example script has a simple function, ``foo``, which just cycles through |
| 3059 |
all the logging levels, writing to ``sys.stderr`` to say what level it's about |
| 3060 |
to log at, and then actually logging a message at that level. You can pass a |
| 3061 |
parameter to ``foo`` which, if true, will log at ERROR and CRITICAL levels - |
| 3062 |
otherwise, it only logs at DEBUG, INFO and WARNING levels. |
| 3063 |
|
| 3064 |
The script just arranges to decorate ``foo`` with a decorator which will do the |
| 3065 |
conditional logging that's required. The decorator takes a logger as a parameter |
| 3066 |
and attaches a memory handler for the duration of the call to the decorated |
| 3067 |
function. The decorator can be additionally parameterised using a target handler, |
| 3068 |
a level at which flushing should occur, and a capacity for the buffer (number of |
| 3069 |
records buffered). These default to a :class:`~logging.StreamHandler` which |
| 3070 |
writes to ``sys.stderr``, ``logging.ERROR`` and ``100`` respectively. |
| 3071 |
|
| 3072 |
Here's the script:: |
| 3073 |
|
| 3074 |
import logging |
| 3075 |
from logging.handlers import MemoryHandler |
| 3076 |
import sys |
| 3077 |
|
| 3078 |
logger = logging.getLogger(__name__) |
| 3079 |
logger.addHandler(logging.NullHandler()) |
| 3080 |
|
| 3081 |
def log_if_errors(logger, target_handler=None, flush_level=None, capacity=None): |
| 3082 |
if target_handler is None: |
| 3083 |
target_handler = logging.StreamHandler() |
| 3084 |
if flush_level is None: |
| 3085 |
flush_level = logging.ERROR |
| 3086 |
if capacity is None: |
| 3087 |
capacity = 100 |
| 3088 |
handler = MemoryHandler(capacity, flushLevel=flush_level, target=target_handler) |
| 3089 |
|
| 3090 |
def decorator(fn): |
| 3091 |
def wrapper(*args, **kwargs): |
| 3092 |
logger.addHandler(handler) |
| 3093 |
try: |
| 3094 |
return fn(*args, **kwargs) |
| 3095 |
except Exception: |
| 3096 |
logger.exception('call failed') |
| 3097 |
raise |
| 3098 |
finally: |
| 3099 |
super(MemoryHandler, handler).flush() |
| 3100 |
logger.removeHandler(handler) |
| 3101 |
return wrapper |
| 3102 |
|
| 3103 |
return decorator |
| 3104 |
|
| 3105 |
def write_line(s): |
| 3106 |
sys.stderr.write('%s\n' % s) |
| 3107 |
|
| 3108 |
def foo(fail=False): |
| 3109 |
write_line('about to log at DEBUG ...') |
| 3110 |
logger.debug('Actually logged at DEBUG') |
| 3111 |
write_line('about to log at INFO ...') |
| 3112 |
logger.info('Actually logged at INFO') |
| 3113 |
write_line('about to log at WARNING ...') |
| 3114 |
logger.warning('Actually logged at WARNING') |
| 3115 |
if fail: |
| 3116 |
write_line('about to log at ERROR ...') |
| 3117 |
logger.error('Actually logged at ERROR') |
| 3118 |
write_line('about to log at CRITICAL ...') |
| 3119 |
logger.critical('Actually logged at CRITICAL') |
| 3120 |
return fail |
| 3121 |
|
| 3122 |
decorated_foo = log_if_errors(logger)(foo) |
| 3123 |
|
| 3124 |
if __name__ == '__main__': |
| 3125 |
logger.setLevel(logging.DEBUG) |
| 3126 |
write_line('Calling undecorated foo with False') |
| 3127 |
assert not foo(False) |
| 3128 |
write_line('Calling undecorated foo with True') |
| 3129 |
assert foo(True) |
| 3130 |
write_line('Calling decorated foo with False') |
| 3131 |
assert not decorated_foo(False) |
| 3132 |
write_line('Calling decorated foo with True') |
| 3133 |
assert decorated_foo(True) |
| 3134 |
|
| 3135 |
When this script is run, the following output should be observed: |
| 3136 |
|
| 3137 |
.. code-block:: none |
| 3138 |
|
| 3139 |
Calling undecorated foo with False |
| 3140 |
about to log at DEBUG ... |
| 3141 |
about to log at INFO ... |
| 3142 |
about to log at WARNING ... |
| 3143 |
Calling undecorated foo with True |
| 3144 |
about to log at DEBUG ... |
| 3145 |
about to log at INFO ... |
| 3146 |
about to log at WARNING ... |
| 3147 |
about to log at ERROR ... |
| 3148 |
about to log at CRITICAL ... |
| 3149 |
Calling decorated foo with False |
| 3150 |
about to log at DEBUG ... |
| 3151 |
about to log at INFO ... |
| 3152 |
about to log at WARNING ... |
| 3153 |
Calling decorated foo with True |
| 3154 |
about to log at DEBUG ... |
| 3155 |
about to log at INFO ... |
| 3156 |
about to log at WARNING ... |
| 3157 |
about to log at ERROR ... |
| 3158 |
Actually logged at DEBUG |
| 3159 |
Actually logged at INFO |
| 3160 |
Actually logged at WARNING |
| 3161 |
Actually logged at ERROR |
| 3162 |
about to log at CRITICAL ... |
| 3163 |
Actually logged at CRITICAL |
| 3164 |
|
| 3165 |
As you can see, actual logging output only occurs when an event is logged whose |
| 3166 |
severity is ERROR or greater, but in that case, any previous events at lower |
| 3167 |
severities are also logged. |
| 3168 |
|
| 3169 |
You can of course use the conventional means of decoration:: |
| 3170 |
|
| 3171 |
@log_if_errors(logger) |
| 3172 |
def foo(fail=False): |
| 3173 |
... |
| 3174 |
|
| 3175 |
|
| 3176 |
.. _buffered-smtp: |
| 3177 |
|
| 3178 |
Sending logging messages to email, with buffering |
| 3179 |
------------------------------------------------- |
| 3180 |
|
| 3181 |
To illustrate how you can send log messages via email, so that a set number of |
| 3182 |
messages are sent per email, you can subclass |
| 3183 |
:class:`~logging.handlers.BufferingHandler`. In the following example, which you can |
| 3184 |
adapt to suit your specific needs, a simple test harness is provided which allows you |
| 3185 |
to run the script with command line arguments specifying what you typically need to |
| 3186 |
send things via SMTP. (Run the downloaded script with the ``-h`` argument to see the |
| 3187 |
required and optional arguments.) |
| 3188 |
|
| 3189 |
.. code-block:: python |
| 3190 |
|
| 3191 |
import logging |
| 3192 |
import logging.handlers |
| 3193 |
import smtplib |
| 3194 |
|
| 3195 |
class BufferingSMTPHandler(logging.handlers.BufferingHandler): |
| 3196 |
def __init__(self, mailhost, port, username, password, fromaddr, toaddrs, |
| 3197 |
subject, capacity): |
| 3198 |
logging.handlers.BufferingHandler.__init__(self, capacity) |
| 3199 |
self.mailhost = mailhost |
| 3200 |
self.mailport = port |
| 3201 |
self.username = username |
| 3202 |
self.password = password |
| 3203 |
self.fromaddr = fromaddr |
| 3204 |
if isinstance(toaddrs, str): |
| 3205 |
toaddrs = [toaddrs] |
| 3206 |
self.toaddrs = toaddrs |
| 3207 |
self.subject = subject |
| 3208 |
self.setFormatter(logging.Formatter("%(asctime)s %(levelname)-5s %(message)s")) |
| 3209 |
|
| 3210 |
def flush(self): |
| 3211 |
if len(self.buffer) > 0: |
| 3212 |
try: |
| 3213 |
smtp = smtplib.SMTP(self.mailhost, self.mailport) |
| 3214 |
smtp.starttls() |
| 3215 |
smtp.login(self.username, self.password) |
| 3216 |
msg = "From: %s\r\nTo: %s\r\nSubject: %s\r\n\r\n" % (self.fromaddr, ','.join(self.toaddrs), self.subject) |
| 3217 |
for record in self.buffer: |
| 3218 |
s = self.format(record) |
| 3219 |
msg = msg + s + "\r\n" |
| 3220 |
smtp.sendmail(self.fromaddr, self.toaddrs, msg) |
| 3221 |
smtp.quit() |
| 3222 |
except Exception: |
| 3223 |
if logging.raiseExceptions: |
| 3224 |
raise |
| 3225 |
self.buffer = [] |
| 3226 |
|
| 3227 |
if __name__ == '__main__': |
| 3228 |
import argparse |
| 3229 |
|
| 3230 |
ap = argparse.ArgumentParser() |
| 3231 |
aa = ap.add_argument |
| 3232 |
aa('host', metavar='HOST', help='SMTP server') |
| 3233 |
aa('--port', '-p', type=int, default=587, help='SMTP port') |
| 3234 |
aa('user', metavar='USER', help='SMTP username') |
| 3235 |
aa('password', metavar='PASSWORD', help='SMTP password') |
| 3236 |
aa('to', metavar='TO', help='Addressee for emails') |
| 3237 |
aa('sender', metavar='SENDER', help='Sender email address') |
| 3238 |
aa('--subject', '-s', |
| 3239 |
default='Test Logging email from Python logging module (buffering)', |
| 3240 |
help='Subject of email') |
| 3241 |
options = ap.parse_args() |
| 3242 |
logger = logging.getLogger() |
| 3243 |
logger.setLevel(logging.DEBUG) |
| 3244 |
h = BufferingSMTPHandler(options.host, options.port, options.user, |
| 3245 |
options.password, options.sender, |
| 3246 |
options.to, options.subject, 10) |
| 3247 |
logger.addHandler(h) |
| 3248 |
for i in range(102): |
| 3249 |
logger.info("Info index = %d", i) |
| 3250 |
h.flush() |
| 3251 |
h.close() |
| 3252 |
|
| 3253 |
If you run this script and your SMTP server is correctly set up, you should find that |
| 3254 |
it sends eleven emails to the addressee you specify. The first ten emails will each |
| 3255 |
have ten log messages, and the eleventh will have two messages. That makes up 102 |
| 3256 |
messages as specified in the script. |
| 3257 |
|
| 3258 |
.. _utc-formatting: |
| 3259 |
|
| 3260 |
Formatting times using UTC (GMT) via configuration |
| 3261 |
-------------------------------------------------- |
| 3262 |
|
| 3263 |
Sometimes you want to format times using UTC, which can be done using a class |
| 3264 |
such as ``UTCFormatter``, shown below:: |
| 3265 |
|
| 3266 |
import logging |
| 3267 |
import time |
| 3268 |
|
| 3269 |
class UTCFormatter(logging.Formatter): |
| 3270 |
converter = time.gmtime |
| 3271 |
|
| 3272 |
and you can then use the ``UTCFormatter`` in your code instead of |
| 3273 |
:class:`~logging.Formatter`. If you want to do that via configuration, you can |
| 3274 |
use the :func:`~logging.config.dictConfig` API with an approach illustrated by |
| 3275 |
the following complete example:: |
| 3276 |
|
| 3277 |
import logging |
| 3278 |
import logging.config |
| 3279 |
import time |
| 3280 |
|
| 3281 |
class UTCFormatter(logging.Formatter): |
| 3282 |
converter = time.gmtime |
| 3283 |
|
| 3284 |
LOGGING = { |
| 3285 |
'version': 1, |
| 3286 |
'disable_existing_loggers': False, |
| 3287 |
'formatters': { |
| 3288 |
'utc': { |
| 3289 |
'()': UTCFormatter, |
| 3290 |
'format': '%(asctime)s %(message)s', |
| 3291 |
}, |
| 3292 |
'local': { |
| 3293 |
'format': '%(asctime)s %(message)s', |
| 3294 |
} |
| 3295 |
}, |
| 3296 |
'handlers': { |
| 3297 |
'console1': { |
| 3298 |
'class': 'logging.StreamHandler', |
| 3299 |
'formatter': 'utc', |
| 3300 |
}, |
| 3301 |
'console2': { |
| 3302 |
'class': 'logging.StreamHandler', |
| 3303 |
'formatter': 'local', |
| 3304 |
}, |
| 3305 |
}, |
| 3306 |
'root': { |
| 3307 |
'handlers': ['console1', 'console2'], |
| 3308 |
} |
| 3309 |
} |
| 3310 |
|
| 3311 |
if __name__ == '__main__': |
| 3312 |
logging.config.dictConfig(LOGGING) |
| 3313 |
logging.warning('The local time is %s', time.asctime()) |
| 3314 |
|
| 3315 |
When this script is run, it should print something like: |
| 3316 |
|
| 3317 |
.. code-block:: none |
| 3318 |
|
| 3319 |
2015-10-17 12:53:29,501 The local time is Sat Oct 17 13:53:29 2015 |
| 3320 |
2015-10-17 13:53:29,501 The local time is Sat Oct 17 13:53:29 2015 |
| 3321 |
|
| 3322 |
showing how the time is formatted both as local time and UTC, one for each |
| 3323 |
handler. |
| 3324 |
|
| 3325 |
|
| 3326 |
.. _context-manager: |
| 3327 |
|
| 3328 |
Using a context manager for selective logging |
| 3329 |
--------------------------------------------- |
| 3330 |
|
| 3331 |
There are times when it would be useful to temporarily change the logging |
| 3332 |
configuration and revert it back after doing something. For this, a context |
| 3333 |
manager is the most obvious way of saving and restoring the logging context. |
| 3334 |
Here is a simple example of such a context manager, which allows you to |
| 3335 |
optionally change the logging level and add a logging handler purely in the |
| 3336 |
scope of the context manager:: |
| 3337 |
|
| 3338 |
import logging |
| 3339 |
import sys |
| 3340 |
|
| 3341 |
class LoggingContext: |
| 3342 |
def __init__(self, logger, level=None, handler=None, close=True): |
| 3343 |
self.logger = logger |
| 3344 |
self.level = level |
| 3345 |
self.handler = handler |
| 3346 |
self.close = close |
| 3347 |
|
| 3348 |
def __enter__(self): |
| 3349 |
if self.level is not None: |
| 3350 |
self.old_level = self.logger.level |
| 3351 |
self.logger.setLevel(self.level) |
| 3352 |
if self.handler: |
| 3353 |
self.logger.addHandler(self.handler) |
| 3354 |
|
| 3355 |
def __exit__(self, et, ev, tb): |
| 3356 |
if self.level is not None: |
| 3357 |
self.logger.setLevel(self.old_level) |
| 3358 |
if self.handler: |
| 3359 |
self.logger.removeHandler(self.handler) |
| 3360 |
if self.handler and self.close: |
| 3361 |
self.handler.close() |
| 3362 |
# implicit return of None => don't swallow exceptions |
| 3363 |
|
| 3364 |
If you specify a level value, the logger's level is set to that value in the |
| 3365 |
scope of the with block covered by the context manager. If you specify a |
| 3366 |
handler, it is added to the logger on entry to the block and removed on exit |
| 3367 |
from the block. You can also ask the manager to close the handler for you on |
| 3368 |
block exit - you could do this if you don't need the handler any more. |
| 3369 |
|
| 3370 |
To illustrate how it works, we can add the following block of code to the |
| 3371 |
above:: |
| 3372 |
|
| 3373 |
if __name__ == '__main__': |
| 3374 |
logger = logging.getLogger('foo') |
| 3375 |
logger.addHandler(logging.StreamHandler()) |
| 3376 |
logger.setLevel(logging.INFO) |
| 3377 |
logger.info('1. This should appear just once on stderr.') |
| 3378 |
logger.debug('2. This should not appear.') |
| 3379 |
with LoggingContext(logger, level=logging.DEBUG): |
| 3380 |
logger.debug('3. This should appear once on stderr.') |
| 3381 |
logger.debug('4. This should not appear.') |
| 3382 |
h = logging.StreamHandler(sys.stdout) |
| 3383 |
with LoggingContext(logger, level=logging.DEBUG, handler=h, close=True): |
| 3384 |
logger.debug('5. This should appear twice - once on stderr and once on stdout.') |
| 3385 |
logger.info('6. This should appear just once on stderr.') |
| 3386 |
logger.debug('7. This should not appear.') |
| 3387 |
|
| 3388 |
We initially set the logger's level to ``INFO``, so message #1 appears and |
| 3389 |
message #2 doesn't. We then change the level to ``DEBUG`` temporarily in the |
| 3390 |
following ``with`` block, and so message #3 appears. After the block exits, the |
| 3391 |
logger's level is restored to ``INFO`` and so message #4 doesn't appear. In the |
| 3392 |
next ``with`` block, we set the level to ``DEBUG`` again but also add a handler |
| 3393 |
writing to ``sys.stdout``. Thus, message #5 appears twice on the console (once |
| 3394 |
via ``stderr`` and once via ``stdout``). After the ``with`` statement's |
| 3395 |
completion, the status is as it was before so message #6 appears (like message |
| 3396 |
#1) whereas message #7 doesn't (just like message #2). |
| 3397 |
|
| 3398 |
If we run the resulting script, the result is as follows: |
| 3399 |
|
| 3400 |
.. code-block:: shell-session |
| 3401 |
|
| 3402 |
$ python logctx.py |
| 3403 |
1. This should appear just once on stderr. |
| 3404 |
3. This should appear once on stderr. |
| 3405 |
5. This should appear twice - once on stderr and once on stdout. |
| 3406 |
5. This should appear twice - once on stderr and once on stdout. |
| 3407 |
6. This should appear just once on stderr. |
| 3408 |
|
| 3409 |
If we run it again, but pipe ``stderr`` to ``/dev/null``, we see the following, |
| 3410 |
which is the only message written to ``stdout``: |
| 3411 |
|
| 3412 |
.. code-block:: shell-session |
| 3413 |
|
| 3414 |
$ python logctx.py 2>/dev/null |
| 3415 |
5. This should appear twice - once on stderr and once on stdout. |
| 3416 |
|
| 3417 |
Once again, but piping ``stdout`` to ``/dev/null``, we get: |
| 3418 |
|
| 3419 |
.. code-block:: shell-session |
| 3420 |
|
| 3421 |
$ python logctx.py >/dev/null |
| 3422 |
1. This should appear just once on stderr. |
| 3423 |
3. This should appear once on stderr. |
| 3424 |
5. This should appear twice - once on stderr and once on stdout. |
| 3425 |
6. This should appear just once on stderr. |
| 3426 |
|
| 3427 |
In this case, the message #5 printed to ``stdout`` doesn't appear, as expected. |
| 3428 |
|
| 3429 |
Of course, the approach described here can be generalised, for example to attach |
| 3430 |
logging filters temporarily. Note that the above code works in Python 2 as well |
| 3431 |
as Python 3. |
| 3432 |
|
| 3433 |
|
| 3434 |
.. _starter-template: |
| 3435 |
|
| 3436 |
A CLI application starter template |
| 3437 |
---------------------------------- |
| 3438 |
|
| 3439 |
Here's an example which shows how you can: |
| 3440 |
|
| 3441 |
* Use a logging level based on command-line arguments |
| 3442 |
* Dispatch to multiple subcommands in separate files, all logging at the same |
| 3443 |
level in a consistent way |
| 3444 |
* Make use of simple, minimal configuration |
| 3445 |
|
| 3446 |
Suppose we have a command-line application whose job is to stop, start or |
| 3447 |
restart some services. This could be organised for the purposes of illustration |
| 3448 |
as a file ``app.py`` that is the main script for the application, with individual |
| 3449 |
commands implemented in ``start.py``, ``stop.py`` and ``restart.py``. Suppose |
| 3450 |
further that we want to control the verbosity of the application via a |
| 3451 |
command-line argument, defaulting to ``logging.INFO``. Here's one way that |
| 3452 |
``app.py`` could be written:: |
| 3453 |
|
| 3454 |
import argparse |
| 3455 |
import importlib |
| 3456 |
import logging |
| 3457 |
import os |
| 3458 |
import sys |
| 3459 |
|
| 3460 |
def main(args=None): |
| 3461 |
scriptname = os.path.basename(__file__) |
| 3462 |
parser = argparse.ArgumentParser(scriptname) |
| 3463 |
levels = ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL') |
| 3464 |
parser.add_argument('--log-level', default='INFO', choices=levels) |
| 3465 |
subparsers = parser.add_subparsers(dest='command', |
| 3466 |
help='Available commands:') |
| 3467 |
start_cmd = subparsers.add_parser('start', help='Start a service') |
| 3468 |
start_cmd.add_argument('name', metavar='NAME', |
| 3469 |
help='Name of service to start') |
| 3470 |
stop_cmd = subparsers.add_parser('stop', |
| 3471 |
help='Stop one or more services') |
| 3472 |
stop_cmd.add_argument('names', metavar='NAME', nargs='+', |
| 3473 |
help='Name of service to stop') |
| 3474 |
restart_cmd = subparsers.add_parser('restart', |
| 3475 |
help='Restart one or more services') |
| 3476 |
restart_cmd.add_argument('names', metavar='NAME', nargs='+', |
| 3477 |
help='Name of service to restart') |
| 3478 |
options = parser.parse_args() |
| 3479 |
# the code to dispatch commands could all be in this file. For the purposes |
| 3480 |
# of illustration only, we implement each command in a separate module. |
| 3481 |
try: |
| 3482 |
mod = importlib.import_module(options.command) |
| 3483 |
cmd = getattr(mod, 'command') |
| 3484 |
except (ImportError, AttributeError): |
| 3485 |
print('Unable to find the code for command \'%s\'' % options.command) |
| 3486 |
return 1 |
| 3487 |
# Could get fancy here and load configuration from file or dictionary |
| 3488 |
logging.basicConfig(level=options.log_level, |
| 3489 |
format='%(levelname)s %(name)s %(message)s') |
| 3490 |
cmd(options) |
| 3491 |
|
| 3492 |
if __name__ == '__main__': |
| 3493 |
sys.exit(main()) |
| 3494 |
|
| 3495 |
And the ``start``, ``stop`` and ``restart`` commands can be implemented in |
| 3496 |
separate modules, like so for starting:: |
| 3497 |
|
| 3498 |
# start.py |
| 3499 |
import logging |
| 3500 |
|
| 3501 |
logger = logging.getLogger(__name__) |
| 3502 |
|
| 3503 |
def command(options): |
| 3504 |
logger.debug('About to start %s', options.name) |
| 3505 |
# actually do the command processing here ... |
| 3506 |
logger.info('Started the \'%s\' service.', options.name) |
| 3507 |
|
| 3508 |
and thus for stopping:: |
| 3509 |
|
| 3510 |
# stop.py |
| 3511 |
import logging |
| 3512 |
|
| 3513 |
logger = logging.getLogger(__name__) |
| 3514 |
|
| 3515 |
def command(options): |
| 3516 |
n = len(options.names) |
| 3517 |
if n == 1: |
| 3518 |
plural = '' |
| 3519 |
services = '\'%s\'' % options.names[0] |
| 3520 |
else: |
| 3521 |
plural = 's' |
| 3522 |
services = ', '.join('\'%s\'' % name for name in options.names) |
| 3523 |
i = services.rfind(', ') |
| 3524 |
services = services[:i] + ' and ' + services[i + 2:] |
| 3525 |
logger.debug('About to stop %s', services) |
| 3526 |
# actually do the command processing here ... |
| 3527 |
logger.info('Stopped the %s service%s.', services, plural) |
| 3528 |
|
| 3529 |
and similarly for restarting:: |
| 3530 |
|
| 3531 |
# restart.py |
| 3532 |
import logging |
| 3533 |
|
| 3534 |
logger = logging.getLogger(__name__) |
| 3535 |
|
| 3536 |
def command(options): |
| 3537 |
n = len(options.names) |
| 3538 |
if n == 1: |
| 3539 |
plural = '' |
| 3540 |
services = '\'%s\'' % options.names[0] |
| 3541 |
else: |
| 3542 |
plural = 's' |
| 3543 |
services = ', '.join('\'%s\'' % name for name in options.names) |
| 3544 |
i = services.rfind(', ') |
| 3545 |
services = services[:i] + ' and ' + services[i + 2:] |
| 3546 |
logger.debug('About to restart %s', services) |
| 3547 |
# actually do the command processing here ... |
| 3548 |
logger.info('Restarted the %s service%s.', services, plural) |
| 3549 |
|
| 3550 |
If we run this application with the default log level, we get output like this: |
| 3551 |
|
| 3552 |
.. code-block:: shell-session |
| 3553 |
|
| 3554 |
$ python app.py start foo |
| 3555 |
INFO start Started the 'foo' service. |
| 3556 |
|
| 3557 |
$ python app.py stop foo bar |
| 3558 |
INFO stop Stopped the 'foo' and 'bar' services. |
| 3559 |
|
| 3560 |
$ python app.py restart foo bar baz |
| 3561 |
INFO restart Restarted the 'foo', 'bar' and 'baz' services. |
| 3562 |
|
| 3563 |
The first word is the logging level, and the second word is the module or |
| 3564 |
package name of the place where the event was logged. |
| 3565 |
|
| 3566 |
If we change the logging level, then we can change the information sent to the |
| 3567 |
log. For example, if we want more information: |
| 3568 |
|
| 3569 |
.. code-block:: shell-session |
| 3570 |
|
| 3571 |
$ python app.py --log-level DEBUG start foo |
| 3572 |
DEBUG start About to start foo |
| 3573 |
INFO start Started the 'foo' service. |
| 3574 |
|
| 3575 |
$ python app.py --log-level DEBUG stop foo bar |
| 3576 |
DEBUG stop About to stop 'foo' and 'bar' |
| 3577 |
INFO stop Stopped the 'foo' and 'bar' services. |
| 3578 |
|
| 3579 |
$ python app.py --log-level DEBUG restart foo bar baz |
| 3580 |
DEBUG restart About to restart 'foo', 'bar' and 'baz' |
| 3581 |
INFO restart Restarted the 'foo', 'bar' and 'baz' services. |
| 3582 |
|
| 3583 |
And if we want less: |
| 3584 |
|
| 3585 |
.. code-block:: shell-session |
| 3586 |
|
| 3587 |
$ python app.py --log-level WARNING start foo |
| 3588 |
$ python app.py --log-level WARNING stop foo bar |
| 3589 |
$ python app.py --log-level WARNING restart foo bar baz |
| 3590 |
|
| 3591 |
In this case, the commands don't print anything to the console, since nothing |
| 3592 |
at ``WARNING`` level or above is logged by them. |
| 3593 |
|
| 3594 |
.. _qt-gui: |
| 3595 |
|
| 3596 |
A Qt GUI for logging |
| 3597 |
-------------------- |
| 3598 |
|
| 3599 |
A question that comes up from time to time is about how to log to a GUI |
| 3600 |
application. The `Qt <https://www.qt.io/>`_ framework is a popular |
| 3601 |
cross-platform UI framework with Python bindings using :pypi:`PySide2` |
| 3602 |
or :pypi:`PyQt5` libraries. |
| 3603 |
|
| 3604 |
The following example shows how to log to a Qt GUI. This introduces a simple |
| 3605 |
``QtHandler`` class which takes a callable, which should be a slot in the main |
| 3606 |
thread that does GUI updates. A worker thread is also created to show how you |
| 3607 |
can log to the GUI from both the UI itself (via a button for manual logging) |
| 3608 |
as well as a worker thread doing work in the background (here, just logging |
| 3609 |
messages at random levels with random short delays in between). |
| 3610 |
|
| 3611 |
The worker thread is implemented using Qt's ``QThread`` class rather than the |
| 3612 |
:mod:`threading` module, as there are circumstances where one has to use |
| 3613 |
``QThread``, which offers better integration with other ``Qt`` components. |
| 3614 |
|
| 3615 |
The code should work with recent releases of any of ``PySide6``, ``PyQt6``, |
| 3616 |
``PySide2`` or ``PyQt5``. You should be able to adapt the approach to earlier |
| 3617 |
versions of Qt. Please refer to the comments in the code snippet for more |
| 3618 |
detailed information. |
| 3619 |
|
| 3620 |
.. code-block:: python3 |
| 3621 |
|
| 3622 |
import logging |
| 3623 |
import random |
| 3624 |
import sys |
| 3625 |
import time |
| 3626 |
|
| 3627 |
# Deal with minor differences between different Qt packages |
| 3628 |
try: |
| 3629 |
from PySide6 import QtCore, QtGui, QtWidgets |
| 3630 |
Signal = QtCore.Signal |
| 3631 |
Slot = QtCore.Slot |
| 3632 |
except ImportError: |
| 3633 |
try: |
| 3634 |
from PyQt6 import QtCore, QtGui, QtWidgets |
| 3635 |
Signal = QtCore.pyqtSignal |
| 3636 |
Slot = QtCore.pyqtSlot |
| 3637 |
except ImportError: |
| 3638 |
try: |
| 3639 |
from PySide2 import QtCore, QtGui, QtWidgets |
| 3640 |
Signal = QtCore.Signal |
| 3641 |
Slot = QtCore.Slot |
| 3642 |
except ImportError: |
| 3643 |
from PyQt5 import QtCore, QtGui, QtWidgets |
| 3644 |
Signal = QtCore.pyqtSignal |
| 3645 |
Slot = QtCore.pyqtSlot |
| 3646 |
|
| 3647 |
logger = logging.getLogger(__name__) |
| 3648 |
|
| 3649 |
|
| 3650 |
# |
| 3651 |
# Signals need to be contained in a QObject or subclass in order to be correctly |
| 3652 |
# initialized. |
| 3653 |
# |
| 3654 |
class Signaller(QtCore.QObject): |
| 3655 |
signal = Signal(str, logging.LogRecord) |
| 3656 |
|
| 3657 |
# |
| 3658 |
# Output to a Qt GUI is only supposed to happen on the main thread. So, this |
| 3659 |
# handler is designed to take a slot function which is set up to run in the main |
| 3660 |
# thread. In this example, the function takes a string argument which is a |
| 3661 |
# formatted log message, and the log record which generated it. The formatted |
| 3662 |
# string is just a convenience - you could format a string for output any way |
| 3663 |
# you like in the slot function itself. |
| 3664 |
# |
| 3665 |
# You specify the slot function to do whatever GUI updates you want. The handler |
| 3666 |
# doesn't know or care about specific UI elements. |
| 3667 |
# |
| 3668 |
class QtHandler(logging.Handler): |
| 3669 |
def __init__(self, slotfunc, *args, **kwargs): |
| 3670 |
super().__init__(*args, **kwargs) |
| 3671 |
self.signaller = Signaller() |
| 3672 |
self.signaller.signal.connect(slotfunc) |
| 3673 |
|
| 3674 |
def emit(self, record): |
| 3675 |
s = self.format(record) |
| 3676 |
self.signaller.signal.emit(s, record) |
| 3677 |
|
| 3678 |
# |
| 3679 |
# This example uses QThreads, which means that the threads at the Python level |
| 3680 |
# are named something like "Dummy-1". The function below gets the Qt name of the |
| 3681 |
# current thread. |
| 3682 |
# |
| 3683 |
def ctname(): |
| 3684 |
return QtCore.QThread.currentThread().objectName() |
| 3685 |
|
| 3686 |
|
| 3687 |
# |
| 3688 |
# Used to generate random levels for logging. |
| 3689 |
# |
| 3690 |
LEVELS = (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, |
| 3691 |
logging.CRITICAL) |
| 3692 |
|
| 3693 |
# |
| 3694 |
# This worker class represents work that is done in a thread separate to the |
| 3695 |
# main thread. The way the thread is kicked off to do work is via a button press |
| 3696 |
# that connects to a slot in the worker. |
| 3697 |
# |
| 3698 |
# Because the default threadName value in the LogRecord isn't much use, we add |
| 3699 |
# a qThreadName which contains the QThread name as computed above, and pass that |
| 3700 |
# value in an "extra" dictionary which is used to update the LogRecord with the |
| 3701 |
# QThread name. |
| 3702 |
# |
| 3703 |
# This example worker just outputs messages sequentially, interspersed with |
| 3704 |
# random delays of the order of a few seconds. |
| 3705 |
# |
| 3706 |
class Worker(QtCore.QObject): |
| 3707 |
@Slot() |
| 3708 |
def start(self): |
| 3709 |
extra = {'qThreadName': ctname() } |
| 3710 |
logger.debug('Started work', extra=extra) |
| 3711 |
i = 1 |
| 3712 |
# Let the thread run until interrupted. This allows reasonably clean |
| 3713 |
# thread termination. |
| 3714 |
while not QtCore.QThread.currentThread().isInterruptionRequested(): |
| 3715 |
delay = 0.5 + random.random() * 2 |
| 3716 |
time.sleep(delay) |
| 3717 |
try: |
| 3718 |
if random.random() < 0.1: |
| 3719 |
raise ValueError('Exception raised: %d' % i) |
| 3720 |
else: |
| 3721 |
level = random.choice(LEVELS) |
| 3722 |
logger.log(level, 'Message after delay of %3.1f: %d', delay, i, extra=extra) |
| 3723 |
except ValueError as e: |
| 3724 |
logger.exception('Failed: %s', e, extra=extra) |
| 3725 |
i += 1 |
| 3726 |
|
| 3727 |
# |
| 3728 |
# Implement a simple UI for this cookbook example. This contains: |
| 3729 |
# |
| 3730 |
# * A read-only text edit window which holds formatted log messages |
| 3731 |
# * A button to start work and log stuff in a separate thread |
| 3732 |
# * A button to log something from the main thread |
| 3733 |
# * A button to clear the log window |
| 3734 |
# |
| 3735 |
class Window(QtWidgets.QWidget): |
| 3736 |
|
| 3737 |
COLORS = { |
| 3738 |
logging.DEBUG: 'black', |
| 3739 |
logging.INFO: 'blue', |
| 3740 |
logging.WARNING: 'orange', |
| 3741 |
logging.ERROR: 'red', |
| 3742 |
logging.CRITICAL: 'purple', |
| 3743 |
} |
| 3744 |
|
| 3745 |
def __init__(self, app): |
| 3746 |
super().__init__() |
| 3747 |
self.app = app |
| 3748 |
self.textedit = te = QtWidgets.QPlainTextEdit(self) |
| 3749 |
# Set whatever the default monospace font is for the platform |
| 3750 |
f = QtGui.QFont('nosuchfont') |
| 3751 |
if hasattr(f, 'Monospace'): |
| 3752 |
f.setStyleHint(f.Monospace) |
| 3753 |
else: |
| 3754 |
f.setStyleHint(f.StyleHint.Monospace) # for Qt6 |
| 3755 |
te.setFont(f) |
| 3756 |
te.setReadOnly(True) |
| 3757 |
PB = QtWidgets.QPushButton |
| 3758 |
self.work_button = PB('Start background work', self) |
| 3759 |
self.log_button = PB('Log a message at a random level', self) |
| 3760 |
self.clear_button = PB('Clear log window', self) |
| 3761 |
self.handler = h = QtHandler(self.update_status) |
| 3762 |
# Remember to use qThreadName rather than threadName in the format string. |
| 3763 |
fs = '%(asctime)s %(qThreadName)-12s %(levelname)-8s %(message)s' |
| 3764 |
formatter = logging.Formatter(fs) |
| 3765 |
h.setFormatter(formatter) |
| 3766 |
logger.addHandler(h) |
| 3767 |
# Set up to terminate the QThread when we exit |
| 3768 |
app.aboutToQuit.connect(self.force_quit) |
| 3769 |
|
| 3770 |
# Lay out all the widgets |
| 3771 |
layout = QtWidgets.QVBoxLayout(self) |
| 3772 |
layout.addWidget(te) |
| 3773 |
layout.addWidget(self.work_button) |
| 3774 |
layout.addWidget(self.log_button) |
| 3775 |
layout.addWidget(self.clear_button) |
| 3776 |
self.setFixedSize(900, 400) |
| 3777 |
|
| 3778 |
# Connect the non-worker slots and signals |
| 3779 |
self.log_button.clicked.connect(self.manual_update) |
| 3780 |
self.clear_button.clicked.connect(self.clear_display) |
| 3781 |
|
| 3782 |
# Start a new worker thread and connect the slots for the worker |
| 3783 |
self.start_thread() |
| 3784 |
self.work_button.clicked.connect(self.worker.start) |
| 3785 |
# Once started, the button should be disabled |
| 3786 |
self.work_button.clicked.connect(lambda : self.work_button.setEnabled(False)) |
| 3787 |
|
| 3788 |
def start_thread(self): |
| 3789 |
self.worker = Worker() |
| 3790 |
self.worker_thread = QtCore.QThread() |
| 3791 |
self.worker.setObjectName('Worker') |
| 3792 |
self.worker_thread.setObjectName('WorkerThread') # for qThreadName |
| 3793 |
self.worker.moveToThread(self.worker_thread) |
| 3794 |
# This will start an event loop in the worker thread |
| 3795 |
self.worker_thread.start() |
| 3796 |
|
| 3797 |
def kill_thread(self): |
| 3798 |
# Just tell the worker to stop, then tell it to quit and wait for that |
| 3799 |
# to happen |
| 3800 |
self.worker_thread.requestInterruption() |
| 3801 |
if self.worker_thread.isRunning(): |
| 3802 |
self.worker_thread.quit() |
| 3803 |
self.worker_thread.wait() |
| 3804 |
else: |
| 3805 |
print('worker has already exited.') |
| 3806 |
|
| 3807 |
def force_quit(self): |
| 3808 |
# For use when the window is closed |
| 3809 |
if self.worker_thread.isRunning(): |
| 3810 |
self.kill_thread() |
| 3811 |
|
| 3812 |
# The functions below update the UI and run in the main thread because |
| 3813 |
# that's where the slots are set up |
| 3814 |
|
| 3815 |
@Slot(str, logging.LogRecord) |
| 3816 |
def update_status(self, status, record): |
| 3817 |
color = self.COLORS.get(record.levelno, 'black') |
| 3818 |
s = '<pre><font color="%s">%s</font></pre>' % (color, status) |
| 3819 |
self.textedit.appendHtml(s) |
| 3820 |
|
| 3821 |
@Slot() |
| 3822 |
def manual_update(self): |
| 3823 |
# This function uses the formatted message passed in, but also uses |
| 3824 |
# information from the record to format the message in an appropriate |
| 3825 |
# color according to its severity (level). |
| 3826 |
level = random.choice(LEVELS) |
| 3827 |
extra = {'qThreadName': ctname() } |
| 3828 |
logger.log(level, 'Manually logged!', extra=extra) |
| 3829 |
|
| 3830 |
@Slot() |
| 3831 |
def clear_display(self): |
| 3832 |
self.textedit.clear() |
| 3833 |
|
| 3834 |
|
| 3835 |
def main(): |
| 3836 |
QtCore.QThread.currentThread().setObjectName('MainThread') |
| 3837 |
logging.getLogger().setLevel(logging.DEBUG) |
| 3838 |
app = QtWidgets.QApplication(sys.argv) |
| 3839 |
example = Window(app) |
| 3840 |
example.show() |
| 3841 |
if hasattr(app, 'exec'): |
| 3842 |
rc = app.exec() |
| 3843 |
else: |
| 3844 |
rc = app.exec_() |
| 3845 |
sys.exit(rc) |
| 3846 |
|
| 3847 |
if __name__=='__main__': |
| 3848 |
main() |
| 3849 |
|
| 3850 |
Logging to syslog with RFC5424 support |
| 3851 |
-------------------------------------- |
| 3852 |
|
| 3853 |
Although :rfc:`5424` dates from 2009, most syslog servers are configured by default to |
| 3854 |
use the older :rfc:`3164`, which hails from 2001. When ``logging`` was added to Python |
| 3855 |
in 2003, it supported the earlier (and only existing) protocol at the time. Since |
| 3856 |
RFC 5424 came out, as there has not been widespread deployment of it in syslog |
| 3857 |
servers, the :class:`~logging.handlers.SysLogHandler` functionality has not been |
| 3858 |
updated. |
| 3859 |
|
| 3860 |
RFC 5424 contains some useful features such as support for structured data, and if you |
| 3861 |
need to be able to log to a syslog server with support for it, you can do so with a |
| 3862 |
subclassed handler which looks something like this:: |
| 3863 |
|
| 3864 |
import datetime as dt |
| 3865 |
import logging.handlers |
| 3866 |
import re |
| 3867 |
import socket |
| 3868 |
import time |
| 3869 |
|
| 3870 |
class SysLogHandler5424(logging.handlers.SysLogHandler): |
| 3871 |
|
| 3872 |
tz_offset = re.compile(r'([+-]\d{2})(\d{2})$') |
| 3873 |
escaped = re.compile(r'([\]"\\])') |
| 3874 |
|
| 3875 |
def __init__(self, *args, **kwargs): |
| 3876 |
self.msgid = kwargs.pop('msgid', None) |
| 3877 |
self.appname = kwargs.pop('appname', None) |
| 3878 |
super().__init__(*args, **kwargs) |
| 3879 |
|
| 3880 |
def format(self, record): |
| 3881 |
version = 1 |
| 3882 |
asctime = dt.datetime.fromtimestamp(record.created).isoformat() |
| 3883 |
m = self.tz_offset.match(time.strftime('%z')) |
| 3884 |
has_offset = False |
| 3885 |
if m and time.timezone: |
| 3886 |
hrs, mins = m.groups() |
| 3887 |
if int(hrs) or int(mins): |
| 3888 |
has_offset = True |
| 3889 |
if not has_offset: |
| 3890 |
asctime += 'Z' |
| 3891 |
else: |
| 3892 |
asctime += f'{hrs}:{mins}' |
| 3893 |
try: |
| 3894 |
hostname = socket.gethostname() |
| 3895 |
except Exception: |
| 3896 |
hostname = '-' |
| 3897 |
appname = self.appname or '-' |
| 3898 |
procid = record.process |
| 3899 |
msgid = '-' |
| 3900 |
msg = super().format(record) |
| 3901 |
sdata = '-' |
| 3902 |
if hasattr(record, 'structured_data'): |
| 3903 |
sd = record.structured_data |
| 3904 |
# This should be a dict where the keys are SD-ID and the value is a |
| 3905 |
# dict mapping PARAM-NAME to PARAM-VALUE (refer to the RFC for what these |
| 3906 |
# mean) |
| 3907 |
# There's no error checking here - it's purely for illustration, and you |
| 3908 |
# can adapt this code for use in production environments |
| 3909 |
parts = [] |
| 3910 |
|
| 3911 |
def replacer(m): |
| 3912 |
g = m.groups() |
| 3913 |
return '\\' + g[0] |
| 3914 |
|
| 3915 |
for sdid, dv in sd.items(): |
| 3916 |
part = f'[{sdid}' |
| 3917 |
for k, v in dv.items(): |
| 3918 |
s = str(v) |
| 3919 |
s = self.escaped.sub(replacer, s) |
| 3920 |
part += f' {k}="{s}"' |
| 3921 |
part += ']' |
| 3922 |
parts.append(part) |
| 3923 |
sdata = ''.join(parts) |
| 3924 |
return f'{version} {asctime} {hostname} {appname} {procid} {msgid} {sdata} {msg}' |
| 3925 |
|
| 3926 |
You'll need to be familiar with RFC 5424 to fully understand the above code, and it |
| 3927 |
may be that you have slightly different needs (e.g. for how you pass structural data |
| 3928 |
to the log). Nevertheless, the above should be adaptable to your speciric needs. With |
| 3929 |
the above handler, you'd pass structured data using something like this:: |
| 3930 |
|
| 3931 |
sd = { |
| 3932 |
'foo@12345': {'bar': 'baz', 'baz': 'bozz', 'fizz': r'buzz'}, |
| 3933 |
'foo@54321': {'rab': 'baz', 'zab': 'bozz', 'zzif': r'buzz'} |
| 3934 |
} |
| 3935 |
extra = {'structured_data': sd} |
| 3936 |
i = 1 |
| 3937 |
logger.debug('Message %d', i, extra=extra) |
| 3938 |
|
| 3939 |
How to treat a logger like an output stream |
| 3940 |
------------------------------------------- |
| 3941 |
|
| 3942 |
Sometimes, you need to interface to a third-party API which expects a file-like |
| 3943 |
object to write to, but you want to direct the API's output to a logger. You |
| 3944 |
can do this using a class which wraps a logger with a file-like API. |
| 3945 |
Here's a short script illustrating such a class: |
| 3946 |
|
| 3947 |
.. code-block:: python |
| 3948 |
|
| 3949 |
import logging |
| 3950 |
|
| 3951 |
class LoggerWriter: |
| 3952 |
def __init__(self, logger, level): |
| 3953 |
self.logger = logger |
| 3954 |
self.level = level |
| 3955 |
|
| 3956 |
def write(self, message): |
| 3957 |
if message != '\n': # avoid printing bare newlines, if you like |
| 3958 |
self.logger.log(self.level, message) |
| 3959 |
|
| 3960 |
def flush(self): |
| 3961 |
# doesn't actually do anything, but might be expected of a file-like |
| 3962 |
# object - so optional depending on your situation |
| 3963 |
pass |
| 3964 |
|
| 3965 |
def close(self): |
| 3966 |
# doesn't actually do anything, but might be expected of a file-like |
| 3967 |
# object - so optional depending on your situation. You might want |
| 3968 |
# to set a flag so that later calls to write raise an exception |
| 3969 |
pass |
| 3970 |
|
| 3971 |
def main(): |
| 3972 |
logging.basicConfig(level=logging.DEBUG) |
| 3973 |
logger = logging.getLogger('demo') |
| 3974 |
info_fp = LoggerWriter(logger, logging.INFO) |
| 3975 |
debug_fp = LoggerWriter(logger, logging.DEBUG) |
| 3976 |
print('An INFO message', file=info_fp) |
| 3977 |
print('A DEBUG message', file=debug_fp) |
| 3978 |
|
| 3979 |
if __name__ == "__main__": |
| 3980 |
main() |
| 3981 |
|
| 3982 |
When this script is run, it prints |
| 3983 |
|
| 3984 |
.. code-block:: text |
| 3985 |
|
| 3986 |
INFO:demo:An INFO message |
| 3987 |
DEBUG:demo:A DEBUG message |
| 3988 |
|
| 3989 |
You could also use ``LoggerWriter`` to redirect ``sys.stdout`` and |
| 3990 |
``sys.stderr`` by doing something like this: |
| 3991 |
|
| 3992 |
.. code-block:: python |
| 3993 |
|
| 3994 |
import sys |
| 3995 |
|
| 3996 |
sys.stdout = LoggerWriter(logger, logging.INFO) |
| 3997 |
sys.stderr = LoggerWriter(logger, logging.WARNING) |
| 3998 |
|
| 3999 |
You should do this *after* configuring logging for your needs. In the above |
| 4000 |
example, the :func:`~logging.basicConfig` call does this (using the |
| 4001 |
``sys.stderr`` value *before* it is overwritten by a ``LoggerWriter`` |
| 4002 |
instance). Then, you'd get this kind of result: |
| 4003 |
|
| 4004 |
.. code-block:: pycon |
| 4005 |
|
| 4006 |
>>> print('Foo') |
| 4007 |
INFO:demo:Foo |
| 4008 |
>>> print('Bar', file=sys.stderr) |
| 4009 |
WARNING:demo:Bar |
| 4010 |
>>> |
| 4011 |
|
| 4012 |
Of course, the examples above show output according to the format used by |
| 4013 |
:func:`~logging.basicConfig`, but you can use a different formatter when you |
| 4014 |
configure logging. |
| 4015 |
|
| 4016 |
Note that with the above scheme, you are somewhat at the mercy of buffering and |
| 4017 |
the sequence of write calls which you are intercepting. For example, with the |
| 4018 |
definition of ``LoggerWriter`` above, if you have the snippet |
| 4019 |
|
| 4020 |
.. code-block:: python |
| 4021 |
|
| 4022 |
sys.stderr = LoggerWriter(logger, logging.WARNING) |
| 4023 |
1 / 0 |
| 4024 |
|
| 4025 |
then running the script results in |
| 4026 |
|
| 4027 |
.. code-block:: text |
| 4028 |
|
| 4029 |
WARNING:demo:Traceback (most recent call last): |
| 4030 |
|
| 4031 |
WARNING:demo: File "/home/runner/cookbook-loggerwriter/test.py", line 53, in <module> |
| 4032 |
|
| 4033 |
WARNING:demo: |
| 4034 |
WARNING:demo:main() |
| 4035 |
WARNING:demo: File "/home/runner/cookbook-loggerwriter/test.py", line 49, in main |
| 4036 |
|
| 4037 |
WARNING:demo: |
| 4038 |
WARNING:demo:1 / 0 |
| 4039 |
WARNING:demo:ZeroDivisionError |
| 4040 |
WARNING:demo:: |
| 4041 |
WARNING:demo:division by zero |
| 4042 |
|
| 4043 |
As you can see, this output isn't ideal. That's because the underlying code |
| 4044 |
which writes to ``sys.stderr`` makes multiple writes, each of which results in a |
| 4045 |
separate logged line (for example, the last three lines above). To get around |
| 4046 |
this problem, you need to buffer things and only output log lines when newlines |
| 4047 |
are seen. Let's use a slightly better implementation of ``LoggerWriter``: |
| 4048 |
|
| 4049 |
.. code-block:: python |
| 4050 |
|
| 4051 |
class BufferingLoggerWriter(LoggerWriter): |
| 4052 |
def __init__(self, logger, level): |
| 4053 |
super().__init__(logger, level) |
| 4054 |
self.buffer = '' |
| 4055 |
|
| 4056 |
def write(self, message): |
| 4057 |
if '\n' not in message: |
| 4058 |
self.buffer += message |
| 4059 |
else: |
| 4060 |
parts = message.split('\n') |
| 4061 |
if self.buffer: |
| 4062 |
s = self.buffer + parts.pop(0) |
| 4063 |
self.logger.log(self.level, s) |
| 4064 |
self.buffer = parts.pop() |
| 4065 |
for part in parts: |
| 4066 |
self.logger.log(self.level, part) |
| 4067 |
|
| 4068 |
This just buffers up stuff until a newline is seen, and then logs complete |
| 4069 |
lines. With this approach, you get better output: |
| 4070 |
|
| 4071 |
.. code-block:: text |
| 4072 |
|
| 4073 |
WARNING:demo:Traceback (most recent call last): |
| 4074 |
WARNING:demo: File "/home/runner/cookbook-loggerwriter/main.py", line 55, in <module> |
| 4075 |
WARNING:demo: main() |
| 4076 |
WARNING:demo: File "/home/runner/cookbook-loggerwriter/main.py", line 52, in main |
| 4077 |
WARNING:demo: 1/0 |
| 4078 |
WARNING:demo:ZeroDivisionError: division by zero |
| 4079 |
|
| 4080 |
How to uniformly handle newlines in logging output |
| 4081 |
-------------------------------------------------- |
| 4082 |
|
| 4083 |
Usually, messages that are logged (say to console or file) consist of a single |
| 4084 |
line of text. However, sometimes there is a need to handle messages with |
| 4085 |
multiple lines - whether because a logging format string contains newlines, or |
| 4086 |
logged data contains newlines. If you want to handle such messages uniformly, so |
| 4087 |
that each line in the logged message appears uniformly formatted as if it was |
| 4088 |
logged separately, you can do this using a handler mixin, as in the following |
| 4089 |
snippet: |
| 4090 |
|
| 4091 |
.. code-block:: python |
| 4092 |
|
| 4093 |
# Assume this is in a module mymixins.py |
| 4094 |
import copy |
| 4095 |
|
| 4096 |
class MultilineMixin: |
| 4097 |
def emit(self, record): |
| 4098 |
s = record.getMessage() |
| 4099 |
if '\n' not in s: |
| 4100 |
super().emit(record) |
| 4101 |
else: |
| 4102 |
lines = s.splitlines() |
| 4103 |
rec = copy.copy(record) |
| 4104 |
rec.args = None |
| 4105 |
for line in lines: |
| 4106 |
rec.msg = line |
| 4107 |
super().emit(rec) |
| 4108 |
|
| 4109 |
You can use the mixin as in the following script: |
| 4110 |
|
| 4111 |
.. code-block:: python |
| 4112 |
|
| 4113 |
import logging |
| 4114 |
|
| 4115 |
from mymixins import MultilineMixin |
| 4116 |
|
| 4117 |
logger = logging.getLogger(__name__) |
| 4118 |
|
| 4119 |
class StreamHandler(MultilineMixin, logging.StreamHandler): |
| 4120 |
pass |
| 4121 |
|
| 4122 |
if __name__ == '__main__': |
| 4123 |
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)-9s %(message)s', |
| 4124 |
handlers = [StreamHandler()]) |
| 4125 |
logger.debug('Single line') |
| 4126 |
logger.debug('Multiple lines:\nfool me once ...') |
| 4127 |
logger.debug('Another single line') |
| 4128 |
logger.debug('Multiple lines:\n%s', 'fool me ...\ncan\'t get fooled again') |
| 4129 |
|
| 4130 |
The script, when run, prints something like: |
| 4131 |
|
| 4132 |
.. code-block:: text |
| 4133 |
|
| 4134 |
2025-07-02 13:54:47,234 DEBUG Single line |
| 4135 |
2025-07-02 13:54:47,234 DEBUG Multiple lines: |
| 4136 |
2025-07-02 13:54:47,234 DEBUG fool me once ... |
| 4137 |
2025-07-02 13:54:47,234 DEBUG Another single line |
| 4138 |
2025-07-02 13:54:47,234 DEBUG Multiple lines: |
| 4139 |
2025-07-02 13:54:47,234 DEBUG fool me ... |
| 4140 |
2025-07-02 13:54:47,234 DEBUG can't get fooled again |
| 4141 |
|
| 4142 |
If, on the other hand, you are concerned about `log injection |
| 4143 |
<https://owasp.org/www-community/attacks/Log_Injection>`_, you can use a |
| 4144 |
formatter which escapes newlines, as per the following example: |
| 4145 |
|
| 4146 |
.. code-block:: python |
| 4147 |
|
| 4148 |
import logging |
| 4149 |
|
| 4150 |
logger = logging.getLogger(__name__) |
| 4151 |
|
| 4152 |
class EscapingFormatter(logging.Formatter): |
| 4153 |
def format(self, record): |
| 4154 |
s = super().format(record) |
| 4155 |
return s.replace('\n', r'\n') |
| 4156 |
|
| 4157 |
if __name__ == '__main__': |
| 4158 |
h = logging.StreamHandler() |
| 4159 |
h.setFormatter(EscapingFormatter('%(asctime)s %(levelname)-9s %(message)s')) |
| 4160 |
logging.basicConfig(level=logging.DEBUG, handlers = [h]) |
| 4161 |
logger.debug('Single line') |
| 4162 |
logger.debug('Multiple lines:\nfool me once ...') |
| 4163 |
logger.debug('Another single line') |
| 4164 |
logger.debug('Multiple lines:\n%s', 'fool me ...\ncan\'t get fooled again') |
| 4165 |
|
| 4166 |
You can, of course, use whatever escaping scheme makes the most sense for you. |
| 4167 |
The script, when run, should produce output like this: |
| 4168 |
|
| 4169 |
.. code-block:: text |
| 4170 |
|
| 4171 |
2025-07-09 06:47:33,783 DEBUG Single line |
| 4172 |
2025-07-09 06:47:33,783 DEBUG Multiple lines:\nfool me once ... |
| 4173 |
2025-07-09 06:47:33,783 DEBUG Another single line |
| 4174 |
2025-07-09 06:47:33,783 DEBUG Multiple lines:\nfool me ...\ncan't get fooled again |
| 4175 |
|
| 4176 |
Escaping behaviour can't be the stdlib default , as it would break backwards |
| 4177 |
compatibility. |
| 4178 |
|
| 4179 |
.. patterns-to-avoid: |
| 4180 |
|
| 4181 |
Patterns to avoid |
| 4182 |
----------------- |
| 4183 |
|
| 4184 |
Although the preceding sections have described ways of doing things you might |
| 4185 |
need to do or deal with, it is worth mentioning some usage patterns which are |
| 4186 |
*unhelpful*, and which should therefore be avoided in most cases. The following |
| 4187 |
sections are in no particular order. |
| 4188 |
|
| 4189 |
Opening the same log file multiple times |
| 4190 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 4191 |
|
| 4192 |
On Windows, you will generally not be able to open the same file multiple times |
| 4193 |
as this will lead to a "file is in use by another process" error. However, on |
| 4194 |
POSIX platforms you'll not get any errors if you open the same file multiple |
| 4195 |
times. This could be done accidentally, for example by: |
| 4196 |
|
| 4197 |
* Adding a file handler more than once which references the same file (e.g. by |
| 4198 |
a copy/paste/forget-to-change error). |
| 4199 |
|
| 4200 |
* Opening two files that look different, as they have different names, but are |
| 4201 |
the same because one is a symbolic link to the other. |
| 4202 |
|
| 4203 |
* Forking a process, following which both parent and child have a reference to |
| 4204 |
the same file. This might be through use of the :mod:`multiprocessing` module, |
| 4205 |
for example. |
| 4206 |
|
| 4207 |
Opening a file multiple times might *appear* to work most of the time, but can |
| 4208 |
lead to a number of problems in practice: |
| 4209 |
|
| 4210 |
* Logging output can be garbled because multiple threads or processes try to |
| 4211 |
write to the same file. Although logging guards against concurrent use of the |
| 4212 |
same handler instance by multiple threads, there is no such protection if |
| 4213 |
concurrent writes are attempted by two different threads using two different |
| 4214 |
handler instances which happen to point to the same file. |
| 4215 |
|
| 4216 |
* An attempt to delete a file (e.g. during file rotation) silently fails, |
| 4217 |
because there is another reference pointing to it. This can lead to confusion |
| 4218 |
and wasted debugging time - log entries end up in unexpected places, or are |
| 4219 |
lost altogether. Or a file that was supposed to be moved remains in place, |
| 4220 |
and grows in size unexpectedly despite size-based rotation being supposedly |
| 4221 |
in place. |
| 4222 |
|
| 4223 |
Use the techniques outlined in :ref:`multiple-processes` to circumvent such |
| 4224 |
issues. |
| 4225 |
|
| 4226 |
Using loggers as attributes in a class or passing them as parameters |
| 4227 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 4228 |
|
| 4229 |
While there might be unusual cases where you'll need to do this, in general |
| 4230 |
there is no point because loggers are singletons. Code can always access a |
| 4231 |
given logger instance by name using ``logging.getLogger(name)``, so passing |
| 4232 |
instances around and holding them as instance attributes is pointless. Note |
| 4233 |
that in other languages such as Java and C#, loggers are often static class |
| 4234 |
attributes. However, this pattern doesn't make sense in Python, where the |
| 4235 |
module (and not the class) is the unit of software decomposition. |
| 4236 |
|
| 4237 |
Adding handlers other than :class:`~logging.NullHandler` to a logger in a library |
| 4238 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 4239 |
|
| 4240 |
Configuring logging by adding handlers, formatters and filters is the |
| 4241 |
responsibility of the application developer, not the library developer. If you |
| 4242 |
are maintaining a library, ensure that you don't add handlers to any of your |
| 4243 |
loggers other than a :class:`~logging.NullHandler` instance. |
| 4244 |
|
| 4245 |
Creating a lot of loggers |
| 4246 |
^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 4247 |
|
| 4248 |
Loggers are singletons that are never freed during a script execution, and so |
| 4249 |
creating lots of loggers will use up memory which can't then be freed. Rather |
| 4250 |
than create a logger per e.g. file processed or network connection made, use |
| 4251 |
the :ref:`existing mechanisms <context-info>` for passing contextual |
| 4252 |
information into your logs and restrict the loggers created to those describing |
| 4253 |
areas within your application (generally modules, but occasionally slightly |
| 4254 |
more fine-grained than that). |
| 4255 |
|
| 4256 |
.. _cookbook-ref-links: |
| 4257 |
|
| 4258 |
Other resources |
| 4259 |
--------------- |
| 4260 |
|
| 4261 |
.. seealso:: |
| 4262 |
|
| 4263 |
Module :mod:`logging` |
| 4264 |
API reference for the logging module. |
| 4265 |
|
| 4266 |
Module :mod:`logging.config` |
| 4267 |
Configuration API for the logging module. |
| 4268 |
|
| 4269 |
Module :mod:`logging.handlers` |
| 4270 |
Useful handlers included with the logging module. |
| 4271 |
|
| 4272 |
:ref:`Basic Tutorial <logging-basic-tutorial>` |
| 4273 |
|
| 4274 |
:ref:`Advanced Tutorial <logging-advanced-tutorial>` |