| 1 |
.. _profile: |
| 2 |
|
| 3 |
******************** |
| 4 |
The Python Profilers |
| 5 |
******************** |
| 6 |
|
| 7 |
**Source code:** :source:`Lib/profile.py` and :source:`Lib/pstats.py` |
| 8 |
|
| 9 |
-------------- |
| 10 |
|
| 11 |
.. _profiler-introduction: |
| 12 |
|
| 13 |
Introduction to the profilers |
| 14 |
============================= |
| 15 |
|
| 16 |
.. index:: |
| 17 |
single: deterministic profiling |
| 18 |
single: profiling, deterministic |
| 19 |
|
| 20 |
:mod:`cProfile` and :mod:`profile` provide :dfn:`deterministic profiling` of |
| 21 |
Python programs. A :dfn:`profile` is a set of statistics that describes how |
| 22 |
often and for how long various parts of the program executed. These statistics |
| 23 |
can be formatted into reports via the :mod:`pstats` module. |
| 24 |
|
| 25 |
The Python standard library provides two different implementations of the same |
| 26 |
profiling interface: |
| 27 |
|
| 28 |
1. :mod:`cProfile` is recommended for most users; it's a C extension with |
| 29 |
reasonable overhead that makes it suitable for profiling long-running |
| 30 |
programs. Based on :mod:`lsprof`, contributed by Brett Rosen and Ted |
| 31 |
Czotter. |
| 32 |
|
| 33 |
2. :mod:`profile`, a pure Python module whose interface is imitated by |
| 34 |
:mod:`cProfile`, but which adds significant overhead to profiled programs. |
| 35 |
If you're trying to extend the profiler in some way, the task might be easier |
| 36 |
with this module. Originally designed and written by Jim Roskind. |
| 37 |
|
| 38 |
.. note:: |
| 39 |
|
| 40 |
The profiler modules are designed to provide an execution profile for a given |
| 41 |
program, not for benchmarking purposes (for that, there is :mod:`timeit` for |
| 42 |
reasonably accurate results). This particularly applies to benchmarking |
| 43 |
Python code against C code: the profilers introduce overhead for Python code, |
| 44 |
but not for C-level functions, and so the C code would seem faster than any |
| 45 |
Python one. |
| 46 |
|
| 47 |
|
| 48 |
.. _profile-instant: |
| 49 |
|
| 50 |
Instant User's Manual |
| 51 |
===================== |
| 52 |
|
| 53 |
This section is provided for users that "don't want to read the manual." It |
| 54 |
provides a very brief overview, and allows a user to rapidly perform profiling |
| 55 |
on an existing application. |
| 56 |
|
| 57 |
To profile a function that takes a single argument, you can do:: |
| 58 |
|
| 59 |
import cProfile |
| 60 |
import re |
| 61 |
cProfile.run('re.compile("foo|bar")') |
| 62 |
|
| 63 |
(Use :mod:`profile` instead of :mod:`cProfile` if the latter is not available on |
| 64 |
your system.) |
| 65 |
|
| 66 |
The above action would run :func:`re.compile` and print profile results like |
| 67 |
the following:: |
| 68 |
|
| 69 |
214 function calls (207 primitive calls) in 0.002 seconds |
| 70 |
|
| 71 |
Ordered by: cumulative time |
| 72 |
|
| 73 |
ncalls tottime percall cumtime percall filename:lineno(function) |
| 74 |
1 0.000 0.000 0.002 0.002 {built-in method builtins.exec} |
| 75 |
1 0.000 0.000 0.001 0.001 <string>:1(<module>) |
| 76 |
1 0.000 0.000 0.001 0.001 __init__.py:250(compile) |
| 77 |
1 0.000 0.000 0.001 0.001 __init__.py:289(_compile) |
| 78 |
1 0.000 0.000 0.000 0.000 _compiler.py:759(compile) |
| 79 |
1 0.000 0.000 0.000 0.000 _parser.py:937(parse) |
| 80 |
1 0.000 0.000 0.000 0.000 _compiler.py:598(_code) |
| 81 |
1 0.000 0.000 0.000 0.000 _parser.py:435(_parse_sub) |
| 82 |
|
| 83 |
The first line indicates that 214 calls were monitored. Of those calls, 207 |
| 84 |
were :dfn:`primitive`, meaning that the call was not induced via recursion. The |
| 85 |
next line: ``Ordered by: cumulative time`` indicates the output is sorted |
| 86 |
by the ``cumtime`` values. The column headings include: |
| 87 |
|
| 88 |
ncalls |
| 89 |
for the number of calls. |
| 90 |
|
| 91 |
tottime |
| 92 |
for the total time spent in the given function (and excluding time made in |
| 93 |
calls to sub-functions) |
| 94 |
|
| 95 |
percall |
| 96 |
is the quotient of ``tottime`` divided by ``ncalls`` |
| 97 |
|
| 98 |
cumtime |
| 99 |
is the cumulative time spent in this and all subfunctions (from invocation |
| 100 |
till exit). This figure is accurate *even* for recursive functions. |
| 101 |
|
| 102 |
percall |
| 103 |
is the quotient of ``cumtime`` divided by primitive calls |
| 104 |
|
| 105 |
filename:lineno(function) |
| 106 |
provides the respective data of each function |
| 107 |
|
| 108 |
When there are two numbers in the first column (for example ``3/1``), it means |
| 109 |
that the function recursed. The second value is the number of primitive calls |
| 110 |
and the former is the total number of calls. Note that when the function does |
| 111 |
not recurse, these two values are the same, and only the single figure is |
| 112 |
printed. |
| 113 |
|
| 114 |
Instead of printing the output at the end of the profile run, you can save the |
| 115 |
results to a file by specifying a filename to the :func:`run` function:: |
| 116 |
|
| 117 |
import cProfile |
| 118 |
import re |
| 119 |
cProfile.run('re.compile("foo|bar")', 'restats') |
| 120 |
|
| 121 |
The :class:`pstats.Stats` class reads profile results from a file and formats |
| 122 |
them in various ways. |
| 123 |
|
| 124 |
.. _profile-cli: |
| 125 |
|
| 126 |
.. program:: cProfile |
| 127 |
|
| 128 |
The files :mod:`cProfile` and :mod:`profile` can also be invoked as a script to |
| 129 |
profile another script. For example:: |
| 130 |
|
| 131 |
python -m cProfile [-o output_file] [-s sort_order] (-m module | myscript.py) |
| 132 |
|
| 133 |
.. option:: -o <output_file> |
| 134 |
|
| 135 |
Writes the profile results to a file instead of to stdout. |
| 136 |
|
| 137 |
.. option:: -s <sort_order> |
| 138 |
|
| 139 |
Specifies one of the :func:`~pstats.Stats.sort_stats` sort values |
| 140 |
to sort the output by. |
| 141 |
This only applies when :option:`-o <cProfile -o>` is not supplied. |
| 142 |
|
| 143 |
.. option:: -m <module> |
| 144 |
|
| 145 |
Specifies that a module is being profiled instead of a script. |
| 146 |
|
| 147 |
.. versionadded:: 3.7 |
| 148 |
Added the ``-m`` option to :mod:`cProfile`. |
| 149 |
|
| 150 |
.. versionadded:: 3.8 |
| 151 |
Added the ``-m`` option to :mod:`profile`. |
| 152 |
|
| 153 |
The :mod:`pstats` module's :class:`~pstats.Stats` class has a variety of methods |
| 154 |
for manipulating and printing the data saved into a profile results file:: |
| 155 |
|
| 156 |
import pstats |
| 157 |
from pstats import SortKey |
| 158 |
p = pstats.Stats('restats') |
| 159 |
p.strip_dirs().sort_stats(-1).print_stats() |
| 160 |
|
| 161 |
The :meth:`~pstats.Stats.strip_dirs` method removed the extraneous path from all |
| 162 |
the module names. The :meth:`~pstats.Stats.sort_stats` method sorted all the |
| 163 |
entries according to the standard module/line/name string that is printed. The |
| 164 |
:meth:`~pstats.Stats.print_stats` method printed out all the statistics. You |
| 165 |
might try the following sort calls:: |
| 166 |
|
| 167 |
p.sort_stats(SortKey.NAME) |
| 168 |
p.print_stats() |
| 169 |
|
| 170 |
The first call will actually sort the list by function name, and the second call |
| 171 |
will print out the statistics. The following are some interesting calls to |
| 172 |
experiment with:: |
| 173 |
|
| 174 |
p.sort_stats(SortKey.CUMULATIVE).print_stats(10) |
| 175 |
|
| 176 |
This sorts the profile by cumulative time in a function, and then only prints |
| 177 |
the ten most significant lines. If you want to understand what algorithms are |
| 178 |
taking time, the above line is what you would use. |
| 179 |
|
| 180 |
If you were looking to see what functions were looping a lot, and taking a lot |
| 181 |
of time, you would do:: |
| 182 |
|
| 183 |
p.sort_stats(SortKey.TIME).print_stats(10) |
| 184 |
|
| 185 |
to sort according to time spent within each function, and then print the |
| 186 |
statistics for the top ten functions. |
| 187 |
|
| 188 |
You might also try:: |
| 189 |
|
| 190 |
p.sort_stats(SortKey.FILENAME).print_stats('__init__') |
| 191 |
|
| 192 |
This will sort all the statistics by file name, and then print out statistics |
| 193 |
for only the class init methods (since they are spelled with ``__init__`` in |
| 194 |
them). As one final example, you could try:: |
| 195 |
|
| 196 |
p.sort_stats(SortKey.TIME, SortKey.CUMULATIVE).print_stats(.5, 'init') |
| 197 |
|
| 198 |
This line sorts statistics with a primary key of time, and a secondary key of |
| 199 |
cumulative time, and then prints out some of the statistics. To be specific, the |
| 200 |
list is first culled down to 50% (re: ``.5``) of its original size, then only |
| 201 |
lines containing ``init`` are maintained, and that sub-sub-list is printed. |
| 202 |
|
| 203 |
If you wondered what functions called the above functions, you could now (``p`` |
| 204 |
is still sorted according to the last criteria) do:: |
| 205 |
|
| 206 |
p.print_callers(.5, 'init') |
| 207 |
|
| 208 |
and you would get a list of callers for each of the listed functions. |
| 209 |
|
| 210 |
If you want more functionality, you're going to have to read the manual, or |
| 211 |
guess what the following functions do:: |
| 212 |
|
| 213 |
p.print_callees() |
| 214 |
p.add('restats') |
| 215 |
|
| 216 |
Invoked as a script, the :mod:`pstats` module is a statistics browser for |
| 217 |
reading and examining profile dumps. It has a simple line-oriented interface |
| 218 |
(implemented using :mod:`cmd`) and interactive help. |
| 219 |
|
| 220 |
:mod:`profile` and :mod:`!cProfile` Module Reference |
| 221 |
==================================================== |
| 222 |
|
| 223 |
.. module:: cProfile |
| 224 |
.. module:: profile |
| 225 |
:synopsis: Python source profiler. |
| 226 |
|
| 227 |
Both the :mod:`profile` and :mod:`!cProfile` modules provide the following |
| 228 |
functions: |
| 229 |
|
| 230 |
.. function:: run(command, filename=None, sort=-1) |
| 231 |
|
| 232 |
This function takes a single argument that can be passed to the :func:`exec` |
| 233 |
function, and an optional file name. In all cases this routine executes:: |
| 234 |
|
| 235 |
exec(command, __main__.__dict__, __main__.__dict__) |
| 236 |
|
| 237 |
and gathers profiling statistics from the execution. If no file name is |
| 238 |
present, then this function automatically creates a :class:`~pstats.Stats` |
| 239 |
instance and prints a simple profiling report. If the sort value is specified, |
| 240 |
it is passed to this :class:`~pstats.Stats` instance to control how the |
| 241 |
results are sorted. |
| 242 |
|
| 243 |
.. function:: runctx(command, globals, locals, filename=None, sort=-1) |
| 244 |
|
| 245 |
This function is similar to :func:`run`, with added arguments to supply the |
| 246 |
globals and locals mappings for the *command* string. This routine |
| 247 |
executes:: |
| 248 |
|
| 249 |
exec(command, globals, locals) |
| 250 |
|
| 251 |
and gathers profiling statistics as in the :func:`run` function above. |
| 252 |
|
| 253 |
.. class:: Profile(timer=None, timeunit=0.0, subcalls=True, builtins=True) |
| 254 |
|
| 255 |
This class is normally only used if more precise control over profiling is |
| 256 |
needed than what the :func:`cProfile.run` function provides. |
| 257 |
|
| 258 |
A custom timer can be supplied for measuring how long code takes to run via |
| 259 |
the *timer* argument. This must be a function that returns a single number |
| 260 |
representing the current time. If the number is an integer, the *timeunit* |
| 261 |
specifies a multiplier that specifies the duration of each unit of time. For |
| 262 |
example, if the timer returns times measured in thousands of seconds, the |
| 263 |
time unit would be ``.001``. |
| 264 |
|
| 265 |
Directly using the :class:`Profile` class allows formatting profile results |
| 266 |
without writing the profile data to a file:: |
| 267 |
|
| 268 |
import cProfile, pstats, io |
| 269 |
from pstats import SortKey |
| 270 |
pr = cProfile.Profile() |
| 271 |
pr.enable() |
| 272 |
# ... do something ... |
| 273 |
pr.disable() |
| 274 |
s = io.StringIO() |
| 275 |
sortby = SortKey.CUMULATIVE |
| 276 |
ps = pstats.Stats(pr, stream=s).sort_stats(sortby) |
| 277 |
ps.print_stats() |
| 278 |
print(s.getvalue()) |
| 279 |
|
| 280 |
The :class:`Profile` class can also be used as a context manager (supported |
| 281 |
only in :mod:`!cProfile` module. see :ref:`typecontextmanager`):: |
| 282 |
|
| 283 |
import cProfile |
| 284 |
|
| 285 |
with cProfile.Profile() as pr: |
| 286 |
# ... do something ... |
| 287 |
|
| 288 |
pr.print_stats() |
| 289 |
|
| 290 |
.. versionchanged:: 3.8 |
| 291 |
Added context manager support. |
| 292 |
|
| 293 |
.. method:: enable() |
| 294 |
|
| 295 |
Start collecting profiling data. Only in :mod:`!cProfile`. |
| 296 |
|
| 297 |
.. method:: disable() |
| 298 |
|
| 299 |
Stop collecting profiling data. Only in :mod:`!cProfile`. |
| 300 |
|
| 301 |
.. method:: create_stats() |
| 302 |
|
| 303 |
Stop collecting profiling data and record the results internally |
| 304 |
as the current profile. |
| 305 |
|
| 306 |
.. method:: print_stats(sort=-1) |
| 307 |
|
| 308 |
Create a :class:`~pstats.Stats` object based on the current |
| 309 |
profile and print the results to stdout. |
| 310 |
|
| 311 |
The *sort* parameter specifies the sorting order of the displayed |
| 312 |
statistics. It accepts a single key or a tuple of keys to enable |
| 313 |
multi-level sorting, as in :func:`Stats.sort_stats <pstats.Stats.sort_stats>`. |
| 314 |
|
| 315 |
.. versionadded:: 3.13 |
| 316 |
:meth:`~Profile.print_stats` now accepts a tuple of keys. |
| 317 |
|
| 318 |
.. method:: dump_stats(filename) |
| 319 |
|
| 320 |
Write the results of the current profile to *filename*. |
| 321 |
|
| 322 |
.. method:: run(cmd) |
| 323 |
|
| 324 |
Profile the cmd via :func:`exec`. |
| 325 |
|
| 326 |
.. method:: runctx(cmd, globals, locals) |
| 327 |
|
| 328 |
Profile the cmd via :func:`exec` with the specified global and |
| 329 |
local environment. |
| 330 |
|
| 331 |
.. method:: runcall(func, /, *args, **kwargs) |
| 332 |
|
| 333 |
Profile ``func(*args, **kwargs)`` |
| 334 |
|
| 335 |
Note that profiling will only work if the called command/function actually |
| 336 |
returns. If the interpreter is terminated (e.g. via a :func:`sys.exit` call |
| 337 |
during the called command/function execution) no profiling results will be |
| 338 |
printed. |
| 339 |
|
| 340 |
.. _profile-stats: |
| 341 |
|
| 342 |
The :class:`Stats` Class |
| 343 |
======================== |
| 344 |
|
| 345 |
Analysis of the profiler data is done using the :class:`~pstats.Stats` class. |
| 346 |
|
| 347 |
.. module:: pstats |
| 348 |
:synopsis: Statistics object for use with the profiler. |
| 349 |
|
| 350 |
.. class:: Stats(*filenames or profile, stream=sys.stdout) |
| 351 |
|
| 352 |
This class constructor creates an instance of a "statistics object" from a |
| 353 |
*filename* (or list of filenames) or from a :class:`Profile` instance. Output |
| 354 |
will be printed to the stream specified by *stream*. |
| 355 |
|
| 356 |
The file selected by the above constructor must have been created by the |
| 357 |
corresponding version of :mod:`profile` or :mod:`cProfile`. To be specific, |
| 358 |
there is *no* file compatibility guaranteed with future versions of this |
| 359 |
profiler, and there is no compatibility with files produced by other |
| 360 |
profilers, or the same profiler run on a different operating system. If |
| 361 |
several files are provided, all the statistics for identical functions will |
| 362 |
be coalesced, so that an overall view of several processes can be considered |
| 363 |
in a single report. If additional files need to be combined with data in an |
| 364 |
existing :class:`~pstats.Stats` object, the :meth:`~pstats.Stats.add` method |
| 365 |
can be used. |
| 366 |
|
| 367 |
Instead of reading the profile data from a file, a :class:`cProfile.Profile` |
| 368 |
or :class:`profile.Profile` object can be used as the profile data source. |
| 369 |
|
| 370 |
:class:`Stats` objects have the following methods: |
| 371 |
|
| 372 |
.. method:: strip_dirs() |
| 373 |
|
| 374 |
This method for the :class:`Stats` class removes all leading path |
| 375 |
information from file names. It is very useful in reducing the size of |
| 376 |
the printout to fit within (close to) 80 columns. This method modifies |
| 377 |
the object, and the stripped information is lost. After performing a |
| 378 |
strip operation, the object is considered to have its entries in a |
| 379 |
"random" order, as it was just after object initialization and loading. |
| 380 |
If :meth:`~pstats.Stats.strip_dirs` causes two function names to be |
| 381 |
indistinguishable (they are on the same line of the same filename, and |
| 382 |
have the same function name), then the statistics for these two entries |
| 383 |
are accumulated into a single entry. |
| 384 |
|
| 385 |
|
| 386 |
.. method:: add(*filenames) |
| 387 |
|
| 388 |
This method of the :class:`Stats` class accumulates additional profiling |
| 389 |
information into the current profiling object. Its arguments should refer |
| 390 |
to filenames created by the corresponding version of :func:`profile.run` |
| 391 |
or :func:`cProfile.run`. Statistics for identically named (re: file, line, |
| 392 |
name) functions are automatically accumulated into single function |
| 393 |
statistics. |
| 394 |
|
| 395 |
|
| 396 |
.. method:: dump_stats(filename) |
| 397 |
|
| 398 |
Save the data loaded into the :class:`Stats` object to a file named |
| 399 |
*filename*. The file is created if it does not exist, and is overwritten |
| 400 |
if it already exists. This is equivalent to the method of the same name |
| 401 |
on the :class:`profile.Profile` and :class:`cProfile.Profile` classes. |
| 402 |
|
| 403 |
|
| 404 |
.. method:: sort_stats(*keys) |
| 405 |
|
| 406 |
This method modifies the :class:`Stats` object by sorting it according to |
| 407 |
the supplied criteria. The argument can be either a string or a SortKey |
| 408 |
enum identifying the basis of a sort (example: ``'time'``, ``'name'``, |
| 409 |
``SortKey.TIME`` or ``SortKey.NAME``). The SortKey enums argument have |
| 410 |
advantage over the string argument in that it is more robust and less |
| 411 |
error prone. |
| 412 |
|
| 413 |
When more than one key is provided, then additional keys are used as |
| 414 |
secondary criteria when there is equality in all keys selected before |
| 415 |
them. For example, ``sort_stats(SortKey.NAME, SortKey.FILE)`` will sort |
| 416 |
all the entries according to their function name, and resolve all ties |
| 417 |
(identical function names) by sorting by file name. |
| 418 |
|
| 419 |
For the string argument, abbreviations can be used for any key names, as |
| 420 |
long as the abbreviation is unambiguous. |
| 421 |
|
| 422 |
The following are the valid string and SortKey: |
| 423 |
|
| 424 |
+------------------+---------------------+----------------------+ |
| 425 |
| Valid String Arg | Valid enum Arg | Meaning | |
| 426 |
+==================+=====================+======================+ |
| 427 |
| ``'calls'`` | SortKey.CALLS | call count | |
| 428 |
+------------------+---------------------+----------------------+ |
| 429 |
| ``'cumulative'`` | SortKey.CUMULATIVE | cumulative time | |
| 430 |
+------------------+---------------------+----------------------+ |
| 431 |
| ``'cumtime'`` | N/A | cumulative time | |
| 432 |
+------------------+---------------------+----------------------+ |
| 433 |
| ``'file'`` | N/A | file name | |
| 434 |
+------------------+---------------------+----------------------+ |
| 435 |
| ``'filename'`` | SortKey.FILENAME | file name | |
| 436 |
+------------------+---------------------+----------------------+ |
| 437 |
| ``'module'`` | N/A | file name | |
| 438 |
+------------------+---------------------+----------------------+ |
| 439 |
| ``'ncalls'`` | N/A | call count | |
| 440 |
+------------------+---------------------+----------------------+ |
| 441 |
| ``'pcalls'`` | SortKey.PCALLS | primitive call count | |
| 442 |
+------------------+---------------------+----------------------+ |
| 443 |
| ``'line'`` | SortKey.LINE | line number | |
| 444 |
+------------------+---------------------+----------------------+ |
| 445 |
| ``'name'`` | SortKey.NAME | function name | |
| 446 |
+------------------+---------------------+----------------------+ |
| 447 |
| ``'nfl'`` | SortKey.NFL | name/file/line | |
| 448 |
+------------------+---------------------+----------------------+ |
| 449 |
| ``'stdname'`` | SortKey.STDNAME | standard name | |
| 450 |
+------------------+---------------------+----------------------+ |
| 451 |
| ``'time'`` | SortKey.TIME | internal time | |
| 452 |
+------------------+---------------------+----------------------+ |
| 453 |
| ``'tottime'`` | N/A | internal time | |
| 454 |
+------------------+---------------------+----------------------+ |
| 455 |
|
| 456 |
Note that all sorts on statistics are in descending order (placing most |
| 457 |
time consuming items first), where as name, file, and line number searches |
| 458 |
are in ascending order (alphabetical). The subtle distinction between |
| 459 |
``SortKey.NFL`` and ``SortKey.STDNAME`` is that the standard name is a |
| 460 |
sort of the name as printed, which means that the embedded line numbers |
| 461 |
get compared in an odd way. For example, lines 3, 20, and 40 would (if |
| 462 |
the file names were the same) appear in the string order 20, 3 and 40. |
| 463 |
In contrast, ``SortKey.NFL`` does a numeric compare of the line numbers. |
| 464 |
In fact, ``sort_stats(SortKey.NFL)`` is the same as |
| 465 |
``sort_stats(SortKey.NAME, SortKey.FILENAME, SortKey.LINE)``. |
| 466 |
|
| 467 |
For backward-compatibility reasons, the numeric arguments ``-1``, ``0``, |
| 468 |
``1``, and ``2`` are permitted. They are interpreted as ``'stdname'``, |
| 469 |
``'calls'``, ``'time'``, and ``'cumulative'`` respectively. If this old |
| 470 |
style format (numeric) is used, only one sort key (the numeric key) will |
| 471 |
be used, and additional arguments will be silently ignored. |
| 472 |
|
| 473 |
.. For compatibility with the old profiler. |
| 474 |
|
| 475 |
.. versionadded:: 3.7 |
| 476 |
Added the SortKey enum. |
| 477 |
|
| 478 |
.. method:: reverse_order() |
| 479 |
|
| 480 |
This method for the :class:`Stats` class reverses the ordering of the |
| 481 |
basic list within the object. Note that by default ascending vs |
| 482 |
descending order is properly selected based on the sort key of choice. |
| 483 |
|
| 484 |
.. This method is provided primarily for compatibility with the old |
| 485 |
profiler. |
| 486 |
|
| 487 |
|
| 488 |
.. method:: print_stats(*restrictions) |
| 489 |
|
| 490 |
This method for the :class:`Stats` class prints out a report as described |
| 491 |
in the :func:`profile.run` definition. |
| 492 |
|
| 493 |
The order of the printing is based on the last |
| 494 |
:meth:`~pstats.Stats.sort_stats` operation done on the object (subject to |
| 495 |
caveats in :meth:`~pstats.Stats.add` and |
| 496 |
:meth:`~pstats.Stats.strip_dirs`). |
| 497 |
|
| 498 |
The arguments provided (if any) can be used to limit the list down to the |
| 499 |
significant entries. Initially, the list is taken to be the complete set |
| 500 |
of profiled functions. Each restriction is either an integer (to select a |
| 501 |
count of lines), or a decimal fraction between 0.0 and 1.0 inclusive (to |
| 502 |
select a percentage of lines), or a string that will be interpreted as a |
| 503 |
regular expression (to pattern match the standard name that is printed). |
| 504 |
If several restrictions are provided, then they are applied sequentially. |
| 505 |
For example:: |
| 506 |
|
| 507 |
print_stats(.1, 'foo:') |
| 508 |
|
| 509 |
would first limit the printing to first 10% of list, and then only print |
| 510 |
functions that were part of filename :file:`.\*foo:`. In contrast, the |
| 511 |
command:: |
| 512 |
|
| 513 |
print_stats('foo:', .1) |
| 514 |
|
| 515 |
would limit the list to all functions having file names :file:`.\*foo:`, |
| 516 |
and then proceed to only print the first 10% of them. |
| 517 |
|
| 518 |
|
| 519 |
.. method:: print_callers(*restrictions) |
| 520 |
|
| 521 |
This method for the :class:`Stats` class prints a list of all functions |
| 522 |
that called each function in the profiled database. The ordering is |
| 523 |
identical to that provided by :meth:`~pstats.Stats.print_stats`, and the |
| 524 |
definition of the restricting argument is also identical. Each caller is |
| 525 |
reported on its own line. The format differs slightly depending on the |
| 526 |
profiler that produced the stats: |
| 527 |
|
| 528 |
* With :mod:`profile`, a number is shown in parentheses after each caller |
| 529 |
to show how many times this specific call was made. For convenience, a |
| 530 |
second non-parenthesized number repeats the cumulative time spent in the |
| 531 |
function at the right. |
| 532 |
|
| 533 |
* With :mod:`cProfile`, each caller is preceded by three numbers: the |
| 534 |
number of times this specific call was made, and the total and |
| 535 |
cumulative times spent in the current function while it was invoked by |
| 536 |
this specific caller. |
| 537 |
|
| 538 |
|
| 539 |
.. method:: print_callees(*restrictions) |
| 540 |
|
| 541 |
This method for the :class:`Stats` class prints a list of all function |
| 542 |
that were called by the indicated function. Aside from this reversal of |
| 543 |
direction of calls (re: called vs was called by), the arguments and |
| 544 |
ordering are identical to the :meth:`~pstats.Stats.print_callers` method. |
| 545 |
|
| 546 |
|
| 547 |
.. method:: get_stats_profile() |
| 548 |
|
| 549 |
This method returns an instance of StatsProfile, which contains a mapping |
| 550 |
of function names to instances of FunctionProfile. Each FunctionProfile |
| 551 |
instance holds information related to the function's profile such as how |
| 552 |
long the function took to run, how many times it was called, etc... |
| 553 |
|
| 554 |
.. versionadded:: 3.9 |
| 555 |
Added the following dataclasses: StatsProfile, FunctionProfile. |
| 556 |
Added the following function: get_stats_profile. |
| 557 |
|
| 558 |
.. _deterministic-profiling: |
| 559 |
|
| 560 |
What Is Deterministic Profiling? |
| 561 |
================================ |
| 562 |
|
| 563 |
:dfn:`Deterministic profiling` is meant to reflect the fact that all *function |
| 564 |
call*, *function return*, and *exception* events are monitored, and precise |
| 565 |
timings are made for the intervals between these events (during which time the |
| 566 |
user's code is executing). In contrast, :dfn:`statistical profiling` (which is |
| 567 |
not done by this module) randomly samples the effective instruction pointer, and |
| 568 |
deduces where time is being spent. The latter technique traditionally involves |
| 569 |
less overhead (as the code does not need to be instrumented), but provides only |
| 570 |
relative indications of where time is being spent. |
| 571 |
|
| 572 |
In Python, since there is an interpreter active during execution, the presence |
| 573 |
of instrumented code is not required in order to do deterministic profiling. |
| 574 |
Python automatically provides a :dfn:`hook` (optional callback) for each event. |
| 575 |
In addition, the interpreted nature of Python tends to add so much overhead to |
| 576 |
execution, that deterministic profiling tends to only add small processing |
| 577 |
overhead in typical applications. The result is that deterministic profiling is |
| 578 |
not that expensive, yet provides extensive run time statistics about the |
| 579 |
execution of a Python program. |
| 580 |
|
| 581 |
Call count statistics can be used to identify bugs in code (surprising counts), |
| 582 |
and to identify possible inline-expansion points (high call counts). Internal |
| 583 |
time statistics can be used to identify "hot loops" that should be carefully |
| 584 |
optimized. Cumulative time statistics should be used to identify high level |
| 585 |
errors in the selection of algorithms. Note that the unusual handling of |
| 586 |
cumulative times in this profiler allows statistics for recursive |
| 587 |
implementations of algorithms to be directly compared to iterative |
| 588 |
implementations. |
| 589 |
|
| 590 |
|
| 591 |
.. _profile-limitations: |
| 592 |
|
| 593 |
Limitations |
| 594 |
=========== |
| 595 |
|
| 596 |
One limitation has to do with accuracy of timing information. There is a |
| 597 |
fundamental problem with deterministic profilers involving accuracy. The most |
| 598 |
obvious restriction is that the underlying "clock" is only ticking at a rate |
| 599 |
(typically) of about .001 seconds. Hence no measurements will be more accurate |
| 600 |
than the underlying clock. If enough measurements are taken, then the "error" |
| 601 |
will tend to average out. Unfortunately, removing this first error induces a |
| 602 |
second source of error. |
| 603 |
|
| 604 |
The second problem is that it "takes a while" from when an event is dispatched |
| 605 |
until the profiler's call to get the time actually *gets* the state of the |
| 606 |
clock. Similarly, there is a certain lag when exiting the profiler event |
| 607 |
handler from the time that the clock's value was obtained (and then squirreled |
| 608 |
away), until the user's code is once again executing. As a result, functions |
| 609 |
that are called many times, or call many functions, will typically accumulate |
| 610 |
this error. The error that accumulates in this fashion is typically less than |
| 611 |
the accuracy of the clock (less than one clock tick), but it *can* accumulate |
| 612 |
and become very significant. |
| 613 |
|
| 614 |
The problem is more important with :mod:`profile` than with the lower-overhead |
| 615 |
:mod:`cProfile`. For this reason, :mod:`profile` provides a means of |
| 616 |
calibrating itself for a given platform so that this error can be |
| 617 |
probabilistically (on the average) removed. After the profiler is calibrated, it |
| 618 |
will be more accurate (in a least square sense), but it will sometimes produce |
| 619 |
negative numbers (when call counts are exceptionally low, and the gods of |
| 620 |
probability work against you :-). ) Do *not* be alarmed by negative numbers in |
| 621 |
the profile. They should *only* appear if you have calibrated your profiler, |
| 622 |
and the results are actually better than without calibration. |
| 623 |
|
| 624 |
|
| 625 |
.. _profile-calibration: |
| 626 |
|
| 627 |
Calibration |
| 628 |
=========== |
| 629 |
|
| 630 |
The profiler of the :mod:`profile` module subtracts a constant from each event |
| 631 |
handling time to compensate for the overhead of calling the time function, and |
| 632 |
socking away the results. By default, the constant is 0. The following |
| 633 |
procedure can be used to obtain a better constant for a given platform (see |
| 634 |
:ref:`profile-limitations`). :: |
| 635 |
|
| 636 |
import profile |
| 637 |
pr = profile.Profile() |
| 638 |
for i in range(5): |
| 639 |
print(pr.calibrate(10000)) |
| 640 |
|
| 641 |
The method executes the number of Python calls given by the argument, directly |
| 642 |
and again under the profiler, measuring the time for both. It then computes the |
| 643 |
hidden overhead per profiler event, and returns that as a float. For example, |
| 644 |
on a 1.8Ghz Intel Core i5 running macOS, and using Python's time.process_time() as |
| 645 |
the timer, the magical number is about 4.04e-6. |
| 646 |
|
| 647 |
The object of this exercise is to get a fairly consistent result. If your |
| 648 |
computer is *very* fast, or your timer function has poor resolution, you might |
| 649 |
have to pass 100000, or even 1000000, to get consistent results. |
| 650 |
|
| 651 |
When you have a consistent answer, there are three ways you can use it:: |
| 652 |
|
| 653 |
import profile |
| 654 |
|
| 655 |
# 1. Apply computed bias to all Profile instances created hereafter. |
| 656 |
profile.Profile.bias = your_computed_bias |
| 657 |
|
| 658 |
# 2. Apply computed bias to a specific Profile instance. |
| 659 |
pr = profile.Profile() |
| 660 |
pr.bias = your_computed_bias |
| 661 |
|
| 662 |
# 3. Specify computed bias in instance constructor. |
| 663 |
pr = profile.Profile(bias=your_computed_bias) |
| 664 |
|
| 665 |
If you have a choice, you are better off choosing a smaller constant, and then |
| 666 |
your results will "less often" show up as negative in profile statistics. |
| 667 |
|
| 668 |
.. _profile-timers: |
| 669 |
|
| 670 |
Using a custom timer |
| 671 |
==================== |
| 672 |
|
| 673 |
If you want to change how current time is determined (for example, to force use |
| 674 |
of wall-clock time or elapsed process time), pass the timing function you want |
| 675 |
to the :class:`Profile` class constructor:: |
| 676 |
|
| 677 |
pr = profile.Profile(your_time_func) |
| 678 |
|
| 679 |
The resulting profiler will then call ``your_time_func``. Depending on whether |
| 680 |
you are using :class:`profile.Profile` or :class:`cProfile.Profile`, |
| 681 |
``your_time_func``'s return value will be interpreted differently: |
| 682 |
|
| 683 |
:class:`profile.Profile` |
| 684 |
``your_time_func`` should return a single number, or a list of numbers whose |
| 685 |
sum is the current time (like what :func:`os.times` returns). If the |
| 686 |
function returns a single time number, or the list of returned numbers has |
| 687 |
length 2, then you will get an especially fast version of the dispatch |
| 688 |
routine. |
| 689 |
|
| 690 |
Be warned that you should calibrate the profiler class for the timer function |
| 691 |
that you choose (see :ref:`profile-calibration`). For most machines, a timer |
| 692 |
that returns a lone integer value will provide the best results in terms of |
| 693 |
low overhead during profiling. (:func:`os.times` is *pretty* bad, as it |
| 694 |
returns a tuple of floating-point values). If you want to substitute a |
| 695 |
better timer in the cleanest fashion, derive a class and hardwire a |
| 696 |
replacement dispatch method that best handles your timer call, along with the |
| 697 |
appropriate calibration constant. |
| 698 |
|
| 699 |
:class:`cProfile.Profile` |
| 700 |
``your_time_func`` should return a single number. If it returns integers, |
| 701 |
you can also invoke the class constructor with a second argument specifying |
| 702 |
the real duration of one unit of time. For example, if |
| 703 |
``your_integer_time_func`` returns times measured in thousands of seconds, |
| 704 |
you would construct the :class:`Profile` instance as follows:: |
| 705 |
|
| 706 |
pr = cProfile.Profile(your_integer_time_func, 0.001) |
| 707 |
|
| 708 |
As the :class:`cProfile.Profile` class cannot be calibrated, custom timer |
| 709 |
functions should be used with care and should be as fast as possible. For |
| 710 |
the best results with a custom timer, it might be necessary to hard-code it |
| 711 |
in the C source of the internal :mod:`!_lsprof` module. |
| 712 |
|
| 713 |
Python 3.3 adds several new functions in :mod:`time` that can be used to make |
| 714 |
precise measurements of process or wall-clock time. For example, see |
| 715 |
:func:`time.perf_counter`. |