Debian Codegraph @

Doc/library/doctest.rst

raw
1 :mod:`!doctest` --- Test interactive Python examples
2 ====================================================
3
4 .. module:: doctest
5 :synopsis: Test pieces of code within docstrings.
6
7 .. moduleauthor:: Tim Peters <[email protected]>
8 .. sectionauthor:: Tim Peters <[email protected]>
9 .. sectionauthor:: Moshe Zadka <[email protected]>
10 .. sectionauthor:: Edward Loper <[email protected]>
11
12 **Source code:** :source:`Lib/doctest.py`
13
14 --------------
15
16 The :mod:`doctest` module searches for pieces of text that look like interactive
17 Python sessions, and then executes those sessions to verify that they work
18 exactly as shown. There are several common ways to use doctest:
19
20 * To check that a module's docstrings are up-to-date by verifying that all
21 interactive examples still work as documented.
22
23 * To perform regression testing by verifying that interactive examples from a
24 test file or a test object work as expected.
25
26 * To write tutorial documentation for a package, liberally illustrated with
27 input-output examples. Depending on whether the examples or the expository text
28 are emphasized, this has the flavor of "literate testing" or "executable
29 documentation".
30
31 Here's a complete but small example module::
32
33 """
34 This is the "example" module.
35
36 The example module supplies one function, factorial(). For example,
37
38 >>> factorial(5)
39 120
40 """
41
42 def factorial(n):
43 """Return the factorial of n, an exact integer >= 0.
44
45 >>> [factorial(n) for n in range(6)]
46 [1, 1, 2, 6, 24, 120]
47 >>> factorial(30)
48 265252859812191058636308480000000
49 >>> factorial(-1)
50 Traceback (most recent call last):
51 ...
52 ValueError: n must be >= 0
53
54 Factorials of floats are OK, but the float must be an exact integer:
55 >>> factorial(30.1)
56 Traceback (most recent call last):
57 ...
58 ValueError: n must be exact integer
59 >>> factorial(30.0)
60 265252859812191058636308480000000
61
62 It must also not be ridiculously large:
63 >>> factorial(1e100)
64 Traceback (most recent call last):
65 ...
66 OverflowError: n too large
67 """
68
69 import math
70 if not n >= 0:
71 raise ValueError("n must be >= 0")
72 if math.floor(n) != n:
73 raise ValueError("n must be exact integer")
74 if n+1 == n: # catch a value like 1e300
75 raise OverflowError("n too large")
76 result = 1
77 factor = 2
78 while factor <= n:
79 result *= factor
80 factor += 1
81 return result
82
83
84 if __name__ == "__main__":
85 import doctest
86 doctest.testmod()
87
88 If you run :file:`example.py` directly from the command line, :mod:`doctest`
89 works its magic:
90
91 .. code-block:: shell-session
92
93 $ python example.py
94 $
95
96 There's no output! That's normal, and it means all the examples worked. Pass
97 ``-v`` to the script, and :mod:`doctest` prints a detailed log of what
98 it's trying, and prints a summary at the end:
99
100 .. code-block:: shell-session
101
102 $ python example.py -v
103 Trying:
104 factorial(5)
105 Expecting:
106 120
107 ok
108 Trying:
109 [factorial(n) for n in range(6)]
110 Expecting:
111 [1, 1, 2, 6, 24, 120]
112 ok
113
114 And so on, eventually ending with:
115
116 .. code-block:: none
117
118 Trying:
119 factorial(1e100)
120 Expecting:
121 Traceback (most recent call last):
122 ...
123 OverflowError: n too large
124 ok
125 2 items passed all tests:
126 1 test in __main__
127 6 tests in __main__.factorial
128 7 tests in 2 items.
129 7 passed.
130 Test passed.
131 $
132
133 That's all you need to know to start making productive use of :mod:`doctest`!
134 Jump in. The following sections provide full details. Note that there are many
135 examples of doctests in the standard Python test suite and libraries.
136 Especially useful examples can be found in the standard test file
137 :file:`Lib/test/test_doctest/test_doctest.py`.
138
139
140 .. _doctest-simple-testmod:
141
142 Simple Usage: Checking Examples in Docstrings
143 ---------------------------------------------
144
145 The simplest way to start using doctest (but not necessarily the way you'll
146 continue to do it) is to end each module :mod:`!M` with::
147
148 if __name__ == "__main__":
149 import doctest
150 doctest.testmod()
151
152 :mod:`!doctest` then examines docstrings in module :mod:`!M`.
153
154 Running the module as a script causes the examples in the docstrings to get
155 executed and verified::
156
157 python M.py
158
159 This won't display anything unless an example fails, in which case the failing
160 example(s) and the cause(s) of the failure(s) are printed to stdout, and the
161 final line of output is ``***Test Failed*** N failures.``, where *N* is the
162 number of examples that failed.
163
164 Run it with the ``-v`` switch instead::
165
166 python M.py -v
167
168 and a detailed report of all examples tried is printed to standard output, along
169 with assorted summaries at the end.
170
171 You can force verbose mode by passing ``verbose=True`` to :func:`testmod`, or
172 prohibit it by passing ``verbose=False``. In either of those cases,
173 :data:`sys.argv` is not examined by :func:`testmod` (so passing ``-v`` or not
174 has no effect).
175
176 There is also a command line shortcut for running :func:`testmod`, see section
177 :ref:`doctest-cli`.
178
179 For more information on :func:`testmod`, see section :ref:`doctest-basic-api`.
180
181
182 .. _doctest-simple-testfile:
183
184 Simple Usage: Checking Examples in a Text File
185 ----------------------------------------------
186
187 Another simple application of doctest is testing interactive examples in a text
188 file. This can be done with the :func:`testfile` function::
189
190 import doctest
191 doctest.testfile("example.txt")
192
193 That short script executes and verifies any interactive Python examples
194 contained in the file :file:`example.txt`. The file content is treated as if it
195 were a single giant docstring; the file doesn't need to contain a Python
196 program! For example, perhaps :file:`example.txt` contains this:
197
198 .. code-block:: none
199
200 The ``example`` module
201 ======================
202
203 Using ``factorial``
204 -------------------
205
206 This is an example text file in reStructuredText format. First import
207 ``factorial`` from the ``example`` module:
208
209 >>> from example import factorial
210
211 Now use it:
212
213 >>> factorial(6)
214 120
215
216 Running ``doctest.testfile("example.txt")`` then finds the error in this
217 documentation::
218
219 File "./example.txt", line 14, in example.txt
220 Failed example:
221 factorial(6)
222 Expected:
223 120
224 Got:
225 720
226
227 As with :func:`testmod`, :func:`testfile` won't display anything unless an
228 example fails. If an example does fail, then the failing example(s) and the
229 cause(s) of the failure(s) are printed to stdout, using the same format as
230 :func:`!testmod`.
231
232 By default, :func:`testfile` looks for files in the calling module's directory.
233 See section :ref:`doctest-basic-api` for a description of the optional arguments
234 that can be used to tell it to look for files in other locations.
235
236 Like :func:`testmod`, :func:`testfile`'s verbosity can be set with the
237 ``-v`` command-line switch or with the optional keyword argument
238 *verbose*.
239
240 There is also a command line shortcut for running :func:`testfile`, see section
241 :ref:`doctest-cli`.
242
243 For more information on :func:`testfile`, see section :ref:`doctest-basic-api`.
244
245
246 .. _doctest-cli:
247
248 Command-line Usage
249 ------------------
250
251 The :mod:`doctest` module can be invoked as a script from the command line:
252
253 .. code-block:: bash
254
255 python -m doctest [-v] [-o OPTION] [-f] file [file ...]
256
257 .. program:: doctest
258
259 .. option:: -v, --verbose
260
261 Detailed report of all examples tried is printed to standard output,
262 along with assorted summaries at the end::
263
264 python -m doctest -v example.py
265
266 This will import :file:`example.py` as a standalone module and run
267 :func:`testmod` on it. Note that this may not work correctly if the
268 file is part of a package and imports other submodules from that package.
269
270 If the file name does not end with :file:`.py`, :mod:`!doctest` infers
271 that it must be run with :func:`testfile` instead::
272
273 python -m doctest -v example.txt
274
275 .. option:: -o, --option <option>
276
277 Option flags control various aspects of doctest's behavior, see section
278 :ref:`doctest-options`.
279
280 .. versionadded:: 3.4
281
282 .. option:: -f, --fail-fast
283
284 This is shorthand for ``-o FAIL_FAST``.
285
286 .. versionadded:: 3.4
287
288
289 .. _doctest-how-it-works:
290
291 How It Works
292 ------------
293
294 This section examines in detail how doctest works: which docstrings it looks at,
295 how it finds interactive examples, what execution context it uses, how it
296 handles exceptions, and how option flags can be used to control its behavior.
297 This is the information that you need to know to write doctest examples; for
298 information about actually running doctest on these examples, see the following
299 sections.
300
301
302 .. _doctest-which-docstrings:
303
304 Which Docstrings Are Examined?
305 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
306
307 The module docstring, and all function, class and method docstrings are
308 searched. Objects imported into the module are not searched.
309
310 .. currentmodule:: None
311
312 .. attribute:: module.__test__
313 :no-typesetting:
314
315 .. currentmodule:: doctest
316
317 In addition, there are cases when you want tests to be part of a module but not part
318 of the help text, which requires that the tests not be included in the docstring.
319 Doctest looks for a module-level variable called ``__test__`` and uses it to locate other
320 tests. If ``M.__test__`` exists, it must be a dict, and each
321 entry maps a (string) name to a function object, class object, or string.
322 Function and class object docstrings found from ``M.__test__`` are searched, and
323 strings are treated as if they were docstrings. In output, a key ``K`` in
324 ``M.__test__`` appears with name ``M.__test__.K``.
325
326 For example, place this block of code at the top of :file:`example.py`:
327
328 .. code-block:: python
329
330 __test__ = {
331 'numbers': """
332 >>> factorial(6)
333 720
334
335 >>> [factorial(n) for n in range(6)]
336 [1, 1, 2, 6, 24, 120]
337 """
338 }
339
340 The value of ``example.__test__["numbers"]`` will be treated as a
341 docstring and all the tests inside it will be run. It is
342 important to note that the value can be mapped to a function,
343 class object, or module; if so, :mod:`!doctest`
344 searches them recursively for docstrings, which are then scanned for tests.
345
346 Any classes found are recursively searched similarly, to test docstrings in
347 their contained methods and nested classes.
348
349 .. note::
350
351 ``doctest`` can only automatically discover classes and functions that are
352 defined at the module level or inside other classes.
353
354 Since nested classes and functions only exist when an outer function
355 is called, they cannot be discovered. Define them outside to make them visible.
356
357 .. _doctest-finding-examples:
358
359 How are Docstring Examples Recognized?
360 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
361
362 In most cases a copy-and-paste of an interactive console session works fine,
363 but doctest isn't trying to do an exact emulation of any specific Python shell.
364
365 ::
366
367 >>> # comments are ignored
368 >>> x = 12
369 >>> x
370 12
371 >>> if x == 13:
372 ... print("yes")
373 ... else:
374 ... print("no")
375 ... print("NO")
376 ... print("NO!!!")
377 ...
378 no
379 NO
380 NO!!!
381 >>>
382
383 .. index::
384 single: >>>; interpreter prompt
385 single: ...; interpreter prompt
386
387 Any expected output must immediately follow the final ``'>>> '`` or ``'... '``
388 line containing the code, and the expected output (if any) extends to the next
389 ``'>>> '`` or all-whitespace line.
390
391 The fine print:
392
393 * Expected output cannot contain an all-whitespace line, since such a line is
394 taken to signal the end of expected output. If expected output does contain a
395 blank line, put ``<BLANKLINE>`` in your doctest example each place a blank line
396 is expected.
397
398 * All hard tab characters are expanded to spaces, using 8-column tab stops.
399 Tabs in output generated by the tested code are not modified. Because any
400 hard tabs in the sample output *are* expanded, this means that if the code
401 output includes hard tabs, the only way the doctest can pass is if the
402 :const:`NORMALIZE_WHITESPACE` option or :ref:`directive <doctest-directives>`
403 is in effect.
404 Alternatively, the test can be rewritten to capture the output and compare it
405 to an expected value as part of the test. This handling of tabs in the
406 source was arrived at through trial and error, and has proven to be the least
407 error prone way of handling them. It is possible to use a different
408 algorithm for handling tabs by writing a custom :class:`DocTestParser` class.
409
410 * Output to stdout is captured, but not output to stderr (exception tracebacks
411 are captured via a different means).
412
413 * If you continue a line via backslashing in an interactive session, or for any
414 other reason use a backslash, you should use a raw docstring, which will
415 preserve your backslashes exactly as you type them::
416
417 >>> def f(x):
418 ... r'''Backslashes in a raw docstring: m\n'''
419 ...
420 >>> print(f.__doc__)
421 Backslashes in a raw docstring: m\n
422
423 Otherwise, the backslash will be interpreted as part of the string. For example,
424 the ``\n`` above would be interpreted as a newline character. Alternatively, you
425 can double each backslash in the doctest version (and not use a raw string)::
426
427 >>> def f(x):
428 ... '''Backslashes in a raw docstring: m\\n'''
429 ...
430 >>> print(f.__doc__)
431 Backslashes in a raw docstring: m\n
432
433 * The starting column doesn't matter::
434
435 >>> assert "Easy!"
436 >>> import math
437 >>> math.floor(1.9)
438 1
439
440 and as many leading whitespace characters are stripped from the expected output
441 as appeared in the initial ``'>>> '`` line that started the example.
442
443
444 .. _doctest-execution-context:
445
446 What's the Execution Context?
447 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
448
449 By default, each time :mod:`doctest` finds a docstring to test, it uses a
450 *shallow copy* of :mod:`!M`'s globals, so that running tests doesn't change the
451 module's real globals, and so that one test in :mod:`!M` can't leave behind
452 crumbs that accidentally allow another test to work. This means examples can
453 freely use any names defined at top-level in :mod:`!M`, and names defined earlier
454 in the docstring being run. Examples cannot see names defined in other
455 docstrings.
456
457 You can force use of your own dict as the execution context by passing
458 ``globs=your_dict`` to :func:`testmod` or :func:`testfile` instead.
459
460
461 .. _doctest-exceptions:
462
463 What About Exceptions?
464 ^^^^^^^^^^^^^^^^^^^^^^
465
466 No problem, provided that the traceback is the only output produced by the
467 example: just paste in the traceback. [#]_ Since tracebacks contain details
468 that are likely to change rapidly (for example, exact file paths and line
469 numbers), this is one case where doctest works hard to be flexible in what it
470 accepts.
471
472 Simple example::
473
474 >>> [1, 2, 3].remove(42)
475 Traceback (most recent call last):
476 File "<stdin>", line 1, in <module>
477 ValueError: list.remove(x): x not in list
478
479 That doctest succeeds if :exc:`ValueError` is raised, with the ``list.remove(x):
480 x not in list`` detail as shown.
481
482 The expected output for an exception must start with a traceback header, which
483 may be either of the following two lines, indented the same as the first line of
484 the example::
485
486 Traceback (most recent call last):
487 Traceback (innermost last):
488
489 The traceback header is followed by an optional traceback stack, whose contents
490 are ignored by doctest. The traceback stack is typically omitted, or copied
491 verbatim from an interactive session.
492
493 The traceback stack is followed by the most interesting part: the line(s)
494 containing the exception type and detail. This is usually the last line of a
495 traceback, but can extend across multiple lines if the exception has a
496 multi-line detail::
497
498 >>> raise ValueError('multi\n line\ndetail')
499 Traceback (most recent call last):
500 File "<stdin>", line 1, in <module>
501 ValueError: multi
502 line
503 detail
504
505 The last three lines (starting with :exc:`ValueError`) are compared against the
506 exception's type and detail, and the rest are ignored.
507
508 Best practice is to omit the traceback stack, unless it adds significant
509 documentation value to the example. So the last example is probably better as::
510
511 >>> raise ValueError('multi\n line\ndetail')
512 Traceback (most recent call last):
513 ...
514 ValueError: multi
515 line
516 detail
517
518 Note that tracebacks are treated very specially. In particular, in the
519 rewritten example, the use of ``...`` is independent of doctest's
520 :const:`ELLIPSIS` option. The ellipsis in that example could be left out, or
521 could just as well be three (or three hundred) commas or digits, or an indented
522 transcript of a Monty Python skit.
523
524 Some details you should read once, but won't need to remember:
525
526 * Doctest can't guess whether your expected output came from an exception
527 traceback or from ordinary printing. So, e.g., an example that expects
528 ``ValueError: 42 is prime`` will pass whether :exc:`ValueError` is actually
529 raised or if the example merely prints that traceback text. In practice,
530 ordinary output rarely begins with a traceback header line, so this doesn't
531 create real problems.
532
533 * Each line of the traceback stack (if present) must be indented further than
534 the first line of the example, *or* start with a non-alphanumeric character.
535 The first line following the traceback header indented the same and starting
536 with an alphanumeric is taken to be the start of the exception detail. Of
537 course this does the right thing for genuine tracebacks.
538
539 * When the :const:`IGNORE_EXCEPTION_DETAIL` doctest option is specified,
540 everything following the leftmost colon and any module information in the
541 exception name is ignored.
542
543 * The interactive shell omits the traceback header line for some
544 :exc:`SyntaxError`\ s. But doctest uses the traceback header line to
545 distinguish exceptions from non-exceptions. So in the rare case where you need
546 to test a :exc:`!SyntaxError` that omits the traceback header, you will need to
547 manually add the traceback header line to your test example.
548
549 .. index:: single: ^ (caret); marker
550
551 * For some exceptions, Python displays the position of the error using ``^``
552 markers and tildes::
553
554 >>> 1 + None
555 File "<stdin>", line 1
556 1 + None
557 ~~^~~~~~
558 TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
559
560 Since the lines showing the position of the error come before the exception type
561 and detail, they are not checked by doctest. For example, the following test
562 would pass, even though it puts the ``^`` marker in the wrong location::
563
564 >>> 1 + None
565 File "<stdin>", line 1
566 1 + None
567 ^~~~~~~~
568 TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
569
570
571 .. _option-flags-and-directives:
572 .. _doctest-options:
573
574 Option Flags
575 ^^^^^^^^^^^^
576
577 A number of option flags control various aspects of doctest's behavior.
578 Symbolic names for the flags are supplied as module constants, which can be
579 :ref:`bitwise ORed <bitwise>` together and passed to various functions.
580 The names can also be used in :ref:`doctest directives <doctest-directives>`,
581 and may be passed to the doctest command line interface via the ``-o`` option.
582
583 The first group of options define test semantics, controlling aspects of how
584 doctest decides whether actual output matches an example's expected output:
585
586
587 .. data:: DONT_ACCEPT_TRUE_FOR_1
588
589 By default, if an expected output block contains just ``1``, an actual output
590 block containing just ``1`` or just ``True`` is considered to be a match, and
591 similarly for ``0`` versus ``False``. When :const:`DONT_ACCEPT_TRUE_FOR_1` is
592 specified, neither substitution is allowed. The default behavior caters to that
593 Python changed the return type of many functions from integer to boolean;
594 doctests expecting "little integer" output still work in these cases. This
595 option will probably go away, but not for several years.
596
597
598 .. index:: single: <BLANKLINE>
599 .. data:: DONT_ACCEPT_BLANKLINE
600
601 By default, if an expected output block contains a line containing only the
602 string ``<BLANKLINE>``, then that line will match a blank line in the actual
603 output. Because a genuinely blank line delimits the expected output, this is
604 the only way to communicate that a blank line is expected. When
605 :const:`DONT_ACCEPT_BLANKLINE` is specified, this substitution is not allowed.
606
607
608 .. data:: NORMALIZE_WHITESPACE
609
610 When specified, all sequences of whitespace (blanks and newlines) are treated as
611 equal. Any sequence of whitespace within the expected output will match any
612 sequence of whitespace within the actual output. By default, whitespace must
613 match exactly. :const:`NORMALIZE_WHITESPACE` is especially useful when a line of
614 expected output is very long, and you want to wrap it across multiple lines in
615 your source.
616
617
618 .. index:: single: ...; in doctests
619 .. data:: ELLIPSIS
620
621 When specified, an ellipsis marker (``...``) in the expected output can match
622 any substring in the actual output. This includes substrings that span line
623 boundaries, and empty substrings, so it's best to keep usage of this simple.
624 Complicated uses can lead to the same kinds of "oops, it matched too much!"
625 surprises that ``.*`` is prone to in regular expressions.
626
627
628 .. data:: IGNORE_EXCEPTION_DETAIL
629
630 When specified, doctests expecting exceptions pass so long as an exception
631 of the expected type is raised, even if the details
632 (message and fully qualified exception name) don't match.
633
634 For example, an example expecting ``ValueError: 42`` will pass if the actual
635 exception raised is ``ValueError: 3*14``, but will fail if, say, a
636 :exc:`TypeError` is raised instead.
637 It will also ignore any fully qualified name included before the
638 exception class, which can vary between implementations and versions
639 of Python and the code/libraries in use.
640 Hence, all three of these variations will work with the flag specified:
641
642 .. code-block:: pycon
643
644 >>> raise Exception('message')
645 Traceback (most recent call last):
646 Exception: message
647
648 >>> raise Exception('message')
649 Traceback (most recent call last):
650 builtins.Exception: message
651
652 >>> raise Exception('message')
653 Traceback (most recent call last):
654 __main__.Exception: message
655
656 Note that :const:`ELLIPSIS` can also be used to ignore the
657 details of the exception message, but such a test may still fail based
658 on whether the module name is present or matches exactly.
659
660 .. versionchanged:: 3.2
661 :const:`IGNORE_EXCEPTION_DETAIL` now also ignores any information relating
662 to the module containing the exception under test.
663
664
665 .. data:: SKIP
666
667 When specified, do not run the example at all. This can be useful in contexts
668 where doctest examples serve as both documentation and test cases, and an
669 example should be included for documentation purposes, but should not be
670 checked. E.g., the example's output might be random; or the example might
671 depend on resources which would be unavailable to the test driver.
672
673 The SKIP flag can also be used for temporarily "commenting out" examples.
674
675
676 .. data:: COMPARISON_FLAGS
677
678 A bitmask or'ing together all the comparison flags above.
679
680 The second group of options controls how test failures are reported:
681
682
683 .. data:: REPORT_UDIFF
684
685 When specified, failures that involve multi-line expected and actual outputs are
686 displayed using a unified diff.
687
688
689 .. data:: REPORT_CDIFF
690
691 When specified, failures that involve multi-line expected and actual outputs
692 will be displayed using a context diff.
693
694
695 .. data:: REPORT_NDIFF
696
697 When specified, differences are computed by ``difflib.Differ``, using the same
698 algorithm as the popular :file:`ndiff.py` utility. This is the only method that
699 marks differences within lines as well as across lines. For example, if a line
700 of expected output contains digit ``1`` where actual output contains letter
701 ``l``, a line is inserted with a caret marking the mismatching column positions.
702
703
704 .. data:: REPORT_ONLY_FIRST_FAILURE
705
706 When specified, display the first failing example in each doctest, but suppress
707 output for all remaining examples. This will prevent doctest from reporting
708 correct examples that break because of earlier failures; but it might also hide
709 incorrect examples that fail independently of the first failure. When
710 :const:`REPORT_ONLY_FIRST_FAILURE` is specified, the remaining examples are
711 still run, and still count towards the total number of failures reported; only
712 the output is suppressed.
713
714
715 .. data:: FAIL_FAST
716
717 When specified, exit after the first failing example and don't attempt to run
718 the remaining examples. Thus, the number of failures reported will be at most
719 1. This flag may be useful during debugging, since examples after the first
720 failure won't even produce debugging output.
721
722
723 .. data:: REPORTING_FLAGS
724
725 A bitmask or'ing together all the reporting flags above.
726
727
728 There is also a way to register new option flag names, though this isn't
729 useful unless you intend to extend :mod:`doctest` internals via subclassing:
730
731
732 .. function:: register_optionflag(name)
733
734 Create a new option flag with a given name, and return the new flag's integer
735 value. :func:`register_optionflag` can be used when subclassing
736 :class:`OutputChecker` or :class:`DocTestRunner` to create new options that are
737 supported by your subclasses. :func:`register_optionflag` should always be
738 called using the following idiom::
739
740 MY_FLAG = register_optionflag('MY_FLAG')
741
742
743 .. index::
744 single: # (hash); in doctests
745 single: + (plus); in doctests
746 single: - (minus); in doctests
747 .. _doctest-directives:
748
749 Directives
750 ^^^^^^^^^^
751
752 Doctest directives may be used to modify the :ref:`option flags
753 <doctest-options>` for an individual example. Doctest directives are
754 special Python comments following an example's source code:
755
756 .. productionlist:: doctest
757 directive: "#" "doctest:" `directive_options`
758 directive_options: `directive_option` ("," `directive_option`)*
759 directive_option: `on_or_off` `directive_option_name`
760 on_or_off: "+" | "-"
761 directive_option_name: "DONT_ACCEPT_BLANKLINE" | "NORMALIZE_WHITESPACE" | ...
762
763 Whitespace is not allowed between the ``+`` or ``-`` and the directive option
764 name. The directive option name can be any of the option flag names explained
765 above.
766
767 An example's doctest directives modify doctest's behavior for that single
768 example. Use ``+`` to enable the named behavior, or ``-`` to disable it.
769
770 For example, this test passes:
771
772 .. doctest::
773 :no-trim-doctest-flags:
774
775 >>> print(list(range(20))) # doctest: +NORMALIZE_WHITESPACE
776 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
777 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
778
779 Without the directive it would fail, both because the actual output doesn't have
780 two blanks before the single-digit list elements, and because the actual output
781 is on a single line. This test also passes, and also requires a directive to do
782 so:
783
784 .. doctest::
785 :no-trim-doctest-flags:
786
787 >>> print(list(range(20))) # doctest: +ELLIPSIS
788 [0, 1, ..., 18, 19]
789
790 Multiple directives can be used on a single physical line, separated by
791 commas:
792
793 .. doctest::
794 :no-trim-doctest-flags:
795
796 >>> print(list(range(20))) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
797 [0, 1, ..., 18, 19]
798
799 If multiple directive comments are used for a single example, then they are
800 combined:
801
802 .. doctest::
803 :no-trim-doctest-flags:
804
805 >>> print(list(range(20))) # doctest: +ELLIPSIS
806 ... # doctest: +NORMALIZE_WHITESPACE
807 [0, 1, ..., 18, 19]
808
809 As the previous example shows, you can add ``...`` lines to your example
810 containing only directives. This can be useful when an example is too long for
811 a directive to comfortably fit on the same line:
812
813 .. doctest::
814 :no-trim-doctest-flags:
815
816 >>> print(list(range(5)) + list(range(10, 20)) + list(range(30, 40)))
817 ... # doctest: +ELLIPSIS
818 [0, ..., 4, 10, ..., 19, 30, ..., 39]
819
820 Note that since all options are disabled by default, and directives apply only
821 to the example they appear in, enabling options (via ``+`` in a directive) is
822 usually the only meaningful choice. However, option flags can also be passed to
823 functions that run doctests, establishing different defaults. In such cases,
824 disabling an option via ``-`` in a directive can be useful.
825
826
827 .. _doctest-warnings:
828
829 Warnings
830 ^^^^^^^^
831
832 :mod:`doctest` is serious about requiring exact matches in expected output. If
833 even a single character doesn't match, the test fails. This will probably
834 surprise you a few times, as you learn exactly what Python does and doesn't
835 guarantee about output. For example, when printing a set, Python doesn't
836 guarantee that the element is printed in any particular order, so a test like ::
837
838 >>> foo()
839 {"spam", "eggs"}
840
841 is vulnerable! One workaround is to do ::
842
843 >>> foo() == {"spam", "eggs"}
844 True
845
846 instead. Another is to do ::
847
848 >>> d = sorted(foo())
849 >>> d
850 ['eggs', 'spam']
851
852 There are others, but you get the idea.
853
854 Another bad idea is to print things that embed an object address, like
855
856 .. doctest::
857
858 >>> id(1.0) # certain to fail some of the time # doctest: +SKIP
859 7948648
860 >>> class C: pass
861 >>> C() # the default repr() for instances embeds an address # doctest: +SKIP
862 <C object at 0x00AC18F0>
863
864 The :const:`ELLIPSIS` directive gives a nice approach for the last example:
865
866 .. doctest::
867 :no-trim-doctest-flags:
868
869 >>> C() # doctest: +ELLIPSIS
870 <C object at 0x...>
871
872 Floating-point numbers are also subject to small output variations across
873 platforms, because Python defers to the platform C library for some
874 floating-point calculations, and C libraries vary widely in quality here. ::
875
876 >>> 1000**0.1 # risky
877 1.9952623149688797
878 >>> round(1000**0.1, 9) # safer
879 1.995262315
880 >>> print(f'{1000**0.1:.4f}') # much safer
881 1.9953
882
883 Numbers of the form ``I/2.**J`` are safe across all platforms, and I often
884 contrive doctest examples to produce numbers of that form::
885
886 >>> 3./4 # utterly safe
887 0.75
888
889 Simple fractions are also easier for people to understand, and that makes for
890 better documentation.
891
892
893 .. _doctest-basic-api:
894
895 Basic API
896 ---------
897
898 The functions :func:`testmod` and :func:`testfile` provide a simple interface to
899 doctest that should be sufficient for most basic uses. For a less formal
900 introduction to these two functions, see sections :ref:`doctest-simple-testmod`
901 and :ref:`doctest-simple-testfile`.
902
903
904 .. function:: testfile(filename, module_relative=True, name=None, package=None, globs=None, verbose=None, report=True, optionflags=0, extraglobs=None, raise_on_error=False, parser=DocTestParser(), encoding=None)
905
906 All arguments except *filename* are optional, and should be specified in keyword
907 form.
908
909 Test examples in the file named *filename*. Return ``(failure_count,
910 test_count)``.
911
912 Optional argument *module_relative* specifies how the filename should be
913 interpreted:
914
915 * If *module_relative* is ``True`` (the default), then *filename* specifies an
916 OS-independent module-relative path. By default, this path is relative to the
917 calling module's directory; but if the *package* argument is specified, then it
918 is relative to that package. To ensure OS-independence, *filename* should use
919 ``/`` characters to separate path segments, and may not be an absolute path
920 (i.e., it may not begin with ``/``).
921
922 * If *module_relative* is ``False``, then *filename* specifies an OS-specific
923 path. The path may be absolute or relative; relative paths are resolved with
924 respect to the current working directory.
925
926 Optional argument *name* gives the name of the test; by default, or if ``None``,
927 ``os.path.basename(filename)`` is used.
928
929 Optional argument *package* is a Python package or the name of a Python package
930 whose directory should be used as the base directory for a module-relative
931 filename. If no package is specified, then the calling module's directory is
932 used as the base directory for module-relative filenames. It is an error to
933 specify *package* if *module_relative* is ``False``.
934
935 Optional argument *globs* gives a dict to be used as the globals when executing
936 examples. A new shallow copy of this dict is created for the doctest, so its
937 examples start with a clean slate. By default, or if ``None``, a new empty dict
938 is used.
939
940 Optional argument *extraglobs* gives a dict merged into the globals used to
941 execute examples. This works like :meth:`dict.update`: if *globs* and
942 *extraglobs* have a common key, the associated value in *extraglobs* appears in
943 the combined dict. By default, or if ``None``, no extra globals are used. This
944 is an advanced feature that allows parameterization of doctests. For example, a
945 doctest can be written for a base class, using a generic name for the class,
946 then reused to test any number of subclasses by passing an *extraglobs* dict
947 mapping the generic name to the subclass to be tested.
948
949 Optional argument *verbose* prints lots of stuff if true, and prints only
950 failures if false; by default, or if ``None``, it's true if and only if ``'-v'``
951 is in :data:`sys.argv`.
952
953 Optional argument *report* prints a summary at the end when true, else prints
954 nothing at the end. In verbose mode, the summary is detailed, else the summary
955 is very brief (in fact, empty if all tests passed).
956
957 Optional argument *optionflags* (default value ``0``) takes the
958 :ref:`bitwise OR <bitwise>` of option flags.
959 See section :ref:`doctest-options`.
960
961 Optional argument *raise_on_error* defaults to false. If true, an exception is
962 raised upon the first failure or unexpected exception in an example. This
963 allows failures to be post-mortem debugged. Default behavior is to continue
964 running examples.
965
966 Optional argument *parser* specifies a :class:`DocTestParser` (or subclass) that
967 should be used to extract tests from the files. It defaults to a normal parser
968 (i.e., ``DocTestParser()``).
969
970 Optional argument *encoding* specifies an encoding that should be used to
971 convert the file to unicode.
972
973
974 .. function:: testmod(m=None, name=None, globs=None, verbose=None, report=True, optionflags=0, extraglobs=None, raise_on_error=False, exclude_empty=False)
975
976 All arguments are optional, and all except for *m* should be specified in
977 keyword form.
978
979 Test examples in docstrings in functions and classes reachable from module *m*
980 (or module :mod:`__main__` if *m* is not supplied or is ``None``), starting with
981 ``m.__doc__``.
982
983 Also test examples reachable from dict ``m.__test__``, if it exists.
984 ``m.__test__`` maps names (strings) to functions, classes and
985 strings; function and class docstrings are searched for examples; strings are
986 searched directly, as if they were docstrings.
987
988 Only docstrings attached to objects belonging to module *m* are searched.
989
990 Return ``(failure_count, test_count)``.
991
992 Optional argument *name* gives the name of the module; by default, or if
993 ``None``, ``m.__name__`` is used.
994
995 Optional argument *exclude_empty* defaults to false. If true, objects for which
996 no doctests are found are excluded from consideration. The default is a backward
997 compatibility hack, so that code still using
998 :meth:`doctest.master.summarize <DocTestRunner.summarize>` in
999 conjunction with :func:`testmod` continues to get output for objects with no
1000 tests. The *exclude_empty* argument to the newer :class:`DocTestFinder`
1001 constructor defaults to true.
1002
1003 Optional arguments *extraglobs*, *verbose*, *report*, *optionflags*,
1004 *raise_on_error*, and *globs* are the same as for function :func:`testfile`
1005 above, except that *globs* defaults to ``m.__dict__``.
1006
1007
1008 .. function:: run_docstring_examples(f, globs, verbose=False, name="NoName", compileflags=None, optionflags=0)
1009
1010 Test examples associated with object *f*; for example, *f* may be a string,
1011 a module, a function, or a class object.
1012
1013 A shallow copy of dictionary argument *globs* is used for the execution context.
1014
1015 Optional argument *name* is used in failure messages, and defaults to
1016 ``"NoName"``.
1017
1018 If optional argument *verbose* is true, output is generated even if there are no
1019 failures. By default, output is generated only in case of an example failure.
1020
1021 Optional argument *compileflags* gives the set of flags that should be used by
1022 the Python compiler when running the examples. By default, or if ``None``,
1023 flags are deduced corresponding to the set of future features found in *globs*.
1024
1025 Optional argument *optionflags* works as for function :func:`testfile` above.
1026
1027
1028 .. _doctest-unittest-api:
1029
1030 Unittest API
1031 ------------
1032
1033 As your collection of doctest'ed modules grows, you'll want a way to run all
1034 their doctests systematically. :mod:`doctest` provides two functions that can
1035 be used to create :mod:`unittest` test suites from modules and text files
1036 containing doctests. To integrate with :mod:`unittest` test discovery, include
1037 a :ref:`load_tests <load_tests-protocol>` function in your test module::
1038
1039 import unittest
1040 import doctest
1041 import my_module_with_doctests
1042
1043 def load_tests(loader, tests, ignore):
1044 tests.addTests(doctest.DocTestSuite(my_module_with_doctests))
1045 return tests
1046
1047 There are two main functions for creating :class:`unittest.TestSuite` instances
1048 from text files and modules with doctests:
1049
1050
1051 .. function:: DocFileSuite(*paths, module_relative=True, package=None, setUp=None, tearDown=None, globs=None, optionflags=0, parser=DocTestParser(), encoding=None)
1052
1053 Convert doctest tests from one or more text files to a
1054 :class:`unittest.TestSuite`.
1055
1056 The returned :class:`unittest.TestSuite` is to be run by the unittest framework
1057 and runs the interactive examples in each file. If an example in any file
1058 fails, then the synthesized unit test fails, and a :exc:`~unittest.TestCase.failureException`
1059 exception is raised showing the name of the file containing the test and a
1060 (sometimes approximate) line number. If all the examples in a file are
1061 skipped, then the synthesized unit test is also marked as skipped.
1062
1063 Pass one or more paths (as strings) to text files to be examined.
1064
1065 Options may be provided as keyword arguments:
1066
1067 Optional argument *module_relative* specifies how the filenames in *paths*
1068 should be interpreted:
1069
1070 * If *module_relative* is ``True`` (the default), then each filename in
1071 *paths* specifies an OS-independent module-relative path. By default, this
1072 path is relative to the calling module's directory; but if the *package*
1073 argument is specified, then it is relative to that package. To ensure
1074 OS-independence, each filename should use ``/`` characters to separate path
1075 segments, and may not be an absolute path (i.e., it may not begin with
1076 ``/``).
1077
1078 * If *module_relative* is ``False``, then each filename in *paths* specifies
1079 an OS-specific path. The path may be absolute or relative; relative paths
1080 are resolved with respect to the current working directory.
1081
1082 Optional argument *package* is a Python package or the name of a Python
1083 package whose directory should be used as the base directory for
1084 module-relative filenames in *paths*. If no package is specified, then the
1085 calling module's directory is used as the base directory for module-relative
1086 filenames. It is an error to specify *package* if *module_relative* is
1087 ``False``.
1088
1089 Optional argument *setUp* specifies a set-up function for the test suite.
1090 This is called before running the tests in each file. The *setUp* function
1091 will be passed a :class:`DocTest` object. The *setUp* function can access the
1092 test globals as the :attr:`~DocTest.globs` attribute of the test passed.
1093
1094 Optional argument *tearDown* specifies a tear-down function for the test
1095 suite. This is called after running the tests in each file. The *tearDown*
1096 function will be passed a :class:`DocTest` object. The *tearDown* function can
1097 access the test globals as the :attr:`~DocTest.globs` attribute of the test
1098 passed.
1099
1100 Optional argument *globs* is a dictionary containing the initial global
1101 variables for the tests. A new copy of this dictionary is created for each
1102 test. By default, *globs* is a new empty dictionary.
1103
1104 Optional argument *optionflags* specifies the default doctest options for the
1105 tests, created by or-ing together individual option flags. See section
1106 :ref:`doctest-options`. See function :func:`set_unittest_reportflags` below
1107 for a better way to set reporting options.
1108
1109 Optional argument *parser* specifies a :class:`DocTestParser` (or subclass)
1110 that should be used to extract tests from the files. It defaults to a normal
1111 parser (i.e., ``DocTestParser()``).
1112
1113 Optional argument *encoding* specifies an encoding that should be used to
1114 convert the file to unicode.
1115
1116 The global ``__file__`` is added to the globals provided to doctests loaded
1117 from a text file using :func:`DocFileSuite`.
1118
1119
1120 .. function:: DocTestSuite(module=None, globs=None, extraglobs=None, test_finder=None, setUp=None, tearDown=None, optionflags=0, checker=None)
1121
1122 Convert doctest tests for a module to a :class:`unittest.TestSuite`.
1123
1124 The returned :class:`unittest.TestSuite` is to be run by the unittest framework
1125 and runs each doctest in the module.
1126 Each docstring is run as a separate unit test.
1127 If any of the doctests fail, then the synthesized unit test fails,
1128 and a :exc:`unittest.TestCase.failureException` exception is raised
1129 showing the name of the file containing the test and a (sometimes approximate)
1130 line number. If all the examples in a docstring are skipped, then the
1131
1132 Optional argument *module* provides the module to be tested. It can be a module
1133 object or a (possibly dotted) module name. If not specified, the module calling
1134 this function is used.
1135
1136 Optional argument *globs* is a dictionary containing the initial global
1137 variables for the tests. A new copy of this dictionary is created for each
1138 test. By default, *globs* is the module's :attr:`~module.__dict__`.
1139
1140 Optional argument *extraglobs* specifies an extra set of global variables, which
1141 is merged into *globs*. By default, no extra globals are used.
1142
1143 Optional argument *test_finder* is the :class:`DocTestFinder` object (or a
1144 drop-in replacement) that is used to extract doctests from the module.
1145
1146 Optional arguments *setUp*, *tearDown*, and *optionflags* are the same as for
1147 function :func:`DocFileSuite` above, but they are called for each docstring.
1148
1149 This function uses the same search technique as :func:`testmod`.
1150
1151 .. versionchanged:: 3.5
1152 :func:`DocTestSuite` returns an empty :class:`unittest.TestSuite` if *module*
1153 contains no docstrings instead of raising :exc:`ValueError`.
1154
1155 Under the covers, :func:`DocTestSuite` creates a :class:`unittest.TestSuite` out
1156 of :class:`!doctest.DocTestCase` instances, and :class:`!DocTestCase` is a
1157 subclass of :class:`unittest.TestCase`. :class:`!DocTestCase` isn't documented
1158 here (it's an internal detail), but studying its code can answer questions about
1159 the exact details of :mod:`unittest` integration.
1160
1161 Similarly, :func:`DocFileSuite` creates a :class:`unittest.TestSuite` out of
1162 :class:`!doctest.DocFileCase` instances, and :class:`!DocFileCase` is a subclass
1163 of :class:`!DocTestCase`.
1164
1165 So both ways of creating a :class:`unittest.TestSuite` run instances of
1166 :class:`!DocTestCase`. This is important for a subtle reason: when you run
1167 :mod:`doctest` functions yourself, you can control the :mod:`!doctest` options in
1168 use directly, by passing option flags to :mod:`!doctest` functions. However, if
1169 you're writing a :mod:`unittest` framework, :mod:`!unittest` ultimately controls
1170 when and how tests get run. The framework author typically wants to control
1171 :mod:`!doctest` reporting options (perhaps, e.g., specified by command line
1172 options), but there's no way to pass options through :mod:`!unittest` to
1173 :mod:`!doctest` test runners.
1174
1175 For this reason, :mod:`doctest` also supports a notion of :mod:`!doctest`
1176 reporting flags specific to :mod:`unittest` support, via this function:
1177
1178
1179 .. function:: set_unittest_reportflags(flags)
1180
1181 Set the :mod:`doctest` reporting flags to use.
1182
1183 Argument *flags* takes the :ref:`bitwise OR <bitwise>` of option flags. See
1184 section :ref:`doctest-options`. Only "reporting flags" can be used.
1185
1186 This is a module-global setting, and affects all future doctests run by module
1187 :mod:`unittest`: the :meth:`!runTest` method of :class:`!DocTestCase` looks at
1188 the option flags specified for the test case when the :class:`!DocTestCase`
1189 instance was constructed. If no reporting flags were specified (which is the
1190 typical and expected case), :mod:`!doctest`'s :mod:`!unittest` reporting flags are
1191 :ref:`bitwise ORed <bitwise>` into the option flags, and the option flags
1192 so augmented are passed to the :class:`DocTestRunner` instance created to
1193 run the doctest. If any reporting flags were specified when the
1194 :class:`!DocTestCase` instance was constructed, :mod:`!doctest`'s
1195 :mod:`!unittest` reporting flags are ignored.
1196
1197 The value of the :mod:`unittest` reporting flags in effect before the function
1198 was called is returned by the function.
1199
1200
1201 .. _doctest-advanced-api:
1202
1203 Advanced API
1204 ------------
1205
1206 The basic API is a simple wrapper that's intended to make doctest easy to use.
1207 It is fairly flexible, and should meet most users' needs; however, if you
1208 require more fine-grained control over testing, or wish to extend doctest's
1209 capabilities, then you should use the advanced API.
1210
1211 The advanced API revolves around two container classes, which are used to store
1212 the interactive examples extracted from doctest cases:
1213
1214 * :class:`Example`: A single Python :term:`statement`, paired with its expected
1215 output.
1216
1217 * :class:`DocTest`: A collection of :class:`Example`\ s, typically extracted
1218 from a single docstring or text file.
1219
1220 Additional processing classes are defined to find, parse, and run, and check
1221 doctest examples:
1222
1223 * :class:`DocTestFinder`: Finds all docstrings in a given module, and uses a
1224 :class:`DocTestParser` to create a :class:`DocTest` from every docstring that
1225 contains interactive examples.
1226
1227 * :class:`DocTestParser`: Creates a :class:`DocTest` object from a string (such
1228 as an object's docstring).
1229
1230 * :class:`DocTestRunner`: Executes the examples in a :class:`DocTest`, and uses
1231 an :class:`OutputChecker` to verify their output.
1232
1233 * :class:`OutputChecker`: Compares the actual output from a doctest example with
1234 the expected output, and decides whether they match.
1235
1236 The relationships among these processing classes are summarized in the following
1237 diagram::
1238
1239 list of:
1240 +------+ +---------+
1241 |module| --DocTestFinder-> | DocTest | --DocTestRunner-> results
1242 +------+ | ^ +---------+ | ^ (printed)
1243 | | | Example | | |
1244 v | | ... | v |
1245 DocTestParser | Example | OutputChecker
1246 +---------+
1247
1248
1249 .. _doctest-doctest:
1250
1251 DocTest Objects
1252 ^^^^^^^^^^^^^^^
1253
1254
1255 .. class:: DocTest(examples, globs, name, filename, lineno, docstring)
1256
1257 A collection of doctest examples that should be run in a single namespace. The
1258 constructor arguments are used to initialize the attributes of the same names.
1259
1260
1261 :class:`DocTest` defines the following attributes. They are initialized by
1262 the constructor, and should not be modified directly.
1263
1264
1265 .. attribute:: examples
1266
1267 A list of :class:`Example` objects encoding the individual interactive Python
1268 examples that should be run by this test.
1269
1270
1271 .. attribute:: globs
1272
1273 The namespace (aka globals) that the examples should be run in. This is a
1274 dictionary mapping names to values. Any changes to the namespace made by the
1275 examples (such as binding new variables) will be reflected in :attr:`globs`
1276 after the test is run.
1277
1278
1279 .. attribute:: name
1280
1281 A string name identifying the :class:`DocTest`. Typically, this is the name
1282 of the object or file that the test was extracted from.
1283
1284
1285 .. attribute:: filename
1286
1287 The name of the file that this :class:`DocTest` was extracted from; or
1288 ``None`` if the filename is unknown, or if the :class:`!DocTest` was not
1289 extracted from a file.
1290
1291
1292 .. attribute:: lineno
1293
1294 The line number within :attr:`filename` where this :class:`DocTest` begins, or
1295 ``None`` if the line number is unavailable. This line number is zero-based
1296 with respect to the beginning of the file.
1297
1298
1299 .. attribute:: docstring
1300
1301 The string that the test was extracted from, or ``None`` if the string is
1302 unavailable, or if the test was not extracted from a string.
1303
1304
1305 .. _doctest-example:
1306
1307 Example Objects
1308 ^^^^^^^^^^^^^^^
1309
1310
1311 .. class:: Example(source, want, exc_msg=None, lineno=0, indent=0, options=None)
1312
1313 A single interactive example, consisting of a Python statement and its expected
1314 output. The constructor arguments are used to initialize the attributes of
1315 the same names.
1316
1317
1318 :class:`Example` defines the following attributes. They are initialized by
1319 the constructor, and should not be modified directly.
1320
1321
1322 .. attribute:: source
1323
1324 A string containing the example's source code. This source code consists of a
1325 single Python statement, and always ends with a newline; the constructor adds
1326 a newline when necessary.
1327
1328
1329 .. attribute:: want
1330
1331 The expected output from running the example's source code (either from
1332 stdout, or a traceback in case of exception). :attr:`want` ends with a
1333 newline unless no output is expected, in which case it's an empty string. The
1334 constructor adds a newline when necessary.
1335
1336
1337 .. attribute:: exc_msg
1338
1339 The exception message generated by the example, if the example is expected to
1340 generate an exception; or ``None`` if it is not expected to generate an
1341 exception. This exception message is compared against the return value of
1342 :func:`traceback.format_exception_only`. :attr:`exc_msg` ends with a newline
1343 unless it's ``None``. The constructor adds a newline if needed.
1344
1345
1346 .. attribute:: lineno
1347
1348 The line number within the string containing this example where the example
1349 begins. This line number is zero-based with respect to the beginning of the
1350 containing string.
1351
1352
1353 .. attribute:: indent
1354
1355 The example's indentation in the containing string, i.e., the number of space
1356 characters that precede the example's first prompt.
1357
1358
1359 .. attribute:: options
1360
1361 A dictionary mapping from option flags to ``True`` or ``False``, which is used
1362 to override default options for this example. Any option flags not contained
1363 in this dictionary are left at their default value (as specified by the
1364 :class:`DocTestRunner`'s :ref:`optionflags <doctest-options>`).
1365 By default, no options are set.
1366
1367
1368 .. _doctest-doctestfinder:
1369
1370 DocTestFinder objects
1371 ^^^^^^^^^^^^^^^^^^^^^
1372
1373
1374 .. class:: DocTestFinder(verbose=False, parser=DocTestParser(), recurse=True, exclude_empty=True)
1375
1376 A processing class used to extract the :class:`DocTest`\ s that are relevant to
1377 a given object, from its docstring and the docstrings of its contained objects.
1378 :class:`DocTest`\ s can be extracted from modules, classes, functions,
1379 methods, staticmethods, classmethods, and properties.
1380
1381 The optional argument *verbose* can be used to display the objects searched by
1382 the finder. It defaults to ``False`` (no output).
1383
1384 The optional argument *parser* specifies the :class:`DocTestParser` object (or a
1385 drop-in replacement) that is used to extract doctests from docstrings.
1386
1387 If the optional argument *recurse* is false, then :meth:`DocTestFinder.find`
1388 will only examine the given object, and not any contained objects.
1389
1390 If the optional argument *exclude_empty* is false, then
1391 :meth:`DocTestFinder.find` will include tests for objects with empty docstrings.
1392
1393
1394 :class:`DocTestFinder` defines the following method:
1395
1396
1397 .. method:: find(obj[, name][, module][, globs][, extraglobs])
1398
1399 Return a list of the :class:`DocTest`\ s that are defined by *obj*'s
1400 docstring, or by any of its contained objects' docstrings.
1401
1402 The optional argument *name* specifies the object's name; this name will be
1403 used to construct names for the returned :class:`DocTest`\ s. If *name* is
1404 not specified, then ``obj.__name__`` is used.
1405
1406 The optional parameter *module* is the module that contains the given object.
1407 If the module is not specified or is ``None``, then the test finder will attempt
1408 to automatically determine the correct module. The object's module is used:
1409
1410 * As a default namespace, if *globs* is not specified.
1411
1412 * To prevent the DocTestFinder from extracting DocTests from objects that are
1413 imported from other modules. (Contained objects with modules other than
1414 *module* are ignored.)
1415
1416 * To find the name of the file containing the object.
1417
1418 * To help find the line number of the object within its file.
1419
1420 If *module* is ``False``, no attempt to find the module will be made. This is
1421 obscure, of use mostly in testing doctest itself: if *module* is ``False``, or
1422 is ``None`` but cannot be found automatically, then all objects are considered
1423 to belong to the (non-existent) module, so all contained objects will
1424 (recursively) be searched for doctests.
1425
1426 The globals for each :class:`DocTest` is formed by combining *globs* and
1427 *extraglobs* (bindings in *extraglobs* override bindings in *globs*). A new
1428 shallow copy of the globals dictionary is created for each :class:`!DocTest`.
1429 If *globs* is not specified, then it defaults to the module's
1430 :attr:`~module.__dict__`, if specified, or ``{}`` otherwise.
1431 If *extraglobs* is not specified, then it defaults to ``{}``.
1432
1433
1434 .. _doctest-doctestparser:
1435
1436 DocTestParser objects
1437 ^^^^^^^^^^^^^^^^^^^^^
1438
1439
1440 .. class:: DocTestParser()
1441
1442 A processing class used to extract interactive examples from a string, and use
1443 them to create a :class:`DocTest` object.
1444
1445
1446 :class:`DocTestParser` defines the following methods:
1447
1448
1449 .. method:: get_doctest(string, globs, name, filename, lineno)
1450
1451 Extract all doctest examples from the given string, and collect them into a
1452 :class:`DocTest` object.
1453
1454 *globs*, *name*, *filename*, and *lineno* are attributes for the new
1455 :class:`!DocTest` object. See the documentation for :class:`DocTest` for more
1456 information.
1457
1458
1459 .. method:: get_examples(string, name='<string>')
1460
1461 Extract all doctest examples from the given string, and return them as a list
1462 of :class:`Example` objects. Line numbers are 0-based. The optional argument
1463 *name* is a name identifying this string, and is only used for error messages.
1464
1465
1466 .. method:: parse(string, name='<string>')
1467
1468 Divide the given string into examples and intervening text, and return them as
1469 a list of alternating :class:`Example`\ s and strings. Line numbers for the
1470 :class:`!Example`\ s are 0-based. The optional argument *name* is a name
1471 identifying this string, and is only used for error messages.
1472
1473
1474 TestResults objects
1475 ^^^^^^^^^^^^^^^^^^^
1476
1477
1478 .. class:: TestResults(failed, attempted)
1479
1480 .. attribute:: failed
1481
1482 Number of failed tests.
1483
1484 .. attribute:: attempted
1485
1486 Number of attempted tests.
1487
1488 .. attribute:: skipped
1489
1490 Number of skipped tests.
1491
1492 .. versionadded:: 3.13
1493
1494
1495 .. _doctest-doctestrunner:
1496
1497 DocTestRunner objects
1498 ^^^^^^^^^^^^^^^^^^^^^
1499
1500
1501 .. class:: DocTestRunner(checker=None, verbose=None, optionflags=0)
1502
1503 A processing class used to execute and verify the interactive examples in a
1504 :class:`DocTest`.
1505
1506 The comparison between expected outputs and actual outputs is done by an
1507 :class:`OutputChecker`. This comparison may be customized with a number of
1508 option flags; see section :ref:`doctest-options` for more information. If the
1509 option flags are insufficient, then the comparison may also be customized by
1510 passing a subclass of :class:`!OutputChecker` to the constructor.
1511
1512 The test runner's display output can be controlled in two ways. First, an output
1513 function can be passed to :meth:`run`; this function will be called
1514 with strings that should be displayed. It defaults to ``sys.stdout.write``. If
1515 capturing the output is not sufficient, then the display output can be also
1516 customized by subclassing DocTestRunner, and overriding the methods
1517 :meth:`report_start`, :meth:`report_success`,
1518 :meth:`report_unexpected_exception`, and :meth:`report_failure`.
1519
1520 The optional keyword argument *checker* specifies the :class:`OutputChecker`
1521 object (or drop-in replacement) that should be used to compare the expected
1522 outputs to the actual outputs of doctest examples.
1523
1524 The optional keyword argument *verbose* controls the :class:`DocTestRunner`'s
1525 verbosity. If *verbose* is ``True``, then information is printed about each
1526 example, as it is run. If *verbose* is ``False``, then only failures are
1527 printed. If *verbose* is unspecified, or ``None``, then verbose output is used
1528 iff the command-line switch ``-v`` is used.
1529
1530 The optional keyword argument *optionflags* can be used to control how the test
1531 runner compares expected output to actual output, and how it displays failures.
1532 For more information, see section :ref:`doctest-options`.
1533
1534 The test runner accumulates statistics. The aggregated number of attempted,
1535 failed and skipped examples is also available via the :attr:`tries`,
1536 :attr:`failures` and :attr:`skips` attributes. The :meth:`run` and
1537 :meth:`summarize` methods return a :class:`TestResults` instance.
1538
1539 :class:`DocTestRunner` defines the following methods:
1540
1541
1542 .. method:: report_start(out, test, example)
1543
1544 Report that the test runner is about to process the given example. This method
1545 is provided to allow subclasses of :class:`DocTestRunner` to customize their
1546 output; it should not be called directly.
1547
1548 *example* is the example about to be processed. *test* is the test
1549 containing *example*. *out* is the output function that was passed to
1550 :meth:`DocTestRunner.run`.
1551
1552
1553 .. method:: report_success(out, test, example, got)
1554
1555 Report that the given example ran successfully. This method is provided to
1556 allow subclasses of :class:`DocTestRunner` to customize their output; it
1557 should not be called directly.
1558
1559 *example* is the example about to be processed. *got* is the actual output
1560 from the example. *test* is the test containing *example*. *out* is the
1561 output function that was passed to :meth:`DocTestRunner.run`.
1562
1563
1564 .. method:: report_failure(out, test, example, got)
1565
1566 Report that the given example failed. This method is provided to allow
1567 subclasses of :class:`DocTestRunner` to customize their output; it should not
1568 be called directly.
1569
1570 *example* is the example about to be processed. *got* is the actual output
1571 from the example. *test* is the test containing *example*. *out* is the
1572 output function that was passed to :meth:`DocTestRunner.run`.
1573
1574
1575 .. method:: report_unexpected_exception(out, test, example, exc_info)
1576
1577 Report that the given example raised an unexpected exception. This method is
1578 provided to allow subclasses of :class:`DocTestRunner` to customize their
1579 output; it should not be called directly.
1580
1581 *example* is the example about to be processed. *exc_info* is a tuple
1582 containing information about the unexpected exception (as returned by
1583 :func:`sys.exc_info`). *test* is the test containing *example*. *out* is the
1584 output function that was passed to :meth:`DocTestRunner.run`.
1585
1586
1587 .. method:: run(test, compileflags=None, out=None, clear_globs=True)
1588
1589 Run the examples in *test* (a :class:`DocTest` object), and display the
1590 results using the writer function *out*. Return a :class:`TestResults`
1591 instance.
1592
1593 The examples are run in the namespace ``test.globs``. If *clear_globs* is
1594 true (the default), then this namespace will be cleared after the test runs,
1595 to help with garbage collection. If you would like to examine the namespace
1596 after the test completes, then use *clear_globs=False*.
1597
1598 *compileflags* gives the set of flags that should be used by the Python
1599 compiler when running the examples. If not specified, then it will default to
1600 the set of future-import flags that apply to *globs*.
1601
1602 The output of each example is checked using the :class:`DocTestRunner`'s
1603 output checker, and the results are formatted by the
1604 :meth:`!DocTestRunner.report_\*` methods.
1605
1606
1607 .. method:: summarize(verbose=None)
1608
1609 Print a summary of all the test cases that have been run by this DocTestRunner,
1610 and return a :class:`TestResults` instance.
1611
1612 The optional *verbose* argument controls how detailed the summary is. If the
1613 verbosity is not specified, then the :class:`DocTestRunner`'s verbosity is
1614 used.
1615
1616 :class:`DocTestParser` has the following attributes:
1617
1618 .. attribute:: tries
1619
1620 Number of attempted examples.
1621
1622 .. attribute:: failures
1623
1624 Number of failed examples.
1625
1626 .. attribute:: skips
1627
1628 Number of skipped examples.
1629
1630 .. versionadded:: 3.13
1631
1632
1633 .. _doctest-outputchecker:
1634
1635 OutputChecker objects
1636 ^^^^^^^^^^^^^^^^^^^^^
1637
1638
1639 .. class:: OutputChecker()
1640
1641 A class used to check the whether the actual output from a doctest example
1642 matches the expected output. :class:`OutputChecker` defines two methods:
1643 :meth:`check_output`, which compares a given pair of outputs, and returns ``True``
1644 if they match; and :meth:`output_difference`, which returns a string describing
1645 the differences between two outputs.
1646
1647
1648 :class:`OutputChecker` defines the following methods:
1649
1650 .. method:: check_output(want, got, optionflags)
1651
1652 Return ``True`` iff the actual output from an example (*got*) matches the
1653 expected output (*want*). These strings are always considered to match if
1654 they are identical; but depending on what option flags the test runner is
1655 using, several non-exact match types are also possible. See section
1656 :ref:`doctest-options` for more information about option flags.
1657
1658
1659 .. method:: output_difference(example, got, optionflags)
1660
1661 Return a string describing the differences between the expected output for a
1662 given example (*example*) and the actual output (*got*). *optionflags* is the
1663 set of option flags used to compare *want* and *got*.
1664
1665
1666 .. _doctest-debugging:
1667
1668 Debugging
1669 ---------
1670
1671 Doctest provides several mechanisms for debugging doctest examples:
1672
1673 * Several functions convert doctests to executable Python programs, which can be
1674 run under the Python debugger, :mod:`pdb`.
1675
1676 * The :class:`DebugRunner` class is a subclass of :class:`DocTestRunner` that
1677 raises an exception for the first failing example, containing information about
1678 that example. This information can be used to perform post-mortem debugging on
1679 the example.
1680
1681 * The :mod:`unittest` cases generated by :func:`DocTestSuite` support the
1682 :meth:`debug` method defined by :class:`unittest.TestCase`.
1683
1684 * You can add a call to :func:`pdb.set_trace` in a doctest example, and you'll
1685 drop into the Python debugger when that line is executed. Then you can inspect
1686 current values of variables, and so on. For example, suppose :file:`a.py`
1687 contains just this module docstring::
1688
1689 """
1690 >>> def f(x):
1691 ... g(x*2)
1692 >>> def g(x):
1693 ... print(x+3)
1694 ... import pdb; pdb.set_trace()
1695 >>> f(3)
1696 9
1697 """
1698
1699 Then an interactive Python session may look like this::
1700
1701 >>> import a, doctest
1702 >>> doctest.testmod(a)
1703 --Return--
1704 > <doctest a[1]>(3)g()->None
1705 -> import pdb; pdb.set_trace()
1706 (Pdb) list
1707 1 def g(x):
1708 2 print(x+3)
1709 3 -> import pdb; pdb.set_trace()
1710 [EOF]
1711 (Pdb) p x
1712 6
1713 (Pdb) step
1714 --Return--
1715 > <doctest a[0]>(2)f()->None
1716 -> g(x*2)
1717 (Pdb) list
1718 1 def f(x):
1719 2 -> g(x*2)
1720 [EOF]
1721 (Pdb) p x
1722 3
1723 (Pdb) step
1724 --Return--
1725 > <doctest a[2]>(1)?()->None
1726 -> f(3)
1727 (Pdb) cont
1728 (0, 3)
1729 >>>
1730
1731
1732 Functions that convert doctests to Python code, and possibly run the synthesized
1733 code under the debugger:
1734
1735
1736 .. function:: script_from_examples(s)
1737
1738 Convert text with examples to a script.
1739
1740 Argument *s* is a string containing doctest examples. The string is converted
1741 to a Python script, where doctest examples in *s* are converted to regular code,
1742 and everything else is converted to Python comments. The generated script is
1743 returned as a string. For example, ::
1744
1745 import doctest
1746 print(doctest.script_from_examples(r"""
1747 Set x and y to 1 and 2.
1748 >>> x, y = 1, 2
1749
1750 Print their sum:
1751 >>> print(x+y)
1752 3
1753 """))
1754
1755 displays::
1756
1757 # Set x and y to 1 and 2.
1758 x, y = 1, 2
1759 #
1760 # Print their sum:
1761 print(x+y)
1762 # Expected:
1763 ## 3
1764
1765 This function is used internally by other functions (see below), but can also be
1766 useful when you want to transform an interactive Python session into a Python
1767 script.
1768
1769
1770 .. function:: testsource(module, name)
1771
1772 Convert the doctest for an object to a script.
1773
1774 Argument *module* is a module object, or dotted name of a module, containing the
1775 object whose doctests are of interest. Argument *name* is the name (within the
1776 module) of the object with the doctests of interest. The result is a string,
1777 containing the object's docstring converted to a Python script, as described for
1778 :func:`script_from_examples` above. For example, if module :file:`a.py`
1779 contains a top-level function :func:`!f`, then ::
1780
1781 import a, doctest
1782 print(doctest.testsource(a, "a.f"))
1783
1784 prints a script version of function :func:`!f`'s docstring, with doctests
1785 converted to code, and the rest placed in comments.
1786
1787
1788 .. function:: debug(module, name, pm=False)
1789
1790 Debug the doctests for an object.
1791
1792 The *module* and *name* arguments are the same as for function
1793 :func:`testsource` above. The synthesized Python script for the named object's
1794 docstring is written to a temporary file, and then that file is run under the
1795 control of the Python debugger, :mod:`pdb`.
1796
1797 A shallow copy of ``module.__dict__`` is used for both local and global
1798 execution context.
1799
1800 Optional argument *pm* controls whether post-mortem debugging is used. If *pm*
1801 has a true value, the script file is run directly, and the debugger gets
1802 involved only if the script terminates via raising an unhandled exception. If
1803 it does, then post-mortem debugging is invoked, via :func:`pdb.post_mortem`,
1804 passing the traceback object from the unhandled exception. If *pm* is not
1805 specified, or is false, the script is run under the debugger from the start, via
1806 passing an appropriate :func:`exec` call to :func:`pdb.run`.
1807
1808
1809 .. function:: debug_src(src, pm=False, globs=None)
1810
1811 Debug the doctests in a string.
1812
1813 This is like function :func:`debug` above, except that a string containing
1814 doctest examples is specified directly, via the *src* argument.
1815
1816 Optional argument *pm* has the same meaning as in function :func:`debug` above.
1817
1818 Optional argument *globs* gives a dictionary to use as both local and global
1819 execution context. If not specified, or ``None``, an empty dictionary is used.
1820 If specified, a shallow copy of the dictionary is used.
1821
1822
1823 The :class:`DebugRunner` class, and the special exceptions it may raise, are of
1824 most interest to testing framework authors, and will only be sketched here. See
1825 the source code, and especially :class:`DebugRunner`'s docstring (which is a
1826 doctest!) for more details:
1827
1828
1829 .. class:: DebugRunner(checker=None, verbose=None, optionflags=0)
1830
1831 A subclass of :class:`DocTestRunner` that raises an exception as soon as a
1832 failure is encountered. If an unexpected exception occurs, an
1833 :exc:`UnexpectedException` exception is raised, containing the test, the
1834 example, and the original exception. If the output doesn't match, then a
1835 :exc:`DocTestFailure` exception is raised, containing the test, the example, and
1836 the actual output.
1837
1838 For information about the constructor parameters and methods, see the
1839 documentation for :class:`DocTestRunner` in section :ref:`doctest-advanced-api`.
1840
1841 There are two exceptions that may be raised by :class:`DebugRunner` instances:
1842
1843
1844 .. exception:: DocTestFailure(test, example, got)
1845
1846 An exception raised by :class:`DocTestRunner` to signal that a doctest example's
1847 actual output did not match its expected output. The constructor arguments are
1848 used to initialize the attributes of the same names.
1849
1850 :exc:`DocTestFailure` defines the following attributes:
1851
1852
1853 .. attribute:: DocTestFailure.test
1854
1855 The :class:`DocTest` object that was being run when the example failed.
1856
1857
1858 .. attribute:: DocTestFailure.example
1859
1860 The :class:`Example` that failed.
1861
1862
1863 .. attribute:: DocTestFailure.got
1864
1865 The example's actual output.
1866
1867
1868 .. exception:: UnexpectedException(test, example, exc_info)
1869
1870 An exception raised by :class:`DocTestRunner` to signal that a doctest
1871 example raised an unexpected exception. The constructor arguments are used
1872 to initialize the attributes of the same names.
1873
1874 :exc:`UnexpectedException` defines the following attributes:
1875
1876
1877 .. attribute:: UnexpectedException.test
1878
1879 The :class:`DocTest` object that was being run when the example failed.
1880
1881
1882 .. attribute:: UnexpectedException.example
1883
1884 The :class:`Example` that failed.
1885
1886
1887 .. attribute:: UnexpectedException.exc_info
1888
1889 A tuple containing information about the unexpected exception, as returned by
1890 :func:`sys.exc_info`.
1891
1892
1893 .. _doctest-soapbox:
1894
1895 Soapbox
1896 -------
1897
1898 As mentioned in the introduction, :mod:`doctest` has grown to have three primary
1899 uses:
1900
1901 #. Checking examples in docstrings.
1902
1903 #. Regression testing.
1904
1905 #. Executable documentation / literate testing.
1906
1907 These uses have different requirements, and it is important to distinguish them.
1908 In particular, filling your docstrings with obscure test cases makes for bad
1909 documentation.
1910
1911 When writing a docstring, choose docstring examples with care. There's an art to
1912 this that needs to be learned---it may not be natural at first. Examples should
1913 add genuine value to the documentation. A good example can often be worth many
1914 words. If done with care, the examples will be invaluable for your users, and
1915 will pay back the time it takes to collect them many times over as the years go
1916 by and things change. I'm still amazed at how often one of my :mod:`doctest`
1917 examples stops working after a "harmless" change.
1918
1919 Doctest also makes an excellent tool for regression testing, especially if you
1920 don't skimp on explanatory text. By interleaving prose and examples, it becomes
1921 much easier to keep track of what's actually being tested, and why. When a test
1922 fails, good prose can make it much easier to figure out what the problem is, and
1923 how it should be fixed. It's true that you could write extensive comments in
1924 code-based testing, but few programmers do. Many have found that using doctest
1925 approaches instead leads to much clearer tests. Perhaps this is simply because
1926 doctest makes writing prose a little easier than writing code, while writing
1927 comments in code is a little harder. I think it goes deeper than just that:
1928 the natural attitude when writing a doctest-based test is that you want to
1929 explain the fine points of your software, and illustrate them with examples.
1930 This in turn naturally leads to test files that start with the simplest
1931 features, and logically progress to complications and edge cases. A coherent
1932 narrative is the result, instead of a collection of isolated functions that test
1933 isolated bits of functionality seemingly at random. It's a different attitude,
1934 and produces different results, blurring the distinction between testing and
1935 explaining.
1936
1937 Regression testing is best confined to dedicated objects or files. There are
1938 several options for organizing tests:
1939
1940 * Write text files containing test cases as interactive examples, and test the
1941 files using :func:`testfile` or :func:`DocFileSuite`. This is recommended,
1942 although is easiest to do for new projects, designed from the start to use
1943 doctest.
1944
1945 * Define functions named ``_regrtest_topic`` that consist of single docstrings,
1946 containing test cases for the named topics. These functions can be included in
1947 the same file as the module, or separated out into a separate test file.
1948
1949 * Define a :attr:`~module.__test__` dictionary mapping from regression test topics to
1950 docstrings containing test cases.
1951
1952 When you have placed your tests in a module, the module can itself be the test
1953 runner. When a test fails, you can arrange for your test runner to re-run only
1954 the failing doctest while you debug the problem. Here is a minimal example of
1955 such a test runner::
1956
1957 if __name__ == '__main__':
1958 import doctest
1959 flags = doctest.REPORT_NDIFF|doctest.FAIL_FAST
1960 if len(sys.argv) > 1:
1961 name = sys.argv[1]
1962 if name in globals():
1963 obj = globals()[name]
1964 else:
1965 obj = __test__[name]
1966 doctest.run_docstring_examples(obj, globals(), name=name,
1967 optionflags=flags)
1968 else:
1969 fail, total = doctest.testmod(optionflags=flags)
1970 print(f"{fail} failures out of {total} tests")
1971
1972
1973 .. rubric:: Footnotes
1974
1975 .. [#] Examples containing both expected output and an exception are not supported.
1976 Trying to guess where one ends and the other begins is too error-prone, and that
1977 also makes for a confusing test.