Debian Codegraph @

Doc/library/doctest.rst

raw
1 :mod:`!doctest` --- Test interactive Python examples
2 ====================================================
3
4 .. module:: doctest
5 :synopsis: Test pieces of code within docstrings.
6
7 .. moduleauthor:: Tim Peters <[email protected]>
8 .. sectionauthor:: Tim Peters <[email protected]>
9 .. sectionauthor:: Moshe Zadka <[email protected]>
10 .. sectionauthor:: Edward Loper <[email protected]>
11
12 **Source code:** :source:`Lib/doctest.py`
13
14 --------------
15
16 The :mod:`!doctest` module searches for pieces of text that look like interactive
17 Python sessions, and then executes those sessions to verify that they work
18 exactly as shown. There are several common ways to use doctest:
19
20 * To check that a module's docstrings are up-to-date by verifying that all
21 interactive examples still work as documented.
22
23 * To perform regression testing by verifying that interactive examples from a
24 test file or a test object work as expected.
25
26 * To write tutorial documentation for a package, liberally illustrated with
27 input-output examples. Depending on whether the examples or the expository text
28 are emphasized, this has the flavor of "literate testing" or "executable
29 documentation".
30
31 Here's a complete but small example module::
32
33 """
34 This is the "example" module.
35
36 The example module supplies one function, factorial(). For example,
37
38 >>> factorial(5)
39 120
40 """
41
42 def factorial(n):
43 """Return the factorial of n, an exact integer >= 0.
44
45 >>> [factorial(n) for n in range(6)]
46 [1, 1, 2, 6, 24, 120]
47 >>> factorial(30)
48 265252859812191058636308480000000
49 >>> factorial(-1)
50 Traceback (most recent call last):
51 ...
52 ValueError: n must be >= 0
53
54 Factorials of floats are OK, but the float must be an exact integer:
55 >>> factorial(30.1)
56 Traceback (most recent call last):
57 ...
58 ValueError: n must be exact integer
59 >>> factorial(30.0)
60 265252859812191058636308480000000
61
62 It must also not be ridiculously large:
63 >>> factorial(1e100)
64 Traceback (most recent call last):
65 ...
66 OverflowError: n too large
67 """
68
69 import math
70 if not n >= 0:
71 raise ValueError("n must be >= 0")
72 if math.floor(n) != n:
73 raise ValueError("n must be exact integer")
74 if n+1 == n: # catch a value like 1e300
75 raise OverflowError("n too large")
76 result = 1
77 factor = 2
78 while factor <= n:
79 result *= factor
80 factor += 1
81 return result
82
83
84 if __name__ == "__main__":
85 import doctest
86 doctest.testmod()
87
88 If you run :file:`example.py` directly from the command line, :mod:`!doctest`
89 works its magic:
90
91 .. code-block:: shell-session
92
93 $ python example.py
94 $
95
96 There's no output! That's normal, and it means all the examples worked. Pass
97 ``-v`` to the script, and :mod:`!doctest` prints a detailed log of what
98 it's trying, and prints a summary at the end:
99
100 .. code-block:: shell-session
101
102 $ python example.py -v
103 Trying:
104 factorial(5)
105 Expecting:
106 120
107 ok
108 Trying:
109 [factorial(n) for n in range(6)]
110 Expecting:
111 [1, 1, 2, 6, 24, 120]
112 ok
113
114 And so on, eventually ending with:
115
116 .. code-block:: none
117
118 Trying:
119 factorial(1e100)
120 Expecting:
121 Traceback (most recent call last):
122 ...
123 OverflowError: n too large
124 ok
125 2 items passed all tests:
126 1 test in __main__
127 6 tests in __main__.factorial
128 7 tests in 2 items.
129 7 passed.
130 Test passed.
131 $
132
133 That's all you need to know to start making productive use of :mod:`!doctest`!
134 Jump in. The following sections provide full details. Note that there are many
135 examples of doctests in the standard Python test suite and libraries.
136 Especially useful examples can be found in the standard test file
137 :file:`Lib/test/test_doctest/test_doctest.py`.
138
139 .. versionadded:: 3.13
140 Output is colorized by default and can be
141 :ref:`controlled using environment variables <using-on-controlling-color>`.
142
143
144 .. _doctest-simple-testmod:
145
146 Simple Usage: Checking Examples in Docstrings
147 ---------------------------------------------
148
149 The simplest way to start using doctest (but not necessarily the way you'll
150 continue to do it) is to end each module :mod:`!M` with::
151
152 if __name__ == "__main__":
153 import doctest
154 doctest.testmod()
155
156 :mod:`!doctest` then examines docstrings in module :mod:`!M`.
157
158 Running the module as a script causes the examples in the docstrings to get
159 executed and verified::
160
161 python M.py
162
163 This won't display anything unless an example fails, in which case the failing
164 example(s) and the cause(s) of the failure(s) are printed to stdout, and the
165 final line of output is ``***Test Failed*** N failures.``, where *N* is the
166 number of examples that failed.
167
168 Run it with the ``-v`` switch instead::
169
170 python M.py -v
171
172 and a detailed report of all examples tried is printed to standard output, along
173 with assorted summaries at the end.
174
175 You can force verbose mode by passing ``verbose=True`` to :func:`testmod`, or
176 prohibit it by passing ``verbose=False``. In either of those cases,
177 :data:`sys.argv` is not examined by :func:`testmod` (so passing ``-v`` or not
178 has no effect).
179
180 There is also a command line shortcut for running :func:`testmod`, see section
181 :ref:`doctest-cli`.
182
183 For more information on :func:`testmod`, see section :ref:`doctest-basic-api`.
184
185
186 .. _doctest-simple-testfile:
187
188 Simple Usage: Checking Examples in a Text File
189 ----------------------------------------------
190
191 Another simple application of doctest is testing interactive examples in a text
192 file. This can be done with the :func:`testfile` function::
193
194 import doctest
195 doctest.testfile("example.txt")
196
197 That short script executes and verifies any interactive Python examples
198 contained in the file :file:`example.txt`. The file content is treated as if it
199 were a single giant docstring; the file doesn't need to contain a Python
200 program! For example, perhaps :file:`example.txt` contains this:
201
202 .. code-block:: none
203
204 The ``example`` module
205 ======================
206
207 Using ``factorial``
208 -------------------
209
210 This is an example text file in reStructuredText format. First import
211 ``factorial`` from the ``example`` module:
212
213 >>> from example import factorial
214
215 Now use it:
216
217 >>> factorial(6)
218 120
219
220 Running ``doctest.testfile("example.txt")`` then finds the error in this
221 documentation::
222
223 File "./example.txt", line 14, in example.txt
224 Failed example:
225 factorial(6)
226 Expected:
227 120
228 Got:
229 720
230
231 As with :func:`testmod`, :func:`testfile` won't display anything unless an
232 example fails. If an example does fail, then the failing example(s) and the
233 cause(s) of the failure(s) are printed to stdout, using the same format as
234 :func:`!testmod`.
235
236 By default, :func:`testfile` looks for files in the calling module's directory.
237 See section :ref:`doctest-basic-api` for a description of the optional arguments
238 that can be used to tell it to look for files in other locations.
239
240 Like :func:`testmod`, :func:`testfile`'s verbosity can be set with the
241 ``-v`` command-line switch or with the optional keyword argument
242 *verbose*.
243
244 There is also a command line shortcut for running :func:`testfile`, see section
245 :ref:`doctest-cli`.
246
247 For more information on :func:`testfile`, see section :ref:`doctest-basic-api`.
248
249
250 .. _doctest-cli:
251
252 Command-line Usage
253 ------------------
254
255 The :mod:`!doctest` module can be invoked as a script from the command line:
256
257 .. code-block:: bash
258
259 python -m doctest [-v] [-o OPTION] [-f] file [file ...]
260
261 .. program:: doctest
262
263 .. option:: -v, --verbose
264
265 Detailed report of all examples tried is printed to standard output,
266 along with assorted summaries at the end::
267
268 python -m doctest -v example.py
269
270 This will import :file:`example.py` as a standalone module and run
271 :func:`testmod` on it. Note that this may not work correctly if the
272 file is part of a package and imports other submodules from that package.
273
274 If the file name does not end with :file:`.py`, :mod:`!doctest` infers
275 that it must be run with :func:`testfile` instead::
276
277 python -m doctest -v example.txt
278
279 .. option:: -o, --option <option>
280
281 Option flags control various aspects of doctest's behavior, see section
282 :ref:`doctest-options`.
283
284 .. versionadded:: 3.4
285
286 .. option:: -f, --fail-fast
287
288 This is shorthand for ``-o FAIL_FAST``.
289
290 .. versionadded:: 3.4
291
292
293 .. _doctest-how-it-works:
294
295 How It Works
296 ------------
297
298 This section examines in detail how doctest works: which docstrings it looks at,
299 how it finds interactive examples, what execution context it uses, how it
300 handles exceptions, and how option flags can be used to control its behavior.
301 This is the information that you need to know to write doctest examples; for
302 information about actually running doctest on these examples, see the following
303 sections.
304
305
306 .. _doctest-which-docstrings:
307
308 Which Docstrings Are Examined?
309 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
310
311 The module docstring, and all function, class and method docstrings are
312 searched. Objects imported into the module are not searched.
313
314 .. currentmodule:: None
315
316 .. attribute:: module.__test__
317 :no-typesetting:
318
319 .. currentmodule:: doctest
320
321 In addition, there are cases when you want tests to be part of a module but not part
322 of the help text, which requires that the tests not be included in the docstring.
323 Doctest looks for a module-level variable called ``__test__`` and uses it to locate other
324 tests. If ``M.__test__`` exists, it must be a dict, and each
325 entry maps a (string) name to a function object, class object, or string.
326 Function and class object docstrings found from ``M.__test__`` are searched, and
327 strings are treated as if they were docstrings. In output, a key ``K`` in
328 ``M.__test__`` appears with name ``M.__test__.K``.
329
330 For example, place this block of code at the top of :file:`example.py`:
331
332 .. code-block:: python
333
334 __test__ = {
335 'numbers': """
336 >>> factorial(6)
337 720
338
339 >>> [factorial(n) for n in range(6)]
340 [1, 1, 2, 6, 24, 120]
341 """
342 }
343
344 The value of ``example.__test__["numbers"]`` will be treated as a
345 docstring and all the tests inside it will be run. It is
346 important to note that the value can be mapped to a function,
347 class object, or module; if so, :mod:`!doctest`
348 searches them recursively for docstrings, which are then scanned for tests.
349
350 Any classes found are recursively searched similarly, to test docstrings in
351 their contained methods and nested classes.
352
353 .. note::
354
355 ``doctest`` can only automatically discover classes and functions that are
356 defined at the module level or inside other classes.
357
358 Since nested classes and functions only exist when an outer function
359 is called, they cannot be discovered. Define them outside to make them visible.
360
361 .. _doctest-finding-examples:
362
363 How are Docstring Examples Recognized?
364 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
365
366 In most cases a copy-and-paste of an interactive console session works fine,
367 but doctest isn't trying to do an exact emulation of any specific Python shell.
368
369 ::
370
371 >>> # comments are ignored
372 >>> x = 12
373 >>> x
374 12
375 >>> if x == 13:
376 ... print("yes")
377 ... else:
378 ... print("no")
379 ... print("NO")
380 ... print("NO!!!")
381 ...
382 no
383 NO
384 NO!!!
385 >>>
386
387 .. index::
388 single: >>>; interpreter prompt
389 single: ...; interpreter prompt
390
391 Any expected output must immediately follow the final ``'>>> '`` or ``'... '``
392 line containing the code, and the expected output (if any) extends to the next
393 ``'>>> '`` or all-whitespace line.
394
395 The fine print:
396
397 * Expected output cannot contain an all-whitespace line, since such a line is
398 taken to signal the end of expected output. If expected output does contain a
399 blank line, put ``<BLANKLINE>`` in your doctest example each place a blank line
400 is expected.
401
402 * All hard tab characters are expanded to spaces, using 8-column tab stops.
403 Tabs in output generated by the tested code are not modified. Because any
404 hard tabs in the sample output *are* expanded, this means that if the code
405 output includes hard tabs, the only way the doctest can pass is if the
406 :const:`NORMALIZE_WHITESPACE` option or :ref:`directive <doctest-directives>`
407 is in effect.
408 Alternatively, the test can be rewritten to capture the output and compare it
409 to an expected value as part of the test. This handling of tabs in the
410 source was arrived at through trial and error, and has proven to be the least
411 error prone way of handling them. It is possible to use a different
412 algorithm for handling tabs by writing a custom :class:`DocTestParser` class.
413
414 * Output to stdout is captured, but not output to stderr (exception tracebacks
415 are captured via a different means).
416
417 * If you continue a line via backslashing in an interactive session, or for any
418 other reason use a backslash, you should use a raw docstring, which will
419 preserve your backslashes exactly as you type them::
420
421 >>> def f(x):
422 ... r'''Backslashes in a raw docstring: m\n'''
423 ...
424 >>> print(f.__doc__)
425 Backslashes in a raw docstring: m\n
426
427 Otherwise, the backslash will be interpreted as part of the string. For example,
428 the ``\n`` above would be interpreted as a newline character. Alternatively, you
429 can double each backslash in the doctest version (and not use a raw string)::
430
431 >>> def f(x):
432 ... '''Backslashes in a raw docstring: m\\n'''
433 ...
434 >>> print(f.__doc__)
435 Backslashes in a raw docstring: m\n
436
437 * The starting column doesn't matter::
438
439 >>> assert "Easy!"
440 >>> import math
441 >>> math.floor(1.9)
442 1
443
444 and as many leading whitespace characters are stripped from the expected output
445 as appeared in the initial ``'>>> '`` line that started the example.
446
447
448 .. _doctest-execution-context:
449
450 What's the Execution Context?
451 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
452
453 By default, each time :mod:`!doctest` finds a docstring to test, it uses a
454 *shallow copy* of :mod:`!M`'s globals, so that running tests doesn't change the
455 module's real globals, and so that one test in :mod:`!M` can't leave behind
456 crumbs that accidentally allow another test to work. This means examples can
457 freely use any names defined at top-level in :mod:`!M`, and names defined earlier
458 in the docstring being run. Examples cannot see names defined in other
459 docstrings.
460
461 You can force use of your own dict as the execution context by passing
462 ``globs=your_dict`` to :func:`testmod` or :func:`testfile` instead.
463
464
465 .. _doctest-exceptions:
466
467 What About Exceptions?
468 ^^^^^^^^^^^^^^^^^^^^^^
469
470 No problem, provided that the traceback is the only output produced by the
471 example: just paste in the traceback. [#]_ Since tracebacks contain details
472 that are likely to change rapidly (for example, exact file paths and line
473 numbers), this is one case where doctest works hard to be flexible in what it
474 accepts.
475
476 Simple example::
477
478 >>> [1, 2, 3].remove(42)
479 Traceback (most recent call last):
480 File "<stdin>", line 1, in <module>
481 ValueError: list.remove(x): x not in list
482
483 That doctest succeeds if :exc:`ValueError` is raised, with the ``list.remove(x):
484 x not in list`` detail as shown.
485
486 The expected output for an exception must start with a traceback header, which
487 may be either of the following two lines, indented the same as the first line of
488 the example::
489
490 Traceback (most recent call last):
491 Traceback (innermost last):
492
493 The traceback header is followed by an optional traceback stack, whose contents
494 are ignored by doctest. The traceback stack is typically omitted, or copied
495 verbatim from an interactive session.
496
497 The traceback stack is followed by the most interesting part: the line(s)
498 containing the exception type and detail. This is usually the last line of a
499 traceback, but can extend across multiple lines if the exception has a
500 multi-line detail::
501
502 >>> raise ValueError('multi\n line\ndetail')
503 Traceback (most recent call last):
504 File "<stdin>", line 1, in <module>
505 ValueError: multi
506 line
507 detail
508
509 The last three lines (starting with :exc:`ValueError`) are compared against the
510 exception's type and detail, and the rest are ignored.
511
512 Best practice is to omit the traceback stack, unless it adds significant
513 documentation value to the example. So the last example is probably better as::
514
515 >>> raise ValueError('multi\n line\ndetail')
516 Traceback (most recent call last):
517 ...
518 ValueError: multi
519 line
520 detail
521
522 Note that tracebacks are treated very specially. In particular, in the
523 rewritten example, the use of ``...`` is independent of doctest's
524 :const:`ELLIPSIS` option. The ellipsis in that example could be left out, or
525 could just as well be three (or three hundred) commas or digits, or an indented
526 transcript of a Monty Python skit.
527
528 Some details you should read once, but won't need to remember:
529
530 * Doctest can't guess whether your expected output came from an exception
531 traceback or from ordinary printing. So, e.g., an example that expects
532 ``ValueError: 42 is prime`` will pass whether :exc:`ValueError` is actually
533 raised or if the example merely prints that traceback text. In practice,
534 ordinary output rarely begins with a traceback header line, so this doesn't
535 create real problems.
536
537 * Each line of the traceback stack (if present) must be indented further than
538 the first line of the example, *or* start with a non-alphanumeric character.
539 The first line following the traceback header indented the same and starting
540 with an alphanumeric is taken to be the start of the exception detail. Of
541 course this does the right thing for genuine tracebacks.
542
543 * When the :const:`IGNORE_EXCEPTION_DETAIL` doctest option is specified,
544 everything following the leftmost colon and any module information in the
545 exception name is ignored.
546
547 * The interactive shell omits the traceback header line for some
548 :exc:`SyntaxError`\ s. But doctest uses the traceback header line to
549 distinguish exceptions from non-exceptions. So in the rare case where you need
550 to test a :exc:`!SyntaxError` that omits the traceback header, you will need to
551 manually add the traceback header line to your test example.
552
553 .. index:: single: ^ (caret); marker
554
555 * For some exceptions, Python displays the position of the error using ``^``
556 markers and tildes::
557
558 >>> 1 + None
559 File "<stdin>", line 1
560 1 + None
561 ~~^~~~~~
562 TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
563
564 Since the lines showing the position of the error come before the exception type
565 and detail, they are not checked by doctest. For example, the following test
566 would pass, even though it puts the ``^`` marker in the wrong location::
567
568 >>> 1 + None
569 File "<stdin>", line 1
570 1 + None
571 ^~~~~~~~
572 TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'
573
574
575 .. _option-flags-and-directives:
576 .. _doctest-options:
577
578 Option Flags
579 ^^^^^^^^^^^^
580
581 A number of option flags control various aspects of doctest's behavior.
582 Symbolic names for the flags are supplied as module constants, which can be
583 :ref:`bitwise ORed <bitwise>` together and passed to various functions.
584 The names can also be used in :ref:`doctest directives <doctest-directives>`,
585 and may be passed to the doctest command line interface via the ``-o`` option.
586
587 The first group of options define test semantics, controlling aspects of how
588 doctest decides whether actual output matches an example's expected output:
589
590
591 .. data:: DONT_ACCEPT_TRUE_FOR_1
592
593 By default, if an expected output block contains just ``1``, an actual output
594 block containing just ``1`` or just ``True`` is considered to be a match, and
595 similarly for ``0`` versus ``False``. When :const:`DONT_ACCEPT_TRUE_FOR_1` is
596 specified, neither substitution is allowed. The default behavior caters to that
597 Python changed the return type of many functions from integer to boolean;
598 doctests expecting "little integer" output still work in these cases. This
599 option will probably go away, but not for several years.
600
601
602 .. index:: single: <BLANKLINE>
603 .. data:: DONT_ACCEPT_BLANKLINE
604
605 By default, if an expected output block contains a line containing only the
606 string ``<BLANKLINE>``, then that line will match a blank line in the actual
607 output. Because a genuinely blank line delimits the expected output, this is
608 the only way to communicate that a blank line is expected. When
609 :const:`DONT_ACCEPT_BLANKLINE` is specified, this substitution is not allowed.
610
611
612 .. data:: NORMALIZE_WHITESPACE
613
614 When specified, all sequences of whitespace (blanks and newlines) are treated as
615 equal. Any sequence of whitespace within the expected output will match any
616 sequence of whitespace within the actual output. By default, whitespace must
617 match exactly. :const:`NORMALIZE_WHITESPACE` is especially useful when a line of
618 expected output is very long, and you want to wrap it across multiple lines in
619 your source.
620
621
622 .. index:: single: ...; in doctests
623 .. data:: ELLIPSIS
624
625 When specified, an ellipsis marker (``...``) in the expected output can match
626 any substring in the actual output. This includes substrings that span line
627 boundaries, and empty substrings, so it's best to keep usage of this simple.
628 Complicated uses can lead to the same kinds of "oops, it matched too much!"
629 surprises that ``.*`` is prone to in regular expressions.
630
631
632 .. data:: IGNORE_EXCEPTION_DETAIL
633
634 When specified, doctests expecting exceptions pass so long as an exception
635 of the expected type is raised, even if the details
636 (message and fully qualified exception name) don't match.
637
638 For example, an example expecting ``ValueError: 42`` will pass if the actual
639 exception raised is ``ValueError: 3*14``, but will fail if, say, a
640 :exc:`TypeError` is raised instead.
641 It will also ignore any fully qualified name included before the
642 exception class, which can vary between implementations and versions
643 of Python and the code/libraries in use.
644 Hence, all three of these variations will work with the flag specified:
645
646 .. code-block:: pycon
647
648 >>> raise Exception('message')
649 Traceback (most recent call last):
650 Exception: message
651
652 >>> raise Exception('message')
653 Traceback (most recent call last):
654 builtins.Exception: message
655
656 >>> raise Exception('message')
657 Traceback (most recent call last):
658 __main__.Exception: message
659
660 Note that :const:`ELLIPSIS` can also be used to ignore the
661 details of the exception message, but such a test may still fail based
662 on whether the module name is present or matches exactly.
663
664 .. versionchanged:: 3.2
665 :const:`IGNORE_EXCEPTION_DETAIL` now also ignores any information relating
666 to the module containing the exception under test.
667
668
669 .. data:: SKIP
670
671 When specified, do not run the example at all. This can be useful in contexts
672 where doctest examples serve as both documentation and test cases, and an
673 example should be included for documentation purposes, but should not be
674 checked. E.g., the example's output might be random; or the example might
675 depend on resources which would be unavailable to the test driver.
676
677 The SKIP flag can also be used for temporarily "commenting out" examples.
678
679
680 .. data:: COMPARISON_FLAGS
681
682 A bitmask or'ing together all the comparison flags above.
683
684 The second group of options controls how test failures are reported:
685
686
687 .. data:: REPORT_UDIFF
688
689 When specified, failures that involve multi-line expected and actual outputs are
690 displayed using a unified diff.
691
692
693 .. data:: REPORT_CDIFF
694
695 When specified, failures that involve multi-line expected and actual outputs
696 will be displayed using a context diff.
697
698
699 .. data:: REPORT_NDIFF
700
701 When specified, differences are computed by ``difflib.Differ``, using the same
702 algorithm as the popular :file:`ndiff.py` utility. This is the only method that
703 marks differences within lines as well as across lines. For example, if a line
704 of expected output contains digit ``1`` where actual output contains letter
705 ``l``, a line is inserted with a caret marking the mismatching column positions.
706
707
708 .. data:: REPORT_ONLY_FIRST_FAILURE
709
710 When specified, display the first failing example in each doctest, but suppress
711 output for all remaining examples. This will prevent doctest from reporting
712 correct examples that break because of earlier failures; but it might also hide
713 incorrect examples that fail independently of the first failure. When
714 :const:`REPORT_ONLY_FIRST_FAILURE` is specified, the remaining examples are
715 still run, and still count towards the total number of failures reported; only
716 the output is suppressed.
717
718
719 .. data:: FAIL_FAST
720
721 When specified, exit after the first failing example and don't attempt to run
722 the remaining examples. Thus, the number of failures reported will be at most
723 1. This flag may be useful during debugging, since examples after the first
724 failure won't even produce debugging output.
725
726
727 .. data:: REPORTING_FLAGS
728
729 A bitmask or'ing together all the reporting flags above.
730
731
732 There is also a way to register new option flag names, though this isn't
733 useful unless you intend to extend :mod:`!doctest` internals via subclassing:
734
735
736 .. function:: register_optionflag(name)
737
738 Create a new option flag with a given name, and return the new flag's integer
739 value. :func:`register_optionflag` can be used when subclassing
740 :class:`OutputChecker` or :class:`DocTestRunner` to create new options that are
741 supported by your subclasses. :func:`register_optionflag` should always be
742 called using the following idiom::
743
744 MY_FLAG = register_optionflag('MY_FLAG')
745
746
747 .. index::
748 single: # (hash); in doctests
749 single: + (plus); in doctests
750 single: - (minus); in doctests
751 .. _doctest-directives:
752
753 Directives
754 ^^^^^^^^^^
755
756 Doctest directives may be used to modify the :ref:`option flags
757 <doctest-options>` for an individual example. Doctest directives are
758 special Python comments following an example's source code:
759
760 .. productionlist:: doctest
761 directive: "#" "doctest:" `directive_options`
762 directive_options: `directive_option` ("," `directive_option`)*
763 directive_option: `on_or_off` `directive_option_name`
764 on_or_off: "+" | "-"
765 directive_option_name: "DONT_ACCEPT_BLANKLINE" | "NORMALIZE_WHITESPACE" | ...
766
767 Whitespace is not allowed between the ``+`` or ``-`` and the directive option
768 name. The directive option name can be any of the option flag names explained
769 above.
770
771 An example's doctest directives modify doctest's behavior for that single
772 example. Use ``+`` to enable the named behavior, or ``-`` to disable it.
773
774 For example, this test passes:
775
776 .. doctest::
777 :no-trim-doctest-flags:
778
779 >>> print(list(range(20))) # doctest: +NORMALIZE_WHITESPACE
780 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
781 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
782
783 Without the directive it would fail, both because the actual output doesn't have
784 two blanks before the single-digit list elements, and because the actual output
785 is on a single line. This test also passes, and also requires a directive to do
786 so:
787
788 .. doctest::
789 :no-trim-doctest-flags:
790
791 >>> print(list(range(20))) # doctest: +ELLIPSIS
792 [0, 1, ..., 18, 19]
793
794 Multiple directives can be used on a single physical line, separated by
795 commas:
796
797 .. doctest::
798 :no-trim-doctest-flags:
799
800 >>> print(list(range(20))) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
801 [0, 1, ..., 18, 19]
802
803 If multiple directive comments are used for a single example, then they are
804 combined:
805
806 .. doctest::
807 :no-trim-doctest-flags:
808
809 >>> print(list(range(20))) # doctest: +ELLIPSIS
810 ... # doctest: +NORMALIZE_WHITESPACE
811 [0, 1, ..., 18, 19]
812
813 As the previous example shows, you can add ``...`` lines to your example
814 containing only directives. This can be useful when an example is too long for
815 a directive to comfortably fit on the same line:
816
817 .. doctest::
818 :no-trim-doctest-flags:
819
820 >>> print(list(range(5)) + list(range(10, 20)) + list(range(30, 40)))
821 ... # doctest: +ELLIPSIS
822 [0, ..., 4, 10, ..., 19, 30, ..., 39]
823
824 Note that since all options are disabled by default, and directives apply only
825 to the example they appear in, enabling options (via ``+`` in a directive) is
826 usually the only meaningful choice. However, option flags can also be passed to
827 functions that run doctests, establishing different defaults. In such cases,
828 disabling an option via ``-`` in a directive can be useful.
829
830
831 .. _doctest-warnings:
832
833 Warnings
834 ^^^^^^^^
835
836 :mod:`!doctest` is serious about requiring exact matches in expected output. If
837 even a single character doesn't match, the test fails. This will probably
838 surprise you a few times, as you learn exactly what Python does and doesn't
839 guarantee about output. For example, when printing a set, Python doesn't
840 guarantee that the element is printed in any particular order, so a test like ::
841
842 >>> foo()
843 {"spam", "eggs"}
844
845 is vulnerable! One workaround is to do ::
846
847 >>> foo() == {"spam", "eggs"}
848 True
849
850 instead. Another is to do ::
851
852 >>> d = sorted(foo())
853 >>> d
854 ['eggs', 'spam']
855
856 There are others, but you get the idea.
857
858 Another bad idea is to print things that embed an object address, like
859
860 .. doctest::
861
862 >>> id(1.0) # certain to fail some of the time # doctest: +SKIP
863 7948648
864 >>> class C: pass
865 >>> C() # the default repr() for instances embeds an address # doctest: +SKIP
866 <C object at 0x00AC18F0>
867
868 The :const:`ELLIPSIS` directive gives a nice approach for the last example:
869
870 .. doctest::
871 :no-trim-doctest-flags:
872
873 >>> C() # doctest: +ELLIPSIS
874 <C object at 0x...>
875
876 Floating-point numbers are also subject to small output variations across
877 platforms, because Python defers to the platform C library for some
878 floating-point calculations, and C libraries vary widely in quality here. ::
879
880 >>> 1000**0.1 # risky
881 1.9952623149688797
882 >>> round(1000**0.1, 9) # safer
883 1.995262315
884 >>> print(f'{1000**0.1:.4f}') # much safer
885 1.9953
886
887 Numbers of the form ``I/2.**J`` are safe across all platforms, and I often
888 contrive doctest examples to produce numbers of that form::
889
890 >>> 3./4 # utterly safe
891 0.75
892
893 Simple fractions are also easier for people to understand, and that makes for
894 better documentation.
895
896
897 .. _doctest-basic-api:
898
899 Basic API
900 ---------
901
902 The functions :func:`testmod` and :func:`testfile` provide a simple interface to
903 doctest that should be sufficient for most basic uses. For a less formal
904 introduction to these two functions, see sections :ref:`doctest-simple-testmod`
905 and :ref:`doctest-simple-testfile`.
906
907
908 .. function:: testfile(filename, module_relative=True, name=None, package=None, globs=None, verbose=None, report=True, optionflags=0, extraglobs=None, raise_on_error=False, parser=DocTestParser(), encoding=None)
909
910 All arguments except *filename* are optional, and should be specified in keyword
911 form.
912
913 Test examples in the file named *filename*. Return ``(failure_count,
914 test_count)``.
915
916 Optional argument *module_relative* specifies how the filename should be
917 interpreted:
918
919 * If *module_relative* is ``True`` (the default), then *filename* specifies an
920 OS-independent module-relative path. By default, this path is relative to the
921 calling module's directory; but if the *package* argument is specified, then it
922 is relative to that package. To ensure OS-independence, *filename* should use
923 ``/`` characters to separate path segments, and may not be an absolute path
924 (i.e., it may not begin with ``/``).
925
926 * If *module_relative* is ``False``, then *filename* specifies an OS-specific
927 path. The path may be absolute or relative; relative paths are resolved with
928 respect to the current working directory.
929
930 Optional argument *name* gives the name of the test; by default, or if ``None``,
931 ``os.path.basename(filename)`` is used.
932
933 Optional argument *package* is a Python package or the name of a Python package
934 whose directory should be used as the base directory for a module-relative
935 filename. If no package is specified, then the calling module's directory is
936 used as the base directory for module-relative filenames. It is an error to
937 specify *package* if *module_relative* is ``False``.
938
939 Optional argument *globs* gives a dict to be used as the globals when executing
940 examples. A new shallow copy of this dict is created for the doctest, so its
941 examples start with a clean slate. By default, or if ``None``, a new empty dict
942 is used.
943
944 Optional argument *extraglobs* gives a dict merged into the globals used to
945 execute examples. This works like :meth:`dict.update`: if *globs* and
946 *extraglobs* have a common key, the associated value in *extraglobs* appears in
947 the combined dict. By default, or if ``None``, no extra globals are used. This
948 is an advanced feature that allows parameterization of doctests. For example, a
949 doctest can be written for a base class, using a generic name for the class,
950 then reused to test any number of subclasses by passing an *extraglobs* dict
951 mapping the generic name to the subclass to be tested.
952
953 Optional argument *verbose* prints lots of stuff if true, and prints only
954 failures if false; by default, or if ``None``, it's true if and only if ``'-v'``
955 is in :data:`sys.argv`.
956
957 Optional argument *report* prints a summary at the end when true, else prints
958 nothing at the end. In verbose mode, the summary is detailed, else the summary
959 is very brief (in fact, empty if all tests passed).
960
961 Optional argument *optionflags* (default value ``0``) takes the
962 :ref:`bitwise OR <bitwise>` of option flags.
963 See section :ref:`doctest-options`.
964
965 Optional argument *raise_on_error* defaults to false. If true, an exception is
966 raised upon the first failure or unexpected exception in an example. This
967 allows failures to be post-mortem debugged. Default behavior is to continue
968 running examples.
969
970 Optional argument *parser* specifies a :class:`DocTestParser` (or subclass) that
971 should be used to extract tests from the files. It defaults to a normal parser
972 (i.e., ``DocTestParser()``).
973
974 Optional argument *encoding* specifies an encoding that should be used to
975 convert the file to unicode.
976
977
978 .. function:: testmod(m=None, name=None, globs=None, verbose=None, report=True, optionflags=0, extraglobs=None, raise_on_error=False, exclude_empty=False)
979
980 All arguments are optional, and all except for *m* should be specified in
981 keyword form.
982
983 Test examples in docstrings in functions and classes reachable from module *m*
984 (or module :mod:`__main__` if *m* is not supplied or is ``None``), starting with
985 ``m.__doc__``.
986
987 Also test examples reachable from dict ``m.__test__``, if it exists.
988 ``m.__test__`` maps names (strings) to functions, classes and
989 strings; function and class docstrings are searched for examples; strings are
990 searched directly, as if they were docstrings.
991
992 Only docstrings attached to objects belonging to module *m* are searched.
993
994 Return ``(failure_count, test_count)``.
995
996 Optional argument *name* gives the name of the module; by default, or if
997 ``None``, ``m.__name__`` is used.
998
999 Optional argument *exclude_empty* defaults to false. If true, objects for which
1000 no doctests are found are excluded from consideration. The default is a backward
1001 compatibility hack, so that code still using
1002 :meth:`doctest.master.summarize <DocTestRunner.summarize>` in
1003 conjunction with :func:`testmod` continues to get output for objects with no
1004 tests. The *exclude_empty* argument to the newer :class:`DocTestFinder`
1005 constructor defaults to true.
1006
1007 Optional arguments *extraglobs*, *verbose*, *report*, *optionflags*,
1008 *raise_on_error*, and *globs* are the same as for function :func:`testfile`
1009 above, except that *globs* defaults to ``m.__dict__``.
1010
1011
1012 .. function:: run_docstring_examples(f, globs, verbose=False, name="NoName", compileflags=None, optionflags=0)
1013
1014 Test examples associated with object *f*; for example, *f* may be a string,
1015 a module, a function, or a class object.
1016
1017 A shallow copy of dictionary argument *globs* is used for the execution context.
1018
1019 Optional argument *name* is used in failure messages, and defaults to
1020 ``"NoName"``.
1021
1022 If optional argument *verbose* is true, output is generated even if there are no
1023 failures. By default, output is generated only in case of an example failure.
1024
1025 Optional argument *compileflags* gives the set of flags that should be used by
1026 the Python compiler when running the examples. By default, or if ``None``,
1027 flags are deduced corresponding to the set of future features found in *globs*.
1028
1029 Optional argument *optionflags* works as for function :func:`testfile` above.
1030
1031
1032 .. _doctest-unittest-api:
1033
1034 Unittest API
1035 ------------
1036
1037 As your collection of doctest'ed modules grows, you'll want a way to run all
1038 their doctests systematically. :mod:`!doctest` provides two functions that can
1039 be used to create :mod:`unittest` test suites from modules and text files
1040 containing doctests. To integrate with :mod:`unittest` test discovery, include
1041 a :ref:`load_tests <load_tests-protocol>` function in your test module::
1042
1043 import unittest
1044 import doctest
1045 import my_module_with_doctests
1046
1047 def load_tests(loader, tests, ignore):
1048 tests.addTests(doctest.DocTestSuite(my_module_with_doctests))
1049 return tests
1050
1051 There are two main functions for creating :class:`unittest.TestSuite` instances
1052 from text files and modules with doctests:
1053
1054
1055 .. function:: DocFileSuite(*paths, module_relative=True, package=None, setUp=None, tearDown=None, globs=None, optionflags=0, parser=DocTestParser(), encoding=None)
1056
1057 Convert doctest tests from one or more text files to a
1058 :class:`unittest.TestSuite`.
1059
1060 The returned :class:`unittest.TestSuite` is to be run by the unittest framework
1061 and runs the interactive examples in each file. If an example in any file
1062 fails, then the synthesized unit test fails, and a :exc:`~unittest.TestCase.failureException`
1063 exception is raised showing the name of the file containing the test and a
1064 (sometimes approximate) line number. If all the examples in a file are
1065 skipped, then the synthesized unit test is also marked as skipped.
1066
1067 Pass one or more paths (as strings) to text files to be examined.
1068
1069 Options may be provided as keyword arguments:
1070
1071 Optional argument *module_relative* specifies how the filenames in *paths*
1072 should be interpreted:
1073
1074 * If *module_relative* is ``True`` (the default), then each filename in
1075 *paths* specifies an OS-independent module-relative path. By default, this
1076 path is relative to the calling module's directory; but if the *package*
1077 argument is specified, then it is relative to that package. To ensure
1078 OS-independence, each filename should use ``/`` characters to separate path
1079 segments, and may not be an absolute path (i.e., it may not begin with
1080 ``/``).
1081
1082 * If *module_relative* is ``False``, then each filename in *paths* specifies
1083 an OS-specific path. The path may be absolute or relative; relative paths
1084 are resolved with respect to the current working directory.
1085
1086 Optional argument *package* is a Python package or the name of a Python
1087 package whose directory should be used as the base directory for
1088 module-relative filenames in *paths*. If no package is specified, then the
1089 calling module's directory is used as the base directory for module-relative
1090 filenames. It is an error to specify *package* if *module_relative* is
1091 ``False``.
1092
1093 Optional argument *setUp* specifies a set-up function for the test suite.
1094 This is called before running the tests in each file. The *setUp* function
1095 will be passed a :class:`DocTest` object. The *setUp* function can access the
1096 test globals as the :attr:`~DocTest.globs` attribute of the test passed.
1097
1098 Optional argument *tearDown* specifies a tear-down function for the test
1099 suite. This is called after running the tests in each file. The *tearDown*
1100 function will be passed a :class:`DocTest` object. The *tearDown* function can
1101 access the test globals as the :attr:`~DocTest.globs` attribute of the test
1102 passed.
1103
1104 Optional argument *globs* is a dictionary containing the initial global
1105 variables for the tests. A new copy of this dictionary is created for each
1106 test. By default, *globs* is a new empty dictionary.
1107
1108 Optional argument *optionflags* specifies the default doctest options for the
1109 tests, created by or-ing together individual option flags. See section
1110 :ref:`doctest-options`. See function :func:`set_unittest_reportflags` below
1111 for a better way to set reporting options.
1112
1113 Optional argument *parser* specifies a :class:`DocTestParser` (or subclass)
1114 that should be used to extract tests from the files. It defaults to a normal
1115 parser (i.e., ``DocTestParser()``).
1116
1117 Optional argument *encoding* specifies an encoding that should be used to
1118 convert the file to unicode.
1119
1120 The global ``__file__`` is added to the globals provided to doctests loaded
1121 from a text file using :func:`DocFileSuite`.
1122
1123
1124 .. function:: DocTestSuite(module=None, globs=None, extraglobs=None, test_finder=None, setUp=None, tearDown=None, optionflags=0, checker=None)
1125
1126 Convert doctest tests for a module to a :class:`unittest.TestSuite`.
1127
1128 The returned :class:`unittest.TestSuite` is to be run by the unittest framework
1129 and runs each doctest in the module.
1130 Each docstring is run as a separate unit test.
1131 If any of the doctests fail, then the synthesized unit test fails,
1132 and a :exc:`unittest.TestCase.failureException` exception is raised
1133 showing the name of the file containing the test and a (sometimes approximate)
1134 line number. If all the examples in a docstring are skipped, then the
1135
1136 Optional argument *module* provides the module to be tested. It can be a module
1137 object or a (possibly dotted) module name. If not specified, the module calling
1138 this function is used.
1139
1140 Optional argument *globs* is a dictionary containing the initial global
1141 variables for the tests. A new copy of this dictionary is created for each
1142 test. By default, *globs* is the module's :attr:`~module.__dict__`.
1143
1144 Optional argument *extraglobs* specifies an extra set of global variables, which
1145 is merged into *globs*. By default, no extra globals are used.
1146
1147 Optional argument *test_finder* is the :class:`DocTestFinder` object (or a
1148 drop-in replacement) that is used to extract doctests from the module.
1149
1150 Optional arguments *setUp*, *tearDown*, and *optionflags* are the same as for
1151 function :func:`DocFileSuite` above, but they are called for each docstring.
1152
1153 This function uses the same search technique as :func:`testmod`.
1154
1155 .. versionchanged:: 3.5
1156 :func:`DocTestSuite` returns an empty :class:`unittest.TestSuite` if *module*
1157 contains no docstrings instead of raising :exc:`ValueError`.
1158
1159 Under the covers, :func:`DocTestSuite` creates a :class:`unittest.TestSuite` out
1160 of :class:`!doctest.DocTestCase` instances, and :class:`!DocTestCase` is a
1161 subclass of :class:`unittest.TestCase`. :class:`!DocTestCase` isn't documented
1162 here (it's an internal detail), but studying its code can answer questions about
1163 the exact details of :mod:`unittest` integration.
1164
1165 Similarly, :func:`DocFileSuite` creates a :class:`unittest.TestSuite` out of
1166 :class:`!doctest.DocFileCase` instances, and :class:`!DocFileCase` is a subclass
1167 of :class:`!DocTestCase`.
1168
1169 So both ways of creating a :class:`unittest.TestSuite` run instances of
1170 :class:`!DocTestCase`. This is important for a subtle reason: when you run
1171 :mod:`!doctest` functions yourself, you can control the :mod:`!doctest` options in
1172 use directly, by passing option flags to :mod:`!doctest` functions. However, if
1173 you're writing a :mod:`unittest` framework, :mod:`!unittest` ultimately controls
1174 when and how tests get run. The framework author typically wants to control
1175 :mod:`!doctest` reporting options (perhaps, e.g., specified by command line
1176 options), but there's no way to pass options through :mod:`!unittest` to
1177 :mod:`!doctest` test runners.
1178
1179 For this reason, :mod:`!doctest` also supports a notion of :mod:`!doctest`
1180 reporting flags specific to :mod:`unittest` support, via this function:
1181
1182
1183 .. function:: set_unittest_reportflags(flags)
1184
1185 Set the :mod:`!doctest` reporting flags to use.
1186
1187 Argument *flags* takes the :ref:`bitwise OR <bitwise>` of option flags. See
1188 section :ref:`doctest-options`. Only "reporting flags" can be used.
1189
1190 This is a module-global setting, and affects all future doctests run by module
1191 :mod:`unittest`: the :meth:`!runTest` method of :class:`!DocTestCase` looks at
1192 the option flags specified for the test case when the :class:`!DocTestCase`
1193 instance was constructed. If no reporting flags were specified (which is the
1194 typical and expected case), :mod:`!doctest`'s :mod:`!unittest` reporting flags are
1195 :ref:`bitwise ORed <bitwise>` into the option flags, and the option flags
1196 so augmented are passed to the :class:`DocTestRunner` instance created to
1197 run the doctest. If any reporting flags were specified when the
1198 :class:`!DocTestCase` instance was constructed, :mod:`!doctest`'s
1199 :mod:`!unittest` reporting flags are ignored.
1200
1201 The value of the :mod:`unittest` reporting flags in effect before the function
1202 was called is returned by the function.
1203
1204
1205 .. _doctest-advanced-api:
1206
1207 Advanced API
1208 ------------
1209
1210 The basic API is a simple wrapper that's intended to make doctest easy to use.
1211 It is fairly flexible, and should meet most users' needs; however, if you
1212 require more fine-grained control over testing, or wish to extend doctest's
1213 capabilities, then you should use the advanced API.
1214
1215 The advanced API revolves around two container classes, which are used to store
1216 the interactive examples extracted from doctest cases:
1217
1218 * :class:`Example`: A single Python :term:`statement`, paired with its expected
1219 output.
1220
1221 * :class:`DocTest`: A collection of :class:`Example`\ s, typically extracted
1222 from a single docstring or text file.
1223
1224 Additional processing classes are defined to find, parse, and run, and check
1225 doctest examples:
1226
1227 * :class:`DocTestFinder`: Finds all docstrings in a given module, and uses a
1228 :class:`DocTestParser` to create a :class:`DocTest` from every docstring that
1229 contains interactive examples.
1230
1231 * :class:`DocTestParser`: Creates a :class:`DocTest` object from a string (such
1232 as an object's docstring).
1233
1234 * :class:`DocTestRunner`: Executes the examples in a :class:`DocTest`, and uses
1235 an :class:`OutputChecker` to verify their output.
1236
1237 * :class:`OutputChecker`: Compares the actual output from a doctest example with
1238 the expected output, and decides whether they match.
1239
1240 The relationships among these processing classes are summarized in the following
1241 diagram::
1242
1243 list of:
1244 +------+ +---------+
1245 |module| --DocTestFinder-> | DocTest | --DocTestRunner-> results
1246 +------+ | ^ +---------+ | ^ (printed)
1247 | | | Example | | |
1248 v | | ... | v |
1249 DocTestParser | Example | OutputChecker
1250 +---------+
1251
1252
1253 .. _doctest-doctest:
1254
1255 DocTest Objects
1256 ^^^^^^^^^^^^^^^
1257
1258
1259 .. class:: DocTest(examples, globs, name, filename, lineno, docstring)
1260
1261 A collection of doctest examples that should be run in a single namespace. The
1262 constructor arguments are used to initialize the attributes of the same names.
1263
1264
1265 :class:`DocTest` defines the following attributes. They are initialized by
1266 the constructor, and should not be modified directly.
1267
1268
1269 .. attribute:: examples
1270
1271 A list of :class:`Example` objects encoding the individual interactive Python
1272 examples that should be run by this test.
1273
1274
1275 .. attribute:: globs
1276
1277 The namespace (aka globals) that the examples should be run in. This is a
1278 dictionary mapping names to values. Any changes to the namespace made by the
1279 examples (such as binding new variables) will be reflected in :attr:`globs`
1280 after the test is run.
1281
1282
1283 .. attribute:: name
1284
1285 A string name identifying the :class:`DocTest`. Typically, this is the name
1286 of the object or file that the test was extracted from.
1287
1288
1289 .. attribute:: filename
1290
1291 The name of the file that this :class:`DocTest` was extracted from; or
1292 ``None`` if the filename is unknown, or if the :class:`!DocTest` was not
1293 extracted from a file.
1294
1295
1296 .. attribute:: lineno
1297
1298 The line number within :attr:`filename` where this :class:`DocTest` begins, or
1299 ``None`` if the line number is unavailable. This line number is zero-based
1300 with respect to the beginning of the file.
1301
1302
1303 .. attribute:: docstring
1304
1305 The string that the test was extracted from, or ``None`` if the string is
1306 unavailable, or if the test was not extracted from a string.
1307
1308
1309 .. _doctest-example:
1310
1311 Example Objects
1312 ^^^^^^^^^^^^^^^
1313
1314
1315 .. class:: Example(source, want, exc_msg=None, lineno=0, indent=0, options=None)
1316
1317 A single interactive example, consisting of a Python statement and its expected
1318 output. The constructor arguments are used to initialize the attributes of
1319 the same names.
1320
1321
1322 :class:`Example` defines the following attributes. They are initialized by
1323 the constructor, and should not be modified directly.
1324
1325
1326 .. attribute:: source
1327
1328 A string containing the example's source code. This source code consists of a
1329 single Python statement, and always ends with a newline; the constructor adds
1330 a newline when necessary.
1331
1332
1333 .. attribute:: want
1334
1335 The expected output from running the example's source code (either from
1336 stdout, or a traceback in case of exception). :attr:`want` ends with a
1337 newline unless no output is expected, in which case it's an empty string. The
1338 constructor adds a newline when necessary.
1339
1340
1341 .. attribute:: exc_msg
1342
1343 The exception message generated by the example, if the example is expected to
1344 generate an exception; or ``None`` if it is not expected to generate an
1345 exception. This exception message is compared against the return value of
1346 :func:`traceback.format_exception_only`. :attr:`exc_msg` ends with a newline
1347 unless it's ``None``. The constructor adds a newline if needed.
1348
1349
1350 .. attribute:: lineno
1351
1352 The line number within the string containing this example where the example
1353 begins. This line number is zero-based with respect to the beginning of the
1354 containing string.
1355
1356
1357 .. attribute:: indent
1358
1359 The example's indentation in the containing string, i.e., the number of space
1360 characters that precede the example's first prompt.
1361
1362
1363 .. attribute:: options
1364
1365 A dictionary mapping from option flags to ``True`` or ``False``, which is used
1366 to override default options for this example. Any option flags not contained
1367 in this dictionary are left at their default value (as specified by the
1368 :class:`DocTestRunner`'s :ref:`optionflags <doctest-options>`).
1369 By default, no options are set.
1370
1371
1372 .. _doctest-doctestfinder:
1373
1374 DocTestFinder objects
1375 ^^^^^^^^^^^^^^^^^^^^^
1376
1377
1378 .. class:: DocTestFinder(verbose=False, parser=DocTestParser(), recurse=True, exclude_empty=True)
1379
1380 A processing class used to extract the :class:`DocTest`\ s that are relevant to
1381 a given object, from its docstring and the docstrings of its contained objects.
1382 :class:`DocTest`\ s can be extracted from modules, classes, functions,
1383 methods, staticmethods, classmethods, and properties.
1384
1385 The optional argument *verbose* can be used to display the objects searched by
1386 the finder. It defaults to ``False`` (no output).
1387
1388 The optional argument *parser* specifies the :class:`DocTestParser` object (or a
1389 drop-in replacement) that is used to extract doctests from docstrings.
1390
1391 If the optional argument *recurse* is false, then :meth:`DocTestFinder.find`
1392 will only examine the given object, and not any contained objects.
1393
1394 If the optional argument *exclude_empty* is false, then
1395 :meth:`DocTestFinder.find` will include tests for objects with empty docstrings.
1396
1397
1398 :class:`DocTestFinder` defines the following method:
1399
1400
1401 .. method:: find(obj[, name][, module][, globs][, extraglobs])
1402
1403 Return a list of the :class:`DocTest`\ s that are defined by *obj*'s
1404 docstring, or by any of its contained objects' docstrings.
1405
1406 The optional argument *name* specifies the object's name; this name will be
1407 used to construct names for the returned :class:`DocTest`\ s. If *name* is
1408 not specified, then ``obj.__name__`` is used.
1409
1410 The optional parameter *module* is the module that contains the given object.
1411 If the module is not specified or is ``None``, then the test finder will attempt
1412 to automatically determine the correct module. The object's module is used:
1413
1414 * As a default namespace, if *globs* is not specified.
1415
1416 * To prevent the DocTestFinder from extracting DocTests from objects that are
1417 imported from other modules. (Contained objects with modules other than
1418 *module* are ignored.)
1419
1420 * To find the name of the file containing the object.
1421
1422 * To help find the line number of the object within its file.
1423
1424 If *module* is ``False``, no attempt to find the module will be made. This is
1425 obscure, of use mostly in testing doctest itself: if *module* is ``False``, or
1426 is ``None`` but cannot be found automatically, then all objects are considered
1427 to belong to the (non-existent) module, so all contained objects will
1428 (recursively) be searched for doctests.
1429
1430 The globals for each :class:`DocTest` is formed by combining *globs* and
1431 *extraglobs* (bindings in *extraglobs* override bindings in *globs*). A new
1432 shallow copy of the globals dictionary is created for each :class:`!DocTest`.
1433 If *globs* is not specified, then it defaults to the module's
1434 :attr:`~module.__dict__`, if specified, or ``{}`` otherwise.
1435 If *extraglobs* is not specified, then it defaults to ``{}``.
1436
1437
1438 .. _doctest-doctestparser:
1439
1440 DocTestParser objects
1441 ^^^^^^^^^^^^^^^^^^^^^
1442
1443
1444 .. class:: DocTestParser()
1445
1446 A processing class used to extract interactive examples from a string, and use
1447 them to create a :class:`DocTest` object.
1448
1449
1450 :class:`DocTestParser` defines the following methods:
1451
1452
1453 .. method:: get_doctest(string, globs, name, filename, lineno)
1454
1455 Extract all doctest examples from the given string, and collect them into a
1456 :class:`DocTest` object.
1457
1458 *globs*, *name*, *filename*, and *lineno* are attributes for the new
1459 :class:`!DocTest` object. See the documentation for :class:`DocTest` for more
1460 information.
1461
1462
1463 .. method:: get_examples(string, name='<string>')
1464
1465 Extract all doctest examples from the given string, and return them as a list
1466 of :class:`Example` objects. Line numbers are 0-based. The optional argument
1467 *name* is a name identifying this string, and is only used for error messages.
1468
1469
1470 .. method:: parse(string, name='<string>')
1471
1472 Divide the given string into examples and intervening text, and return them as
1473 a list of alternating :class:`Example`\ s and strings. Line numbers for the
1474 :class:`!Example`\ s are 0-based. The optional argument *name* is a name
1475 identifying this string, and is only used for error messages.
1476
1477
1478 TestResults objects
1479 ^^^^^^^^^^^^^^^^^^^
1480
1481
1482 .. class:: TestResults(failed, attempted)
1483
1484 .. attribute:: failed
1485
1486 Number of failed tests.
1487
1488 .. attribute:: attempted
1489
1490 Number of attempted tests.
1491
1492 .. attribute:: skipped
1493
1494 Number of skipped tests.
1495
1496 .. versionadded:: 3.13
1497
1498
1499 .. _doctest-doctestrunner:
1500
1501 DocTestRunner objects
1502 ^^^^^^^^^^^^^^^^^^^^^
1503
1504
1505 .. class:: DocTestRunner(checker=None, verbose=None, optionflags=0)
1506
1507 A processing class used to execute and verify the interactive examples in a
1508 :class:`DocTest`.
1509
1510 The comparison between expected outputs and actual outputs is done by an
1511 :class:`OutputChecker`. This comparison may be customized with a number of
1512 option flags; see section :ref:`doctest-options` for more information. If the
1513 option flags are insufficient, then the comparison may also be customized by
1514 passing a subclass of :class:`!OutputChecker` to the constructor.
1515
1516 The test runner's display output can be controlled in two ways. First, an output
1517 function can be passed to :meth:`run`; this function will be called
1518 with strings that should be displayed. It defaults to ``sys.stdout.write``. If
1519 capturing the output is not sufficient, then the display output can be also
1520 customized by subclassing DocTestRunner, and overriding the methods
1521 :meth:`report_start`, :meth:`report_success`,
1522 :meth:`report_unexpected_exception`, and :meth:`report_failure`.
1523
1524 The optional keyword argument *checker* specifies the :class:`OutputChecker`
1525 object (or drop-in replacement) that should be used to compare the expected
1526 outputs to the actual outputs of doctest examples.
1527
1528 The optional keyword argument *verbose* controls the :class:`DocTestRunner`'s
1529 verbosity. If *verbose* is ``True``, then information is printed about each
1530 example, as it is run. If *verbose* is ``False``, then only failures are
1531 printed. If *verbose* is unspecified, or ``None``, then verbose output is used
1532 iff the command-line switch ``-v`` is used.
1533
1534 The optional keyword argument *optionflags* can be used to control how the test
1535 runner compares expected output to actual output, and how it displays failures.
1536 For more information, see section :ref:`doctest-options`.
1537
1538 The test runner accumulates statistics. The aggregated number of attempted,
1539 failed and skipped examples is also available via the :attr:`tries`,
1540 :attr:`failures` and :attr:`skips` attributes. The :meth:`run` and
1541 :meth:`summarize` methods return a :class:`TestResults` instance.
1542
1543 :class:`DocTestRunner` defines the following methods:
1544
1545
1546 .. method:: report_start(out, test, example)
1547
1548 Report that the test runner is about to process the given example. This method
1549 is provided to allow subclasses of :class:`DocTestRunner` to customize their
1550 output; it should not be called directly.
1551
1552 *example* is the example about to be processed. *test* is the test
1553 containing *example*. *out* is the output function that was passed to
1554 :meth:`DocTestRunner.run`.
1555
1556
1557 .. method:: report_success(out, test, example, got)
1558
1559 Report that the given example ran successfully. This method is provided to
1560 allow subclasses of :class:`DocTestRunner` to customize their output; it
1561 should not be called directly.
1562
1563 *example* is the example about to be processed. *got* is the actual output
1564 from the example. *test* is the test containing *example*. *out* is the
1565 output function that was passed to :meth:`DocTestRunner.run`.
1566
1567
1568 .. method:: report_failure(out, test, example, got)
1569
1570 Report that the given example failed. This method is provided to allow
1571 subclasses of :class:`DocTestRunner` to customize their output; it should not
1572 be called directly.
1573
1574 *example* is the example about to be processed. *got* is the actual output
1575 from the example. *test* is the test containing *example*. *out* is the
1576 output function that was passed to :meth:`DocTestRunner.run`.
1577
1578
1579 .. method:: report_unexpected_exception(out, test, example, exc_info)
1580
1581 Report that the given example raised an unexpected exception. This method is
1582 provided to allow subclasses of :class:`DocTestRunner` to customize their
1583 output; it should not be called directly.
1584
1585 *example* is the example about to be processed. *exc_info* is a tuple
1586 containing information about the unexpected exception (as returned by
1587 :func:`sys.exc_info`). *test* is the test containing *example*. *out* is the
1588 output function that was passed to :meth:`DocTestRunner.run`.
1589
1590
1591 .. method:: run(test, compileflags=None, out=None, clear_globs=True)
1592
1593 Run the examples in *test* (a :class:`DocTest` object), and display the
1594 results using the writer function *out*. Return a :class:`TestResults`
1595 instance.
1596
1597 The examples are run in the namespace ``test.globs``. If *clear_globs* is
1598 true (the default), then this namespace will be cleared after the test runs,
1599 to help with garbage collection. If you would like to examine the namespace
1600 after the test completes, then use *clear_globs=False*.
1601
1602 *compileflags* gives the set of flags that should be used by the Python
1603 compiler when running the examples. If not specified, then it will default to
1604 the set of future-import flags that apply to *globs*.
1605
1606 The output of each example is checked using the :class:`DocTestRunner`'s
1607 output checker, and the results are formatted by the
1608 :meth:`!DocTestRunner.report_\*` methods.
1609
1610
1611 .. method:: summarize(verbose=None)
1612
1613 Print a summary of all the test cases that have been run by this DocTestRunner,
1614 and return a :class:`TestResults` instance.
1615
1616 The optional *verbose* argument controls how detailed the summary is. If the
1617 verbosity is not specified, then the :class:`DocTestRunner`'s verbosity is
1618 used.
1619
1620 :class:`DocTestParser` has the following attributes:
1621
1622 .. attribute:: tries
1623
1624 Number of attempted examples.
1625
1626 .. attribute:: failures
1627
1628 Number of failed examples.
1629
1630 .. attribute:: skips
1631
1632 Number of skipped examples.
1633
1634 .. versionadded:: 3.13
1635
1636
1637 .. _doctest-outputchecker:
1638
1639 OutputChecker objects
1640 ^^^^^^^^^^^^^^^^^^^^^
1641
1642
1643 .. class:: OutputChecker()
1644
1645 A class used to check the whether the actual output from a doctest example
1646 matches the expected output. :class:`OutputChecker` defines two methods:
1647 :meth:`check_output`, which compares a given pair of outputs, and returns ``True``
1648 if they match; and :meth:`output_difference`, which returns a string describing
1649 the differences between two outputs.
1650
1651
1652 :class:`OutputChecker` defines the following methods:
1653
1654 .. method:: check_output(want, got, optionflags)
1655
1656 Return ``True`` iff the actual output from an example (*got*) matches the
1657 expected output (*want*). These strings are always considered to match if
1658 they are identical; but depending on what option flags the test runner is
1659 using, several non-exact match types are also possible. See section
1660 :ref:`doctest-options` for more information about option flags.
1661
1662
1663 .. method:: output_difference(example, got, optionflags)
1664
1665 Return a string describing the differences between the expected output for a
1666 given example (*example*) and the actual output (*got*). *optionflags* is the
1667 set of option flags used to compare *want* and *got*.
1668
1669
1670 .. _doctest-debugging:
1671
1672 Debugging
1673 ---------
1674
1675 Doctest provides several mechanisms for debugging doctest examples:
1676
1677 * Several functions convert doctests to executable Python programs, which can be
1678 run under the Python debugger, :mod:`pdb`.
1679
1680 * The :class:`DebugRunner` class is a subclass of :class:`DocTestRunner` that
1681 raises an exception for the first failing example, containing information about
1682 that example. This information can be used to perform post-mortem debugging on
1683 the example.
1684
1685 * The :mod:`unittest` cases generated by :func:`DocTestSuite` support the
1686 :meth:`debug` method defined by :class:`unittest.TestCase`.
1687
1688 * You can add a call to :func:`pdb.set_trace` in a doctest example, and you'll
1689 drop into the Python debugger when that line is executed. Then you can inspect
1690 current values of variables, and so on. For example, suppose :file:`a.py`
1691 contains just this module docstring::
1692
1693 """
1694 >>> def f(x):
1695 ... g(x*2)
1696 >>> def g(x):
1697 ... print(x+3)
1698 ... import pdb; pdb.set_trace()
1699 >>> f(3)
1700 9
1701 """
1702
1703 Then an interactive Python session may look like this::
1704
1705 >>> import a, doctest
1706 >>> doctest.testmod(a)
1707 --Return--
1708 > <doctest a[1]>(3)g()->None
1709 -> import pdb; pdb.set_trace()
1710 (Pdb) list
1711 1 def g(x):
1712 2 print(x+3)
1713 3 -> import pdb; pdb.set_trace()
1714 [EOF]
1715 (Pdb) p x
1716 6
1717 (Pdb) step
1718 --Return--
1719 > <doctest a[0]>(2)f()->None
1720 -> g(x*2)
1721 (Pdb) list
1722 1 def f(x):
1723 2 -> g(x*2)
1724 [EOF]
1725 (Pdb) p x
1726 3
1727 (Pdb) step
1728 --Return--
1729 > <doctest a[2]>(1)?()->None
1730 -> f(3)
1731 (Pdb) cont
1732 (0, 3)
1733 >>>
1734
1735
1736 Functions that convert doctests to Python code, and possibly run the synthesized
1737 code under the debugger:
1738
1739
1740 .. function:: script_from_examples(s)
1741
1742 Convert text with examples to a script.
1743
1744 Argument *s* is a string containing doctest examples. The string is converted
1745 to a Python script, where doctest examples in *s* are converted to regular code,
1746 and everything else is converted to Python comments. The generated script is
1747 returned as a string. For example, ::
1748
1749 import doctest
1750 print(doctest.script_from_examples(r"""
1751 Set x and y to 1 and 2.
1752 >>> x, y = 1, 2
1753
1754 Print their sum:
1755 >>> print(x+y)
1756 3
1757 """))
1758
1759 displays::
1760
1761 # Set x and y to 1 and 2.
1762 x, y = 1, 2
1763 #
1764 # Print their sum:
1765 print(x+y)
1766 # Expected:
1767 ## 3
1768
1769 This function is used internally by other functions (see below), but can also be
1770 useful when you want to transform an interactive Python session into a Python
1771 script.
1772
1773
1774 .. function:: testsource(module, name)
1775
1776 Convert the doctest for an object to a script.
1777
1778 Argument *module* is a module object, or dotted name of a module, containing the
1779 object whose doctests are of interest. Argument *name* is the name (within the
1780 module) of the object with the doctests of interest. The result is a string,
1781 containing the object's docstring converted to a Python script, as described for
1782 :func:`script_from_examples` above. For example, if module :file:`a.py`
1783 contains a top-level function :func:`!f`, then ::
1784
1785 import a, doctest
1786 print(doctest.testsource(a, "a.f"))
1787
1788 prints a script version of function :func:`!f`'s docstring, with doctests
1789 converted to code, and the rest placed in comments.
1790
1791
1792 .. function:: debug(module, name, pm=False)
1793
1794 Debug the doctests for an object.
1795
1796 The *module* and *name* arguments are the same as for function
1797 :func:`testsource` above. The synthesized Python script for the named object's
1798 docstring is written to a temporary file, and then that file is run under the
1799 control of the Python debugger, :mod:`pdb`.
1800
1801 A shallow copy of ``module.__dict__`` is used for both local and global
1802 execution context.
1803
1804 Optional argument *pm* controls whether post-mortem debugging is used. If *pm*
1805 has a true value, the script file is run directly, and the debugger gets
1806 involved only if the script terminates via raising an unhandled exception. If
1807 it does, then post-mortem debugging is invoked, via :func:`pdb.post_mortem`,
1808 passing the traceback object from the unhandled exception. If *pm* is not
1809 specified, or is false, the script is run under the debugger from the start, via
1810 passing an appropriate :func:`exec` call to :func:`pdb.run`.
1811
1812
1813 .. function:: debug_src(src, pm=False, globs=None)
1814
1815 Debug the doctests in a string.
1816
1817 This is like function :func:`debug` above, except that a string containing
1818 doctest examples is specified directly, via the *src* argument.
1819
1820 Optional argument *pm* has the same meaning as in function :func:`debug` above.
1821
1822 Optional argument *globs* gives a dictionary to use as both local and global
1823 execution context. If not specified, or ``None``, an empty dictionary is used.
1824 If specified, a shallow copy of the dictionary is used.
1825
1826
1827 The :class:`DebugRunner` class, and the special exceptions it may raise, are of
1828 most interest to testing framework authors, and will only be sketched here. See
1829 the source code, and especially :class:`DebugRunner`'s docstring (which is a
1830 doctest!) for more details:
1831
1832
1833 .. class:: DebugRunner(checker=None, verbose=None, optionflags=0)
1834
1835 A subclass of :class:`DocTestRunner` that raises an exception as soon as a
1836 failure is encountered. If an unexpected exception occurs, an
1837 :exc:`UnexpectedException` exception is raised, containing the test, the
1838 example, and the original exception. If the output doesn't match, then a
1839 :exc:`DocTestFailure` exception is raised, containing the test, the example, and
1840 the actual output.
1841
1842 For information about the constructor parameters and methods, see the
1843 documentation for :class:`DocTestRunner` in section :ref:`doctest-advanced-api`.
1844
1845 There are two exceptions that may be raised by :class:`DebugRunner` instances:
1846
1847
1848 .. exception:: DocTestFailure(test, example, got)
1849
1850 An exception raised by :class:`DocTestRunner` to signal that a doctest example's
1851 actual output did not match its expected output. The constructor arguments are
1852 used to initialize the attributes of the same names.
1853
1854 :exc:`DocTestFailure` defines the following attributes:
1855
1856
1857 .. attribute:: DocTestFailure.test
1858
1859 The :class:`DocTest` object that was being run when the example failed.
1860
1861
1862 .. attribute:: DocTestFailure.example
1863
1864 The :class:`Example` that failed.
1865
1866
1867 .. attribute:: DocTestFailure.got
1868
1869 The example's actual output.
1870
1871
1872 .. exception:: UnexpectedException(test, example, exc_info)
1873
1874 An exception raised by :class:`DocTestRunner` to signal that a doctest
1875 example raised an unexpected exception. The constructor arguments are used
1876 to initialize the attributes of the same names.
1877
1878 :exc:`UnexpectedException` defines the following attributes:
1879
1880
1881 .. attribute:: UnexpectedException.test
1882
1883 The :class:`DocTest` object that was being run when the example failed.
1884
1885
1886 .. attribute:: UnexpectedException.example
1887
1888 The :class:`Example` that failed.
1889
1890
1891 .. attribute:: UnexpectedException.exc_info
1892
1893 A tuple containing information about the unexpected exception, as returned by
1894 :func:`sys.exc_info`.
1895
1896
1897 .. _doctest-soapbox:
1898
1899 Soapbox
1900 -------
1901
1902 As mentioned in the introduction, :mod:`!doctest` has grown to have three primary
1903 uses:
1904
1905 #. Checking examples in docstrings.
1906
1907 #. Regression testing.
1908
1909 #. Executable documentation / literate testing.
1910
1911 These uses have different requirements, and it is important to distinguish them.
1912 In particular, filling your docstrings with obscure test cases makes for bad
1913 documentation.
1914
1915 When writing a docstring, choose docstring examples with care. There's an art to
1916 this that needs to be learned---it may not be natural at first. Examples should
1917 add genuine value to the documentation. A good example can often be worth many
1918 words. If done with care, the examples will be invaluable for your users, and
1919 will pay back the time it takes to collect them many times over as the years go
1920 by and things change. I'm still amazed at how often one of my :mod:`!doctest`
1921 examples stops working after a "harmless" change.
1922
1923 Doctest also makes an excellent tool for regression testing, especially if you
1924 don't skimp on explanatory text. By interleaving prose and examples, it becomes
1925 much easier to keep track of what's actually being tested, and why. When a test
1926 fails, good prose can make it much easier to figure out what the problem is, and
1927 how it should be fixed. It's true that you could write extensive comments in
1928 code-based testing, but few programmers do. Many have found that using doctest
1929 approaches instead leads to much clearer tests. Perhaps this is simply because
1930 doctest makes writing prose a little easier than writing code, while writing
1931 comments in code is a little harder. I think it goes deeper than just that:
1932 the natural attitude when writing a doctest-based test is that you want to
1933 explain the fine points of your software, and illustrate them with examples.
1934 This in turn naturally leads to test files that start with the simplest
1935 features, and logically progress to complications and edge cases. A coherent
1936 narrative is the result, instead of a collection of isolated functions that test
1937 isolated bits of functionality seemingly at random. It's a different attitude,
1938 and produces different results, blurring the distinction between testing and
1939 explaining.
1940
1941 Regression testing is best confined to dedicated objects or files. There are
1942 several options for organizing tests:
1943
1944 * Write text files containing test cases as interactive examples, and test the
1945 files using :func:`testfile` or :func:`DocFileSuite`. This is recommended,
1946 although is easiest to do for new projects, designed from the start to use
1947 doctest.
1948
1949 * Define functions named ``_regrtest_topic`` that consist of single docstrings,
1950 containing test cases for the named topics. These functions can be included in
1951 the same file as the module, or separated out into a separate test file.
1952
1953 * Define a :attr:`~module.__test__` dictionary mapping from regression test topics to
1954 docstrings containing test cases.
1955
1956 When you have placed your tests in a module, the module can itself be the test
1957 runner. When a test fails, you can arrange for your test runner to re-run only
1958 the failing doctest while you debug the problem. Here is a minimal example of
1959 such a test runner::
1960
1961 if __name__ == '__main__':
1962 import doctest
1963 flags = doctest.REPORT_NDIFF|doctest.FAIL_FAST
1964 if len(sys.argv) > 1:
1965 name = sys.argv[1]
1966 if name in globals():
1967 obj = globals()[name]
1968 else:
1969 obj = __test__[name]
1970 doctest.run_docstring_examples(obj, globals(), name=name,
1971 optionflags=flags)
1972 else:
1973 fail, total = doctest.testmod(optionflags=flags)
1974 print(f"{fail} failures out of {total} tests")
1975
1976
1977 .. rubric:: Footnotes
1978
1979 .. [#] Examples containing both expected output and an exception are not supported.
1980 Trying to guess where one ends and the other begins is too error-prone, and that
1981 also makes for a confusing test.