Debian Codegraph @

docs/Writing_Doctests.rst

raw
1 ========================
2 Writing doctest examples
3 ========================
4
5 Doctest support is provided in Sphinx by the extension
6 `sphinx.ext.doctest`_, and its documentation is one
7 useful resurce for working with the pyparsing doctests.
8
9 .. _sphinx.ext.doctest: https://www.sphinx-doc.org/en/master/usage/extensions/doctest.html
10
11
12 Types of doctests
13 =================
14
15 There are two basic forms of doctest, and both are used extensively
16 in the Pyparsing documentation. Which one to use for a given example
17 is a decision that needs to be made when writing it, but there are
18 some factors that usually make the correct choice an obvious one.
19
20 Doctest type 1: ``testcode`` / ``testoutput`` blocks
21 ----------------------------------------------------
22
23 The first form involves one or potentially two separate code blocks.
24 The ``testcode`` block contains all of the input code in the form of
25 a standard Python script. This can optionally be paired with a
26 second ``testoutput`` block, which if present will contain the output
27 for the preceding ``testcode`` block.
28
29 An example of a ``testcode`` / ``testoutput`` pair, from the docstring
30 for ``ParserElement.__add__``:
31
32
33 .. code-block:: rst
34
35 Example:
36
37 .. testcode::
38
39 greet = Word(alphas) + "," + Word(alphas) + "!"
40 hello = "Hello, World!"
41 print(hello, "->", greet.parse_string(hello))
42
43 prints:
44
45 .. testoutput::
46
47 Hello, World! -> ['Hello', ',', 'World', '!']
48
49 Examples written like this will be formatted in the rendered HTML/Latex/etc.
50 documentation **exactly** as if they'd been written as normal code blocks.
51 There is no visible difference between the code above and this code without
52 doctest support:
53
54 .. code-block:: rst
55
56 Example::
57
58 greet = Word(alphas) + "," + Word(alphas) + "!"
59 hello = "Hello, World!"
60 print(hello, "->", greet.parse_string(hello))
61
62 prints::
63
64 Hello, World! -> ['Hello', ',', 'World', '!']
65
66 However, the advantage to writing doctests is that when ``make doctest``
67 is run from the ``docs/`` directory, the doctest extension will execute
68 each ``testcode`` block, and verify that its output exactly matches the
69 ``testoutput`` block (if present).
70
71 Any deviations will be displayed in "ndiff" format. This enhancement
72 to the standard unified diff will (sometimes) indicate where in each
73 line the differences occur. (The character-difference highlighting is
74 frustratingly inconsistent. But at worst ndiff is equivalent to unified
75 diff, so it's still worth using.)
76
77 Testing examples with doctest allows the code used to demonstrate the
78 pyparsing API to be verified against the *actual* API as it's currently
79 implemented, and ensures that examples stay current and relevant.
80
81 Not all ``testcode`` blocks need a corresponding ``testoutput`` — if a
82 ``testcode`` block is included on its own, the code inside the block will
83 still be executed, but its output won't be verified. This can be useful
84 when displaying code that doesn't require demonstration of its output
85 (or doesn't output anything), as the extension will still verify that
86 the code can be run without error.
87
88 It's also possible to include a *hidden* ``testoutput`` block, which will
89 beverified against the preceding ``testcode`` but won't be displayed in the
90 documentation. To hide a ``testoutput`` block (or a ``testcode`` block,
91 for that matter), add the ``:hide:`` option as an argument to the
92 directive, i.e.:
93
94 .. code-block:: rst
95
96 .. testoutput::
97 :hide:
98
99 """Output that won't be shown, but will be verified against the
100 preceding testcode block."""
101
102 Doctest type 2: ``doctest`` interactive blocks
103 ----------------------------------------------
104
105 The second type of doctest is a ``doctest`` block, which takes the form of
106 an interactive Python REPL session in standard format (using ``>>>`` and
107 ``...`` markers for input lines).
108
109 With these tests, output is interleaved with the code, which can be much
110 easier to follow when there are multiple lines producing output. If an
111 example would contains multiple ``print()`` calls, rather than first
112 displaying all of the code in a ``testcode`` block, then all of the
113 output in a ``testoutput`` block, consider using a ``doctest`` session
114 so that the reader can follow along each step as it occurs.
115
116 A typical ``doctest`` example can be found in the ``ParserElement.ignore``
117 docstring:
118
119 .. code-block:: rst
120
121 Example:
122
123 .. doctest::
124
125 >>> patt = Word(alphas)[...]
126 >>> print(patt.parse_string('ablaj /* comment */ lskjd'))
127 ['ablaj']
128
129 >>> patt = Word(alphas)[...].ignore(c_style_comment)
130 >>> print(patt.parse_string('ablaj /* comment */ lskjd'))
131 ['ablaj', 'lskjd']
132
133
134 Setup code for doctest blocks
135 =============================
136
137 The doctest extension is configured with extensive setup code
138 which is run before each test block. It can be viewed in the
139 :download:`docs/conf.py <../docs/conf.py>` file — look for the
140 ``doctest_global_setup`` variable near the end of the file.
141
142 The setup code is intended to make any useful symbols available
143 to the tests without them having to be included in each and every
144 doctest block. If additional modules are needed, feel free to add
145 them to the global setup. When writing doctests, Pyparsing classes
146 can be invoked directly, or as members of the ``pp`` alias namespace.
147 Either way, the definition of those symbols can be assumed without
148 explicitly importing/defining them.
149
150 When using symbols from other aliased namespaces, however, it's a
151 good idea to establish the alias for the reader at the start of the
152 example code. Even though these are both defined in the global setup,
153 showing the establishing lines before referencing ``ppc`` or ``ppu``
154 in an example makes that example clearer:
155
156 .. code-block: py
157
158 ppc = pp.pyparsing_common
159 ppu = pp.pyparsing_unicode
160
161 However, because those symbols *are* provided by default, they don't
162 need to be explicitly established for **every** example. Feel free
163 to omit them after the first use, when writing multiple examples for
164 a given class or function.
165
166 Documenting exceptions
167 ======================
168
169 Code that will trigger an exception can be both demonstrated and
170 verified using doctests (of either type), although when a ``testoutput``
171 block will demonstrate an exception it should be the only output in
172 that block — doctest does not support mixing regular output and
173 exceptions.
174
175 Both the ``IGNORE_EXCEPTION_DETAIL`` and ``ELLIPSIS`` doctest options
176 are enabled by default, which make demonstrating exceptions far more
177 convenient. Ignoring exception detail means that the full traceback
178 for an exception can be omitted, as well as the fully-qualified name
179 of the exception class. As long as the ``Traceback...`` line and the
180 exception class name match, the doctest will pass. (The exception
181 message is also verified by default, but read on for more about that.)
182
183 This example code, from the ``ParserElement.set_name`` docstring, will
184 actually output a long traceback, followed by an exception of type
185 ``pyparsing.exceptions.ParseException``. But because the ignore-detail
186 option is enabled, the doctest will pass with this abbreviated form:
187
188 .. code-block:: rst
189
190 .. doctest::
191
192 >>> integer = Word(nums)
193 >>> integer.parse_string("ABC")
194 Traceback (most recent call last):
195 ParseException: Expected W:(0-9) (at char 0), (line:1, col:1)
196
197 Relaxing doctest output validation
198 ==================================
199
200 For even more flexibility in demonstrating output, the ``ELLIPSIS``
201 option (enabled by default) means that parts of the output can be
202 replaced with an ellipsis (three periods, ``...``) which will validate
203 against any output.
204
205 This is an extremely useful tool when the exact output of the code is
206 unpredictable (for example, when messages include line and column
207 numbers, or variable data like the current date or a directory path).
208 The code above could also be written like this, and it would still
209 pass the doctest:
210
211 .. code-block:: rst
212
213 .. doctest::
214
215 >>> integer = Word(nums)
216 >>> integer.parse_string("ABC")
217 Traceback (most recent call last):
218 ParseException: Expected W:(0-9) ...
219
220 While this is necessary in some situations, it shouldn't be overused.
221 The more precisely a doctest validates the output of its example,
222 the more useful it is, so think twice before employing an ellipsis in
223 doctest output.
224
225 Normalizing whitespace checks
226 =============================
227
228 Another method of relaxing doctest checks that doesn't impact the
229 test's ability to validate output is the ``NORMALIZE_WHITESPACE``
230 option. This option isn't enabled by default, but can be turned on
231 for any doctest block with a directive argument:
232
233 .. code-block:: rst
234
235 .. testoutput::
236 :options: +NORMALIZE_WHITESPACE
237
238 (Note the preceding ``+`` sign, which adds the option to the default
239 set instead of replacing the default options.)
240
241 With normalization activated, any combination of spaces, tabs, and
242 newlines will compare equal to any other combination.
243
244 One advantage this has is permitting long messages to be wrapped
245 over several lines in the example output. In this example from the
246 ``Keyword`` class docstring, the exception message at the end would
247 normally be printed as one long line. To make the example readable
248 without excessive horizontal scrolling, ``NORMALIZE_WHITESPACE``
249 allows the example output to be broken into multiple lines:
250
251 .. code-block:: rst
252
253 .. doctest::
254 :options: +NORMALIZE_WHITESPACE
255
256 >>> Keyword("start").parse_string("start")
257 ParseResults(['start'], {})
258 >>> Keyword("start").parse_string("starting")
259 Traceback (most recent call last):
260 ParseException: Expected Keyword 'start',
261 keyword was immediately followed by keyword character,
262 found 'ing' (at char 5), (line:1, col:6)
263
264 Doctests in the Pyparsing codebase
265 ==================================
266
267 While the preceding is generally applicable to doctests in any
268 codebase, there are some issues specific to Pyparsing doctests that
269 you should be aware of.
270
271 ``run_tests()`` output
272 ----------------------
273
274 There is one scenario in the pyparsing documentation where the
275 ``NORMALIZE_WHITESPACE`` option *must* be used.
276
277 When the example code uses the ``ParserElement.run_tests()`` method,
278 the output will consist of test strings and matches potentially
279 separated by two blank lines each. (Unless each test is preceded
280 by a comment, then there will be only one blank line.)
281
282 Since ReStructuredText will collapse multiple blank lines in embedded
283 code, the only way to get the ``run_tests`` output to validate against
284 the example is to enable ``NORMALIZE_WHITESPACE`` and collapse the
285 multiple blank lines in the expected output, as well.
286
287 Also, "any whitespace compares equal" doesn't mean that *no*
288 whitespace will be accepted, so the beginning of the ``testoutput``
289 block MUST include an extra blank line at the start, in order
290 to match the leading 2 (or 1) blank lines in the output.
291
292 So, a valid ``run_tests`` output block consists of the ``testoutput``
293 directive, the ``:options: +NORMALIZE_WHITESPACE`` argument, then
294 **TWO blank lines** followed by the output to be verified. This
295 example, from the ``ParserElement.run_tests`` docstring itself,
296 demonstrates the required format:
297
298 .. code-block:: rst
299 :linenos:
300 :emphasize-lines: 17,21,22,26,27
301
302 Failing example:
303
304 .. testcode::
305
306 number_expr = pyparsing_common.number.copy()
307 result = number_expr.run_tests('''
308 100Z
309 3.14.159
310 ''', failure_tests=True)
311 print("Success" if result[0] else "Failed!")
312
313 prints:
314
315 .. testoutput::
316 :options: +NORMALIZE_WHITESPACE
317
318
319 100Z
320 100Z
321 ^
322 ParseException: Expected end of text, found 'Z' ...
323
324 3.14.159
325 3.14.159
326 ^
327 ParseException: Expected end of text, found '.' ...
328 FAIL: Expected end of text, found '.' ...
329 Success
330
331 Note in particular:
332
333 - The extra blank line (line 17) before the first line of output, which
334 is required to match the *two* blank lines in the actual output.
335
336 - Only one blank line (line 22) separating the two tests' output.
337 The real output will again contain two blank lines.
338
339 - The use of ellipses to abbreviate the expected output (lines 21, 26, 27).
340
341 - Exception messages mixed with normal output.
342
343 In this case that presents no problems, because ``run_tests()`` catches
344 any exceptions generated and prints their messages as normal output.
345 Doctest has no restrictions on normal output, only when the exception
346 is raised and a traceback is triggered.
347
348 By the same token, ``IGNORE_EXCEPTION_DETAIL`` is not applicable here
349 (there are no exceptions in the expected string, only regular output),
350 so the normal string-matching rules apply when comparing expected output
351 to actual output.
352
353 Two final notes about failing doctests
354 --------------------------------------
355
356 There are two things to watch out for, when attempting to address
357 doctest failures during a ``make doctest`` run.
358
359 Code location references are not useful
360 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
361
362 Due to the uncommon structure of the pyparsing namespace (with the
363 symbols from all of the package's files imported into the top-level
364 ``pyparsing`` namespace, and documented there rather than at their
365 "home" locations where they're defined), the doctest output for
366 failing test will not display the correct source location for the
367 code. Every failing test will be preceded by a reference similar to:
368
369 .. code-block::
370
371 File "../pyparsing/core.py", line ?, in default
372
373 However, this will be followed by a listing of the code that
374 produced the failing test. So as long as we write examples
375 which are not too generic and are sufficiently distinct from
376 each other (which is good practice anyway), it should be easy
377 enough to find the failing code.
378
379 Diffs on failing tests will include *ALL* differences
380 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
381
382 When ``doctest`` displays the NDIFF-format differences between the
383 expected output and the actual output, it will indicate **EVERY**
384 difference between them — even the differences that would otherwise
385 be ignored. The ``IGNORE_TRACEBACK_DETAILS``, ``ELLIPSIS``, and
386 ``NORMALIZE_WHITESPACE`` options do not apply when NDIFF is generating
387 the comparison ouput for a failed test.
388
389 What this means is that, even though the NDIFF flags an ellipsized
390 section of text as a difference from the actual output, or marks a
391 difference where an output line has been split into two when the
392 ``NORMALIZE_WHITESPACE`` option is enabled, those differences WILL be
393 ignored when the doctest is in a passing state. It's important to
394 focus on the differences that *wouldn't* otherwise be ignored, and
395 just trust that correcting those differences will result in a passing
396 test.
397
398 For example, consider this failing test:
399
400 .. code-block:: shell-session
401 :linenos:
402 :emphasize-lines: 20-23
403
404 $ make doctest
405 ...
406 File "../pyparsing/core.py", line ?, in default
407 Failed example:
408 data_word = Word(alphas)
409 label = data_word + FollowedBy(':')
410
411 attr_expr = (
412 label + Suppress(':')
413 + OneOrMore(data_word, stop_on=label
414 ).set_parse_action(' '.join))
415
416 print(attr_expr.parse_string("color: RED"))
417
418 text = "shape: SQUARE posn: upper left color: light blue texture: burlap"
419
420 # print attributes as plain groups
421 print(attr_expr[1, ...].parse_string(text))
422 Differences (ndiff with -expected +actual):
423 - ['color', "RED"]
424 ? ^ ^
425 + ['color', 'RED']
426 ? ^ ^
427 + ['shape', 'SQUARE', 'posn', 'upper left', 'color', 'light blue', 'texture', 'burlap']
428 - ['shape', 'SQUARE',
429 - ... 'texture', 'burlap']
430 **********************************************************************
431 1 item had failures:
432 1 of 208 in default
433 208 tests in 1 item.
434 207 passed and 1 failed.
435 ***Test Failed*** 1 failure.
436
437 The **only** significant difference is the highlighted one: The wrong
438 quotes used around the word ``"RED"`` in the expected output. Once that's
439 changed to ``'RED'``, the doctest will pass. The remaining diff line(s),
440 where the expected output uses an ellipsis and is split over two lines
441 (with ``NORMALIZE_WHITESPACE`` enabled), will not fail despite being
442 shown as differing from the actual output. (Technically it *does* differ,
443 after all. The configuration simply ignores that difference.)