| 1 |
======================== |
| 2 |
Writing doctest examples |
| 3 |
======================== |
| 4 |
|
| 5 |
Doctest support is provided in Sphinx by the extension |
| 6 |
`sphinx.ext.doctest`_, and its documentation is one |
| 7 |
useful resurce for working with the pyparsing doctests. |
| 8 |
|
| 9 |
.. _sphinx.ext.doctest: https://www.sphinx-doc.org/en/master/usage/extensions/doctest.html |
| 10 |
|
| 11 |
|
| 12 |
Types of doctests |
| 13 |
================= |
| 14 |
|
| 15 |
There are two basic forms of doctest, and both are used extensively |
| 16 |
in the Pyparsing documentation. Which one to use for a given example |
| 17 |
is a decision that needs to be made when writing it, but there are |
| 18 |
some factors that usually make the correct choice an obvious one. |
| 19 |
|
| 20 |
Doctest type 1: ``testcode`` / ``testoutput`` blocks |
| 21 |
---------------------------------------------------- |
| 22 |
|
| 23 |
The first form involves one or potentially two separate code blocks. |
| 24 |
The ``testcode`` block contains all of the input code in the form of |
| 25 |
a standard Python script. This can optionally be paired with a |
| 26 |
second ``testoutput`` block, which if present will contain the output |
| 27 |
for the preceding ``testcode`` block. |
| 28 |
|
| 29 |
An example of a ``testcode`` / ``testoutput`` pair, from the docstring |
| 30 |
for ``ParserElement.__add__``: |
| 31 |
|
| 32 |
|
| 33 |
.. code-block:: rst |
| 34 |
|
| 35 |
Example: |
| 36 |
|
| 37 |
.. testcode:: |
| 38 |
|
| 39 |
greet = Word(alphas) + "," + Word(alphas) + "!" |
| 40 |
hello = "Hello, World!" |
| 41 |
print(hello, "->", greet.parse_string(hello)) |
| 42 |
|
| 43 |
prints: |
| 44 |
|
| 45 |
.. testoutput:: |
| 46 |
|
| 47 |
Hello, World! -> ['Hello', ',', 'World', '!'] |
| 48 |
|
| 49 |
Examples written like this will be formatted in the rendered HTML/Latex/etc. |
| 50 |
documentation **exactly** as if they'd been written as normal code blocks. |
| 51 |
There is no visible difference between the code above and this code without |
| 52 |
doctest support: |
| 53 |
|
| 54 |
.. code-block:: rst |
| 55 |
|
| 56 |
Example:: |
| 57 |
|
| 58 |
greet = Word(alphas) + "," + Word(alphas) + "!" |
| 59 |
hello = "Hello, World!" |
| 60 |
print(hello, "->", greet.parse_string(hello)) |
| 61 |
|
| 62 |
prints:: |
| 63 |
|
| 64 |
Hello, World! -> ['Hello', ',', 'World', '!'] |
| 65 |
|
| 66 |
However, the advantage to writing doctests is that when ``make doctest`` |
| 67 |
is run from the ``docs/`` directory, the doctest extension will execute |
| 68 |
each ``testcode`` block, and verify that its output exactly matches the |
| 69 |
``testoutput`` block (if present). |
| 70 |
|
| 71 |
Any deviations will be displayed in "ndiff" format. This enhancement |
| 72 |
to the standard unified diff will (sometimes) indicate where in each |
| 73 |
line the differences occur. (The character-difference highlighting is |
| 74 |
frustratingly inconsistent. But at worst ndiff is equivalent to unified |
| 75 |
diff, so it's still worth using.) |
| 76 |
|
| 77 |
Testing examples with doctest allows the code used to demonstrate the |
| 78 |
pyparsing API to be verified against the *actual* API as it's currently |
| 79 |
implemented, and ensures that examples stay current and relevant. |
| 80 |
|
| 81 |
Not all ``testcode`` blocks need a corresponding ``testoutput`` — if a |
| 82 |
``testcode`` block is included on its own, the code inside the block will |
| 83 |
still be executed, but its output won't be verified. This can be useful |
| 84 |
when displaying code that doesn't require demonstration of its output |
| 85 |
(or doesn't output anything), as the extension will still verify that |
| 86 |
the code can be run without error. |
| 87 |
|
| 88 |
It's also possible to include a *hidden* ``testoutput`` block, which will |
| 89 |
beverified against the preceding ``testcode`` but won't be displayed in the |
| 90 |
documentation. To hide a ``testoutput`` block (or a ``testcode`` block, |
| 91 |
for that matter), add the ``:hide:`` option as an argument to the |
| 92 |
directive, i.e.: |
| 93 |
|
| 94 |
.. code-block:: rst |
| 95 |
|
| 96 |
.. testoutput:: |
| 97 |
:hide: |
| 98 |
|
| 99 |
"""Output that won't be shown, but will be verified against the |
| 100 |
preceding testcode block.""" |
| 101 |
|
| 102 |
Doctest type 2: ``doctest`` interactive blocks |
| 103 |
---------------------------------------------- |
| 104 |
|
| 105 |
The second type of doctest is a ``doctest`` block, which takes the form of |
| 106 |
an interactive Python REPL session in standard format (using ``>>>`` and |
| 107 |
``...`` markers for input lines). |
| 108 |
|
| 109 |
With these tests, output is interleaved with the code, which can be much |
| 110 |
easier to follow when there are multiple lines producing output. If an |
| 111 |
example would contains multiple ``print()`` calls, rather than first |
| 112 |
displaying all of the code in a ``testcode`` block, then all of the |
| 113 |
output in a ``testoutput`` block, consider using a ``doctest`` session |
| 114 |
so that the reader can follow along each step as it occurs. |
| 115 |
|
| 116 |
A typical ``doctest`` example can be found in the ``ParserElement.ignore`` |
| 117 |
docstring: |
| 118 |
|
| 119 |
.. code-block:: rst |
| 120 |
|
| 121 |
Example: |
| 122 |
|
| 123 |
.. doctest:: |
| 124 |
|
| 125 |
>>> patt = Word(alphas)[...] |
| 126 |
>>> print(patt.parse_string('ablaj /* comment */ lskjd')) |
| 127 |
['ablaj'] |
| 128 |
|
| 129 |
>>> patt = Word(alphas)[...].ignore(c_style_comment) |
| 130 |
>>> print(patt.parse_string('ablaj /* comment */ lskjd')) |
| 131 |
['ablaj', 'lskjd'] |
| 132 |
|
| 133 |
|
| 134 |
Setup code for doctest blocks |
| 135 |
============================= |
| 136 |
|
| 137 |
The doctest extension is configured with extensive setup code |
| 138 |
which is run before each test block. It can be viewed in the |
| 139 |
:download:`docs/conf.py <../docs/conf.py>` file — look for the |
| 140 |
``doctest_global_setup`` variable near the end of the file. |
| 141 |
|
| 142 |
The setup code is intended to make any useful symbols available |
| 143 |
to the tests without them having to be included in each and every |
| 144 |
doctest block. If additional modules are needed, feel free to add |
| 145 |
them to the global setup. When writing doctests, Pyparsing classes |
| 146 |
can be invoked directly, or as members of the ``pp`` alias namespace. |
| 147 |
Either way, the definition of those symbols can be assumed without |
| 148 |
explicitly importing/defining them. |
| 149 |
|
| 150 |
When using symbols from other aliased namespaces, however, it's a |
| 151 |
good idea to establish the alias for the reader at the start of the |
| 152 |
example code. Even though these are both defined in the global setup, |
| 153 |
showing the establishing lines before referencing ``ppc`` or ``ppu`` |
| 154 |
in an example makes that example clearer: |
| 155 |
|
| 156 |
.. code-block: py |
| 157 |
|
| 158 |
ppc = pp.pyparsing_common |
| 159 |
ppu = pp.pyparsing_unicode |
| 160 |
|
| 161 |
However, because those symbols *are* provided by default, they don't |
| 162 |
need to be explicitly established for **every** example. Feel free |
| 163 |
to omit them after the first use, when writing multiple examples for |
| 164 |
a given class or function. |
| 165 |
|
| 166 |
Documenting exceptions |
| 167 |
====================== |
| 168 |
|
| 169 |
Code that will trigger an exception can be both demonstrated and |
| 170 |
verified using doctests (of either type), although when a ``testoutput`` |
| 171 |
block will demonstrate an exception it should be the only output in |
| 172 |
that block — doctest does not support mixing regular output and |
| 173 |
exceptions. |
| 174 |
|
| 175 |
Both the ``IGNORE_EXCEPTION_DETAIL`` and ``ELLIPSIS`` doctest options |
| 176 |
are enabled by default, which make demonstrating exceptions far more |
| 177 |
convenient. Ignoring exception detail means that the full traceback |
| 178 |
for an exception can be omitted, as well as the fully-qualified name |
| 179 |
of the exception class. As long as the ``Traceback...`` line and the |
| 180 |
exception class name match, the doctest will pass. (The exception |
| 181 |
message is also verified by default, but read on for more about that.) |
| 182 |
|
| 183 |
This example code, from the ``ParserElement.set_name`` docstring, will |
| 184 |
actually output a long traceback, followed by an exception of type |
| 185 |
``pyparsing.exceptions.ParseException``. But because the ignore-detail |
| 186 |
option is enabled, the doctest will pass with this abbreviated form: |
| 187 |
|
| 188 |
.. code-block:: rst |
| 189 |
|
| 190 |
.. doctest:: |
| 191 |
|
| 192 |
>>> integer = Word(nums) |
| 193 |
>>> integer.parse_string("ABC") |
| 194 |
Traceback (most recent call last): |
| 195 |
ParseException: Expected W:(0-9) (at char 0), (line:1, col:1) |
| 196 |
|
| 197 |
Relaxing doctest output validation |
| 198 |
================================== |
| 199 |
|
| 200 |
For even more flexibility in demonstrating output, the ``ELLIPSIS`` |
| 201 |
option (enabled by default) means that parts of the output can be |
| 202 |
replaced with an ellipsis (three periods, ``...``) which will validate |
| 203 |
against any output. |
| 204 |
|
| 205 |
This is an extremely useful tool when the exact output of the code is |
| 206 |
unpredictable (for example, when messages include line and column |
| 207 |
numbers, or variable data like the current date or a directory path). |
| 208 |
The code above could also be written like this, and it would still |
| 209 |
pass the doctest: |
| 210 |
|
| 211 |
.. code-block:: rst |
| 212 |
|
| 213 |
.. doctest:: |
| 214 |
|
| 215 |
>>> integer = Word(nums) |
| 216 |
>>> integer.parse_string("ABC") |
| 217 |
Traceback (most recent call last): |
| 218 |
ParseException: Expected W:(0-9) ... |
| 219 |
|
| 220 |
While this is necessary in some situations, it shouldn't be overused. |
| 221 |
The more precisely a doctest validates the output of its example, |
| 222 |
the more useful it is, so think twice before employing an ellipsis in |
| 223 |
doctest output. |
| 224 |
|
| 225 |
Normalizing whitespace checks |
| 226 |
============================= |
| 227 |
|
| 228 |
Another method of relaxing doctest checks that doesn't impact the |
| 229 |
test's ability to validate output is the ``NORMALIZE_WHITESPACE`` |
| 230 |
option. This option isn't enabled by default, but can be turned on |
| 231 |
for any doctest block with a directive argument: |
| 232 |
|
| 233 |
.. code-block:: rst |
| 234 |
|
| 235 |
.. testoutput:: |
| 236 |
:options: +NORMALIZE_WHITESPACE |
| 237 |
|
| 238 |
(Note the preceding ``+`` sign, which adds the option to the default |
| 239 |
set instead of replacing the default options.) |
| 240 |
|
| 241 |
With normalization activated, any combination of spaces, tabs, and |
| 242 |
newlines will compare equal to any other combination. |
| 243 |
|
| 244 |
One advantage this has is permitting long messages to be wrapped |
| 245 |
over several lines in the example output. In this example from the |
| 246 |
``Keyword`` class docstring, the exception message at the end would |
| 247 |
normally be printed as one long line. To make the example readable |
| 248 |
without excessive horizontal scrolling, ``NORMALIZE_WHITESPACE`` |
| 249 |
allows the example output to be broken into multiple lines: |
| 250 |
|
| 251 |
.. code-block:: rst |
| 252 |
|
| 253 |
.. doctest:: |
| 254 |
:options: +NORMALIZE_WHITESPACE |
| 255 |
|
| 256 |
>>> Keyword("start").parse_string("start") |
| 257 |
ParseResults(['start'], {}) |
| 258 |
>>> Keyword("start").parse_string("starting") |
| 259 |
Traceback (most recent call last): |
| 260 |
ParseException: Expected Keyword 'start', |
| 261 |
keyword was immediately followed by keyword character, |
| 262 |
found 'ing' (at char 5), (line:1, col:6) |
| 263 |
|
| 264 |
Doctests in the Pyparsing codebase |
| 265 |
================================== |
| 266 |
|
| 267 |
While the preceding is generally applicable to doctests in any |
| 268 |
codebase, there are some issues specific to Pyparsing doctests that |
| 269 |
you should be aware of. |
| 270 |
|
| 271 |
``run_tests()`` output |
| 272 |
---------------------- |
| 273 |
|
| 274 |
There is one scenario in the pyparsing documentation where the |
| 275 |
``NORMALIZE_WHITESPACE`` option *must* be used. |
| 276 |
|
| 277 |
When the example code uses the ``ParserElement.run_tests()`` method, |
| 278 |
the output will consist of test strings and matches potentially |
| 279 |
separated by two blank lines each. (Unless each test is preceded |
| 280 |
by a comment, then there will be only one blank line.) |
| 281 |
|
| 282 |
Since ReStructuredText will collapse multiple blank lines in embedded |
| 283 |
code, the only way to get the ``run_tests`` output to validate against |
| 284 |
the example is to enable ``NORMALIZE_WHITESPACE`` and collapse the |
| 285 |
multiple blank lines in the expected output, as well. |
| 286 |
|
| 287 |
Also, "any whitespace compares equal" doesn't mean that *no* |
| 288 |
whitespace will be accepted, so the beginning of the ``testoutput`` |
| 289 |
block MUST include an extra blank line at the start, in order |
| 290 |
to match the leading 2 (or 1) blank lines in the output. |
| 291 |
|
| 292 |
So, a valid ``run_tests`` output block consists of the ``testoutput`` |
| 293 |
directive, the ``:options: +NORMALIZE_WHITESPACE`` argument, then |
| 294 |
**TWO blank lines** followed by the output to be verified. This |
| 295 |
example, from the ``ParserElement.run_tests`` docstring itself, |
| 296 |
demonstrates the required format: |
| 297 |
|
| 298 |
.. code-block:: rst |
| 299 |
:linenos: |
| 300 |
:emphasize-lines: 17,21,22,26,27 |
| 301 |
|
| 302 |
Failing example: |
| 303 |
|
| 304 |
.. testcode:: |
| 305 |
|
| 306 |
number_expr = pyparsing_common.number.copy() |
| 307 |
result = number_expr.run_tests(''' |
| 308 |
100Z |
| 309 |
3.14.159 |
| 310 |
''', failure_tests=True) |
| 311 |
print("Success" if result[0] else "Failed!") |
| 312 |
|
| 313 |
prints: |
| 314 |
|
| 315 |
.. testoutput:: |
| 316 |
:options: +NORMALIZE_WHITESPACE |
| 317 |
|
| 318 |
|
| 319 |
100Z |
| 320 |
100Z |
| 321 |
^ |
| 322 |
ParseException: Expected end of text, found 'Z' ... |
| 323 |
|
| 324 |
3.14.159 |
| 325 |
3.14.159 |
| 326 |
^ |
| 327 |
ParseException: Expected end of text, found '.' ... |
| 328 |
FAIL: Expected end of text, found '.' ... |
| 329 |
Success |
| 330 |
|
| 331 |
Note in particular: |
| 332 |
|
| 333 |
- The extra blank line (line 17) before the first line of output, which |
| 334 |
is required to match the *two* blank lines in the actual output. |
| 335 |
|
| 336 |
- Only one blank line (line 22) separating the two tests' output. |
| 337 |
The real output will again contain two blank lines. |
| 338 |
|
| 339 |
- The use of ellipses to abbreviate the expected output (lines 21, 26, 27). |
| 340 |
|
| 341 |
- Exception messages mixed with normal output. |
| 342 |
|
| 343 |
In this case that presents no problems, because ``run_tests()`` catches |
| 344 |
any exceptions generated and prints their messages as normal output. |
| 345 |
Doctest has no restrictions on normal output, only when the exception |
| 346 |
is raised and a traceback is triggered. |
| 347 |
|
| 348 |
By the same token, ``IGNORE_EXCEPTION_DETAIL`` is not applicable here |
| 349 |
(there are no exceptions in the expected string, only regular output), |
| 350 |
so the normal string-matching rules apply when comparing expected output |
| 351 |
to actual output. |
| 352 |
|
| 353 |
Two final notes about failing doctests |
| 354 |
-------------------------------------- |
| 355 |
|
| 356 |
There are two things to watch out for, when attempting to address |
| 357 |
doctest failures during a ``make doctest`` run. |
| 358 |
|
| 359 |
Code location references are not useful |
| 360 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 361 |
|
| 362 |
Due to the uncommon structure of the pyparsing namespace (with the |
| 363 |
symbols from all of the package's files imported into the top-level |
| 364 |
``pyparsing`` namespace, and documented there rather than at their |
| 365 |
"home" locations where they're defined), the doctest output for |
| 366 |
failing test will not display the correct source location for the |
| 367 |
code. Every failing test will be preceded by a reference similar to: |
| 368 |
|
| 369 |
.. code-block:: |
| 370 |
|
| 371 |
File "../pyparsing/core.py", line ?, in default |
| 372 |
|
| 373 |
However, this will be followed by a listing of the code that |
| 374 |
produced the failing test. So as long as we write examples |
| 375 |
which are not too generic and are sufficiently distinct from |
| 376 |
each other (which is good practice anyway), it should be easy |
| 377 |
enough to find the failing code. |
| 378 |
|
| 379 |
Diffs on failing tests will include *ALL* differences |
| 380 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 381 |
|
| 382 |
When ``doctest`` displays the NDIFF-format differences between the |
| 383 |
expected output and the actual output, it will indicate **EVERY** |
| 384 |
difference between them — even the differences that would otherwise |
| 385 |
be ignored. The ``IGNORE_TRACEBACK_DETAILS``, ``ELLIPSIS``, and |
| 386 |
``NORMALIZE_WHITESPACE`` options do not apply when NDIFF is generating |
| 387 |
the comparison ouput for a failed test. |
| 388 |
|
| 389 |
What this means is that, even though the NDIFF flags an ellipsized |
| 390 |
section of text as a difference from the actual output, or marks a |
| 391 |
difference where an output line has been split into two when the |
| 392 |
``NORMALIZE_WHITESPACE`` option is enabled, those differences WILL be |
| 393 |
ignored when the doctest is in a passing state. It's important to |
| 394 |
focus on the differences that *wouldn't* otherwise be ignored, and |
| 395 |
just trust that correcting those differences will result in a passing |
| 396 |
test. |
| 397 |
|
| 398 |
For example, consider this failing test: |
| 399 |
|
| 400 |
.. code-block:: shell-session |
| 401 |
:linenos: |
| 402 |
:emphasize-lines: 20-23 |
| 403 |
|
| 404 |
$ make doctest |
| 405 |
... |
| 406 |
File "../pyparsing/core.py", line ?, in default |
| 407 |
Failed example: |
| 408 |
data_word = Word(alphas) |
| 409 |
label = data_word + FollowedBy(':') |
| 410 |
|
| 411 |
attr_expr = ( |
| 412 |
label + Suppress(':') |
| 413 |
+ OneOrMore(data_word, stop_on=label |
| 414 |
).set_parse_action(' '.join)) |
| 415 |
|
| 416 |
print(attr_expr.parse_string("color: RED")) |
| 417 |
|
| 418 |
text = "shape: SQUARE posn: upper left color: light blue texture: burlap" |
| 419 |
|
| 420 |
# print attributes as plain groups |
| 421 |
print(attr_expr[1, ...].parse_string(text)) |
| 422 |
Differences (ndiff with -expected +actual): |
| 423 |
- ['color', "RED"] |
| 424 |
? ^ ^ |
| 425 |
+ ['color', 'RED'] |
| 426 |
? ^ ^ |
| 427 |
+ ['shape', 'SQUARE', 'posn', 'upper left', 'color', 'light blue', 'texture', 'burlap'] |
| 428 |
- ['shape', 'SQUARE', |
| 429 |
- ... 'texture', 'burlap'] |
| 430 |
********************************************************************** |
| 431 |
1 item had failures: |
| 432 |
1 of 208 in default |
| 433 |
208 tests in 1 item. |
| 434 |
207 passed and 1 failed. |
| 435 |
***Test Failed*** 1 failure. |
| 436 |
|
| 437 |
The **only** significant difference is the highlighted one: The wrong |
| 438 |
quotes used around the word ``"RED"`` in the expected output. Once that's |
| 439 |
changed to ``'RED'``, the doctest will pass. The remaining diff line(s), |
| 440 |
where the expected output uses an ellipsis and is split over two lines |
| 441 |
(with ``NORMALIZE_WHITESPACE`` enabled), will not fail despite being |
| 442 |
shown as differing from the actual output. (Technically it *does* differ, |
| 443 |
after all. The configuration simply ignores that difference.) |