| 1 |
============================= |
| 2 |
What's New in Pyparsing 3.0.0 |
| 3 |
============================= |
| 4 |
|
| 5 |
:author: Paul McGuire |
| 6 |
|
| 7 |
:date: May, 2022 |
| 8 |
|
| 9 |
:abstract: This document summarizes the changes made |
| 10 |
in the 3.0.0 release of pyparsing. |
| 11 |
(Updated to reflect changes up to 3.0.10) |
| 12 |
|
| 13 |
.. contents:: :depth: 4 |
| 14 |
|
| 15 |
|
| 16 |
New Features |
| 17 |
============ |
| 18 |
|
| 19 |
PEP-8 naming |
| 20 |
------------ |
| 21 |
This release of pyparsing will (finally!) include PEP-8 compatible names and arguments. |
| 22 |
Backward-compatibility is maintained by defining synonyms using the old camelCase names |
| 23 |
pointing to the new snake_case names. |
| 24 |
|
| 25 |
This code written using non-PEP8 names:: |
| 26 |
|
| 27 |
wd = pp.Word(pp.printables, excludeChars="$") |
| 28 |
wd_list = pp.delimitedList(wd, delim="$") |
| 29 |
print(wd_list.parseString("dkls$134lkjk$lsd$$").asList()) |
| 30 |
|
| 31 |
can now be written as:: |
| 32 |
|
| 33 |
wd = pp.Word(pp.printables, exclude_chars="$") |
| 34 |
wd_list = pp.delimited_list(wd, delim="$") |
| 35 |
print(wd_list.parse_string("dkls$134lkjk$lsd$$").as_list()) |
| 36 |
|
| 37 |
Pyparsing 3.0 will run both versions of this example. |
| 38 |
|
| 39 |
New code should be written using the PEP-8 compatible names. The compatibility |
| 40 |
synonyms will be removed in a future version of pyparsing. |
| 41 |
|
| 42 |
|
| 43 |
Railroad diagramming |
| 44 |
-------------------- |
| 45 |
An excellent new enhancement is the new railroad diagram |
| 46 |
generator for documenting pyparsing parsers.:: |
| 47 |
|
| 48 |
import pyparsing as pp |
| 49 |
|
| 50 |
# define a simple grammar for parsing street addresses such |
| 51 |
# as "123 Main Street" |
| 52 |
# number word... |
| 53 |
number = pp.Word(pp.nums).set_name("number") |
| 54 |
name = pp.Word(pp.alphas).set_name("word")[1, ...] |
| 55 |
|
| 56 |
parser = number("house_number") + name("street") |
| 57 |
parser.set_name("street address") |
| 58 |
|
| 59 |
# construct railroad track diagram for this parser and |
| 60 |
# save as HTML |
| 61 |
parser.create_diagram('parser_rr_diag.html') |
| 62 |
|
| 63 |
``create_diagram`` accepts these named arguments: |
| 64 |
|
| 65 |
- ``vertical`` (int) - threshold for formatting multiple alternatives vertically |
| 66 |
instead of horizontally (default=3) |
| 67 |
- ``show_results_names`` - bool flag whether diagram should show annotations for |
| 68 |
defined results names |
| 69 |
- ``show_groups`` - bool flag whether groups should be highlighted with an unlabeled surrounding box |
| 70 |
- ``embed`` - bool flag whether generated HTML should omit ``<HEAD>``, ``<BODY>``, and ``<DOCTYPE>`` tags to embed |
| 71 |
the resulting HTML in an enclosing HTML source (new in 3.0.10) |
| 72 |
- ``head`` - str containing additional HTML to insert into the ``<HEAD>`` section of the |
| 73 |
generated code; can be used to insert custom CSS styling |
| 74 |
- ``body`` - str containing additional HTML to insert at the beginning of the ``<BODY>`` section of the |
| 75 |
generated code |
| 76 |
|
| 77 |
To use this new feature, install the supporting diagramming packages using:: |
| 78 |
|
| 79 |
pip install pyparsing[diagrams] |
| 80 |
|
| 81 |
See more in the examples directory: ``make_diagram.py`` and ``railroad_diagram_demo.py``. |
| 82 |
|
| 83 |
(Railroad diagram enhancement contributed by Michael Milton) |
| 84 |
|
| 85 |
Support for left-recursive parsers |
| 86 |
---------------------------------- |
| 87 |
Another significant enhancement in 3.0 is support for left-recursive (LR) |
| 88 |
parsers. Previously, given a left-recursive parser, pyparsing would |
| 89 |
recurse repeatedly until hitting the Python recursion limit. Following |
| 90 |
the methods of the Python PEG parser, pyparsing uses a variation of |
| 91 |
packrat parsing to detect and handle left-recursion during parsing.:: |
| 92 |
|
| 93 |
import pyparsing as pp |
| 94 |
pp.ParserElement.enable_left_recursion() |
| 95 |
|
| 96 |
# a common left-recursion definition |
| 97 |
# define a list of items as 'list + item | item' |
| 98 |
# BNF: |
| 99 |
# item_list := item_list item | item |
| 100 |
# item := word of alphas |
| 101 |
item_list = pp.Forward() |
| 102 |
item = pp.Word(pp.alphas) |
| 103 |
item_list <<= item_list + item | item |
| 104 |
|
| 105 |
item_list.run_tests("""\ |
| 106 |
To parse or not to parse that is the question |
| 107 |
""") |
| 108 |
|
| 109 |
Prints:: |
| 110 |
|
| 111 |
['To', 'parse', 'or', 'not', 'to', 'parse', 'that', 'is', 'the', 'question'] |
| 112 |
|
| 113 |
See more examples in ``left_recursion.py`` in the pyparsing examples directory. |
| 114 |
|
| 115 |
(LR parsing support contributed by Max Fischer) |
| 116 |
|
| 117 |
Packrat/memoization enable and disable methods |
| 118 |
---------------------------------------------- |
| 119 |
As part of the implementation of left-recursion support, new methods have been added |
| 120 |
to enable and disable packrat parsing. |
| 121 |
|
| 122 |
====================== ======================================================= |
| 123 |
Name Description |
| 124 |
---------------------- ------------------------------------------------------- |
| 125 |
enable_packrat Enable packrat parsing (with specified cache size) |
| 126 |
enable_left_recursion Enable left-recursion cache |
| 127 |
disable_memoization Disable all internal parsing caches |
| 128 |
====================== ======================================================= |
| 129 |
|
| 130 |
Type annotations on all public methods |
| 131 |
-------------------------------------- |
| 132 |
Python 3.6 and upward compatible type annotations have been added to most of the |
| 133 |
public methods in pyparsing. This should facilitate developing pyparsing-based |
| 134 |
applications using IDEs for development-time type checking. |
| 135 |
|
| 136 |
New string constants ``identchars`` and ``identbodychars`` to help in defining identifier Word expressions |
| 137 |
---------------------------------------------------------------------------------------------------------- |
| 138 |
Two new module-level strings have been added to help when defining identifiers, |
| 139 |
``identchars`` and ``identbodychars``. |
| 140 |
|
| 141 |
Instead of writing:: |
| 142 |
|
| 143 |
import pyparsing as pp |
| 144 |
identifier = pp.Word(pp.alphas + "_", pp.alphanums + "_") |
| 145 |
|
| 146 |
you will be able to write:: |
| 147 |
|
| 148 |
identifier = pp.Word(pp.identchars, pp.identbodychars) |
| 149 |
|
| 150 |
Those constants have also been added to all the Unicode string classes:: |
| 151 |
|
| 152 |
import pyparsing as pp |
| 153 |
ppu = pp.pyparsing_unicode |
| 154 |
|
| 155 |
cjk_identifier = pp.Word(ppu.CJK.identchars, ppu.CJK.identbodychars) |
| 156 |
greek_identifier = pp.Word(ppu.Greek.identchars, ppu.Greek.identbodychars) |
| 157 |
|
| 158 |
|
| 159 |
Refactored/added diagnostic flags |
| 160 |
--------------------------------- |
| 161 |
Expanded ``__diag__`` and ``__compat__`` to actual classes instead of |
| 162 |
just namespaces, to add some helpful behavior: |
| 163 |
|
| 164 |
- ``pyparsing.enable_diag()`` and ``pyparsing.disable_diag()`` methods to give extra |
| 165 |
help when setting or clearing flags (detects invalid |
| 166 |
flag names, detects when trying to set a ``__compat__`` flag |
| 167 |
that is no longer settable). Use these methods now to |
| 168 |
set or clear flags, instead of directly setting to ``True`` or |
| 169 |
``False``:: |
| 170 |
|
| 171 |
import pyparsing as pp |
| 172 |
pp.enable_diag(pp.Diagnostics.warn_multiple_tokens_in_named_alternation) |
| 173 |
|
| 174 |
- ``pyparsing.enable_all_warnings()`` is another helper that sets |
| 175 |
all "warn*" diagnostics to ``True``:: |
| 176 |
|
| 177 |
pp.enable_all_warnings() |
| 178 |
|
| 179 |
- added support for calling ``enable_all_warnings()`` if warnings are enabled |
| 180 |
using the Python ``-W`` switch, or setting a non-empty value to the environment |
| 181 |
variable ``PYPARSINGENABLEALLWARNINGS``. (If using ``-Wd`` for testing, but |
| 182 |
wishing to disable pyparsing warnings, add ``-Wi:::pyparsing``.) |
| 183 |
|
| 184 |
- added new warning, ``warn_on_match_first_with_lshift_operator`` to |
| 185 |
warn when using ``'<<'`` with a ``'|'`` ``MatchFirst`` operator, |
| 186 |
which will |
| 187 |
create an unintended expression due to precedence of operations. |
| 188 |
|
| 189 |
Example: This statement will erroneously define the ``fwd`` expression |
| 190 |
as just ``expr_a``, even though ``expr_a | expr_b`` was intended, |
| 191 |
since ``'<<'`` operator has precedence over ``'|'``:: |
| 192 |
|
| 193 |
fwd << expr_a | expr_b |
| 194 |
|
| 195 |
To correct this, use the ``'<<='`` operator (preferred) or parentheses |
| 196 |
to override operator precedence:: |
| 197 |
|
| 198 |
fwd <<= expr_a | expr_b |
| 199 |
|
| 200 |
or:: |
| 201 |
|
| 202 |
fwd << (expr_a | expr_b) |
| 203 |
|
| 204 |
- ``warn_on_parse_using_empty_Forward`` - warns that a ``Forward`` |
| 205 |
has been included in a grammar, but no expression was |
| 206 |
attached to it using ``'<<='`` or ``'<<'`` |
| 207 |
|
| 208 |
- ``warn_on_assignment_to_Forward`` - warns that a ``Forward`` has |
| 209 |
been created, but was probably later overwritten by |
| 210 |
erroneously using ``'='`` instead of ``'<<='`` (this is a common |
| 211 |
mistake when using Forwards) |
| 212 |
(**currently not working on PyPy**) |
| 213 |
|
| 214 |
Support for yielding native Python ``list`` and ``dict`` types in place of ``ParseResults`` |
| 215 |
------------------------------------------------------------------------------------------- |
| 216 |
To support parsers that are intended to generate native Python collection |
| 217 |
types such as lists and dicts, the ``Group`` and ``Dict`` classes now accept an |
| 218 |
additional boolean keyword argument ``aslist`` and ``asdict`` respectively. See |
| 219 |
the ``jsonParser.py`` example in the ``pyparsing/examples`` source directory for |
| 220 |
how to return types as ``ParseResults`` and as Python collection types, and the |
| 221 |
distinctions in working with the different types. |
| 222 |
|
| 223 |
In addition parse actions that must return a value of list type (which would |
| 224 |
normally be converted internally to a ``ParseResults``) can override this default |
| 225 |
behavior by returning their list wrapped in the new ``ParseResults.List`` class:: |
| 226 |
|
| 227 |
# this parse action tries to return a list, but pyparsing |
| 228 |
# will convert to a ParseResults |
| 229 |
def return_as_list_but_still_get_parse_results(tokens): |
| 230 |
return tokens.asList() |
| 231 |
|
| 232 |
# this parse action returns the tokens as a list, and pyparsing will |
| 233 |
# maintain its list type in the final parsing results |
| 234 |
def return_as_list(tokens): |
| 235 |
return ParseResults.List(tokens.asList()) |
| 236 |
|
| 237 |
This is the mechanism used internally by the ``Group`` class when defined |
| 238 |
using ``aslist=True``. |
| 239 |
|
| 240 |
New Located class to replace ``locatedExpr`` helper method |
| 241 |
---------------------------------------------------------- |
| 242 |
The new ``Located`` class will replace the current ``locatedExpr`` method for |
| 243 |
marking parsed results with the start and end locations of the parsed data in |
| 244 |
the input string. ``locatedExpr`` had several bugs, and returned its results |
| 245 |
in a hard-to-use format (location data and results names were mixed in with |
| 246 |
the located expression's parsed results, and wrapped in an unnecessary extra |
| 247 |
nesting level). |
| 248 |
|
| 249 |
For this code:: |
| 250 |
|
| 251 |
wd = Word(alphas) |
| 252 |
for match in locatedExpr(wd).search_string("ljsdf123lksdjjf123lkkjj1222"): |
| 253 |
print(match) |
| 254 |
|
| 255 |
the docs for ``locatedExpr`` show this output:: |
| 256 |
|
| 257 |
[[0, 'ljsdf', 5]] |
| 258 |
[[8, 'lksdjjf', 15]] |
| 259 |
[[18, 'lkkjj', 23]] |
| 260 |
|
| 261 |
The parsed values and the start and end locations are merged into a single |
| 262 |
nested ``ParseResults`` (and any results names in the parsed values are also |
| 263 |
merged in with the start and end location names). |
| 264 |
|
| 265 |
Using ``Located``, the output is:: |
| 266 |
|
| 267 |
[0, ['ljsdf'], 5] |
| 268 |
[8, ['lksdjjf'], 15] |
| 269 |
[18, ['lkkjj'], 23] |
| 270 |
|
| 271 |
With ``Located``, the parsed expression values and results names are kept |
| 272 |
separate in the second parsed value, and there is no extra grouping level |
| 273 |
on the whole result. |
| 274 |
|
| 275 |
The existing ``locatedExpr`` is retained for backward-compatibility, but will be |
| 276 |
deprecated in a future release. |
| 277 |
|
| 278 |
New ``AtLineStart`` and ``AtStringStart`` classes |
| 279 |
------------------------------------------------- |
| 280 |
As part of fixing some matching behavior in ``LineStart`` and ``StringStart``, two new |
| 281 |
classes have been added: ``AtLineStart`` and ``AtStringStart``. |
| 282 |
|
| 283 |
``LineStart`` and ``StringStart`` can be treated as separate elements, including whitespace skipping. |
| 284 |
``AtLineStart`` and ``AtStringStart`` enforce that an expression starts exactly at column 1, with no |
| 285 |
leading whitespace.:: |
| 286 |
|
| 287 |
(LineStart() + Word(alphas)).parseString("ABC") # passes |
| 288 |
(LineStart() + Word(alphas)).parseString(" ABC") # passes |
| 289 |
AtLineStart(Word(alphas)).parseString(" ABC") # fails |
| 290 |
|
| 291 |
[This is a fix to behavior that was added in 3.0.0, but was actually a regression from 2.4.x.] |
| 292 |
|
| 293 |
New ``IndentedBlock`` class to replace ``indentedBlock`` helper method |
| 294 |
---------------------------------------------------------------------- |
| 295 |
The new ``IndentedBlock`` class will replace the current ``indentedBlock`` method |
| 296 |
for defining indented blocks of text, similar to Python source code. Using |
| 297 |
``IndentedBlock``, the expression instance itself keeps track of the indent stack, |
| 298 |
so a separate external ``indentStack`` variable is no longer required. |
| 299 |
|
| 300 |
Here is a simple example of an expression containing an alphabetic key, followed |
| 301 |
by an indented list of integers:: |
| 302 |
|
| 303 |
integer = pp.Word(pp.nums) |
| 304 |
group = pp.Group(pp.Char(pp.alphas) + pp.IndentedBlock(integer)) |
| 305 |
|
| 306 |
parses:: |
| 307 |
|
| 308 |
A |
| 309 |
100 |
| 310 |
101 |
| 311 |
B |
| 312 |
200 |
| 313 |
201 |
| 314 |
|
| 315 |
as:: |
| 316 |
|
| 317 |
[['A', [100, 101]], ['B', [200, 201]]] |
| 318 |
|
| 319 |
By default, the results returned from the ``IndentedBlock`` are grouped. |
| 320 |
|
| 321 |
``IndentedBlock`` may also be used to define a recursive indented block (containing nested |
| 322 |
indented blocks). |
| 323 |
|
| 324 |
The existing ``indentedBlock`` is retained for backward-compatibility, but will be |
| 325 |
deprecated in a future release. |
| 326 |
|
| 327 |
Shortened tracebacks |
| 328 |
-------------------- |
| 329 |
Cleaned up default tracebacks when getting a ``ParseException`` when calling |
| 330 |
``parse_string``. Exception traces should now stop at the call in ``parse_string``, |
| 331 |
and not include the internal pyparsing traceback frames. (If the full traceback |
| 332 |
is desired, then set ``ParserElement.verbose_traceback`` to ``True``.) |
| 333 |
|
| 334 |
Improved debug logging |
| 335 |
---------------------- |
| 336 |
Debug logging has been improved by: |
| 337 |
|
| 338 |
- Including ``try/match/fail`` logging when getting results from the |
| 339 |
packrat cache (previously cache hits did not show debug logging). |
| 340 |
Values returned from the packrat cache are marked with an '*'. |
| 341 |
|
| 342 |
- Improved fail logging, showing the failed expression, text line, and marker where |
| 343 |
the failure occurred. |
| 344 |
|
| 345 |
- Adding ``with_line_numbers`` to ``pyparsing_testing``. Use ``with_line_numbers`` |
| 346 |
to visualize the data being parsed, with line and column numbers corresponding |
| 347 |
to the values output when enabling ``set_debug()`` on an expression:: |
| 348 |
|
| 349 |
data = """\ |
| 350 |
A |
| 351 |
100""" |
| 352 |
expr = pp.Word(pp.alphanums).set_name("word").set_debug() |
| 353 |
print(ppt.with_line_numbers(data)) |
| 354 |
expr[...].parseString(data) |
| 355 |
|
| 356 |
prints:: |
| 357 |
|
| 358 |
. 1 |
| 359 |
1234567890 |
| 360 |
1: A |
| 361 |
2: 100 |
| 362 |
Match word at loc 3(1,4) |
| 363 |
A |
| 364 |
^ |
| 365 |
Matched word -> ['A'] |
| 366 |
Match word at loc 11(2,7) |
| 367 |
100 |
| 368 |
^ |
| 369 |
Matched word -> ['100'] |
| 370 |
|
| 371 |
New / improved examples |
| 372 |
----------------------- |
| 373 |
- ``number_words.py`` includes a parser/evaluator to parse ``"forty-two"`` |
| 374 |
and return ``42``. Also includes example code to generate a railroad |
| 375 |
diagram for this parser. |
| 376 |
|
| 377 |
- ``BigQueryViewParser.py`` added to examples directory, submitted |
| 378 |
by Michael Smedberg. |
| 379 |
|
| 380 |
- ``booleansearchparser.py`` added to examples directory, submitted |
| 381 |
by xecgr. Builds on searchparser.py, adding support for '*' |
| 382 |
wildcards and non-Western alphabets. |
| 383 |
|
| 384 |
- Improvements in ``select_parser.py``, to include new SQL syntax |
| 385 |
from SQLite, submitted by Robert Coup. |
| 386 |
|
| 387 |
- Off-by-one bug found in the ``roman_numerals.py`` example, a bug |
| 388 |
that has been there for about 14 years! Submitted by |
| 389 |
Jay Pedersen. |
| 390 |
|
| 391 |
- A simplified Lua parser has been added to the examples |
| 392 |
(``lua_parser.py``). |
| 393 |
|
| 394 |
- Demonstration of defining a custom Unicode set for cuneiform |
| 395 |
symbols, as well as simple Cuneiform->Python conversion is included |
| 396 |
in ``cuneiform_python.py``. |
| 397 |
|
| 398 |
- Fixed bug in ``delta_time.py`` example, when using a quantity |
| 399 |
of seconds/minutes/hours/days > 999. |
| 400 |
|
| 401 |
Other new features |
| 402 |
------------------ |
| 403 |
- ``url`` expression added to ``pyparsing_common``, with named fields for |
| 404 |
common fields in URLs. See the updated ``urlExtractorNew.py`` file in the |
| 405 |
``examples`` directory. Submitted by Wolfgang Fahl. |
| 406 |
|
| 407 |
- ``DelimitedList`` now supports an additional flag ``allow_trailing_delim``, |
| 408 |
to optionally parse an additional delimiter at the end of the list. |
| 409 |
Submitted by Kazantcev Andrey. |
| 410 |
|
| 411 |
- Added global method ``autoname_elements()`` to call ``set_name()`` on all locally |
| 412 |
defined ``ParserElements`` that haven't been explicitly named using ``set_name()``, using |
| 413 |
their local variable name. Useful for setting names on multiple elements when |
| 414 |
creating a railroad diagram:: |
| 415 |
|
| 416 |
a = pp.Literal("a") |
| 417 |
b = pp.Literal("b").set_name("bbb") |
| 418 |
pp.autoname_elements() |
| 419 |
|
| 420 |
``a`` will get named "a", while ``b`` will keep its name "bbb". |
| 421 |
|
| 422 |
- Enhanced default strings created for ``Word`` expressions, now showing |
| 423 |
string ranges if possible. ``Word(alphas)`` would formerly |
| 424 |
print as ``W:(ABCD...)``, now prints as ``W:(A-Za-z)``. |
| 425 |
|
| 426 |
- Better exception messages to show full word where an exception occurred.:: |
| 427 |
|
| 428 |
Word(alphas)[...].parse_string("abc 123", parse_all=True) |
| 429 |
|
| 430 |
Was:: |
| 431 |
|
| 432 |
pyparsing.ParseException: Expected end of text, found '1' (at char 4), (line:1, col:5) |
| 433 |
|
| 434 |
Now:: |
| 435 |
|
| 436 |
pyparsing.exceptions.ParseException: Expected end of text, found '123' (at char 4), (line:1, col:5) |
| 437 |
|
| 438 |
- Using ``...`` for ``SkipTo`` can now be wrapped in ``Suppress`` to suppress |
| 439 |
the skipped text from the returned parse results.:: |
| 440 |
|
| 441 |
source = "lead in START relevant text END trailing text" |
| 442 |
start_marker = Keyword("START") |
| 443 |
end_marker = Keyword("END") |
| 444 |
find_body = Suppress(...) + start_marker + ... + end_marker |
| 445 |
print(find_body.parse_string(source).dump()) |
| 446 |
|
| 447 |
Prints:: |
| 448 |
|
| 449 |
['START', 'relevant text ', 'END'] |
| 450 |
- _skipped: ['relevant text '] |
| 451 |
|
| 452 |
- Added ``ignore_whitespace(recurse:bool = True)`` and added a |
| 453 |
``recurse`` argument to ``leave_whitespace``, both added to provide finer |
| 454 |
control over pyparsing's whitespace skipping. Contributed by |
| 455 |
Michael Milton. |
| 456 |
|
| 457 |
- Added ``ParserElement.recurse()`` method to make it simpler for |
| 458 |
grammar utilities to navigate through the tree of expressions in |
| 459 |
a pyparsing grammar. |
| 460 |
|
| 461 |
- The ``repr()`` string for ``ParseResults`` is now of the form:: |
| 462 |
|
| 463 |
ParseResults([tokens], {named_results}) |
| 464 |
|
| 465 |
The previous form omitted the leading ``ParseResults`` class name, |
| 466 |
and was easily misinterpreted as a ``tuple`` containing a ``list`` and |
| 467 |
a ``dict``. |
| 468 |
|
| 469 |
- Minor reformatting of output from ``run_tests`` to make embedded |
| 470 |
comments more visible. |
| 471 |
|
| 472 |
- New ``pyparsing_test`` namespace, assert methods and classes added to support writing |
| 473 |
unit tests. |
| 474 |
|
| 475 |
- ``assertParseResultsEquals`` |
| 476 |
- ``assertParseAndCheckList`` |
| 477 |
- ``assertParseAndCheckDict`` |
| 478 |
- ``assertRunTestResults`` |
| 479 |
- ``assertRaisesParseException`` |
| 480 |
- ``reset_pyparsing_context`` context manager, to restore pyparsing |
| 481 |
config settings |
| 482 |
|
| 483 |
- Enhanced error messages and error locations when parsing fails on |
| 484 |
the ``Keyword`` or ``CaselessKeyword`` classes due to the presence of a |
| 485 |
preceding or trailing keyword character. |
| 486 |
|
| 487 |
- Enhanced the ``Regex`` class to be compatible with re's compiled with the |
| 488 |
re-equivalent ``regex`` module. Individual expressions can be built with |
| 489 |
regex compiled expressions using:: |
| 490 |
|
| 491 |
import pyparsing as pp |
| 492 |
import regex |
| 493 |
|
| 494 |
# would use regex for this expression |
| 495 |
integer_parser = pp.Regex(regex.compile(r'\d+')) |
| 496 |
|
| 497 |
- Fixed handling of ``ParseSyntaxExceptions`` raised as part of ``Each`` |
| 498 |
expressions, when sub-expressions contain ``'-'`` backtrack |
| 499 |
suppression. |
| 500 |
|
| 501 |
- Potential performance enhancement when parsing ``Word`` |
| 502 |
expressions built from ``pyparsing_unicode`` character sets. ``Word`` now |
| 503 |
internally converts ranges of consecutive characters to regex |
| 504 |
character ranges (converting ``"0123456789"`` to ``"0-9"`` for instance). |
| 505 |
|
| 506 |
- Added a caseless parameter to the ``CloseMatch`` class to allow for casing to be |
| 507 |
ignored when checking for close matches. Contributed by Adrian Edwards. |
| 508 |
|
| 509 |
|
| 510 |
API Changes |
| 511 |
=========== |
| 512 |
|
| 513 |
- [Note added in pyparsing 3.0.7, reflecting a change in 3.0.0] |
| 514 |
Fixed a bug in the ``ParseResults`` class implementation of ``__bool__``, which |
| 515 |
would formerly return ``False`` if the ``ParseResults`` item list was empty, even if it |
| 516 |
contained named results. Now ``ParseResults`` will return ``True`` if either the item |
| 517 |
list is not empty *or* if the named results list is not empty:: |
| 518 |
|
| 519 |
# generate an empty ParseResults by parsing a blank string with a ZeroOrMore |
| 520 |
result = Word(alphas)[...].parse_string("") |
| 521 |
print(result.as_list()) |
| 522 |
print(result.as_dict()) |
| 523 |
print(bool(result)) |
| 524 |
|
| 525 |
# add a results name to the result |
| 526 |
result["name"] = "empty result" |
| 527 |
print(result.as_list()) |
| 528 |
print(result.as_dict()) |
| 529 |
print(bool(result)) |
| 530 |
|
| 531 |
Prints:: |
| 532 |
|
| 533 |
[] |
| 534 |
{} |
| 535 |
False |
| 536 |
|
| 537 |
[] |
| 538 |
{'name': 'empty result'} |
| 539 |
True |
| 540 |
|
| 541 |
In previous versions, the second call to ``bool()`` would return ``False``. |
| 542 |
|
| 543 |
- [Note added in pyparsing 3.0.4, reflecting a change in 3.0.0] |
| 544 |
The ``ParseResults`` class now uses ``__slots__`` to pre-define instance attributes. This |
| 545 |
means that code written like this (which was allowed in pyparsing 2.4.7):: |
| 546 |
|
| 547 |
result = Word(alphas).parseString("abc") |
| 548 |
result.xyz = 100 |
| 549 |
|
| 550 |
now raises this Python exception:: |
| 551 |
|
| 552 |
AttributeError: 'ParseResults' object has no attribute 'xyz' |
| 553 |
|
| 554 |
To add new attribute values to ParseResults object in 3.0.0 and later, you must |
| 555 |
assign them using indexed notation:: |
| 556 |
|
| 557 |
result["xyz"] = 100 |
| 558 |
|
| 559 |
You will still be able to access this new value as an attribute or as an |
| 560 |
indexed item. |
| 561 |
|
| 562 |
- ``enable_diag()`` and ``disable_diag()`` methods to |
| 563 |
enable specific diagnostic values (instead of setting them |
| 564 |
to ``True`` or ``False``). ``enable_all_warnings()`` has |
| 565 |
also been added. |
| 566 |
|
| 567 |
- ``counted_array`` formerly returned its list of items nested |
| 568 |
within another list, so that accessing the items required |
| 569 |
indexing the 0'th element to get the actual list. This |
| 570 |
extra nesting has been removed. In addition, if there are |
| 571 |
other metadata fields parsed between the count and the |
| 572 |
list items, they can be preserved in the resulting list |
| 573 |
if given results names. |
| 574 |
|
| 575 |
- ``ParseException.explain()`` is now an instance method of |
| 576 |
``ParseException``:: |
| 577 |
|
| 578 |
expr = pp.Word(pp.nums) * 3 |
| 579 |
try: |
| 580 |
expr.parse_string("123 456 A789") |
| 581 |
except pp.ParseException as pe: |
| 582 |
print(pe.explain(depth=0)) |
| 583 |
|
| 584 |
prints:: |
| 585 |
|
| 586 |
123 456 A789 |
| 587 |
^ |
| 588 |
ParseException: Expected W:(0-9), found 'A789' (at char 8), (line:1, col:9) |
| 589 |
|
| 590 |
To run explain against other exceptions, use |
| 591 |
``ParseException.explain_exception()``. |
| 592 |
|
| 593 |
- Debug actions now take an added keyword argument ``cache_hit``. |
| 594 |
Now that debug actions are called for expressions matched in the |
| 595 |
packrat parsing cache, debug actions are now called with this extra |
| 596 |
flag, set to ``True``. For custom debug actions, it is necessary to add |
| 597 |
support for this new argument. |
| 598 |
|
| 599 |
- ``ZeroOrMore`` expressions that have results names will now |
| 600 |
include empty lists for their name if no matches are found. |
| 601 |
Previously, no named result would be present. Code that tested |
| 602 |
for the presence of any expressions using ``"if name in results:"`` |
| 603 |
will now always return ``True``. This code will need to change to |
| 604 |
``"if name in results and results[name]:"`` or just |
| 605 |
``"if results[name]:"``. Also, any parser unit tests that check the |
| 606 |
``as_dict()`` contents will now see additional entries for parsers |
| 607 |
having named ``ZeroOrMore`` expressions, whose values will be ``[]``. |
| 608 |
|
| 609 |
- ``ParserElement.set_default_whitespace_chars`` will now update |
| 610 |
whitespace characters on all built-in expressions defined |
| 611 |
in the pyparsing module. |
| 612 |
|
| 613 |
- ``camelCase`` names have been converted to PEP-8 ``snake_case`` names. |
| 614 |
|
| 615 |
Method names and arguments that were camel case (such as ``parseString``) |
| 616 |
have been replaced with PEP-8 snake case versions (``parse_string``). |
| 617 |
|
| 618 |
Backward-compatibility synonyms for all names and arguments have |
| 619 |
been included, to allow parsers written using the old names to run |
| 620 |
without change. The synonyms will be removed in a future release. |
| 621 |
New parser code should be written using the new PEP-8 snake case names. |
| 622 |
|
| 623 |
============================== ================================ |
| 624 |
Name Previous name |
| 625 |
------------------------------ -------------------------------- |
| 626 |
ParserElement |
| 627 |
- parse_string parseString |
| 628 |
- scan_string scanString |
| 629 |
- search_string searchString |
| 630 |
- transform_string transformString |
| 631 |
- add_condition addCondition |
| 632 |
- add_parse_action addParseAction |
| 633 |
- can_parse_next canParseNext |
| 634 |
- default_name defaultName |
| 635 |
- enable_left_recursion enableLeftRecursion |
| 636 |
- enable_packrat enablePackrat |
| 637 |
- ignore_whitespace ignoreWhitespace |
| 638 |
- inline_literals_using inlineLiteralsUsing |
| 639 |
- parse_file parseFile |
| 640 |
- leave_whitespace leaveWhitespace |
| 641 |
- parse_string parseString |
| 642 |
- parse_with_tabs parseWithTabs |
| 643 |
- reset_cache resetCache |
| 644 |
- run_tests runTests |
| 645 |
- scan_string scanString |
| 646 |
- search_string searchString |
| 647 |
- set_break setBreak |
| 648 |
- set_debug setDebug |
| 649 |
- set_debug_actions setDebugActions |
| 650 |
- set_default_whitespace_chars setDefaultWhitespaceChars |
| 651 |
- set_fail_action setFailAction |
| 652 |
- set_name setName |
| 653 |
- set_parse_action setParseAction |
| 654 |
- set_results_name setResultsName |
| 655 |
- set_whitespace_chars setWhitespaceChars |
| 656 |
- transform_string transformString |
| 657 |
- try_parse tryParse |
| 658 |
|
| 659 |
ParseResults |
| 660 |
- as_list asList |
| 661 |
- as_dict asDict |
| 662 |
- get_name getName |
| 663 |
|
| 664 |
ParseBaseException |
| 665 |
- parser_element parserElement |
| 666 |
|
| 667 |
any_open_tag anyOpenTag |
| 668 |
any_close_tag anyCloseTag |
| 669 |
c_style_comment cStyleComment |
| 670 |
common_html_entity commonHTMLEntity |
| 671 |
condition_as_parse_action conditionAsParseAction |
| 672 |
counted_array countedArray |
| 673 |
cpp_style_comment cppStyleComment |
| 674 |
dbl_quoted_string dblQuotedString |
| 675 |
dbl_slash_comment dblSlashComment |
| 676 |
DelimitedList delimitedList |
| 677 |
DelimitedList delimited_list |
| 678 |
dict_of dictOf |
| 679 |
html_comment htmlComment |
| 680 |
infix_notation infixNotation |
| 681 |
java_style_comment javaStyleComment |
| 682 |
line_end lineEnd |
| 683 |
line_start lineStart |
| 684 |
make_html_tags makeHTMLTags |
| 685 |
make_xml_tags makeXMLTags |
| 686 |
match_only_at_col matchOnlyAtCol |
| 687 |
match_previous_expr matchPreviousExpr |
| 688 |
match_previous_literal matchPreviousLiteral |
| 689 |
nested_expr nestedExpr |
| 690 |
null_debug_action nullDebugAction |
| 691 |
one_of oneOf |
| 692 |
OpAssoc opAssoc |
| 693 |
original_text_for originalTextFor |
| 694 |
python_style_comment pythonStyleComment |
| 695 |
quoted_string quotedString |
| 696 |
remove_quotes removeQuotes |
| 697 |
replace_html_entity replaceHTMLEntity |
| 698 |
replace_with replaceWith |
| 699 |
rest_of_line restOfLine |
| 700 |
sgl_quoted_string sglQuotedString |
| 701 |
string_end stringEnd |
| 702 |
string_start stringStart |
| 703 |
token_map tokenMap |
| 704 |
trace_parse_action traceParseAction |
| 705 |
unicode_string unicodeString |
| 706 |
with_attribute withAttribute |
| 707 |
with_class withClass |
| 708 |
============================== ================================ |
| 709 |
|
| 710 |
Discontinued Features |
| 711 |
===================== |
| 712 |
|
| 713 |
Python 2.x no longer supported |
| 714 |
------------------------------ |
| 715 |
Removed Py2.x support and other deprecated features. Pyparsing |
| 716 |
now requires Python 3.6.8 or later. If you are using an earlier |
| 717 |
version of Python, you must use a Pyparsing 2.4.x version. |
| 718 |
|
| 719 |
Other discontinued features |
| 720 |
--------------------------- |
| 721 |
- ``ParseResults.asXML()`` - if used for debugging, switch |
| 722 |
to using ``ParseResults.dump()``; if used for data transfer, |
| 723 |
use ``ParseResults.as_dict()`` to convert to a nested Python |
| 724 |
dict, which can then be converted to XML or JSON or |
| 725 |
other transfer format |
| 726 |
|
| 727 |
- ``operatorPrecedence`` synonym for ``infixNotation`` - |
| 728 |
convert to calling ``infix_notation`` |
| 729 |
|
| 730 |
- ``commaSeparatedList`` - convert to using |
| 731 |
``pyparsing_common.comma_separated_list`` |
| 732 |
|
| 733 |
- ``upcaseTokens`` and ``downcaseTokens`` - convert to using |
| 734 |
``pyparsing_common.upcase_tokens`` and ``downcase_tokens`` |
| 735 |
|
| 736 |
- ``__compat__.collect_all_And_tokens`` will not be settable to |
| 737 |
``False`` to revert to pre-2.3.1 results name behavior - |
| 738 |
review use of names for ``MatchFirst`` and Or expressions |
| 739 |
containing ``And`` expressions, as they will return the |
| 740 |
complete list of parsed tokens, not just the first one. |
| 741 |
Use ``pyparsing.enable_diag(pyparsing.Diagnostics.warn_multiple_tokens_in_named_alternation)`` |
| 742 |
to help identify those expressions in your parsers that |
| 743 |
will have changed as a result. |
| 744 |
|
| 745 |
- Removed support for running ``python setup.py test``. The setuptools |
| 746 |
maintainers consider the ``test`` command deprecated (see |
| 747 |
<https://github.com/pypa/setuptools/issues/1684>). To run the Pyparsing tests, |
| 748 |
use the command ``tox``. |
| 749 |
|
| 750 |
|
| 751 |
Fixed Bugs |
| 752 |
========== |
| 753 |
|
| 754 |
- [Reverted in 3.0.2]Fixed issue when ``LineStart()`` expressions would match input text that was not |
| 755 |
necessarily at the beginning of a line. |
| 756 |
|
| 757 |
[The previous behavior was the correct behavior, since it represents the ``LineStart`` as its own |
| 758 |
matching expression. ``ParserElements`` that must start in column 1 can be wrapped in the new |
| 759 |
``AtLineStart`` class.] |
| 760 |
|
| 761 |
- Fixed bug in regex definitions for ``real`` and ``sci_real`` expressions in |
| 762 |
``pyparsing_common``. |
| 763 |
|
| 764 |
- Fixed ``FutureWarning`` raised beginning in Python 3.7 for ``Regex`` expressions |
| 765 |
containing '[' within a regex set. |
| 766 |
|
| 767 |
- Fixed bug in ``PrecededBy`` which caused infinite recursion. |
| 768 |
|
| 769 |
- Fixed bug in ``CloseMatch`` where end location was incorrectly |
| 770 |
computed; and updated ``partial_gene_match.py`` example. |
| 771 |
|
| 772 |
- Fixed bug in ``indentedBlock`` with a parser using two different |
| 773 |
types of nested indented blocks with different indent values, |
| 774 |
but sharing the same indent stack. |
| 775 |
|
| 776 |
- Fixed bug in ``Each`` when using ``Regex``, when ``Regex`` expression would |
| 777 |
get parsed twice. |
| 778 |
|
| 779 |
- Fixed bugs in ``Each`` when passed ``OneOrMore`` or ``ZeroOrMore`` expressions: |
| 780 |
. first expression match could be enclosed in an extra nesting level |
| 781 |
. out-of-order expressions now handled correctly if mixed with required expressions |
| 782 |
. results names are maintained correctly for these expression |
| 783 |
|
| 784 |
- Fixed ``FutureWarning`` that sometimes is raised when ``'['`` passed as a |
| 785 |
character to ``Word``. |
| 786 |
|
| 787 |
- Fixed debug logging to show failure location after whitespace skipping. |
| 788 |
|
| 789 |
- Fixed ``ParseFatalExceptions`` failing to override normal exceptions or expression |
| 790 |
matches in ``MatchFirst`` expressions. |
| 791 |
|
| 792 |
- Fixed bug in which ``ParseResults`` replaces a collection type value with an invalid |
| 793 |
type annotation (as a result of changed behavior in Python 3.9). |
| 794 |
|
| 795 |
- Fixed bug in ``ParseResults`` when calling ``__getattr__`` for special double-underscored |
| 796 |
methods. Now raises ``AttributeError`` for non-existent results when accessing a |
| 797 |
name starting with '__'. |
| 798 |
|
| 799 |
- Fixed bug in ``Located`` class when used with a results name. |
| 800 |
|
| 801 |
- Fixed bug in ``QuotedString`` class when the escaped quote string is not a |
| 802 |
repeated character. |
| 803 |
|
| 804 |
Acknowledgments |
| 805 |
=============== |
| 806 |
And finally, many thanks to those who helped in the restructuring |
| 807 |
of the pyparsing code base as part of this release. Pyparsing now |
| 808 |
has more standard package structure, more standard unit tests, |
| 809 |
and more standard code formatting (using black). Special thanks |
| 810 |
to jdufresne, klahnakoski, mattcarmody, ckeygusuz, |
| 811 |
tmiguelt, and toonarmycaptain to name just a few. |
| 812 |
|
| 813 |
Thanks also to Michael Milton and Max Fischer, who added some |
| 814 |
significant new features to pyparsing. |