| 1 |
============================= |
| 2 |
What's New in Pyparsing 3.1.0 |
| 3 |
============================= |
| 4 |
|
| 5 |
:author: Paul McGuire |
| 6 |
|
| 7 |
:date: October, 2024 |
| 8 |
|
| 9 |
:abstract: This document summarizes the changes made |
| 10 |
in the 3.1.x releases of pyparsing. |
| 11 |
|
| 12 |
.. contents:: :depth: 4 |
| 13 |
|
| 14 |
|
| 15 |
Supported Python versions |
| 16 |
========================= |
| 17 |
- Added support for Python 3.12. |
| 18 |
|
| 19 |
- All internal string expressions using '%' string interpolation and ``str.format()`` |
| 20 |
converted to f-strings. |
| 21 |
|
| 22 |
|
| 23 |
New Features |
| 24 |
============ |
| 25 |
- Added new ``Tag`` ParserElement, for inserting metadata into the parsed results. |
| 26 |
This allows a parser to add metadata or annotations to the parsed tokens. |
| 27 |
The ``Tag`` element also accepts an optional ``value`` parameter, defaulting to ``True``. |
| 28 |
See the new ``tag_metadata.py`` example in the ``examples`` directory. |
| 29 |
|
| 30 |
Example: |
| 31 |
|
| 32 |
.. doctest:: |
| 33 |
|
| 34 |
>>> # add tag indicating mood |
| 35 |
>>> end_punc = "." | ("!" + Tag("enthusiastic")) |
| 36 |
>>> greeting = "Hello" + Word(alphas) + end_punc |
| 37 |
|
| 38 |
>>> result = greeting.parse_string("Hello World.") |
| 39 |
>>> print(result.dump()) |
| 40 |
['Hello', 'World', '.'] |
| 41 |
|
| 42 |
>>> result = greeting.parse_string("Hello World!") |
| 43 |
>>> print(result.dump()) |
| 44 |
['Hello', 'World', '!'] |
| 45 |
- enthusiastic: True |
| 46 |
|
| 47 |
- Extended ``expr[]`` notation for repetition of ``expr`` to accept a |
| 48 |
slice, where the slice's stop value indicates a ``stop_on`` |
| 49 |
expression: |
| 50 |
|
| 51 |
.. testcode:: |
| 52 |
|
| 53 |
test = "BEGIN aaa bbb ccc END" |
| 54 |
BEGIN, END = Keyword.using_each("BEGIN END".split()) |
| 55 |
body_word = Word(alphas) |
| 56 |
|
| 57 |
# new slice syntax support |
| 58 |
expr = BEGIN + Group(body_word[...:END]) + END |
| 59 |
# equivalent to |
| 60 |
# BEGIN + Group(ZeroOrMore(body_word, stop_on=END)) + END |
| 61 |
|
| 62 |
print(expr.parse_string(test)) |
| 63 |
|
| 64 |
Prints: |
| 65 |
|
| 66 |
.. testoutput:: |
| 67 |
|
| 68 |
['BEGIN', ['aaa', 'bbb', 'ccc'], 'END'] |
| 69 |
|
| 70 |
- Added new class method ``ParserElement.using_each``, to simplify code |
| 71 |
that creates a sequence of ``Literals``, ``Keywords``, or other ``ParserElement`` |
| 72 |
subclasses. |
| 73 |
|
| 74 |
For instance, to define suppressible punctuation, you would previously |
| 75 |
write:: |
| 76 |
|
| 77 |
LPAR, RPAR, LBRACE, RBRACE, SEMI = map(Suppress, "(){};") |
| 78 |
|
| 79 |
You can now write:: |
| 80 |
|
| 81 |
LPAR, RPAR, LBRACE, RBRACE, SEMI = Suppress.using_each("(){};") |
| 82 |
|
| 83 |
``using_each`` will also accept optional keyword args, which it will |
| 84 |
pass through to the class initializer. Here is an expression for |
| 85 |
single-letter variable names that might be used in an algebraic |
| 86 |
expression: |
| 87 |
|
| 88 |
.. testcode:: |
| 89 |
|
| 90 |
algebra_var = MatchFirst( |
| 91 |
Char.using_each(string.ascii_lowercase, as_keyword=True) |
| 92 |
) |
| 93 |
|
| 94 |
- Added new builtin ``python_quoted_string``, which will match any form |
| 95 |
of single-line or multiline quoted strings defined in Python. |
| 96 |
|
| 97 |
- ``Word`` arguments are now validated if ``min`` and ``max`` are both |
| 98 |
given, that ``min`` <= ``max``; raises ``ValueError`` if values are invalid. |
| 99 |
|
| 100 |
- Added '·' (Unicode MIDDLE DOT) to the set of ``Latin1.identbodychars``. |
| 101 |
|
| 102 |
- Added ``ieee_float`` expression to ``pyparsing.common``, which parses float values, |
| 103 |
plus "NaN", "Inf", "Infinity". |
| 104 |
|
| 105 |
- Minor performance speedup in ``trim_arity``, to benefit any parsers using parse actions. |
| 106 |
|
| 107 |
|
| 108 |
API Changes |
| 109 |
=========== |
| 110 |
- ``Optional(expr)`` may now be written as ``expr | ""`` |
| 111 |
|
| 112 |
This will make this code: |
| 113 |
|
| 114 |
.. testcode:: |
| 115 |
|
| 116 |
"{" + Optional(Literal("A") | Literal("a")) + "}" |
| 117 |
|
| 118 |
writable as: |
| 119 |
|
| 120 |
.. testcode:: |
| 121 |
|
| 122 |
"{" + (Literal("A") | Literal("a") | "") + "}" |
| 123 |
|
| 124 |
Some related changes implemented as part of this work: |
| 125 |
- ``Literal("")`` now internally generates an ``Empty()`` (and no longer raises an exception) |
| 126 |
- ``Empty`` is now a subclass of ``Literal`` |
| 127 |
|
| 128 |
- Added new class property ``identifier`` to all Unicode set classes in ``pyparsing.unicode``, |
| 129 |
using the class's values for ``cls.identchars`` and ``cls.identbodychars``. Now Unicode-aware |
| 130 |
parsers that formerly wrote: |
| 131 |
|
| 132 |
.. testcode:: |
| 133 |
|
| 134 |
ppu = pyparsing.unicode |
| 135 |
ident = Word(ppu.Greek.identchars, ppu.Greek.identbodychars) |
| 136 |
|
| 137 |
can now write: |
| 138 |
|
| 139 |
.. testcode:: |
| 140 |
|
| 141 |
ident = ppu.Greek.identifier |
| 142 |
# or |
| 143 |
ident = ppu.Ελληνικά.identifier |
| 144 |
|
| 145 |
- Added bool ``embed`` argument to ``ParserElement.create_diagram()``. |
| 146 |
When passed as True, the resulting diagram will omit the ``<DOCTYPE>``, |
| 147 |
``<HEAD>``, and ``<BODY>`` tags so that it can be embedded in other |
| 148 |
HTML source. (Useful when embedding a call to ``create_diagram()`` in |
| 149 |
a PyScript HTML page.) |
| 150 |
|
| 151 |
- Added ``recurse`` argument to ``ParserElement.set_debug`` to set the |
| 152 |
debug flag on an expression and all of its sub-expressions. |
| 153 |
|
| 154 |
- Reworked ``delimited_list`` function into the new ``DelimitedList`` class. |
| 155 |
``DelimitedList`` has the same constructor interface as ``delimited_list``, and |
| 156 |
in this release, ``delimited_list`` changes from a function to a synonym for |
| 157 |
``DelimitedList``. ``delimited_list`` and the older ``delimitedList`` method will be |
| 158 |
deprecated in a future release, in favor of ``DelimitedList``. |
| 159 |
|
| 160 |
- ``ParseResults`` now has a new method ``deepcopy()``, in addition to the current |
| 161 |
``copy()`` method. ``copy()`` only makes a shallow copy - any contained ``ParseResults`` |
| 162 |
are copied as references - changes in the copy will be seen as changes in the original. |
| 163 |
In many cases, a shallow copy is sufficient, but some applications require a deep copy. |
| 164 |
``deepcopy()`` makes a deeper copy: any contained ``ParseResults`` or other mappings or |
| 165 |
containers are built with copies from the original, and do not get changed if the |
| 166 |
original is later changed. |
| 167 |
|
| 168 |
- Added named field "url" to ``pyparsing.common.url``, returning the entire |
| 169 |
parsed URL string. |
| 170 |
|
| 171 |
- Added exception type to ``trace_parse_action`` exception output. |
| 172 |
|
| 173 |
- Added ``<META>`` tag to HTML generated for railroad diagrams to force UTF-8 encoding |
| 174 |
with older browsers, to better display Unicode parser characters. |
| 175 |
|
| 176 |
- To address a compatibility issue in RDFLib, added a property setter for the |
| 177 |
``ParserElement.name`` property, to call ``ParserElement.set_name``. |
| 178 |
|
| 179 |
- Modified ``ParserElement.set_name()`` to accept a None value, to clear the defined |
| 180 |
name and corresponding error message for a ``ParserElement``. |
| 181 |
|
| 182 |
- Updated railroad diagram generation for ``ZeroOrMore`` and ``OneOrMore`` expressions with |
| 183 |
``stop_on`` expressions. |
| 184 |
|
| 185 |
|
| 186 |
Discontinued Features |
| 187 |
===================== |
| 188 |
|
| 189 |
Python 2.x no longer supported |
| 190 |
------------------------------ |
| 191 |
Removed Py2.x support and other deprecated features. Pyparsing |
| 192 |
now requires Python 3.6.8 or later. If you are using an earlier |
| 193 |
version of Python, you must use a Pyparsing 2.4.x version. |
| 194 |
|
| 195 |
Other discontinued / deprecated features |
| 196 |
---------------------------------------- |
| 197 |
- ``ParserElement.validate()`` is deprecated. It predates the support for left-recursive |
| 198 |
parsers, and was prone to false positives (warning that a grammar was invalid when |
| 199 |
it was in fact valid). It will be removed in a future pyparsing release. In its |
| 200 |
place, developers should use debugging and analytical tools, such as ``ParserElement.set_debug()`` |
| 201 |
and ``ParserElement.create_diagram()``. |
| 202 |
|
| 203 |
|
| 204 |
Fixed Bugs |
| 205 |
========== |
| 206 |
- Updated ``ci.yml`` permissions to limit default access to source. |
| 207 |
|
| 208 |
- Updated ``create_diagram()`` code to be compatible with railroad-diagrams package |
| 209 |
version 3.0. |
| 210 |
|
| 211 |
- Fixed bug in ``pyparsing.common.url``, when input URL is not alone |
| 212 |
on an input line. |
| 213 |
|
| 214 |
- Fixed bug in srange, when parsing escaped '/' and '\' inside a |
| 215 |
range set. |
| 216 |
|
| 217 |
- Fixed exception messages for some ``ParserElements`` with custom names, |
| 218 |
which instead showed their contained expression names. |
| 219 |
|
| 220 |
- Fixed bug in ``Word`` when ``max=2``. Also added performance enhancement |
| 221 |
when specifying ``exact`` argument. |
| 222 |
|
| 223 |
- Fixed bug when parse actions returned an empty string for an expression that |
| 224 |
had a results name, that the results name was not saved. That is: |
| 225 |
|
| 226 |
.. doctest:: |
| 227 |
|
| 228 |
>>> expr = Literal("X").add_parse_action(lambda tokens: "")("value") |
| 229 |
>>> result = expr.parse_string("X") |
| 230 |
>>> result["value"] |
| 231 |
'' |
| 232 |
|
| 233 |
would raise a ``KeyError``. Now empty strings will be saved with the associated |
| 234 |
results name. |
| 235 |
|
| 236 |
- Fixed bug in ``SkipTo`` where ignore expressions were not properly handled while |
| 237 |
scanning for the target expression. |
| 238 |
|
| 239 |
- Fixed bug in ``NotAny``, where parse actions on the negated expr were not being run. |
| 240 |
This could cause ``NotAny`` to incorrectly fail if the expr would normally match, |
| 241 |
but would fail to match if a condition used as a parse action returned False. |
| 242 |
|
| 243 |
- Fixed ``create_diagram()`` to accept keyword args, to be passed through to the |
| 244 |
``template.render()`` method to generate the output HTML. |
| 245 |
|
| 246 |
- Fixed bug in ``python_quoted_string`` regex. |
| 247 |
|
| 248 |
- Fixed regression in Word(min). |
| 249 |
|
| 250 |
- Fixed bug in bad exception messages raised by Forward expressions. |
| 251 |
|
| 252 |
- Fixed regression in SkipTo, where ignored expressions were not checked when looking |
| 253 |
for the target expression. |
| 254 |
|
| 255 |
- Updated pep8 synonym wrappers for better type checking compatibility. |
| 256 |
|
| 257 |
- Fixed empty error message bug. This _should_ return |
| 258 |
pyparsing's exception messages to a former, more helpful form. If you have code that |
| 259 |
parses the exception messages returned by pyparsing, this may require some code |
| 260 |
changes. |
| 261 |
|
| 262 |
- Fixed issue where PEP8 compatibility names for ``ParserElement`` static methods were |
| 263 |
not themselves defined as ``staticmethods``. When called using a ``ParserElement`` instance, |
| 264 |
this resulted in a ``TypeError`` exception. |
| 265 |
|
| 266 |
- Fixed some cosmetics/bugs in railroad diagrams: |
| 267 |
|
| 268 |
- fixed groups being shown even when ``show_groups`` = False |
| 269 |
|
| 270 |
- show results names as quoted strings when ``show_results_names`` = True |
| 271 |
|
| 272 |
- only use integer loop counter if repetition > 2 |
| 273 |
|
| 274 |
|
| 275 |
New / Enhanced Examples |
| 276 |
======================= |
| 277 |
- Added example ``mongodb_query_expression.py``, to convert human-readable infix query |
| 278 |
expressions, such as:: |
| 279 |
|
| 280 |
a==100 and b>=200 |
| 281 |
|
| 282 |
and transform them into an equivalent query argument for the pymongo package:: |
| 283 |
|
| 284 |
{'$and': [{'a': 100}, {'b': {'$gte': 200}}]} |
| 285 |
|
| 286 |
Supports many equality and inequality operators - see the docstring for the |
| 287 |
``transform_query`` function for many more examples. |
| 288 |
|
| 289 |
- ``invRegex.py`` example renamed to ``inv_regex.py`` and updated to PEP-8 |
| 290 |
variable and method naming. |
| 291 |
|
| 292 |
- Removed examples ``sparser.py`` and ``pymicko.py``, since each included its |
| 293 |
own GPL license in the header. Since this conflicts with pyparsing's |
| 294 |
MIT license, they were removed from the distribution to avoid |
| 295 |
confusion among those making use of them in their own projects. |
| 296 |
|
| 297 |
- Updated the ``lucene_grammar.py`` example (better support for '*' and '?' wildcards) |
| 298 |
and corrected the test cases! |
| 299 |
|
| 300 |
- Added ``bf.py`` Brainf*ck parser/executor example. Illustrates using |
| 301 |
a pyparsing grammar to parse language syntax, and attach executable AST nodes to |
| 302 |
the parsed results. |
| 303 |
|
| 304 |
- Added ``tag_emitter.py`` to examples. This example demonstrates how to insert |
| 305 |
tags into your parsed results that are not part of the original parsed text. |
| 306 |
|
| 307 |
- Updated example ``select_parser.py`` to use PEP8 names and added Groups for better retrieval |
| 308 |
of parsed values from multiple SELECT clauses. |
| 309 |
|
| 310 |
- Added example ``email_address_parser.py``. |
| 311 |
|
| 312 |
- Added example ``directx_x_file_parser.py`` to parse DirectX template definitions, and |
| 313 |
generate a Pyparsing parser from a template to parse .x files. |
| 314 |
|
| 315 |
- ``delta_time``, ``lua_parser``, ``decaf_parser``, and ``roman_numerals`` examples cleaned up |
| 316 |
to use latest PEP8 names and add minor enhancements. |
| 317 |
|
| 318 |
- Fixed bug (and corresponding test code) in ``delta_time`` example that did not handle |
| 319 |
weekday references in time expressions (like "Monday at 4pm") when the weekday was |
| 320 |
the same as the current weekday. |
| 321 |
|
| 322 |
|
| 323 |
Acknowledgments |
| 324 |
=============== |
| 325 |
Again, thanks to the many contributors who submitted issues, questions, suggestions, |
| 326 |
and PRs. |