| 1 |
========================== |
| 2 |
Using the pyparsing module |
| 3 |
========================== |
| 4 |
|
| 5 |
:author: Paul McGuire |
| 6 |
:address: [email protected] |
| 7 |
|
| 8 |
:revision: 3.2.0 |
| 9 |
:date: October, 2024 |
| 10 |
|
| 11 |
:copyright: Copyright |copy| 2003-2024 Paul McGuire. |
| 12 |
|
| 13 |
.. |copy| unicode:: 0xA9 |
| 14 |
|
| 15 |
:abstract: This document provides how-to instructions for the |
| 16 |
pyparsing library, an easy-to-use Python module for constructing |
| 17 |
and executing basic text parsers. The pyparsing module is useful |
| 18 |
for evaluating user-definable |
| 19 |
expressions, processing custom application language commands, or |
| 20 |
extracting data from formatted reports. |
| 21 |
|
| 22 |
.. contents:: :depth: 4 |
| 23 |
|
| 24 |
Note: While this content is still valid, there are more detailed |
| 25 |
descriptions and extensive examples at the `online doc server |
| 26 |
<https://pyparsing-docs.readthedocs.io/en/latest/pyparsing.html>`_, and |
| 27 |
in the online help for the various pyparsing classes and methods (viewable |
| 28 |
using the Python interpreter's built-in ``help()`` function). You will also |
| 29 |
find many example scripts in the `examples <https://github.com/pyparsing/pyparsing/tree/master/examples>`_ |
| 30 |
directory of the pyparsing GitHub repo. |
| 31 |
|
| 32 |
----------- |
| 33 |
|
| 34 |
**Note**: *In pyparsing 3.0, many method and function names which were |
| 35 |
originally written using camelCase have been converted to PEP8-compatible |
| 36 |
snake_case. So ``parseString()`` is being renamed to ``parse_string()``, |
| 37 |
``delimitedList`` to DelimitedList_, and so on. You may see the old |
| 38 |
names in legacy parsers, and they will be supported for a time with |
| 39 |
synonyms, but the synonyms will be removed in a future release.* |
| 40 |
|
| 41 |
*If you are using this documentation, but working with a 2.4.x version of pyparsing, |
| 42 |
you'll need to convert methods and arguments from the documented snake_case |
| 43 |
names to the legacy camelCase names. In pyparsing 3.x, both forms are |
| 44 |
supported, but the legacy forms are deprecated; they will be dropped in a |
| 45 |
future 4.0 release.* |
| 46 |
|
| 47 |
----------- |
| 48 |
|
| 49 |
|
| 50 |
Steps to follow |
| 51 |
=============== |
| 52 |
|
| 53 |
To parse an incoming data string, the client code must follow these steps: |
| 54 |
|
| 55 |
1. First define the tokens and patterns to be matched, and assign |
| 56 |
this to a program variable. Optional results names or parse |
| 57 |
actions can also be defined at this time. |
| 58 |
|
| 59 |
2. Call ``parse_string()``, ``scan_string()``, or ``search_string()`` |
| 60 |
on this variable, passing in the string to |
| 61 |
be parsed. During the matching process, whitespace between |
| 62 |
tokens is skipped by default (although this can be changed). |
| 63 |
When token matches occur, any defined parse action methods are |
| 64 |
called. |
| 65 |
|
| 66 |
3. Process the parsed results, returned as a ParseResults_ object. |
| 67 |
The ParseResults_ object can be accessed as if it were a list of |
| 68 |
strings. Matching results may also be accessed as named attributes of |
| 69 |
the returned results, if names are defined in the definition of |
| 70 |
the token pattern, using ``set_results_name()``. |
| 71 |
|
| 72 |
|
| 73 |
Hello, World! |
| 74 |
------------- |
| 75 |
|
| 76 |
The following complete Python program will parse the greeting ``"Hello, World!"``, |
| 77 |
or any other greeting of the form "<salutation>, <addressee>!":: |
| 78 |
|
| 79 |
import pyparsing as pp |
| 80 |
|
| 81 |
greet = pp.Word(pp.alphas) + "," + pp.Word(pp.alphas) + "!" |
| 82 |
for greeting_str in [ |
| 83 |
"Hello, World!", |
| 84 |
"Bonjour, Monde!", |
| 85 |
"Hola, Mundo!", |
| 86 |
"Hallo, Welt!", |
| 87 |
]: |
| 88 |
greeting = greet.parse_string(greeting_str) |
| 89 |
print(greeting) |
| 90 |
|
| 91 |
The parsed tokens are returned in the following form:: |
| 92 |
|
| 93 |
['Hello', ',', 'World', '!'] |
| 94 |
['Bonjour', ',', 'Monde', '!'] |
| 95 |
['Hola', ',', 'Mundo', '!'] |
| 96 |
['Hallo', ',', 'Welt', '!'] |
| 97 |
|
| 98 |
|
| 99 |
Usage notes |
| 100 |
----------- |
| 101 |
|
| 102 |
- The pyparsing module can be used to interpret simple command |
| 103 |
strings or algebraic expressions, or can be used to extract data |
| 104 |
from text reports with complicated format and structure ("screen |
| 105 |
or report scraping"). However, it is possible that your defined |
| 106 |
matching patterns may accept invalid inputs. Use pyparsing to |
| 107 |
extract data from strings assumed to be well-formatted. |
| 108 |
|
| 109 |
- To keep up the readability of your code, use operators_ such as ``+``, ``|``, |
| 110 |
``^``, and ``~`` to combine expressions. You can also combine |
| 111 |
string literals with ``ParseExpressions`` - they will be |
| 112 |
automatically converted to Literal_ objects. For example:: |
| 113 |
|
| 114 |
integer = Word(nums) # simple unsigned integer |
| 115 |
variable = Char(alphas) # single letter variable, such as x, z, m, etc. |
| 116 |
arith_op = one_of("+ - * /") # arithmetic operators |
| 117 |
equation = variable + "=" + integer + arith_op + integer # will match "x=2+2", etc. |
| 118 |
|
| 119 |
In the definition of ``equation``, the string ``"="`` will get added as |
| 120 |
a ``Literal("=")``, but in a more readable way. |
| 121 |
|
| 122 |
- The pyparsing module's default behavior is to ignore whitespace. This is the |
| 123 |
case for 99% of all parsers ever written. This allows you to write simple, clean, |
| 124 |
grammars, such as the above ``equation``, without having to clutter it up with |
| 125 |
extraneous ``ws`` markers. The ``equation`` grammar will successfully parse all of the |
| 126 |
following statements:: |
| 127 |
|
| 128 |
x=2+2 |
| 129 |
x = 2+2 |
| 130 |
a = 10 * 4 |
| 131 |
r= 1234/ 100000 |
| 132 |
|
| 133 |
Of course, it is quite simple to extend this example to support more elaborate expressions, with |
| 134 |
nesting with parentheses, floating point numbers, scientific notation, and named constants |
| 135 |
(such as ``e`` or ``pi``). See `fourFn.py <https://github.com/pyparsing/pyparsing/blob/master/examples/fourFn.py>`_, |
| 136 |
and `simpleArith.py <https://github.com/pyparsing/pyparsing/blob/master/examples/simpleArith.py>`_ |
| 137 |
included in the examples directory. |
| 138 |
|
| 139 |
- To modify pyparsing's default whitespace skipping, you can use one or |
| 140 |
more of the following methods: |
| 141 |
|
| 142 |
- use the static method ``ParserElement.set_default_whitespace_chars`` |
| 143 |
to override the normal set of whitespace chars (``' \t\n'``). For instance |
| 144 |
when defining a grammar in which newlines are significant, you should |
| 145 |
call ``ParserElement.set_default_whitespace_chars(' \t')`` to remove |
| 146 |
newline from the set of skippable whitespace characters. Calling |
| 147 |
this method will affect all pyparsing expressions defined afterward. |
| 148 |
|
| 149 |
- call ``leave_whitespace()`` on individual expressions, to suppress the |
| 150 |
skipping of whitespace before trying to match the expression |
| 151 |
|
| 152 |
- use ``Combine`` to require that successive expressions must be |
| 153 |
adjacent in the input string. For instance, this expression:: |
| 154 |
|
| 155 |
real = Word(nums) + '.' + Word(nums) |
| 156 |
|
| 157 |
will match "3.14159", but will also match "3 . 12". It will also |
| 158 |
return the matched results as ['3', '.', '14159']. By changing this |
| 159 |
expression to:: |
| 160 |
|
| 161 |
real = Combine(Word(nums) + '.' + Word(nums)) |
| 162 |
|
| 163 |
it will not match numbers with embedded spaces, and it will return a |
| 164 |
single concatenated string '3.14159' as the parsed token. |
| 165 |
|
| 166 |
- Repetition of expressions can be indicated using ``*`` or ``[]`` notation. An |
| 167 |
expression may be multiplied by an integer value (to indicate an exact |
| 168 |
repetition count), or indexed with a tuple, representing min and max repetitions |
| 169 |
(with ``...`` representing no min or no max, depending whether it is the first or |
| 170 |
second tuple element). See the following examples, where n is used to |
| 171 |
indicate an integer value: |
| 172 |
|
| 173 |
- ``expr*3`` is equivalent to ``expr + expr + expr`` |
| 174 |
|
| 175 |
- ``expr[2, 3]`` is equivalent to ``expr + expr + Opt(expr)`` |
| 176 |
|
| 177 |
- ``expr[n, ...]`` or ``expr[n,]`` is equivalent |
| 178 |
to ``expr*n + ZeroOrMore(expr)`` (read as "at least n instances of expr") |
| 179 |
|
| 180 |
- ``expr[... ,n]`` is equivalent to ``expr*(0, n)`` |
| 181 |
(read as "0 to n instances of expr") |
| 182 |
|
| 183 |
- ``expr[...]``, ``expr[0, ...]`` and ``expr * ...`` are equivalent to ``ZeroOrMore(expr)`` |
| 184 |
|
| 185 |
- ``expr[1, ...]`` is equivalent to ``OneOrMore(expr)`` |
| 186 |
|
| 187 |
Note that ``expr[..., n]`` does not raise an exception if |
| 188 |
more than n exprs exist in the input stream; that is, |
| 189 |
``expr[..., n]`` does not enforce a maximum number of expr |
| 190 |
occurrences. If this behavior is desired, then write |
| 191 |
``expr[..., n] + ~expr``. |
| 192 |
|
| 193 |
- ``[]`` notation will also accept a stop expression using ':' slice |
| 194 |
notation: |
| 195 |
|
| 196 |
- ``expr[...:end_expr]`` is equivalent to ``ZeroOrMore(expr, stop_on=end_expr)`` |
| 197 |
|
| 198 |
- MatchFirst_ expressions are matched left-to-right, and the first |
| 199 |
match found will skip all later expressions within, so be sure |
| 200 |
to define less-specific patterns after more-specific patterns. |
| 201 |
If you are not sure which expressions are most specific, use Or_ |
| 202 |
expressions (defined using the ``^`` operator) - they will always |
| 203 |
match the longest expression, although they are more |
| 204 |
compute-intensive. |
| 205 |
|
| 206 |
- Or_ expressions will evaluate all of the specified subexpressions |
| 207 |
to determine which is the "best" match, that is, which matches |
| 208 |
the longest string in the input data. In case of a tie, the |
| 209 |
left-most expression in the Or_ list will win. |
| 210 |
|
| 211 |
- If parsing the contents of an entire file, pass it to the |
| 212 |
``parse_file`` method using:: |
| 213 |
|
| 214 |
expr.parse_file(source_file) |
| 215 |
|
| 216 |
- ``ParseExceptions`` will report the location where an expected token |
| 217 |
or expression failed to match. For example, if we tried to use our |
| 218 |
"Hello, World!" parser to parse "Hello World!" (leaving out the separating |
| 219 |
comma), we would get an exception, with the message:: |
| 220 |
|
| 221 |
pyparsing.ParseException: Expected "," (6), (1,7) |
| 222 |
|
| 223 |
In the case of complex |
| 224 |
expressions, the reported location may not be exactly where you |
| 225 |
would expect. See more information under ParseException_ . |
| 226 |
|
| 227 |
- Use the ``Group`` class to enclose logical groups of tokens within a |
| 228 |
sublist. This will help organize your results into more |
| 229 |
hierarchical form (the default behavior is to return matching |
| 230 |
tokens as a flat list of matching input strings). |
| 231 |
|
| 232 |
- Punctuation may be significant for matching, but is rarely of |
| 233 |
much interest in the parsed results. Use the ``suppress()`` method |
| 234 |
to keep these tokens from cluttering up your returned lists of |
| 235 |
tokens. For example, DelimitedList_ matches a succession of |
| 236 |
one or more expressions, separated by delimiters (commas by |
| 237 |
default), but only returns a list of the actual expressions - |
| 238 |
the delimiters are used for parsing, but are suppressed from the |
| 239 |
returned output. |
| 240 |
|
| 241 |
- Parse actions can be used to convert values from strings to |
| 242 |
other data types (ints, floats, booleans, etc.). |
| 243 |
|
| 244 |
- Results names are recommended for retrieving tokens from complex |
| 245 |
expressions. It is much easier to access a token using its field |
| 246 |
name than using a positional index, especially if the expression |
| 247 |
contains optional elements. You can also shortcut |
| 248 |
the ``set_results_name`` call:: |
| 249 |
|
| 250 |
stats = ("AVE:" + real_num.set_results_name("average") |
| 251 |
+ "MIN:" + real_num.set_results_name("min") |
| 252 |
+ "MAX:" + real_num.set_results_name("max")) |
| 253 |
|
| 254 |
can more simply and cleanly be written as this:: |
| 255 |
|
| 256 |
stats = ("AVE:" + real_num("average") |
| 257 |
+ "MIN:" + real_num("min") |
| 258 |
+ "MAX:" + real_num("max")) |
| 259 |
|
| 260 |
- Be careful when defining parse actions that modify global variables or |
| 261 |
data structures (as in fourFn.py_), especially for low level tokens |
| 262 |
or expressions that may occur within an And_ expression; an early element |
| 263 |
of an And_ may match, but the overall expression may fail. |
| 264 |
|
| 265 |
|
| 266 |
Classes |
| 267 |
======= |
| 268 |
|
| 269 |
All the pyparsing classes can be found in this |
| 270 |
`UML class diagram <_static/pyparsingClassDiagram_3.0.9.jpg>`_. |
| 271 |
|
| 272 |
Classes in the pyparsing module |
| 273 |
------------------------------- |
| 274 |
|
| 275 |
``ParserElement`` - abstract base class for all pyparsing classes; |
| 276 |
methods for code to use are: |
| 277 |
|
| 278 |
- ``parse_string(source_string, parse_all=False)`` - only called once, on the overall |
| 279 |
matching pattern; returns a ParseResults_ object that makes the |
| 280 |
matched tokens available as a list, and optionally as a dictionary, |
| 281 |
or as an object with named attributes; if ``parse_all`` is set to True, then |
| 282 |
``parse_string`` will raise a ParseException_ if the grammar does not process |
| 283 |
the complete input string. |
| 284 |
|
| 285 |
- ``parse_file(source_file)`` - a convenience function, that accepts an |
| 286 |
input file object or filename. The file contents are passed as a |
| 287 |
string to ``parse_string()``. ``parse_file`` also supports the ``parse_all`` argument. |
| 288 |
|
| 289 |
- ``scan_string(source_string)`` - generator function, used to find and |
| 290 |
extract matching text in the given source string; for each matched text, |
| 291 |
returns a tuple of: |
| 292 |
|
| 293 |
- matched tokens (packaged as a ParseResults_ object) |
| 294 |
|
| 295 |
- start location of the matched text in the given source string |
| 296 |
|
| 297 |
- end location in the given source string |
| 298 |
|
| 299 |
``scan_string`` allows you to scan through the input source string for |
| 300 |
random matches, instead of exhaustively defining the grammar for the entire |
| 301 |
source text (as would be required with ``parse_string``). |
| 302 |
|
| 303 |
- ``transform_string(source_string)`` - convenience wrapper function for |
| 304 |
``scan_string``, to process the input source string, and replace matching |
| 305 |
text with the tokens returned from parse actions defined in the grammar |
| 306 |
(see set_parse_action_). |
| 307 |
|
| 308 |
- ``search_string(source_string)`` - another convenience wrapper function for |
| 309 |
``scan_string``, returns a list of the matching tokens returned from each |
| 310 |
call to ``scan_string``. |
| 311 |
|
| 312 |
- ``set_name(name)`` - associate a short descriptive name for this |
| 313 |
element, useful in displaying exceptions and trace information |
| 314 |
|
| 315 |
- ``run_tests(tests_string)`` - useful development and testing method on |
| 316 |
expressions, to pass a multiline string of sample strings to test against |
| 317 |
the expression. Comment lines (beginning with ``#``) can be inserted |
| 318 |
and they will be included in the test output:: |
| 319 |
|
| 320 |
digits = Word(nums).set_name("numeric digits") |
| 321 |
real_num = Combine(digits + '.' + digits) |
| 322 |
real_num.run_tests("""\ |
| 323 |
# valid number |
| 324 |
3.14159 |
| 325 |
|
| 326 |
# no integer part |
| 327 |
.00001 |
| 328 |
|
| 329 |
# no decimal |
| 330 |
101 |
| 331 |
|
| 332 |
# no decimal value |
| 333 |
101. |
| 334 |
""") |
| 335 |
|
| 336 |
will print:: |
| 337 |
|
| 338 |
# valid number |
| 339 |
3.14159 |
| 340 |
['3.14159'] |
| 341 |
|
| 342 |
# no integer part |
| 343 |
.00001 |
| 344 |
^ |
| 345 |
FAIL: Expected numeric digits, found '.' (at char 0), (line:1, col:1) |
| 346 |
|
| 347 |
# no decimal |
| 348 |
101 |
| 349 |
^ |
| 350 |
FAIL: Expected ".", found end of text (at char 3), (line:1, col:4) |
| 351 |
|
| 352 |
# no decimal value |
| 353 |
101. |
| 354 |
^ |
| 355 |
FAIL: Expected numeric digits, found end of text (at char 4), (line:1, col:5) |
| 356 |
|
| 357 |
.. _set_results_name: |
| 358 |
|
| 359 |
- ``set_results_name(string, list_all_matches=False)`` - name to be given |
| 360 |
to tokens matching |
| 361 |
the element; if multiple tokens within |
| 362 |
a repetition group (such as ZeroOrMore_ or DelimitedList_) the |
| 363 |
default is to return only the last matching token - if ``list_all_matches`` |
| 364 |
is set to True, then a list of all the matching tokens is returned. |
| 365 |
|
| 366 |
``expr.set_results_name("key")`` can also be written ``expr("key")`` |
| 367 |
(a results name with a trailing '*' character will be |
| 368 |
interpreted as setting ``list_all_matches`` to ``True``). |
| 369 |
|
| 370 |
Note: |
| 371 |
``set_results_name`` returns a *copy* of the element so that a single |
| 372 |
basic element can be referenced multiple times and given |
| 373 |
different names within a complex grammar. |
| 374 |
|
| 375 |
.. _using_each: |
| 376 |
|
| 377 |
- ``using_each(list_of_symbols)`` a short-cut for defining a number of |
| 378 |
symbols of a particular ``ParserElement`` subclass:: |
| 379 |
|
| 380 |
LBRACK, RBRACK, LBRACE, RBRACE, LPAR, RPAR = Suppress.using_each("[]{}()") |
| 381 |
AND, OR, NOT = Keyword.using_each("and or not".split()) |
| 382 |
|
| 383 |
.. _set_parse_action: |
| 384 |
|
| 385 |
- ``set_parse_action(*fn)`` - specify one or more functions to call after successful |
| 386 |
matching of the element; each function is defined as ``fn(s, loc, toks)``, where: |
| 387 |
|
| 388 |
- ``s`` is the original parse string |
| 389 |
|
| 390 |
- ``loc`` is the location in the string where matching started |
| 391 |
|
| 392 |
- ``toks`` is the list of the matched tokens, packaged as a ParseResults_ object |
| 393 |
|
| 394 |
Parse actions can have any of the following signatures:: |
| 395 |
|
| 396 |
fn(s: str, loc: int, tokens: ParseResults) |
| 397 |
fn(loc: int, tokens: ParseResults) |
| 398 |
fn(tokens: ParseResults) |
| 399 |
fn() |
| 400 |
|
| 401 |
Multiple functions can be attached to a ``ParserElement`` by specifying multiple |
| 402 |
arguments to ``set_parse_action``, or by calling ``add_parse_action``. Calls to ``set_parse_action`` |
| 403 |
will replace any previously defined parse actions. ``set_parse_action(None)`` will clear |
| 404 |
all previously defined parse actions. |
| 405 |
|
| 406 |
Each parse action function can return a modified ``toks`` list, to perform conversion, or |
| 407 |
string modifications. For brevity, ``fn`` may also be a |
| 408 |
lambda - here is an example of using a parse action to convert matched |
| 409 |
integer tokens from strings to integers:: |
| 410 |
|
| 411 |
int_number = Word(nums).set_parse_action(lambda s, l, t: [int(t[0])]) |
| 412 |
|
| 413 |
If ``fn`` modifies the ``toks`` list in-place, it does not need to return |
| 414 |
and pyparsing will use the modified ``toks`` list. |
| 415 |
|
| 416 |
If ``set_parse_action`` is called with an argument of ``None``, then this clears all parse actions |
| 417 |
attached to that expression. |
| 418 |
|
| 419 |
A nice short-cut for calling ``set_parse_action`` is to use it as a decorator:: |
| 420 |
|
| 421 |
identifier = Word(alphas, alphanums + "_") |
| 422 |
|
| 423 |
@identifier.set_parse_action |
| 424 |
def resolve_identifier(results: ParseResults): |
| 425 |
return variable_values.get(results[0]) |
| 426 |
|
| 427 |
(Posted by @MisterMiyagi in this SO answer: https://stackoverflow.com/a/63031959/165216) |
| 428 |
|
| 429 |
- ``add_parse_action`` - similar to ``set_parse_action``, but instead of replacing any |
| 430 |
previously defined parse actions, will append the given action or actions to the |
| 431 |
existing defined parse actions. |
| 432 |
|
| 433 |
- ``add_condition`` - a simplified form of ``add_parse_action`` if the purpose |
| 434 |
of the parse action is to simply do some validation, and raise an exception |
| 435 |
if the validation fails. Takes a method that takes the same arguments, |
| 436 |
but simply returns ``True`` or ``False``. If ``False`` is returned, an exception will be |
| 437 |
raised. |
| 438 |
|
| 439 |
- ``set_break(break_flag=True)`` - if ``break_flag`` is ``True``, calls ``pdb.set_break()`` |
| 440 |
as this expression is about to be parsed |
| 441 |
|
| 442 |
- ``copy()`` - returns a copy of a ``ParserElement``; can be used to use the same |
| 443 |
parse expression in different places in a grammar, with different parse actions |
| 444 |
attached to each; a short-form ``expr()`` is equivalent to ``expr.copy()`` |
| 445 |
|
| 446 |
- ``leave_whitespace()`` - change default behavior of skipping |
| 447 |
whitespace before starting matching (mostly used internally to the |
| 448 |
pyparsing module, rarely used by client code) |
| 449 |
|
| 450 |
- ``set_whitespace_chars(chars)`` - define the set of chars to be ignored |
| 451 |
as whitespace before trying to match a specific ``ParserElement``, in place of the |
| 452 |
default set of whitespace (space, tab, newline, and return) |
| 453 |
|
| 454 |
- ``set_default_whitespace_chars(chars)`` - class-level method to override |
| 455 |
the default set of whitespace chars for all subsequently created ParserElements |
| 456 |
(including copies); useful when defining grammars that treat one or more of the |
| 457 |
default whitespace characters as significant (such as a line-sensitive grammar, to |
| 458 |
omit newline from the list of ignorable whitespace) |
| 459 |
|
| 460 |
- ``suppress()`` - convenience function to suppress the output of the |
| 461 |
given element, instead of wrapping it with a ``Suppress`` object. |
| 462 |
|
| 463 |
- ``ignore(expr)`` - function to specify parse expression to be |
| 464 |
ignored while matching defined patterns; can be called |
| 465 |
repeatedly to specify multiple expressions; useful to specify |
| 466 |
patterns of comment syntax, for example |
| 467 |
|
| 468 |
- ``set_debug(flag=True)`` - function to enable/disable tracing output |
| 469 |
when trying to match this element |
| 470 |
|
| 471 |
- ``validate()`` - function to verify that the defined grammar does not |
| 472 |
contain infinitely recursive constructs. |
| 473 |
|
| 474 |
*(``validate()`` is deprecated, and |
| 475 |
will be removed in a future pyparsing release. Pyparsing now supports |
| 476 |
left-recursive parsers, which this function attempted to catch.)* |
| 477 |
|
| 478 |
.. _parse_with_tabs: |
| 479 |
|
| 480 |
- ``parse_with_tabs()`` - function to override default behavior of converting |
| 481 |
tabs to spaces before parsing the input string; rarely used, except when |
| 482 |
specifying whitespace-significant grammars using the White_ class. |
| 483 |
|
| 484 |
- ``enable_packrat()`` - a class-level static method to enable a memoizing |
| 485 |
performance enhancement, known as "packrat parsing". packrat parsing is |
| 486 |
disabled by default, since it may conflict with some user programs that use |
| 487 |
parse actions. To activate the packrat feature, your |
| 488 |
program must call the class method ``ParserElement.enable_packrat()``. For best |
| 489 |
results, call ``enable_packrat()`` immediately after importing pyparsing. |
| 490 |
|
| 491 |
- ``enable_left_recursion()`` - a class-level static method to enable |
| 492 |
pyparsing with left-recursive (LR) parsers. Similar to ``ParserElement.enable_packrat()``, |
| 493 |
your program must call the class method ``ParserElement.enable_left_recursion()`` to |
| 494 |
enable this feature. ``enable_left_recursion()`` uses a separate packrat cache, and so |
| 495 |
is incompatible with ``enable_packrat()``. |
| 496 |
|
| 497 |
Basic ParserElement subclasses |
| 498 |
------------------------------ |
| 499 |
|
| 500 |
.. _Literal: |
| 501 |
|
| 502 |
- ``Literal`` - construct with a string to be matched exactly |
| 503 |
|
| 504 |
.. _CaselessLiteral: |
| 505 |
|
| 506 |
- ``CaselessLiteral`` - construct with a string to be matched, but |
| 507 |
without case checking; results are always returned as the |
| 508 |
defining literal, NOT as they are found in the input string |
| 509 |
|
| 510 |
.. _Keyword: |
| 511 |
|
| 512 |
- ``Keyword`` - similar to Literal_, but must be immediately followed by |
| 513 |
whitespace, punctuation, or other non-keyword characters; prevents |
| 514 |
accidental matching of a non-keyword that happens to begin with a |
| 515 |
defined keyword |
| 516 |
|
| 517 |
- ``CaselessKeyword`` - similar to Keyword_, but with caseless matching |
| 518 |
behavior as described in CaselessLiteral_. |
| 519 |
|
| 520 |
.. _Word: |
| 521 |
|
| 522 |
- ``Word`` - one or more contiguous characters; construct with a |
| 523 |
string containing the set of allowed initial characters, and an |
| 524 |
optional second string of allowed body characters; for instance, |
| 525 |
a common ``Word`` construct is to match a code identifier - in C, a |
| 526 |
valid identifier must start with an alphabetic character or an |
| 527 |
underscore ('_'), followed by a body that can also include numeric |
| 528 |
digits. That is, ``a``, ``i``, ``MAX_LENGTH``, ``_a1``, ``b_109_``, and |
| 529 |
``plan9FromOuterSpace`` |
| 530 |
are all valid identifiers; ``9b7z``, ``$a``, ``.section``, and ``0debug`` |
| 531 |
are not. To |
| 532 |
define an identifier using a ``Word``, use either of the following:: |
| 533 |
|
| 534 |
Word(alphas+"_", alphanums+"_") |
| 535 |
Word(srange("[a-zA-Z_]"), srange("[a-zA-Z0-9_]")) |
| 536 |
|
| 537 |
Pyparsing also provides pre-defined strings ``identchars`` and |
| 538 |
``identbodychars`` so that you can also write:: |
| 539 |
|
| 540 |
Word(identchars, identbodychars) |
| 541 |
|
| 542 |
If only one |
| 543 |
string given, it specifies that the same character set defined |
| 544 |
for the initial character is used for the word body; for instance, to |
| 545 |
define an identifier that can only be composed of capital letters and |
| 546 |
underscores, use one of:: |
| 547 |
|
| 548 |
``Word("ABCDEFGHIJKLMNOPQRSTUVWXYZ_")`` |
| 549 |
``Word(srange("[A-Z_]"))`` |
| 550 |
|
| 551 |
A ``Word`` may |
| 552 |
also be constructed with any of the following optional parameters: |
| 553 |
|
| 554 |
- ``min`` - indicating a minimum length of matching characters |
| 555 |
|
| 556 |
- ``max`` - indicating a maximum length of matching characters |
| 557 |
|
| 558 |
- ``exact`` - indicating an exact length of matching characters; |
| 559 |
if ``exact`` is specified, it will override any values for ``min`` or ``max`` |
| 560 |
|
| 561 |
- ``as_keyword`` - indicating that preceding and following characters must |
| 562 |
be whitespace or non-keyword characters |
| 563 |
|
| 564 |
- ``exclude_chars`` - a string of characters that should be excluded from |
| 565 |
init_chars and body_chars |
| 566 |
|
| 567 |
Sometimes you want to define a word using all |
| 568 |
characters in a range except for one or two of them; you can do this |
| 569 |
with the ``exclude_chars`` argument. This is helpful if you want to define |
| 570 |
a word with all ``printables`` except for a single delimiter character, such |
| 571 |
as '.'. Previously, you would have to create a custom string to pass to Word. |
| 572 |
With this change, you can just create ``Word(printables, exclude_chars='.')``. |
| 573 |
|
| 574 |
- ``Char`` - a convenience form of ``Word`` that will match just a single character from |
| 575 |
a string of matching characters:: |
| 576 |
|
| 577 |
single_digit = Char(nums) |
| 578 |
|
| 579 |
- ``CharsNotIn`` - similar to Word_, but matches characters not |
| 580 |
in the given constructor string (accepts only one string for both |
| 581 |
initial and body characters); also supports ``min``, ``max``, and ``exact`` |
| 582 |
optional parameters. |
| 583 |
|
| 584 |
- ``Regex`` - a powerful construct, that accepts a regular expression |
| 585 |
to be matched at the current parse position; accepts an optional |
| 586 |
``flags`` parameter, corresponding to the flags parameter in the ``re.compile`` |
| 587 |
method; if the expression includes named sub-fields, they will be |
| 588 |
represented in the returned ParseResults_. |
| 589 |
|
| 590 |
- ``QuotedString`` - supports the definition of custom quoted string |
| 591 |
formats, in addition to pyparsing's built-in ``dbl_quoted_string`` and |
| 592 |
``sgl_quoted_string``. ``QuotedString`` allows you to specify the following |
| 593 |
parameters: |
| 594 |
|
| 595 |
- ``quote_char`` - string of one or more characters defining the quote delimiting string |
| 596 |
|
| 597 |
- ``esc_char`` - character to escape quotes, typically backslash (default=None) |
| 598 |
|
| 599 |
- ``esc_quote`` - special quote sequence to escape an embedded quote string (such as SQL's "" to escape an embedded ") (default=None) |
| 600 |
|
| 601 |
- ``multiline`` - boolean indicating whether quotes can span multiple lines (default=False) |
| 602 |
|
| 603 |
- ``unquote_results`` - boolean indicating whether the matched text should be unquoted (default=True) |
| 604 |
|
| 605 |
- ``end_quote_char`` - string of one or more characters defining the end of the quote delimited string (default=None => same as ``quote_char``) |
| 606 |
|
| 607 |
.. _SkipTo: |
| 608 |
|
| 609 |
- ``SkipTo`` - skips ahead in the input string, accepting any |
| 610 |
characters up to the specified pattern; may be constructed with |
| 611 |
the following optional parameters: |
| 612 |
|
| 613 |
- ``include`` - if set to true, also consumes the match expression |
| 614 |
(default is false) |
| 615 |
|
| 616 |
- ``ignore`` - allows the user to specify patterns to not be matched, |
| 617 |
to prevent false matches |
| 618 |
|
| 619 |
- ``fail_on`` - if a literal string or expression is given for this argument, it defines an expression that |
| 620 |
should cause the SkipTo_ expression to fail, and not skip over that expression |
| 621 |
|
| 622 |
``SkipTo`` can also be written using ``...``:: |
| 623 |
|
| 624 |
LBRACE, RBRACE = Literal.using_each("{}") |
| 625 |
|
| 626 |
brace_expr = LBRACE + SkipTo(RBRACE) + RBRACE |
| 627 |
# can also be written as |
| 628 |
brace_expr = LBRACE + ... + RBRACE |
| 629 |
|
| 630 |
.. _White: |
| 631 |
|
| 632 |
- ``White`` - also similar to Word_, but matches whitespace |
| 633 |
characters. Not usually needed, as whitespace is implicitly |
| 634 |
ignored by pyparsing. However, some grammars are whitespace-sensitive, |
| 635 |
such as those that use leading tabs or spaces to indicating grouping |
| 636 |
or hierarchy. (If matching on tab characters, be sure to call |
| 637 |
parse_with_tabs_ on the top-level parse element.) |
| 638 |
|
| 639 |
- ``Empty`` - a null expression, requiring no characters - will always |
| 640 |
match; useful for debugging and for specialized grammars |
| 641 |
|
| 642 |
- ``NoMatch`` - opposite of ``Empty``, will never match; useful for debugging |
| 643 |
and for specialized grammars |
| 644 |
|
| 645 |
|
| 646 |
Expression subclasses |
| 647 |
--------------------- |
| 648 |
|
| 649 |
.. _And: |
| 650 |
|
| 651 |
- ``And`` - construct with a list of ``ParserElements``, all of which must |
| 652 |
match for ``And`` to match; can also be created using the '+' |
| 653 |
operator; multiple expressions can be ``Anded`` together using the '*' |
| 654 |
operator as in:: |
| 655 |
|
| 656 |
ip_address = Word(nums) + ('.' + Word(nums)) * 3 |
| 657 |
|
| 658 |
A tuple can be used as the multiplier, indicating a min/max:: |
| 659 |
|
| 660 |
us_phone_number = Word(nums) + ('-' + Word(nums)) * (1,2) |
| 661 |
|
| 662 |
A special form of ``And`` is created if the '-' operator is used |
| 663 |
instead of the '+' operator. In the ``ip_address`` example above, if |
| 664 |
no trailing '.' and ``Word(nums)`` are found after matching the initial |
| 665 |
``Word(nums)``, then pyparsing will back up in the grammar and try other |
| 666 |
alternatives to ``ip_address``. However, if ``ip_address`` is defined as:: |
| 667 |
|
| 668 |
strict_ip_address = Word(nums) - ('.'+Word(nums))*3 |
| 669 |
|
| 670 |
then no backing up is done. If the first ``Word(nums)`` of ``strict_ip_address`` |
| 671 |
is matched, then any mismatch after that will raise a ``ParseSyntaxException``, |
| 672 |
which will halt the parsing process immediately. By careful use of the |
| 673 |
'-' operator, grammars can provide meaningful error messages close to |
| 674 |
the location where the incoming text does not match the specified |
| 675 |
grammar. |
| 676 |
|
| 677 |
.. _Or: |
| 678 |
|
| 679 |
- ``Or`` - construct with a list of ``ParserElements``, any of which must |
| 680 |
match for ``Or`` to match; if more than one expression matches, the |
| 681 |
expression that makes the longest match will be used; can also |
| 682 |
be created using the '^' operator |
| 683 |
|
| 684 |
.. _MatchFirst: |
| 685 |
|
| 686 |
- ``MatchFirst`` - construct with a list of ``ParserElements``, any of |
| 687 |
which must match for ``MatchFirst`` to match; matching is done |
| 688 |
left-to-right, taking the first expression that matches; can |
| 689 |
also be created using the '|' operator |
| 690 |
|
| 691 |
.. _Each: |
| 692 |
|
| 693 |
- ``Each`` - similar to And_, in that all of the provided expressions |
| 694 |
must match; however, ``Each`` permits matching to be done in any order; |
| 695 |
can also be created using the '&' operator |
| 696 |
|
| 697 |
- ``Opt`` - construct with a ``ParserElement``, but this element is |
| 698 |
not required to match; can be constructed with an optional ``default`` argument, |
| 699 |
containing a default string or object to be supplied if the given optional |
| 700 |
parse element is not found in the input string; parse action will only |
| 701 |
be called if a match is found, or if a default is specified. |
| 702 |
|
| 703 |
An optional element ``expr`` can also be expressed using ``expr | ""``. |
| 704 |
|
| 705 |
(``Opt`` was formerly named ``Optional``, but since the standard Python |
| 706 |
library module ``typing`` now defines ``Optional``, the pyparsing class has |
| 707 |
been renamed to ``Opt``. A compatibility synonym ``Optional`` is defined, |
| 708 |
but will be removed in a future release.) |
| 709 |
|
| 710 |
.. _ZeroOrMore: |
| 711 |
|
| 712 |
- ``ZeroOrMore`` - similar to ``Opt``, but can be repeated; ``ZeroOrMore(expr)`` |
| 713 |
can also be written as ``expr[...]``. |
| 714 |
|
| 715 |
.. _OneOrMore: |
| 716 |
|
| 717 |
- ``OneOrMore`` - similar to ZeroOrMore_, but at least one match must |
| 718 |
be present; ``OneOrMore(expr)`` can also be written as ``expr[1, ...]``. |
| 719 |
|
| 720 |
.. _DelimitedList: |
| 721 |
|
| 722 |
- ``DelimitedList`` - used for |
| 723 |
matching one or more occurrences of ``expr``, separated by ``delim``. |
| 724 |
By default, the delimiters are suppressed, so the returned results contain |
| 725 |
only the separate list elements. Can optionally specify ``combine=True``, |
| 726 |
indicating that the expressions and delimiters should be returned as one |
| 727 |
combined value (useful for scoped variables, such as ``"a.b.c"``, or |
| 728 |
``"a::b::c"``, or paths such as ``"a/b/c"``). Can also optionally specify ``min` and ``max`` |
| 729 |
restrictions on the length of the list, and |
| 730 |
``allow_trailing_delim`` to accept a trailing delimiter at the end of the list. |
| 731 |
|
| 732 |
.. _FollowedBy: |
| 733 |
|
| 734 |
- ``FollowedBy`` - a lookahead expression, requires matching of the given |
| 735 |
expressions, but does not advance the parsing position within the input string |
| 736 |
|
| 737 |
.. _NotAny: |
| 738 |
|
| 739 |
- ``NotAny`` - a negative lookahead expression, prevents matching of named |
| 740 |
expressions, does not advance the parsing position within the input string; |
| 741 |
can also be created using the unary '~' operator |
| 742 |
|
| 743 |
|
| 744 |
.. _operators: |
| 745 |
|
| 746 |
Expression operators |
| 747 |
-------------------- |
| 748 |
|
| 749 |
- ``+`` - creates And_ using the expressions before and after the operator |
| 750 |
|
| 751 |
- ``|`` - creates MatchFirst_ (first left-to-right match) using the expressions before and after the operator |
| 752 |
|
| 753 |
- ``^`` - creates Or_ (longest match) using the expressions before and after the operator |
| 754 |
|
| 755 |
- ``&`` - creates Each_ using the expressions before and after the operator |
| 756 |
|
| 757 |
- ``*`` - creates And_ by multiplying the expression by the integer operand; if |
| 758 |
expression is multiplied by a 2-tuple, creates an And_ of ``(min,max)`` |
| 759 |
expressions (similar to ``{min,max}`` form in regular expressions); if |
| 760 |
``min`` is ``None``, interpret as ``(0,max)``; if ``max`` is ``None``, interpret as |
| 761 |
``expr*min + ZeroOrMore(expr)`` |
| 762 |
|
| 763 |
- ``-`` - like ``+`` but with no backup and retry of alternatives |
| 764 |
|
| 765 |
- ``~`` - creates NotAny_ using the expression after the operator |
| 766 |
|
| 767 |
- ``==`` - matching expression to string; returns ``True`` if the string matches the given expression |
| 768 |
|
| 769 |
- ``<<=`` - inserts the expression following the operator as the body of the |
| 770 |
``Forward`` expression before the operator (``<<`` can also be used, but ``<<=`` is preferred |
| 771 |
to avoid operator precedence misinterpretation of the pyparsing expression) |
| 772 |
|
| 773 |
- ``...`` - inserts a SkipTo_ expression leading to the next expression, as in |
| 774 |
``Keyword("start") + ... + Keyword("end")``. |
| 775 |
|
| 776 |
- ``[min, max]`` - specifies repetition similar to ``*`` with ``min`` and ``max`` specified |
| 777 |
as the minimum and maximum number of repetitions. ``...`` can be used in place of ``None``. |
| 778 |
For example ``expr[...]`` is equivalent to ``ZeroOrMore(expr)``, ``expr[1, ...]`` is |
| 779 |
equivalent to ``OneOrMore(expr)``, and ``expr[..., 3]`` is equivalent to "up to 3 instances |
| 780 |
of ``expr``". |
| 781 |
|
| 782 |
- ``[:stop_on]`` - specifies a stopping expression for the current repetition (may be combined |
| 783 |
with ``...`` or ``min, max``), as in ``Keyword("start") + Word(alphas)[...:Keyword("end")] + Keyword("end")`` |
| 784 |
|
| 785 |
Positional subclasses |
| 786 |
--------------------- |
| 787 |
|
| 788 |
- ``StringStart`` - matches beginning of the text |
| 789 |
|
| 790 |
- ``StringEnd`` - matches the end of the text |
| 791 |
|
| 792 |
- ``LineStart`` - matches beginning of a line (lines delimited by ``\n`` characters) |
| 793 |
|
| 794 |
- ``LineEnd`` - matches the end of a line |
| 795 |
|
| 796 |
- ``WordStart`` - matches a leading word boundary |
| 797 |
|
| 798 |
- ``WordEnd`` - matches a trailing word boundary |
| 799 |
|
| 800 |
|
| 801 |
|
| 802 |
Converter subclasses |
| 803 |
-------------------- |
| 804 |
|
| 805 |
- ``Combine`` - joins all matched tokens into a single string, using |
| 806 |
specified ``join_string`` (default ``join_string=""``); expects |
| 807 |
all matching tokens to be adjacent, with no intervening |
| 808 |
whitespace (can be overridden by specifying ``adjacent=False`` in constructor) |
| 809 |
|
| 810 |
- ``Suppress`` - clears matched tokens; useful to keep returned |
| 811 |
results from being cluttered with required but uninteresting |
| 812 |
tokens (such as list delimiters) |
| 813 |
|
| 814 |
|
| 815 |
Special subclasses |
| 816 |
------------------ |
| 817 |
|
| 818 |
- ``Group`` - causes the matched tokens to be enclosed in a list; |
| 819 |
useful in repeated elements like ZeroOrMore_ and OneOrMore_ to |
| 820 |
break up matched tokens into groups for each repeated pattern |
| 821 |
|
| 822 |
- ``Dict`` - like ``Group``, but also constructs a dictionary, using the |
| 823 |
``[0]``'th elements of all enclosed token lists as the keys, and |
| 824 |
each token list as the value |
| 825 |
|
| 826 |
- ``Forward`` - placeholder token used to define recursive token |
| 827 |
patterns; when defining the actual expression later in the |
| 828 |
program, insert it into the ``Forward`` object using the ``<<=`` |
| 829 |
operator (see fourFn.py_ for an example). |
| 830 |
|
| 831 |
- ``Tag`` - a non-parsing token that always matches, and inserts |
| 832 |
a tag and value into the current parsed tokens; useful for adding |
| 833 |
metadata or annotations to parsed results (see `examples/tag_example.py <../examples/tag_example.py>`_). |
| 834 |
|
| 835 |
|
| 836 |
Other classes |
| 837 |
------------- |
| 838 |
.. _ParseResults: |
| 839 |
|
| 840 |
- ``ParseResults`` - class used to contain and manage the lists of tokens |
| 841 |
created from parsing the input using the user-defined parse |
| 842 |
expression. ``ParseResults`` can be accessed in a number of ways: |
| 843 |
|
| 844 |
- as a list |
| 845 |
|
| 846 |
- total list of elements can be found using ``len()`` |
| 847 |
|
| 848 |
- individual elements can be found using ``[0], [1], [-1],`` etc., |
| 849 |
or retrieved using slices |
| 850 |
|
| 851 |
- elements can be deleted using ``del`` |
| 852 |
|
| 853 |
- the last element can be extracted and removed in a single operation |
| 854 |
using ``pop()``, or any element can be extracted and removed |
| 855 |
using ``pop(n)`` |
| 856 |
|
| 857 |
- a nested ParseResults_ can be created by using the pyparsing ``Group`` class |
| 858 |
around elements in an expression:: |
| 859 |
|
| 860 |
Word(alphas) + Group(Word(nums)[...]) + Word(alphas) |
| 861 |
|
| 862 |
will parse the string "abc 100 200 300 end" as:: |
| 863 |
|
| 864 |
['abc', ['100', '200', '300'], 'end'] |
| 865 |
|
| 866 |
If the ``Group`` is constructed using ``aslist=True``, the resulting tokens |
| 867 |
will be a Python list instead of a ParseResults_. In this case, the returned value will |
| 868 |
no longer support the extended features or methods of a ParseResults_. |
| 869 |
|
| 870 |
- as a dictionary |
| 871 |
|
| 872 |
- if ``set_results_name()`` is used to name elements within the |
| 873 |
overall parse expression, then these fields can be referenced |
| 874 |
as dictionary elements or as attributes |
| 875 |
|
| 876 |
- the ``Dict`` class generates dictionary entries using the data of the |
| 877 |
input text - in addition to ParseResults_ listed as ``[ [ a1, b1, c1, ...], [ a2, b2, c2, ...] ]`` |
| 878 |
it also acts as a dictionary with entries defined as ``{ a1 : [ b1, c1, ... ] }, { a2 : [ b2, c2, ... ] }``; |
| 879 |
this is especially useful when processing tabular data where the first column contains a key |
| 880 |
value for that line of data; when constructed with ``asdict=True``, will |
| 881 |
return an actual Python ``dict`` instead of a ParseResults_. In this case, the returned value will |
| 882 |
no longer support the extended features or methods of a ParseResults_. |
| 883 |
|
| 884 |
- list elements that are deleted using ``del`` will still be accessible by their |
| 885 |
dictionary keys |
| 886 |
|
| 887 |
- supports ``get()``, ``items()`` and ``keys()`` methods, similar to a dictionary |
| 888 |
|
| 889 |
- a keyed item can be extracted and removed using ``pop(key)``. Here |
| 890 |
``key`` must be non-numeric (such as a string), in order to use dict |
| 891 |
extraction instead of list extraction. |
| 892 |
|
| 893 |
- new named elements can be added (in a parse action, for instance), using the same |
| 894 |
syntax as adding an item to a dict (``parse_results["X"] = "new item"``); |
| 895 |
named elements can be removed using ``del parse_results["X"]`` |
| 896 |
|
| 897 |
- as a nested list |
| 898 |
|
| 899 |
- results returned from the Group class are encapsulated within their |
| 900 |
own list structure, so that the tokens can be handled as a hierarchical |
| 901 |
tree |
| 902 |
|
| 903 |
- as an object |
| 904 |
|
| 905 |
- named elements can be accessed as if they were attributes of an object: |
| 906 |
if an element is referenced that does not exist, it will return ``""``. |
| 907 |
|
| 908 |
ParseResults_ can also be converted to an ordinary list of strings |
| 909 |
by calling ``as_list()``. Note that this will strip the results of any |
| 910 |
field names that have been defined for any embedded parse elements. |
| 911 |
(The ``pprint`` module is especially good at printing out the nested contents |
| 912 |
given by ``as_list()``.) |
| 913 |
|
| 914 |
If a ParseResults_ is built with expressions that use results names (see _set_results_name) or |
| 915 |
using the ``Dict`` class, then those names and values can be extracted as a Python |
| 916 |
dict using ``as_dict()``. |
| 917 |
|
| 918 |
Finally, ParseResults_ can be viewed by calling ``dump()``. ``dump()`` will first show |
| 919 |
the ``as_list()`` output, followed by an indented structure listing parsed tokens that |
| 920 |
have been assigned results names. |
| 921 |
|
| 922 |
Here is sample code illustrating some of these methods:: |
| 923 |
|
| 924 |
>>> number = Word(nums) |
| 925 |
>>> name = Combine(Word(alphas)[...], adjacent=False, join_string=" ") |
| 926 |
>>> parser = number("house_number") + name("street_name") |
| 927 |
>>> result = parser.parse_string("123 Main St") |
| 928 |
>>> print(result) |
| 929 |
['123', 'Main St'] |
| 930 |
>>> print(type(result)) |
| 931 |
<class 'pyparsing.ParseResults'> |
| 932 |
>>> print(repr(result)) |
| 933 |
(['123', 'Main St'], {'house_number': ['123'], 'street_name': ['Main St']}) |
| 934 |
>>> result.house_number |
| 935 |
'123' |
| 936 |
>>> result["street_name"] |
| 937 |
'Main St' |
| 938 |
>>> result.as_list() |
| 939 |
['123', 'Main St'] |
| 940 |
>>> result.as_dict() |
| 941 |
{'house_number': '123', 'street_name': 'Main St'} |
| 942 |
>>> print(result.dump()) |
| 943 |
['123', 'Main St'] |
| 944 |
- house_number: '123' |
| 945 |
- street_name: 'Main St' |
| 946 |
|
| 947 |
|
| 948 |
Exception classes and Troubleshooting |
| 949 |
------------------------------------- |
| 950 |
|
| 951 |
.. _ParseException: |
| 952 |
|
| 953 |
- ``ParseException`` - exception returned when a grammar parse fails; |
| 954 |
``ParseExceptions`` have attributes ``loc``, ``msg``, ``line``, ``lineno``, and ``column``; to view the |
| 955 |
text line and location where the reported ParseException occurs, use:: |
| 956 |
|
| 957 |
except ParseException as err: |
| 958 |
print(err.line) |
| 959 |
print(" " * (err.column - 1) + "^") |
| 960 |
print(err) |
| 961 |
|
| 962 |
``ParseExceptions`` also have an ``explain()`` method that gives this same information:: |
| 963 |
|
| 964 |
except ParseException as err: |
| 965 |
print(err.explain()) |
| 966 |
|
| 967 |
- ``RecursiveGrammarException`` - exception returned by ``validate()`` if |
| 968 |
the grammar contains a recursive infinite loop, such as:: |
| 969 |
|
| 970 |
bad_grammar = Forward() |
| 971 |
good_token = Literal("A") |
| 972 |
bad_grammar <<= Opt(good_token) + bad_grammar |
| 973 |
|
| 974 |
- ``ParseFatalException`` - exception that parse actions can raise to stop parsing |
| 975 |
immediately. Should be used when a semantic error is found in the input text, such |
| 976 |
as a mismatched XML tag. |
| 977 |
|
| 978 |
- ``ParseSyntaxException`` - subclass of ``ParseFatalException`` raised when a |
| 979 |
syntax error is found, based on the use of the '-' operator when defining |
| 980 |
a sequence of expressions in an And_ expression. |
| 981 |
|
| 982 |
- You can also get some insights into the parsing logic using diagnostic parse actions, |
| 983 |
and ``set_debug()``, or test the matching of expression fragments by testing them using |
| 984 |
``search_string()`` or ``scan_string()``. |
| 985 |
|
| 986 |
- Use ``with_line_numbers`` from ``pyparsing_testing`` to display the input string |
| 987 |
being parsed, with line and column numbers that correspond to the values reported |
| 988 |
in set_debug() output:: |
| 989 |
|
| 990 |
import pyparsing as pp |
| 991 |
ppt = pp.testing |
| 992 |
|
| 993 |
data = """\ |
| 994 |
A |
| 995 |
100""" |
| 996 |
|
| 997 |
expr = pp.Word(pp.alphanums).set_name("word").set_debug() |
| 998 |
print(ppt.with_line_numbers(data)) |
| 999 |
expr[...].parse_string(data) |
| 1000 |
|
| 1001 |
prints:: |
| 1002 |
|
| 1003 |
. 1 |
| 1004 |
1234567890 |
| 1005 |
1: A| |
| 1006 |
2: 100| |
| 1007 |
|
| 1008 |
Match word at loc 3(1,4) |
| 1009 |
A |
| 1010 |
^ |
| 1011 |
Matched word -> ['A'] |
| 1012 |
Match word at loc 11(2,7) |
| 1013 |
100 |
| 1014 |
^ |
| 1015 |
Matched word -> ['100'] |
| 1016 |
|
| 1017 |
`with_line_numbers` has several options for displaying control characters, end-of-line |
| 1018 |
and space markers, Unicode symbols for control characters - these are documented in the |
| 1019 |
function's docstring. |
| 1020 |
|
| 1021 |
- Diagnostics can be enabled using ``pyparsing.enable_diag`` and passing |
| 1022 |
one of the following enum values defined in ``pyparsing.Diagnostics`` |
| 1023 |
|
| 1024 |
- ``warn_multiple_tokens_in_named_alternation`` - flag to enable warnings when a results |
| 1025 |
name is defined on a MatchFirst_ or Or_ expression with one or more And_ subexpressions |
| 1026 |
|
| 1027 |
- ``warn_ungrouped_named_tokens_in_collection`` - flag to enable warnings when a results |
| 1028 |
name is defined on a containing expression with ungrouped subexpressions that also |
| 1029 |
have results names |
| 1030 |
|
| 1031 |
- ``warn_name_set_on_empty_Forward`` - flag to enable warnings when a ``Forward`` is defined |
| 1032 |
with a results name, but has no contents defined |
| 1033 |
|
| 1034 |
- ``warn_on_parse_using_empty_Forward`` - flag to enable warnings when a ``Forward`` is |
| 1035 |
defined in a grammar but has never had an expression attached to it |
| 1036 |
|
| 1037 |
- ``warn_on_assignment_to_Forward`` - flag to enable warnings when a ``Forward`` is defined |
| 1038 |
but is overwritten by assigning using ``'='`` instead of ``'<<='`` or ``'<<'`` |
| 1039 |
|
| 1040 |
- ``warn_on_multiple_string_args_to_oneof`` - flag to enable warnings when ``one_of`` is |
| 1041 |
incorrectly called with multiple str arguments |
| 1042 |
|
| 1043 |
- ``enable_debug_on_named_expressions`` - flag to auto-enable debug on all subsequent |
| 1044 |
calls to ``ParserElement.set_name`` |
| 1045 |
|
| 1046 |
All warnings can be enabled by calling ``pyparsing.enable_all_warnings()``. |
| 1047 |
Sample:: |
| 1048 |
|
| 1049 |
import pyparsing as pp |
| 1050 |
pp.enable_all_warnings() |
| 1051 |
|
| 1052 |
fwd = pp.Forward().set_results_name("recursive_expr") |
| 1053 |
|
| 1054 |
>>> UserWarning: warn_name_set_on_empty_Forward: setting results name 'recursive_expr' |
| 1055 |
on Forward expression that has no contained expression |
| 1056 |
|
| 1057 |
Warnings can also be enabled using the Python ``-W`` switch (using ``-Wd`` or |
| 1058 |
``-Wd:::pyparsing``) or setting a non-empty value to the environment variable |
| 1059 |
``PYPARSINGENABLEALLWARNINGS``. (If using ``-Wd`` for testing, but wishing to |
| 1060 |
disable pyparsing warnings, add ``-Wi:::pyparsing``.) |
| 1061 |
|
| 1062 |
|
| 1063 |
Miscellaneous attributes and methods |
| 1064 |
==================================== |
| 1065 |
|
| 1066 |
Helper methods |
| 1067 |
-------------- |
| 1068 |
|
| 1069 |
- ``counted_array(expr)`` - convenience function for a pattern where an list of |
| 1070 |
instances of the given expression are preceded by an integer giving the count of |
| 1071 |
elements in the list. Returns an expression that parses the leading integer, |
| 1072 |
reads exactly that many expressions, and returns the array of expressions in the |
| 1073 |
parse results - the leading integer is suppressed from the results (although it |
| 1074 |
is easily reconstructed by using len on the returned array). |
| 1075 |
|
| 1076 |
- ``one_of(choices, caseless=False, as_keyword=False)`` - convenience function for quickly declaring an |
| 1077 |
alternative set of Literal_ expressions. ``choices`` can be passed as a list of strings |
| 1078 |
or as a single string of values separated by spaces. The values are sorted so that longer |
| 1079 |
matches are attempted first; this ensures that a short value does |
| 1080 |
not mask a longer one that starts with the same characters. If ``caseless=True``, |
| 1081 |
will create an alternative set of CaselessLiteral_ tokens. If ``as_keyword=True``, |
| 1082 |
``one_of`` will declare Keyword_ expressions instead of Literal_ expressions. |
| 1083 |
|
| 1084 |
- ``dict_of(key, value)`` - convenience function for quickly declaring a |
| 1085 |
dictionary pattern of ``Dict(ZeroOrMore(Group(key + value)))``. |
| 1086 |
|
| 1087 |
- ``make_html_tags(tag_str)`` and ``make_xml_tags(tag_str)`` - convenience |
| 1088 |
functions to create definitions of opening and closing tag expressions. Returns |
| 1089 |
a pair of expressions, for the corresponding ``<tag>`` and ``</tag>`` strings. Includes |
| 1090 |
support for attributes in the opening tag, such as ``<tag attr1="abc">`` - attributes |
| 1091 |
are returned as named results in the returned ParseResults_. ``make_html_tags`` is less |
| 1092 |
restrictive than ``make_xml_tags``, especially with respect to case sensitivity. |
| 1093 |
|
| 1094 |
- ``infix_notation(base_operand, operator_list)`` - |
| 1095 |
convenience function to define a grammar for parsing infix notation |
| 1096 |
expressions with a hierarchical precedence of operators. To use the ``infix_notation`` |
| 1097 |
helper: |
| 1098 |
|
| 1099 |
1. Define the base "atom" operand term of the grammar. |
| 1100 |
For this simple grammar, the smallest operand is either |
| 1101 |
an integer or a variable. This will be the first argument |
| 1102 |
to the ``infix_notation`` method. |
| 1103 |
|
| 1104 |
2. Define a list of tuples for each level of operator |
| 1105 |
precedence. Each tuple is of the form |
| 1106 |
``(operand_expr, num_operands, right_left_assoc, parse_action)``, where: |
| 1107 |
|
| 1108 |
- ``operand_expr`` - the pyparsing expression for the operator; |
| 1109 |
may also be a string, which will be converted to a Literal_; if |
| 1110 |
``None``, indicates an empty operator, such as the implied |
| 1111 |
multiplication operation between 'm' and 'x' in "y = mx + b". |
| 1112 |
|
| 1113 |
- ``num_operands`` - the number of terms for this operator (must |
| 1114 |
be 1, 2, or 3) |
| 1115 |
|
| 1116 |
- ``right_left_assoc`` is the indicator whether the operator is |
| 1117 |
right or left associative, using the pyparsing-defined |
| 1118 |
constants ``OpAssoc.RIGHT`` and ``OpAssoc.LEFT``. |
| 1119 |
|
| 1120 |
- ``parse_action`` is the parse action to be associated with |
| 1121 |
expressions matching this operator expression (the |
| 1122 |
``parse_action`` tuple member may be omitted) |
| 1123 |
|
| 1124 |
3. Call ``infix_notation`` passing the operand expression and |
| 1125 |
the operator precedence list, and save the returned value |
| 1126 |
as the generated pyparsing expression. You can then use |
| 1127 |
this expression to parse input strings, or incorporate it |
| 1128 |
into a larger, more complex grammar. |
| 1129 |
|
| 1130 |
Here is an ``infix_notation`` definition for 4-function arithmetic, |
| 1131 |
taking numbers or variables as operands. The order of definition of |
| 1132 |
the operators follows the standard precedence of operations for |
| 1133 |
arithmetic:: |
| 1134 |
|
| 1135 |
number = pp.common.number() |
| 1136 |
variable = pp.common.identifier() |
| 1137 |
arithmetic_expression = pp.infix_notation( |
| 1138 |
integer | variable, |
| 1139 |
[ |
| 1140 |
("-", 1, pp.OpAssoc.RIGHT), |
| 1141 |
(pp.one_of("* /"), 2, pp.OpAssoc.LEFT), |
| 1142 |
(pp.one_of("+ -"), 2, pp.OpAssoc.LEFT), |
| 1143 |
] |
| 1144 |
) |
| 1145 |
|
| 1146 |
``infix_notation`` also supports optional arguments ``lpar`` and ``rpar``, to |
| 1147 |
parse groups with symbols other than "(" and ")". They may be passed as strings |
| 1148 |
(in which case they will be converted to ``Suppress`` objects, and suppressed from |
| 1149 |
the parsed results), or passed as pyparsing expressions, in which case they will |
| 1150 |
be kept as-is, and grouped with their contents. |
| 1151 |
|
| 1152 |
For instance, to use "<" and ">" for grouping symbols, you could write:: |
| 1153 |
|
| 1154 |
expr = infix_notation(int_expr, |
| 1155 |
[ |
| 1156 |
(one_of("+ -"), 2, OpAssoc.LEFT), |
| 1157 |
], |
| 1158 |
lpar="<", |
| 1159 |
rpar=">" |
| 1160 |
) |
| 1161 |
expr.parse_string("3 - <2 + 11>") |
| 1162 |
|
| 1163 |
returning:: |
| 1164 |
|
| 1165 |
[3, '-', [2, '+', 11]] |
| 1166 |
|
| 1167 |
If the grouping symbols are to be retained, then pass them as pyparsing ``Literals``:: |
| 1168 |
|
| 1169 |
expr = infix_notation(int_expr, |
| 1170 |
[ |
| 1171 |
(one_of("+ -"), 2, OpAssoc.LEFT), |
| 1172 |
], |
| 1173 |
lpar=Literal("<"), |
| 1174 |
rpar=Literal(">") |
| 1175 |
) |
| 1176 |
expr.parse_string("3 - <2 + 11>") |
| 1177 |
|
| 1178 |
returning:: |
| 1179 |
|
| 1180 |
[3, '-', ['<', [2, '+', 11], '>']] |
| 1181 |
|
| 1182 |
- ``match_previous_literal`` and ``match_previous_expr`` - function to define an |
| 1183 |
expression that matches the same content |
| 1184 |
as was parsed in a previous parse expression. For instance:: |
| 1185 |
|
| 1186 |
first = Word(nums) |
| 1187 |
match_expr = first + ":" + match_previous_literal(first) |
| 1188 |
|
| 1189 |
will match "1:1", but not "1:2". Since this matches at the literal |
| 1190 |
level, this will also match the leading "1:1" in "1:10". |
| 1191 |
|
| 1192 |
In contrast:: |
| 1193 |
|
| 1194 |
first = Word(nums) |
| 1195 |
match_expr = first + ":" + match_previous_expr(first) |
| 1196 |
|
| 1197 |
will *not* match the leading "1:1" in "1:10"; the expressions are |
| 1198 |
evaluated first, and then compared, so "1" is compared with "10". |
| 1199 |
|
| 1200 |
- ``nested_expr(opener, closer, content=None, ignore_expr=quoted_string)`` - method for defining nested |
| 1201 |
lists enclosed in opening and closing delimiters. |
| 1202 |
|
| 1203 |
- ``opener`` - opening character for a nested list (default="("); can also be a pyparsing expression |
| 1204 |
|
| 1205 |
- ``closer`` - closing character for a nested list (default=")"); can also be a pyparsing expression |
| 1206 |
|
| 1207 |
- ``content`` - expression for items within the nested lists (default=None) |
| 1208 |
|
| 1209 |
- ``ignore_expr`` - expression for ignoring opening and closing delimiters (default=``quoted_string``) |
| 1210 |
|
| 1211 |
If an expression is not provided for the content argument, the nested |
| 1212 |
expression will capture all whitespace-delimited content between delimiters |
| 1213 |
as a list of separate values. |
| 1214 |
|
| 1215 |
Use the ``ignore_expr`` argument to define expressions that may contain |
| 1216 |
opening or closing characters that should not be treated as opening |
| 1217 |
or closing characters for nesting, such as ``quoted_string`` or a comment |
| 1218 |
expression. Specify multiple expressions using an Or_ or MatchFirst_. |
| 1219 |
The default is ``quoted_string``, but if no expressions are to be ignored, |
| 1220 |
then pass ``None`` for this argument. |
| 1221 |
|
| 1222 |
|
| 1223 |
- ``IndentedBlock(statement_expr, recursive=False, grouped=True)`` - |
| 1224 |
function to define an indented block of statements, similar to |
| 1225 |
indentation-based blocking in Python source code: |
| 1226 |
|
| 1227 |
- ``statement_expr`` - the expression defining a statement that |
| 1228 |
will be found in the indented block; a valid ``IndentedBlock`` |
| 1229 |
must contain at least 1 matching ``statement_expr`` |
| 1230 |
|
| 1231 |
- ``recursive`` - flag indicating whether the IndentedBlock can |
| 1232 |
itself contain nested sub-blocks of the same type of expression |
| 1233 |
(default=False) |
| 1234 |
|
| 1235 |
- ``grouped`` - flag indicating whether the tokens returned from |
| 1236 |
parsing the IndentedBlock should be grouped (default=True) |
| 1237 |
|
| 1238 |
.. _originalTextFor: |
| 1239 |
|
| 1240 |
- ``original_text_for(expr)`` - helper function to preserve the originally parsed text, regardless of any |
| 1241 |
token processing or conversion done by the contained expression. For instance, the following expression:: |
| 1242 |
|
| 1243 |
full_name = Word(alphas) + Word(alphas) |
| 1244 |
|
| 1245 |
will return the parse of "John Smith" as ['John', 'Smith']. In some applications, the actual name as it |
| 1246 |
was given in the input string is what is desired. To do this, use ``original_text_for``:: |
| 1247 |
|
| 1248 |
full_name = original_text_for(Word(alphas) + Word(alphas)) |
| 1249 |
|
| 1250 |
- ``ungroup(expr)`` - function to "ungroup" returned tokens; useful |
| 1251 |
to undo the default behavior of And_ to always group the returned tokens, even |
| 1252 |
if there is only one in the list. |
| 1253 |
|
| 1254 |
- ``lineno(loc, string)`` - function to give the line number of the |
| 1255 |
location within the string; the first line is line 1, newlines |
| 1256 |
start new rows |
| 1257 |
|
| 1258 |
- ``col(loc, string)`` - function to give the column number of the |
| 1259 |
location within the string; the first column is column 1, |
| 1260 |
newlines reset the column number to 1 |
| 1261 |
|
| 1262 |
- ``line(loc, string)`` - function to retrieve the line of text |
| 1263 |
representing ``lineno(loc, string)``; useful when printing out diagnostic |
| 1264 |
messages for exceptions |
| 1265 |
|
| 1266 |
- ``srange(range_spec)`` - function to define a string of characters, |
| 1267 |
given a string of the form used by regexp string ranges, such as ``"[0-9]"`` for |
| 1268 |
all numeric digits, ``"[A-Z_]"`` for uppercase characters plus underscore, and |
| 1269 |
so on (note that ``range_spec`` does not include support for generic regular |
| 1270 |
expressions, just string range specs) |
| 1271 |
|
| 1272 |
- ``trace_parse_action(fn)`` - decorator function to debug parse actions. Lists |
| 1273 |
each call, called arguments, and return value or exception |
| 1274 |
|
| 1275 |
|
| 1276 |
|
| 1277 |
Helper parse actions |
| 1278 |
-------------------- |
| 1279 |
|
| 1280 |
- ``remove_quotes`` - removes the first and last characters of a quoted string; |
| 1281 |
useful to remove the delimiting quotes from quoted strings |
| 1282 |
|
| 1283 |
- ``replace_with(repl_string)`` - returns a parse action that simply returns the |
| 1284 |
``repl_string``; useful when using ``transform_string``, or converting HTML entities, as in:: |
| 1285 |
|
| 1286 |
nbsp = Literal(" ").set_parse_action(replace_with("<BLANK>")) |
| 1287 |
|
| 1288 |
- ``original_text_for``- restores any internal whitespace or suppressed |
| 1289 |
text within the tokens for a matched parse |
| 1290 |
expression. This is especially useful when defining expressions |
| 1291 |
for ``scan_string`` or ``transform_string`` applications. |
| 1292 |
|
| 1293 |
- ``with_attribute(*args, **kwargs)`` - helper to create a validating parse action to be used with start tags created |
| 1294 |
with ``make_xml_tags`` or ``make_html_tags``. Use ``with_attribute`` to qualify a starting tag |
| 1295 |
with a required attribute value, to avoid false matches on common tags such as |
| 1296 |
``<TD>`` or ``<DIV>``. |
| 1297 |
|
| 1298 |
``with_attribute`` can be called with: |
| 1299 |
|
| 1300 |
- keyword arguments, as in ``(class="Customer", align="right")``, or |
| 1301 |
|
| 1302 |
- a list of name-value tuples, as in ``(("ns1:class", "Customer"), ("ns2:align", "right"))`` |
| 1303 |
|
| 1304 |
An attribute can be specified to have the special value |
| 1305 |
``with_attribute.ANY_VALUE``, which will match any value - use this to |
| 1306 |
ensure that an attribute is present but any attribute value is |
| 1307 |
acceptable. |
| 1308 |
|
| 1309 |
- ``match_only_at_col(column_number)`` - a parse action that verifies that |
| 1310 |
an expression was matched at a particular column, raising a |
| 1311 |
``ParseException`` if matching at a different column number; useful when parsing |
| 1312 |
tabular data |
| 1313 |
|
| 1314 |
- ``common.convert_to_integer()`` - converts all matched tokens to int |
| 1315 |
|
| 1316 |
- ``common.convert_to_float()`` - converts all matched tokens to float |
| 1317 |
|
| 1318 |
- ``common.convert_to_date()`` - converts matched token to a datetime.date |
| 1319 |
|
| 1320 |
- ``common.convert_to_datetime()`` - converts matched token to a datetime.datetime |
| 1321 |
|
| 1322 |
- ``common.strip_html_tags()`` - removes HTML tags from matched token |
| 1323 |
|
| 1324 |
- ``common.downcase_tokens()`` - converts all matched tokens to lowercase |
| 1325 |
|
| 1326 |
- ``common.upcase_tokens()`` - converts all matched tokens to uppercase |
| 1327 |
|
| 1328 |
|
| 1329 |
Common string and token constants |
| 1330 |
--------------------------------- |
| 1331 |
|
| 1332 |
- ``alphas`` - same as ``string.ascii_letters`` |
| 1333 |
|
| 1334 |
- ``nums`` - same as ``string.digits`` |
| 1335 |
|
| 1336 |
- ``alphanums`` - a string containing ``alphas + nums`` |
| 1337 |
|
| 1338 |
- ``alphas8bit`` - a string containing alphabetic 8-bit characters:: |
| 1339 |
|
| 1340 |
ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþ |
| 1341 |
|
| 1342 |
.. _identchars: |
| 1343 |
|
| 1344 |
- ``identchars`` - a string containing characters that are valid as initial identifier characters:: |
| 1345 |
|
| 1346 |
ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyzª |
| 1347 |
µºÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ |
| 1348 |
|
| 1349 |
- ``identbodychars`` - a string containing characters that are valid as identifier body characters (those following a |
| 1350 |
valid leading identifier character as given in identchars_):: |
| 1351 |
|
| 1352 |
0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyzª |
| 1353 |
µ·ºÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ |
| 1354 |
|
| 1355 |
- ``printables`` - same as ``string.printable``, minus the space (``' '``) character |
| 1356 |
|
| 1357 |
- ``empty`` - a global ``Empty()``; will always match |
| 1358 |
|
| 1359 |
- ``sgl_quoted_string`` - a string of characters enclosed in 's; may |
| 1360 |
include whitespace, but not newlines |
| 1361 |
|
| 1362 |
- ``dbl_quoted_string`` - a string of characters enclosed in "s; may |
| 1363 |
include whitespace, but not newlines |
| 1364 |
|
| 1365 |
- ``quoted_string`` - ``sgl_quoted_string | dbl_quoted_string`` |
| 1366 |
|
| 1367 |
- ``python_quoted_string`` - ``quoted_string | multiline quoted string`` |
| 1368 |
|
| 1369 |
- ``c_style_comment`` - a comment block delimited by ``'/*'`` and ``'*/'`` sequences; can span |
| 1370 |
multiple lines, but does not support nesting of comments |
| 1371 |
|
| 1372 |
- ``html_comment`` - a comment block delimited by ``'<!--'`` and ``'-->'`` sequences; can span |
| 1373 |
multiple lines, but does not support nesting of comments |
| 1374 |
|
| 1375 |
- ``comma_separated_list`` - similar to DelimitedList_, except that the |
| 1376 |
list expressions can be any text value, or a quoted string; quoted strings can |
| 1377 |
safely include commas without incorrectly breaking the string into two tokens |
| 1378 |
|
| 1379 |
- ``rest_of_line`` - all remaining printable characters up to but not including the next |
| 1380 |
newline |
| 1381 |
|
| 1382 |
- ``common.integer`` - an integer with no leading sign; parsed token is converted to int |
| 1383 |
|
| 1384 |
- ``common.hex_integer`` - a hexadecimal integer; parsed token is converted to int |
| 1385 |
|
| 1386 |
- ``common.signed_integer`` - an integer with optional leading sign; parsed token is converted to int |
| 1387 |
|
| 1388 |
- ``common.fraction`` - signed_integer '/' signed_integer; parsed tokens are converted to float |
| 1389 |
|
| 1390 |
- ``common.mixed_integer`` - signed_integer '-' fraction; parsed tokens are converted to float |
| 1391 |
|
| 1392 |
- ``common.real`` - real number; parsed tokens are converted to float |
| 1393 |
|
| 1394 |
- ``common.sci_real`` - real number with optional scientific notation; parsed tokens are convert to float |
| 1395 |
|
| 1396 |
- ``common.number`` - any numeric expression; parsed tokens are returned as converted by the matched expression |
| 1397 |
|
| 1398 |
- ``common.fnumber`` - any numeric expression; parsed tokens are converted to float |
| 1399 |
|
| 1400 |
- ``common.ieee_float`` - any floating-point literal (int, real number, infinity, or NaN), returned as float |
| 1401 |
|
| 1402 |
- ``common.identifier`` - a programming identifier (follows Python's syntax convention of leading alpha or "_", |
| 1403 |
followed by 0 or more alpha, num, or "_") |
| 1404 |
|
| 1405 |
- ``common.ipv4_address`` - IPv4 address |
| 1406 |
|
| 1407 |
- ``common.ipv6_address`` - IPv6 address |
| 1408 |
|
| 1409 |
- ``common.mac_address`` - MAC address (with ":", "-", or "." delimiters) |
| 1410 |
|
| 1411 |
- ``common.iso8601_date`` - date in ``YYYY-MM-DD`` format |
| 1412 |
|
| 1413 |
- ``common.iso8601_datetime`` - datetime in ``YYYY-MM-DDThh:mm:ss.s(Z|+-00:00)`` format; trailing seconds, |
| 1414 |
milliseconds, and timezone optional; accepts separating ``'T'`` or ``' '`` |
| 1415 |
|
| 1416 |
- ``common.url`` - matches URL strings and returns a ParseResults with named fields like those returned |
| 1417 |
by ``urllib.parse.urlparse()`` |
| 1418 |
|
| 1419 |
|
| 1420 |
Unicode character sets for international parsing |
| 1421 |
------------------------------------------------ |
| 1422 |
Pyparsing includes the ``unicode`` namespace that contains definitions for ``alphas``, ``nums``, ``alphanums``, |
| 1423 |
``identchars``, ``identbodychars``, and ``printables`` for character ranges besides 7- or 8-bit ASCII. You can |
| 1424 |
access them using code like the following:: |
| 1425 |
|
| 1426 |
import pyparsing as pp |
| 1427 |
ppu = pp.unicode |
| 1428 |
|
| 1429 |
greek_word = pp.Word(ppu.Greek.alphas) |
| 1430 |
greek_word[...].parse_string("Καλημέρα κόσμε") |
| 1431 |
|
| 1432 |
The following language ranges are defined. |
| 1433 |
|
| 1434 |
========================== ================= ======================================================== |
| 1435 |
Unicode set Alternate names Description |
| 1436 |
-------------------------- ----------------- -------------------------------------------------------- |
| 1437 |
``Arabic`` العربية |
| 1438 |
``Chinese`` 中文 |
| 1439 |
``CJK`` Union of Chinese, Japanese, and Korean sets |
| 1440 |
``Cyrillic`` кириллица |
| 1441 |
``Devanagari`` देवनागरी |
| 1442 |
``Greek`` Ελληνικά |
| 1443 |
``Hangul`` Korean, 한국어 |
| 1444 |
``Hebrew`` עִברִית |
| 1445 |
``Japanese`` 日本語 Union of Kanji, Katakana, and Hiragana sets |
| 1446 |
``Japanese.Hiragana`` ひらがな |
| 1447 |
``Japanese.Kanji`` 漢字 |
| 1448 |
``Japanese.Katakana`` カタカナ |
| 1449 |
``Latin1`` All Unicode characters up to code point 0x7f (255) |
| 1450 |
``LatinA`` Unicode characters for code points 0x100-0x17f (256-383) |
| 1451 |
``LatinB`` Unicode characters for code points 0x180-0x24f (384-591) |
| 1452 |
``Thai`` ไทย |
| 1453 |
``BasicMultilingualPlane`` BMP All Unicode characters up to code point 0xffff (65535) |
| 1454 |
========================== ================= ======================================================== |
| 1455 |
|
| 1456 |
The base ``unicode`` class also includes definitions based on all Unicode code points up to ``sys.maxunicode``. This |
| 1457 |
set will include emojis, wingdings, and many other specialized and typographical variant characters. |
| 1458 |
|
| 1459 |
|
| 1460 |
Generating Railroad Diagrams |
| 1461 |
============================ |
| 1462 |
Grammars are conventionally represented in what are called "railroad diagrams", which allow you to visually follow |
| 1463 |
the sequence of tokens in a grammar along lines which are a bit like train tracks. You might want to generate a |
| 1464 |
railroad diagram for your grammar in order to better understand it yourself, or maybe to communicate it to others. |
| 1465 |
|
| 1466 |
Usage |
| 1467 |
----- |
| 1468 |
To generate a railroad diagram in pyparsing, you first have to install pyparsing with the ``diagrams`` extra. |
| 1469 |
To do this, just run ``pip install pyparsing[diagrams]``, and make sure you add ``pyparsing[diagrams]`` to any |
| 1470 |
``setup.py`` or ``requirements.txt`` that specifies pyparsing as a dependency. |
| 1471 |
|
| 1472 |
Create your parser as you normally would. Then call ``create_diagram()``, passing the name of an output HTML file.:: |
| 1473 |
|
| 1474 |
street_address = Word(nums).set_name("house_number") + Word(alphas)[1, ...].set_name("street_name") |
| 1475 |
street_address.set_name("street_address") |
| 1476 |
street_address.create_diagram("street_address_diagram.html") |
| 1477 |
|
| 1478 |
This will result in the railroad diagram being written to ``street_address_diagram.html``. |
| 1479 |
|
| 1480 |
`create_diagram` takes the following arguments: |
| 1481 |
|
| 1482 |
- ``output_html`` (str or file-like object) - output target for generated diagram HTML |
| 1483 |
|
| 1484 |
- ``vertical`` (int) - threshold for formatting multiple alternatives vertically instead of horizontally (default=3) |
| 1485 |
|
| 1486 |
- ``show_results_names`` - bool flag whether diagram should show annotations for defined results names |
| 1487 |
|
| 1488 |
- ``show_groups`` - bool flag whether groups should be highlighted with an unlabeled surrounding box |
| 1489 |
|
| 1490 |
- ``show_hidden`` - bool flag whether internal pyparsing elements that are normally omitted in diagrams should be shown (default=False) |
| 1491 |
|
| 1492 |
- ``embed`` - bool flag whether generated HTML should omit <HEAD>, <BODY>, and <DOCTYPE> tags to embed |
| 1493 |
the resulting HTML in an enclosing HTML source (such as PyScript HTML) |
| 1494 |
|
| 1495 |
- ``head`` - str containing additional HTML to insert into the <HEAD> section of the generated code; |
| 1496 |
can be used to insert custom CSS styling |
| 1497 |
|
| 1498 |
- ``body`` - str containing additional HTML to insert at the beginning of the <BODY> section of the |
| 1499 |
generated code |
| 1500 |
|
| 1501 |
|
| 1502 |
Example |
| 1503 |
------- |
| 1504 |
You can view an example railroad diagram generated from `a pyparsing grammar for |
| 1505 |
SQL SELECT statements <_static/sql_railroad.html>`_ (generated from |
| 1506 |
`examples/select_parser.py <https://github.com/pyparsing/pyparsing/blob/master/examples/select_parser.py>`_). |
| 1507 |
|
| 1508 |
Naming tip |
| 1509 |
---------- |
| 1510 |
Parser elements that are separately named will be broken out as their own sub-diagrams. As a short-cut alternative |
| 1511 |
to going through and adding ``.set_name()`` calls on all your sub-expressions, you can use ``autoname_elements()`` after |
| 1512 |
defining your complete grammar. For example:: |
| 1513 |
|
| 1514 |
a = pp.Literal("a") |
| 1515 |
b = pp.Literal("b").set_name("bbb") |
| 1516 |
pp.autoname_elements() |
| 1517 |
|
| 1518 |
`a` will get named "a", while `b` will keep its name "bbb". |
| 1519 |
|
| 1520 |
Customization |
| 1521 |
------------- |
| 1522 |
You can customize the resulting diagram in a few ways. |
| 1523 |
To do so, run ``pyparsing.diagrams.to_railroad`` to convert your grammar into a form understood by the |
| 1524 |
`railroad-diagrams <https://github.com/tabatkins/railroad-diagrams/blob/gh-pages/README-py.md>`_ module, and |
| 1525 |
then ``pyparsing.diagrams.railroad_to_html`` to convert that into an HTML document. For example:: |
| 1526 |
|
| 1527 |
from pyparsing.diagram import to_railroad, railroad_to_html |
| 1528 |
|
| 1529 |
with open('output.html', 'w') as fp: |
| 1530 |
railroad = to_railroad(my_grammar) |
| 1531 |
fp.write(railroad_to_html(railroad)) |
| 1532 |
|
| 1533 |
This will result in the railroad diagram being written to ``output.html`` |
| 1534 |
|
| 1535 |
You can then pass in additional keyword arguments to ``pyparsing.diagrams.to_railroad``, which will be passed |
| 1536 |
into the ``Diagram()`` constructor of the underlying library, |
| 1537 |
`as explained here <https://github.com/tabatkins/railroad-diagrams/blob/gh-pages/README-py.md#diagrams>`_. |
| 1538 |
|
| 1539 |
In addition, you can edit global options in the underlying library, by editing constants:: |
| 1540 |
|
| 1541 |
from pyparsing.diagram import to_railroad, railroad_to_html |
| 1542 |
import railroad |
| 1543 |
|
| 1544 |
railroad.DIAGRAM_CLASS = "my-custom-class" |
| 1545 |
my_railroad = to_railroad(my_grammar) |
| 1546 |
|
| 1547 |
These options `are documented here <https://github.com/tabatkins/railroad-diagrams/blob/gh-pages/README-py.md#options>`_. |
| 1548 |
|
| 1549 |
Finally, you can edit the HTML produced by ``pyparsing.diagrams.railroad_to_html`` by passing in certain keyword |
| 1550 |
arguments that will be used in the HTML template. Currently, these are: |
| 1551 |
|
| 1552 |
- ``head``: A string containing HTML to use in the ``<head>`` tag. This might be a stylesheet or other metadata |
| 1553 |
|
| 1554 |
- ``body``: A string containing HTML to use in the ``<body>`` tag, above the actual diagram. This might consist of a |
| 1555 |
heading, description, or JavaScript. |
| 1556 |
|
| 1557 |
If you want to provide a custom stylesheet using the ``head`` keyword, you can make use of the following CSS classes: |
| 1558 |
|
| 1559 |
- ``railroad-group``: A group containing everything relating to a given element group (ie something with a heading) |
| 1560 |
|
| 1561 |
- ``railroad-heading``: The title for each group |
| 1562 |
|
| 1563 |
- ``railroad-svg``: A div containing only the diagram SVG for each group |
| 1564 |
|
| 1565 |
- ``railroad-description``: A div containing the group description (unused) |