Debian Codegraph @

docs/HowToUsePyparsing.rst

raw
1 ==========================
2 Using the pyparsing module
3 ==========================
4
5 :author: Paul McGuire
6 :address: [email protected]
7
8 :revision: 3.2.0
9 :date: October, 2024
10
11 :copyright: Copyright |copy| 2003-2024 Paul McGuire.
12
13 .. |copy| unicode:: 0xA9
14
15 :abstract: This document provides how-to instructions for the
16 pyparsing library, an easy-to-use Python module for constructing
17 and executing basic text parsers. The pyparsing module is useful
18 for evaluating user-definable
19 expressions, processing custom application language commands, or
20 extracting data from formatted reports.
21
22 .. contents:: :depth: 4
23
24 Note: While this content is still valid, there are more detailed
25 descriptions and extensive examples at the `online doc server
26 <https://pyparsing-docs.readthedocs.io/en/latest/pyparsing.html>`_, and
27 in the online help for the various pyparsing classes and methods (viewable
28 using the Python interpreter's built-in ``help()`` function). You will also
29 find many example scripts in the `examples <https://github.com/pyparsing/pyparsing/tree/master/examples>`_
30 directory of the pyparsing GitHub repo.
31
32 -----------
33
34 **Note**: *In pyparsing 3.0, many method and function names which were
35 originally written using camelCase have been converted to PEP8-compatible
36 snake_case. So ``parseString()`` is being renamed to ``parse_string()``,
37 ``delimitedList`` to DelimitedList_, and so on. You may see the old
38 names in legacy parsers, and they will be supported for a time with
39 synonyms, but the synonyms will be removed in a future release.*
40
41 *If you are using this documentation, but working with a 2.4.x version of pyparsing,
42 you'll need to convert methods and arguments from the documented snake_case
43 names to the legacy camelCase names. In pyparsing 3.x, both forms are
44 supported, but the legacy forms are deprecated; they will be dropped in a
45 future 4.0 release.*
46
47 -----------
48
49
50 Steps to follow
51 ===============
52
53 To parse an incoming data string, the client code must follow these steps:
54
55 1. First define the tokens and patterns to be matched, and assign
56 this to a program variable. Optional results names or parse
57 actions can also be defined at this time.
58
59 2. Call ``parse_string()``, ``scan_string()``, or ``search_string()``
60 on this variable, passing in the string to
61 be parsed. During the matching process, whitespace between
62 tokens is skipped by default (although this can be changed).
63 When token matches occur, any defined parse action methods are
64 called.
65
66 3. Process the parsed results, returned as a ParseResults_ object.
67 The ParseResults_ object can be accessed as if it were a list of
68 strings. Matching results may also be accessed as named attributes of
69 the returned results, if names are defined in the definition of
70 the token pattern, using ``set_results_name()``.
71
72
73 Hello, World!
74 -------------
75
76 The following complete Python program will parse the greeting ``"Hello, World!"``,
77 or any other greeting of the form "<salutation>, <addressee>!"::
78
79 import pyparsing as pp
80
81 greet = pp.Word(pp.alphas) + "," + pp.Word(pp.alphas) + "!"
82 for greeting_str in [
83 "Hello, World!",
84 "Bonjour, Monde!",
85 "Hola, Mundo!",
86 "Hallo, Welt!",
87 ]:
88 greeting = greet.parse_string(greeting_str)
89 print(greeting)
90
91 The parsed tokens are returned in the following form::
92
93 ['Hello', ',', 'World', '!']
94 ['Bonjour', ',', 'Monde', '!']
95 ['Hola', ',', 'Mundo', '!']
96 ['Hallo', ',', 'Welt', '!']
97
98
99 Usage notes
100 -----------
101
102 - The pyparsing module can be used to interpret simple command
103 strings or algebraic expressions, or can be used to extract data
104 from text reports with complicated format and structure ("screen
105 or report scraping"). However, it is possible that your defined
106 matching patterns may accept invalid inputs. Use pyparsing to
107 extract data from strings assumed to be well-formatted.
108
109 - To keep up the readability of your code, use operators_ such as ``+``, ``|``,
110 ``^``, and ``~`` to combine expressions. You can also combine
111 string literals with ``ParseExpressions`` - they will be
112 automatically converted to Literal_ objects. For example::
113
114 integer = Word(nums) # simple unsigned integer
115 variable = Char(alphas) # single letter variable, such as x, z, m, etc.
116 arith_op = one_of("+ - * /") # arithmetic operators
117 equation = variable + "=" + integer + arith_op + integer # will match "x=2+2", etc.
118
119 In the definition of ``equation``, the string ``"="`` will get added as
120 a ``Literal("=")``, but in a more readable way.
121
122 - The pyparsing module's default behavior is to ignore whitespace. This is the
123 case for 99% of all parsers ever written. This allows you to write simple, clean,
124 grammars, such as the above ``equation``, without having to clutter it up with
125 extraneous ``ws`` markers. The ``equation`` grammar will successfully parse all of the
126 following statements::
127
128 x=2+2
129 x = 2+2
130 a = 10 * 4
131 r= 1234/ 100000
132
133 Of course, it is quite simple to extend this example to support more elaborate expressions, with
134 nesting with parentheses, floating point numbers, scientific notation, and named constants
135 (such as ``e`` or ``pi``). See `fourFn.py <https://github.com/pyparsing/pyparsing/blob/master/examples/fourFn.py>`_,
136 and `simpleArith.py <https://github.com/pyparsing/pyparsing/blob/master/examples/simpleArith.py>`_
137 included in the examples directory.
138
139 - To modify pyparsing's default whitespace skipping, you can use one or
140 more of the following methods:
141
142 - use the static method ``ParserElement.set_default_whitespace_chars``
143 to override the normal set of whitespace chars (``' \t\n'``). For instance
144 when defining a grammar in which newlines are significant, you should
145 call ``ParserElement.set_default_whitespace_chars(' \t')`` to remove
146 newline from the set of skippable whitespace characters. Calling
147 this method will affect all pyparsing expressions defined afterward.
148
149 - call ``leave_whitespace()`` on individual expressions, to suppress the
150 skipping of whitespace before trying to match the expression
151
152 - use ``Combine`` to require that successive expressions must be
153 adjacent in the input string. For instance, this expression::
154
155 real = Word(nums) + '.' + Word(nums)
156
157 will match "3.14159", but will also match "3 . 12". It will also
158 return the matched results as ['3', '.', '14159']. By changing this
159 expression to::
160
161 real = Combine(Word(nums) + '.' + Word(nums))
162
163 it will not match numbers with embedded spaces, and it will return a
164 single concatenated string '3.14159' as the parsed token.
165
166 - Repetition of expressions can be indicated using ``*`` or ``[]`` notation. An
167 expression may be multiplied by an integer value (to indicate an exact
168 repetition count), or indexed with a tuple, representing min and max repetitions
169 (with ``...`` representing no min or no max, depending whether it is the first or
170 second tuple element). See the following examples, where n is used to
171 indicate an integer value:
172
173 - ``expr*3`` is equivalent to ``expr + expr + expr``
174
175 - ``expr[2, 3]`` is equivalent to ``expr + expr + Opt(expr)``
176
177 - ``expr[n, ...]`` or ``expr[n,]`` is equivalent
178 to ``expr*n + ZeroOrMore(expr)`` (read as "at least n instances of expr")
179
180 - ``expr[... ,n]`` is equivalent to ``expr*(0, n)``
181 (read as "0 to n instances of expr")
182
183 - ``expr[...]``, ``expr[0, ...]`` and ``expr * ...`` are equivalent to ``ZeroOrMore(expr)``
184
185 - ``expr[1, ...]`` is equivalent to ``OneOrMore(expr)``
186
187 Note that ``expr[..., n]`` does not raise an exception if
188 more than n exprs exist in the input stream; that is,
189 ``expr[..., n]`` does not enforce a maximum number of expr
190 occurrences. If this behavior is desired, then write
191 ``expr[..., n] + ~expr``.
192
193 - ``[]`` notation will also accept a stop expression using ':' slice
194 notation:
195
196 - ``expr[...:end_expr]`` is equivalent to ``ZeroOrMore(expr, stop_on=end_expr)``
197
198 - MatchFirst_ expressions are matched left-to-right, and the first
199 match found will skip all later expressions within, so be sure
200 to define less-specific patterns after more-specific patterns.
201 If you are not sure which expressions are most specific, use Or_
202 expressions (defined using the ``^`` operator) - they will always
203 match the longest expression, although they are more
204 compute-intensive.
205
206 - Or_ expressions will evaluate all of the specified subexpressions
207 to determine which is the "best" match, that is, which matches
208 the longest string in the input data. In case of a tie, the
209 left-most expression in the Or_ list will win.
210
211 - If parsing the contents of an entire file, pass it to the
212 ``parse_file`` method using::
213
214 expr.parse_file(source_file)
215
216 - ``ParseExceptions`` will report the location where an expected token
217 or expression failed to match. For example, if we tried to use our
218 "Hello, World!" parser to parse "Hello World!" (leaving out the separating
219 comma), we would get an exception, with the message::
220
221 pyparsing.ParseException: Expected "," (6), (1,7)
222
223 In the case of complex
224 expressions, the reported location may not be exactly where you
225 would expect. See more information under ParseException_ .
226
227 - Use the ``Group`` class to enclose logical groups of tokens within a
228 sublist. This will help organize your results into more
229 hierarchical form (the default behavior is to return matching
230 tokens as a flat list of matching input strings).
231
232 - Punctuation may be significant for matching, but is rarely of
233 much interest in the parsed results. Use the ``suppress()`` method
234 to keep these tokens from cluttering up your returned lists of
235 tokens. For example, DelimitedList_ matches a succession of
236 one or more expressions, separated by delimiters (commas by
237 default), but only returns a list of the actual expressions -
238 the delimiters are used for parsing, but are suppressed from the
239 returned output.
240
241 - Parse actions can be used to convert values from strings to
242 other data types (ints, floats, booleans, etc.).
243
244 - Results names are recommended for retrieving tokens from complex
245 expressions. It is much easier to access a token using its field
246 name than using a positional index, especially if the expression
247 contains optional elements. You can also shortcut
248 the ``set_results_name`` call::
249
250 stats = ("AVE:" + real_num.set_results_name("average")
251 + "MIN:" + real_num.set_results_name("min")
252 + "MAX:" + real_num.set_results_name("max"))
253
254 can more simply and cleanly be written as this::
255
256 stats = ("AVE:" + real_num("average")
257 + "MIN:" + real_num("min")
258 + "MAX:" + real_num("max"))
259
260 - Be careful when defining parse actions that modify global variables or
261 data structures (as in fourFn.py_), especially for low level tokens
262 or expressions that may occur within an And_ expression; an early element
263 of an And_ may match, but the overall expression may fail.
264
265
266 Classes
267 =======
268
269 All the pyparsing classes can be found in this
270 `UML class diagram <_static/pyparsingClassDiagram_3.0.9.jpg>`_.
271
272 Classes in the pyparsing module
273 -------------------------------
274
275 ``ParserElement`` - abstract base class for all pyparsing classes;
276 methods for code to use are:
277
278 - ``parse_string(source_string, parse_all=False)`` - only called once, on the overall
279 matching pattern; returns a ParseResults_ object that makes the
280 matched tokens available as a list, and optionally as a dictionary,
281 or as an object with named attributes; if ``parse_all`` is set to True, then
282 ``parse_string`` will raise a ParseException_ if the grammar does not process
283 the complete input string.
284
285 - ``parse_file(source_file)`` - a convenience function, that accepts an
286 input file object or filename. The file contents are passed as a
287 string to ``parse_string()``. ``parse_file`` also supports the ``parse_all`` argument.
288
289 - ``scan_string(source_string)`` - generator function, used to find and
290 extract matching text in the given source string; for each matched text,
291 returns a tuple of:
292
293 - matched tokens (packaged as a ParseResults_ object)
294
295 - start location of the matched text in the given source string
296
297 - end location in the given source string
298
299 ``scan_string`` allows you to scan through the input source string for
300 random matches, instead of exhaustively defining the grammar for the entire
301 source text (as would be required with ``parse_string``).
302
303 - ``transform_string(source_string)`` - convenience wrapper function for
304 ``scan_string``, to process the input source string, and replace matching
305 text with the tokens returned from parse actions defined in the grammar
306 (see set_parse_action_).
307
308 - ``search_string(source_string)`` - another convenience wrapper function for
309 ``scan_string``, returns a list of the matching tokens returned from each
310 call to ``scan_string``.
311
312 - ``set_name(name)`` - associate a short descriptive name for this
313 element, useful in displaying exceptions and trace information
314
315 - ``run_tests(tests_string)`` - useful development and testing method on
316 expressions, to pass a multiline string of sample strings to test against
317 the expression. Comment lines (beginning with ``#``) can be inserted
318 and they will be included in the test output::
319
320 digits = Word(nums).set_name("numeric digits")
321 real_num = Combine(digits + '.' + digits)
322 real_num.run_tests("""\
323 # valid number
324 3.14159
325
326 # no integer part
327 .00001
328
329 # no decimal
330 101
331
332 # no decimal value
333 101.
334 """)
335
336 will print::
337
338 # valid number
339 3.14159
340 ['3.14159']
341
342 # no integer part
343 .00001
344 ^
345 FAIL: Expected numeric digits, found '.' (at char 0), (line:1, col:1)
346
347 # no decimal
348 101
349 ^
350 FAIL: Expected ".", found end of text (at char 3), (line:1, col:4)
351
352 # no decimal value
353 101.
354 ^
355 FAIL: Expected numeric digits, found end of text (at char 4), (line:1, col:5)
356
357 .. _set_results_name:
358
359 - ``set_results_name(string, list_all_matches=False)`` - name to be given
360 to tokens matching
361 the element; if multiple tokens within
362 a repetition group (such as ZeroOrMore_ or DelimitedList_) the
363 default is to return only the last matching token - if ``list_all_matches``
364 is set to True, then a list of all the matching tokens is returned.
365
366 ``expr.set_results_name("key")`` can also be written ``expr("key")``
367 (a results name with a trailing '*' character will be
368 interpreted as setting ``list_all_matches`` to ``True``).
369
370 Note:
371 ``set_results_name`` returns a *copy* of the element so that a single
372 basic element can be referenced multiple times and given
373 different names within a complex grammar.
374
375 .. _using_each:
376
377 - ``using_each(list_of_symbols)`` a short-cut for defining a number of
378 symbols of a particular ``ParserElement`` subclass::
379
380 LBRACK, RBRACK, LBRACE, RBRACE, LPAR, RPAR = Suppress.using_each("[]{}()")
381 AND, OR, NOT = Keyword.using_each("and or not".split())
382
383 .. _set_parse_action:
384
385 - ``set_parse_action(*fn)`` - specify one or more functions to call after successful
386 matching of the element; each function is defined as ``fn(s, loc, toks)``, where:
387
388 - ``s`` is the original parse string
389
390 - ``loc`` is the location in the string where matching started
391
392 - ``toks`` is the list of the matched tokens, packaged as a ParseResults_ object
393
394 Parse actions can have any of the following signatures::
395
396 fn(s: str, loc: int, tokens: ParseResults)
397 fn(loc: int, tokens: ParseResults)
398 fn(tokens: ParseResults)
399 fn()
400
401 Multiple functions can be attached to a ``ParserElement`` by specifying multiple
402 arguments to ``set_parse_action``, or by calling ``add_parse_action``. Calls to ``set_parse_action``
403 will replace any previously defined parse actions. ``set_parse_action(None)`` will clear
404 all previously defined parse actions.
405
406 Each parse action function can return a modified ``toks`` list, to perform conversion, or
407 string modifications. For brevity, ``fn`` may also be a
408 lambda - here is an example of using a parse action to convert matched
409 integer tokens from strings to integers::
410
411 int_number = Word(nums).set_parse_action(lambda s, l, t: [int(t[0])])
412
413 If ``fn`` modifies the ``toks`` list in-place, it does not need to return
414 and pyparsing will use the modified ``toks`` list.
415
416 If ``set_parse_action`` is called with an argument of ``None``, then this clears all parse actions
417 attached to that expression.
418
419 A nice short-cut for calling ``set_parse_action`` is to use it as a decorator::
420
421 identifier = Word(alphas, alphanums + "_")
422
423 @identifier.set_parse_action
424 def resolve_identifier(results: ParseResults):
425 return variable_values.get(results[0])
426
427 (Posted by @MisterMiyagi in this SO answer: https://stackoverflow.com/a/63031959/165216)
428
429 - ``add_parse_action`` - similar to ``set_parse_action``, but instead of replacing any
430 previously defined parse actions, will append the given action or actions to the
431 existing defined parse actions.
432
433 - ``add_condition`` - a simplified form of ``add_parse_action`` if the purpose
434 of the parse action is to simply do some validation, and raise an exception
435 if the validation fails. Takes a method that takes the same arguments,
436 but simply returns ``True`` or ``False``. If ``False`` is returned, an exception will be
437 raised.
438
439 - ``set_break(break_flag=True)`` - if ``break_flag`` is ``True``, calls ``pdb.set_break()``
440 as this expression is about to be parsed
441
442 - ``copy()`` - returns a copy of a ``ParserElement``; can be used to use the same
443 parse expression in different places in a grammar, with different parse actions
444 attached to each; a short-form ``expr()`` is equivalent to ``expr.copy()``
445
446 - ``leave_whitespace()`` - change default behavior of skipping
447 whitespace before starting matching (mostly used internally to the
448 pyparsing module, rarely used by client code)
449
450 - ``set_whitespace_chars(chars)`` - define the set of chars to be ignored
451 as whitespace before trying to match a specific ``ParserElement``, in place of the
452 default set of whitespace (space, tab, newline, and return)
453
454 - ``set_default_whitespace_chars(chars)`` - class-level method to override
455 the default set of whitespace chars for all subsequently created ParserElements
456 (including copies); useful when defining grammars that treat one or more of the
457 default whitespace characters as significant (such as a line-sensitive grammar, to
458 omit newline from the list of ignorable whitespace)
459
460 - ``suppress()`` - convenience function to suppress the output of the
461 given element, instead of wrapping it with a ``Suppress`` object.
462
463 - ``ignore(expr)`` - function to specify parse expression to be
464 ignored while matching defined patterns; can be called
465 repeatedly to specify multiple expressions; useful to specify
466 patterns of comment syntax, for example
467
468 - ``set_debug(flag=True)`` - function to enable/disable tracing output
469 when trying to match this element
470
471 - ``validate()`` - function to verify that the defined grammar does not
472 contain infinitely recursive constructs.
473
474 *(``validate()`` is deprecated, and
475 will be removed in a future pyparsing release. Pyparsing now supports
476 left-recursive parsers, which this function attempted to catch.)*
477
478 .. _parse_with_tabs:
479
480 - ``parse_with_tabs()`` - function to override default behavior of converting
481 tabs to spaces before parsing the input string; rarely used, except when
482 specifying whitespace-significant grammars using the White_ class.
483
484 - ``enable_packrat()`` - a class-level static method to enable a memoizing
485 performance enhancement, known as "packrat parsing". packrat parsing is
486 disabled by default, since it may conflict with some user programs that use
487 parse actions. To activate the packrat feature, your
488 program must call the class method ``ParserElement.enable_packrat()``. For best
489 results, call ``enable_packrat()`` immediately after importing pyparsing.
490
491 - ``enable_left_recursion()`` - a class-level static method to enable
492 pyparsing with left-recursive (LR) parsers. Similar to ``ParserElement.enable_packrat()``,
493 your program must call the class method ``ParserElement.enable_left_recursion()`` to
494 enable this feature. ``enable_left_recursion()`` uses a separate packrat cache, and so
495 is incompatible with ``enable_packrat()``.
496
497 Basic ParserElement subclasses
498 ------------------------------
499
500 .. _Literal:
501
502 - ``Literal`` - construct with a string to be matched exactly
503
504 .. _CaselessLiteral:
505
506 - ``CaselessLiteral`` - construct with a string to be matched, but
507 without case checking; results are always returned as the
508 defining literal, NOT as they are found in the input string
509
510 .. _Keyword:
511
512 - ``Keyword`` - similar to Literal_, but must be immediately followed by
513 whitespace, punctuation, or other non-keyword characters; prevents
514 accidental matching of a non-keyword that happens to begin with a
515 defined keyword
516
517 - ``CaselessKeyword`` - similar to Keyword_, but with caseless matching
518 behavior as described in CaselessLiteral_.
519
520 .. _Word:
521
522 - ``Word`` - one or more contiguous characters; construct with a
523 string containing the set of allowed initial characters, and an
524 optional second string of allowed body characters; for instance,
525 a common ``Word`` construct is to match a code identifier - in C, a
526 valid identifier must start with an alphabetic character or an
527 underscore ('_'), followed by a body that can also include numeric
528 digits. That is, ``a``, ``i``, ``MAX_LENGTH``, ``_a1``, ``b_109_``, and
529 ``plan9FromOuterSpace``
530 are all valid identifiers; ``9b7z``, ``$a``, ``.section``, and ``0debug``
531 are not. To
532 define an identifier using a ``Word``, use either of the following::
533
534 Word(alphas+"_", alphanums+"_")
535 Word(srange("[a-zA-Z_]"), srange("[a-zA-Z0-9_]"))
536
537 Pyparsing also provides pre-defined strings ``identchars`` and
538 ``identbodychars`` so that you can also write::
539
540 Word(identchars, identbodychars)
541
542 If only one
543 string given, it specifies that the same character set defined
544 for the initial character is used for the word body; for instance, to
545 define an identifier that can only be composed of capital letters and
546 underscores, use one of::
547
548 ``Word("ABCDEFGHIJKLMNOPQRSTUVWXYZ_")``
549 ``Word(srange("[A-Z_]"))``
550
551 A ``Word`` may
552 also be constructed with any of the following optional parameters:
553
554 - ``min`` - indicating a minimum length of matching characters
555
556 - ``max`` - indicating a maximum length of matching characters
557
558 - ``exact`` - indicating an exact length of matching characters;
559 if ``exact`` is specified, it will override any values for ``min`` or ``max``
560
561 - ``as_keyword`` - indicating that preceding and following characters must
562 be whitespace or non-keyword characters
563
564 - ``exclude_chars`` - a string of characters that should be excluded from
565 init_chars and body_chars
566
567 Sometimes you want to define a word using all
568 characters in a range except for one or two of them; you can do this
569 with the ``exclude_chars`` argument. This is helpful if you want to define
570 a word with all ``printables`` except for a single delimiter character, such
571 as '.'. Previously, you would have to create a custom string to pass to Word.
572 With this change, you can just create ``Word(printables, exclude_chars='.')``.
573
574 - ``Char`` - a convenience form of ``Word`` that will match just a single character from
575 a string of matching characters::
576
577 single_digit = Char(nums)
578
579 - ``CharsNotIn`` - similar to Word_, but matches characters not
580 in the given constructor string (accepts only one string for both
581 initial and body characters); also supports ``min``, ``max``, and ``exact``
582 optional parameters.
583
584 - ``Regex`` - a powerful construct, that accepts a regular expression
585 to be matched at the current parse position; accepts an optional
586 ``flags`` parameter, corresponding to the flags parameter in the ``re.compile``
587 method; if the expression includes named sub-fields, they will be
588 represented in the returned ParseResults_.
589
590 - ``QuotedString`` - supports the definition of custom quoted string
591 formats, in addition to pyparsing's built-in ``dbl_quoted_string`` and
592 ``sgl_quoted_string``. ``QuotedString`` allows you to specify the following
593 parameters:
594
595 - ``quote_char`` - string of one or more characters defining the quote delimiting string
596
597 - ``esc_char`` - character to escape quotes, typically backslash (default=None)
598
599 - ``esc_quote`` - special quote sequence to escape an embedded quote string (such as SQL's "" to escape an embedded ") (default=None)
600
601 - ``multiline`` - boolean indicating whether quotes can span multiple lines (default=False)
602
603 - ``unquote_results`` - boolean indicating whether the matched text should be unquoted (default=True)
604
605 - ``end_quote_char`` - string of one or more characters defining the end of the quote delimited string (default=None => same as ``quote_char``)
606
607 .. _SkipTo:
608
609 - ``SkipTo`` - skips ahead in the input string, accepting any
610 characters up to the specified pattern; may be constructed with
611 the following optional parameters:
612
613 - ``include`` - if set to true, also consumes the match expression
614 (default is false)
615
616 - ``ignore`` - allows the user to specify patterns to not be matched,
617 to prevent false matches
618
619 - ``fail_on`` - if a literal string or expression is given for this argument, it defines an expression that
620 should cause the SkipTo_ expression to fail, and not skip over that expression
621
622 ``SkipTo`` can also be written using ``...``::
623
624 LBRACE, RBRACE = Literal.using_each("{}")
625
626 brace_expr = LBRACE + SkipTo(RBRACE) + RBRACE
627 # can also be written as
628 brace_expr = LBRACE + ... + RBRACE
629
630 .. _White:
631
632 - ``White`` - also similar to Word_, but matches whitespace
633 characters. Not usually needed, as whitespace is implicitly
634 ignored by pyparsing. However, some grammars are whitespace-sensitive,
635 such as those that use leading tabs or spaces to indicating grouping
636 or hierarchy. (If matching on tab characters, be sure to call
637 parse_with_tabs_ on the top-level parse element.)
638
639 - ``Empty`` - a null expression, requiring no characters - will always
640 match; useful for debugging and for specialized grammars
641
642 - ``NoMatch`` - opposite of ``Empty``, will never match; useful for debugging
643 and for specialized grammars
644
645
646 Expression subclasses
647 ---------------------
648
649 .. _And:
650
651 - ``And`` - construct with a list of ``ParserElements``, all of which must
652 match for ``And`` to match; can also be created using the '+'
653 operator; multiple expressions can be ``Anded`` together using the '*'
654 operator as in::
655
656 ip_address = Word(nums) + ('.' + Word(nums)) * 3
657
658 A tuple can be used as the multiplier, indicating a min/max::
659
660 us_phone_number = Word(nums) + ('-' + Word(nums)) * (1,2)
661
662 A special form of ``And`` is created if the '-' operator is used
663 instead of the '+' operator. In the ``ip_address`` example above, if
664 no trailing '.' and ``Word(nums)`` are found after matching the initial
665 ``Word(nums)``, then pyparsing will back up in the grammar and try other
666 alternatives to ``ip_address``. However, if ``ip_address`` is defined as::
667
668 strict_ip_address = Word(nums) - ('.'+Word(nums))*3
669
670 then no backing up is done. If the first ``Word(nums)`` of ``strict_ip_address``
671 is matched, then any mismatch after that will raise a ``ParseSyntaxException``,
672 which will halt the parsing process immediately. By careful use of the
673 '-' operator, grammars can provide meaningful error messages close to
674 the location where the incoming text does not match the specified
675 grammar.
676
677 .. _Or:
678
679 - ``Or`` - construct with a list of ``ParserElements``, any of which must
680 match for ``Or`` to match; if more than one expression matches, the
681 expression that makes the longest match will be used; can also
682 be created using the '^' operator
683
684 .. _MatchFirst:
685
686 - ``MatchFirst`` - construct with a list of ``ParserElements``, any of
687 which must match for ``MatchFirst`` to match; matching is done
688 left-to-right, taking the first expression that matches; can
689 also be created using the '|' operator
690
691 .. _Each:
692
693 - ``Each`` - similar to And_, in that all of the provided expressions
694 must match; however, ``Each`` permits matching to be done in any order;
695 can also be created using the '&' operator
696
697 - ``Opt`` - construct with a ``ParserElement``, but this element is
698 not required to match; can be constructed with an optional ``default`` argument,
699 containing a default string or object to be supplied if the given optional
700 parse element is not found in the input string; parse action will only
701 be called if a match is found, or if a default is specified.
702
703 An optional element ``expr`` can also be expressed using ``expr | ""``.
704
705 (``Opt`` was formerly named ``Optional``, but since the standard Python
706 library module ``typing`` now defines ``Optional``, the pyparsing class has
707 been renamed to ``Opt``. A compatibility synonym ``Optional`` is defined,
708 but will be removed in a future release.)
709
710 .. _ZeroOrMore:
711
712 - ``ZeroOrMore`` - similar to ``Opt``, but can be repeated; ``ZeroOrMore(expr)``
713 can also be written as ``expr[...]``.
714
715 .. _OneOrMore:
716
717 - ``OneOrMore`` - similar to ZeroOrMore_, but at least one match must
718 be present; ``OneOrMore(expr)`` can also be written as ``expr[1, ...]``.
719
720 .. _DelimitedList:
721
722 - ``DelimitedList`` - used for
723 matching one or more occurrences of ``expr``, separated by ``delim``.
724 By default, the delimiters are suppressed, so the returned results contain
725 only the separate list elements. Can optionally specify ``combine=True``,
726 indicating that the expressions and delimiters should be returned as one
727 combined value (useful for scoped variables, such as ``"a.b.c"``, or
728 ``"a::b::c"``, or paths such as ``"a/b/c"``). Can also optionally specify ``min` and ``max``
729 restrictions on the length of the list, and
730 ``allow_trailing_delim`` to accept a trailing delimiter at the end of the list.
731
732 .. _FollowedBy:
733
734 - ``FollowedBy`` - a lookahead expression, requires matching of the given
735 expressions, but does not advance the parsing position within the input string
736
737 .. _NotAny:
738
739 - ``NotAny`` - a negative lookahead expression, prevents matching of named
740 expressions, does not advance the parsing position within the input string;
741 can also be created using the unary '~' operator
742
743
744 .. _operators:
745
746 Expression operators
747 --------------------
748
749 - ``+`` - creates And_ using the expressions before and after the operator
750
751 - ``|`` - creates MatchFirst_ (first left-to-right match) using the expressions before and after the operator
752
753 - ``^`` - creates Or_ (longest match) using the expressions before and after the operator
754
755 - ``&`` - creates Each_ using the expressions before and after the operator
756
757 - ``*`` - creates And_ by multiplying the expression by the integer operand; if
758 expression is multiplied by a 2-tuple, creates an And_ of ``(min,max)``
759 expressions (similar to ``{min,max}`` form in regular expressions); if
760 ``min`` is ``None``, interpret as ``(0,max)``; if ``max`` is ``None``, interpret as
761 ``expr*min + ZeroOrMore(expr)``
762
763 - ``-`` - like ``+`` but with no backup and retry of alternatives
764
765 - ``~`` - creates NotAny_ using the expression after the operator
766
767 - ``==`` - matching expression to string; returns ``True`` if the string matches the given expression
768
769 - ``<<=`` - inserts the expression following the operator as the body of the
770 ``Forward`` expression before the operator (``<<`` can also be used, but ``<<=`` is preferred
771 to avoid operator precedence misinterpretation of the pyparsing expression)
772
773 - ``...`` - inserts a SkipTo_ expression leading to the next expression, as in
774 ``Keyword("start") + ... + Keyword("end")``.
775
776 - ``[min, max]`` - specifies repetition similar to ``*`` with ``min`` and ``max`` specified
777 as the minimum and maximum number of repetitions. ``...`` can be used in place of ``None``.
778 For example ``expr[...]`` is equivalent to ``ZeroOrMore(expr)``, ``expr[1, ...]`` is
779 equivalent to ``OneOrMore(expr)``, and ``expr[..., 3]`` is equivalent to "up to 3 instances
780 of ``expr``".
781
782 - ``[:stop_on]`` - specifies a stopping expression for the current repetition (may be combined
783 with ``...`` or ``min, max``), as in ``Keyword("start") + Word(alphas)[...:Keyword("end")] + Keyword("end")``
784
785 Positional subclasses
786 ---------------------
787
788 - ``StringStart`` - matches beginning of the text
789
790 - ``StringEnd`` - matches the end of the text
791
792 - ``LineStart`` - matches beginning of a line (lines delimited by ``\n`` characters)
793
794 - ``LineEnd`` - matches the end of a line
795
796 - ``WordStart`` - matches a leading word boundary
797
798 - ``WordEnd`` - matches a trailing word boundary
799
800
801
802 Converter subclasses
803 --------------------
804
805 - ``Combine`` - joins all matched tokens into a single string, using
806 specified ``join_string`` (default ``join_string=""``); expects
807 all matching tokens to be adjacent, with no intervening
808 whitespace (can be overridden by specifying ``adjacent=False`` in constructor)
809
810 - ``Suppress`` - clears matched tokens; useful to keep returned
811 results from being cluttered with required but uninteresting
812 tokens (such as list delimiters)
813
814
815 Special subclasses
816 ------------------
817
818 - ``Group`` - causes the matched tokens to be enclosed in a list;
819 useful in repeated elements like ZeroOrMore_ and OneOrMore_ to
820 break up matched tokens into groups for each repeated pattern
821
822 - ``Dict`` - like ``Group``, but also constructs a dictionary, using the
823 ``[0]``'th elements of all enclosed token lists as the keys, and
824 each token list as the value
825
826 - ``Forward`` - placeholder token used to define recursive token
827 patterns; when defining the actual expression later in the
828 program, insert it into the ``Forward`` object using the ``<<=``
829 operator (see fourFn.py_ for an example).
830
831 - ``Tag`` - a non-parsing token that always matches, and inserts
832 a tag and value into the current parsed tokens; useful for adding
833 metadata or annotations to parsed results (see `examples/tag_example.py <../examples/tag_example.py>`_).
834
835
836 Other classes
837 -------------
838 .. _ParseResults:
839
840 - ``ParseResults`` - class used to contain and manage the lists of tokens
841 created from parsing the input using the user-defined parse
842 expression. ``ParseResults`` can be accessed in a number of ways:
843
844 - as a list
845
846 - total list of elements can be found using ``len()``
847
848 - individual elements can be found using ``[0], [1], [-1],`` etc.,
849 or retrieved using slices
850
851 - elements can be deleted using ``del``
852
853 - the last element can be extracted and removed in a single operation
854 using ``pop()``, or any element can be extracted and removed
855 using ``pop(n)``
856
857 - a nested ParseResults_ can be created by using the pyparsing ``Group`` class
858 around elements in an expression::
859
860 Word(alphas) + Group(Word(nums)[...]) + Word(alphas)
861
862 will parse the string "abc 100 200 300 end" as::
863
864 ['abc', ['100', '200', '300'], 'end']
865
866 If the ``Group`` is constructed using ``aslist=True``, the resulting tokens
867 will be a Python list instead of a ParseResults_. In this case, the returned value will
868 no longer support the extended features or methods of a ParseResults_.
869
870 - as a dictionary
871
872 - if ``set_results_name()`` is used to name elements within the
873 overall parse expression, then these fields can be referenced
874 as dictionary elements or as attributes
875
876 - the ``Dict`` class generates dictionary entries using the data of the
877 input text - in addition to ParseResults_ listed as ``[ [ a1, b1, c1, ...], [ a2, b2, c2, ...] ]``
878 it also acts as a dictionary with entries defined as ``{ a1 : [ b1, c1, ... ] }, { a2 : [ b2, c2, ... ] }``;
879 this is especially useful when processing tabular data where the first column contains a key
880 value for that line of data; when constructed with ``asdict=True``, will
881 return an actual Python ``dict`` instead of a ParseResults_. In this case, the returned value will
882 no longer support the extended features or methods of a ParseResults_.
883
884 - list elements that are deleted using ``del`` will still be accessible by their
885 dictionary keys
886
887 - supports ``get()``, ``items()`` and ``keys()`` methods, similar to a dictionary
888
889 - a keyed item can be extracted and removed using ``pop(key)``. Here
890 ``key`` must be non-numeric (such as a string), in order to use dict
891 extraction instead of list extraction.
892
893 - new named elements can be added (in a parse action, for instance), using the same
894 syntax as adding an item to a dict (``parse_results["X"] = "new item"``);
895 named elements can be removed using ``del parse_results["X"]``
896
897 - as a nested list
898
899 - results returned from the Group class are encapsulated within their
900 own list structure, so that the tokens can be handled as a hierarchical
901 tree
902
903 - as an object
904
905 - named elements can be accessed as if they were attributes of an object:
906 if an element is referenced that does not exist, it will return ``""``.
907
908 ParseResults_ can also be converted to an ordinary list of strings
909 by calling ``as_list()``. Note that this will strip the results of any
910 field names that have been defined for any embedded parse elements.
911 (The ``pprint`` module is especially good at printing out the nested contents
912 given by ``as_list()``.)
913
914 If a ParseResults_ is built with expressions that use results names (see _set_results_name) or
915 using the ``Dict`` class, then those names and values can be extracted as a Python
916 dict using ``as_dict()``.
917
918 Finally, ParseResults_ can be viewed by calling ``dump()``. ``dump()`` will first show
919 the ``as_list()`` output, followed by an indented structure listing parsed tokens that
920 have been assigned results names.
921
922 Here is sample code illustrating some of these methods::
923
924 >>> number = Word(nums)
925 >>> name = Combine(Word(alphas)[...], adjacent=False, join_string=" ")
926 >>> parser = number("house_number") + name("street_name")
927 >>> result = parser.parse_string("123 Main St")
928 >>> print(result)
929 ['123', 'Main St']
930 >>> print(type(result))
931 <class 'pyparsing.ParseResults'>
932 >>> print(repr(result))
933 (['123', 'Main St'], {'house_number': ['123'], 'street_name': ['Main St']})
934 >>> result.house_number
935 '123'
936 >>> result["street_name"]
937 'Main St'
938 >>> result.as_list()
939 ['123', 'Main St']
940 >>> result.as_dict()
941 {'house_number': '123', 'street_name': 'Main St'}
942 >>> print(result.dump())
943 ['123', 'Main St']
944 - house_number: '123'
945 - street_name: 'Main St'
946
947
948 Exception classes and Troubleshooting
949 -------------------------------------
950
951 .. _ParseException:
952
953 - ``ParseException`` - exception returned when a grammar parse fails;
954 ``ParseExceptions`` have attributes ``loc``, ``msg``, ``line``, ``lineno``, and ``column``; to view the
955 text line and location where the reported ParseException occurs, use::
956
957 except ParseException as err:
958 print(err.line)
959 print(" " * (err.column - 1) + "^")
960 print(err)
961
962 ``ParseExceptions`` also have an ``explain()`` method that gives this same information::
963
964 except ParseException as err:
965 print(err.explain())
966
967 - ``RecursiveGrammarException`` - exception returned by ``validate()`` if
968 the grammar contains a recursive infinite loop, such as::
969
970 bad_grammar = Forward()
971 good_token = Literal("A")
972 bad_grammar <<= Opt(good_token) + bad_grammar
973
974 - ``ParseFatalException`` - exception that parse actions can raise to stop parsing
975 immediately. Should be used when a semantic error is found in the input text, such
976 as a mismatched XML tag.
977
978 - ``ParseSyntaxException`` - subclass of ``ParseFatalException`` raised when a
979 syntax error is found, based on the use of the '-' operator when defining
980 a sequence of expressions in an And_ expression.
981
982 - You can also get some insights into the parsing logic using diagnostic parse actions,
983 and ``set_debug()``, or test the matching of expression fragments by testing them using
984 ``search_string()`` or ``scan_string()``.
985
986 - Use ``with_line_numbers`` from ``pyparsing_testing`` to display the input string
987 being parsed, with line and column numbers that correspond to the values reported
988 in set_debug() output::
989
990 import pyparsing as pp
991 ppt = pp.testing
992
993 data = """\
994 A
995 100"""
996
997 expr = pp.Word(pp.alphanums).set_name("word").set_debug()
998 print(ppt.with_line_numbers(data))
999 expr[...].parse_string(data)
1000
1001 prints::
1002
1003 . 1
1004 1234567890
1005 1: A|
1006 2: 100|
1007
1008 Match word at loc 3(1,4)
1009 A
1010 ^
1011 Matched word -> ['A']
1012 Match word at loc 11(2,7)
1013 100
1014 ^
1015 Matched word -> ['100']
1016
1017 `with_line_numbers` has several options for displaying control characters, end-of-line
1018 and space markers, Unicode symbols for control characters - these are documented in the
1019 function's docstring.
1020
1021 - Diagnostics can be enabled using ``pyparsing.enable_diag`` and passing
1022 one of the following enum values defined in ``pyparsing.Diagnostics``
1023
1024 - ``warn_multiple_tokens_in_named_alternation`` - flag to enable warnings when a results
1025 name is defined on a MatchFirst_ or Or_ expression with one or more And_ subexpressions
1026
1027 - ``warn_ungrouped_named_tokens_in_collection`` - flag to enable warnings when a results
1028 name is defined on a containing expression with ungrouped subexpressions that also
1029 have results names
1030
1031 - ``warn_name_set_on_empty_Forward`` - flag to enable warnings when a ``Forward`` is defined
1032 with a results name, but has no contents defined
1033
1034 - ``warn_on_parse_using_empty_Forward`` - flag to enable warnings when a ``Forward`` is
1035 defined in a grammar but has never had an expression attached to it
1036
1037 - ``warn_on_assignment_to_Forward`` - flag to enable warnings when a ``Forward`` is defined
1038 but is overwritten by assigning using ``'='`` instead of ``'<<='`` or ``'<<'``
1039
1040 - ``warn_on_multiple_string_args_to_oneof`` - flag to enable warnings when ``one_of`` is
1041 incorrectly called with multiple str arguments
1042
1043 - ``enable_debug_on_named_expressions`` - flag to auto-enable debug on all subsequent
1044 calls to ``ParserElement.set_name``
1045
1046 All warnings can be enabled by calling ``pyparsing.enable_all_warnings()``.
1047 Sample::
1048
1049 import pyparsing as pp
1050 pp.enable_all_warnings()
1051
1052 fwd = pp.Forward().set_results_name("recursive_expr")
1053
1054 >>> UserWarning: warn_name_set_on_empty_Forward: setting results name 'recursive_expr'
1055 on Forward expression that has no contained expression
1056
1057 Warnings can also be enabled using the Python ``-W`` switch (using ``-Wd`` or
1058 ``-Wd:::pyparsing``) or setting a non-empty value to the environment variable
1059 ``PYPARSINGENABLEALLWARNINGS``. (If using ``-Wd`` for testing, but wishing to
1060 disable pyparsing warnings, add ``-Wi:::pyparsing``.)
1061
1062
1063 Miscellaneous attributes and methods
1064 ====================================
1065
1066 Helper methods
1067 --------------
1068
1069 - ``counted_array(expr)`` - convenience function for a pattern where an list of
1070 instances of the given expression are preceded by an integer giving the count of
1071 elements in the list. Returns an expression that parses the leading integer,
1072 reads exactly that many expressions, and returns the array of expressions in the
1073 parse results - the leading integer is suppressed from the results (although it
1074 is easily reconstructed by using len on the returned array).
1075
1076 - ``one_of(choices, caseless=False, as_keyword=False)`` - convenience function for quickly declaring an
1077 alternative set of Literal_ expressions. ``choices`` can be passed as a list of strings
1078 or as a single string of values separated by spaces. The values are sorted so that longer
1079 matches are attempted first; this ensures that a short value does
1080 not mask a longer one that starts with the same characters. If ``caseless=True``,
1081 will create an alternative set of CaselessLiteral_ tokens. If ``as_keyword=True``,
1082 ``one_of`` will declare Keyword_ expressions instead of Literal_ expressions.
1083
1084 - ``dict_of(key, value)`` - convenience function for quickly declaring a
1085 dictionary pattern of ``Dict(ZeroOrMore(Group(key + value)))``.
1086
1087 - ``make_html_tags(tag_str)`` and ``make_xml_tags(tag_str)`` - convenience
1088 functions to create definitions of opening and closing tag expressions. Returns
1089 a pair of expressions, for the corresponding ``<tag>`` and ``</tag>`` strings. Includes
1090 support for attributes in the opening tag, such as ``<tag attr1="abc">`` - attributes
1091 are returned as named results in the returned ParseResults_. ``make_html_tags`` is less
1092 restrictive than ``make_xml_tags``, especially with respect to case sensitivity.
1093
1094 - ``infix_notation(base_operand, operator_list)`` -
1095 convenience function to define a grammar for parsing infix notation
1096 expressions with a hierarchical precedence of operators. To use the ``infix_notation``
1097 helper:
1098
1099 1. Define the base "atom" operand term of the grammar.
1100 For this simple grammar, the smallest operand is either
1101 an integer or a variable. This will be the first argument
1102 to the ``infix_notation`` method.
1103
1104 2. Define a list of tuples for each level of operator
1105 precedence. Each tuple is of the form
1106 ``(operand_expr, num_operands, right_left_assoc, parse_action)``, where:
1107
1108 - ``operand_expr`` - the pyparsing expression for the operator;
1109 may also be a string, which will be converted to a Literal_; if
1110 ``None``, indicates an empty operator, such as the implied
1111 multiplication operation between 'm' and 'x' in "y = mx + b".
1112
1113 - ``num_operands`` - the number of terms for this operator (must
1114 be 1, 2, or 3)
1115
1116 - ``right_left_assoc`` is the indicator whether the operator is
1117 right or left associative, using the pyparsing-defined
1118 constants ``OpAssoc.RIGHT`` and ``OpAssoc.LEFT``.
1119
1120 - ``parse_action`` is the parse action to be associated with
1121 expressions matching this operator expression (the
1122 ``parse_action`` tuple member may be omitted)
1123
1124 3. Call ``infix_notation`` passing the operand expression and
1125 the operator precedence list, and save the returned value
1126 as the generated pyparsing expression. You can then use
1127 this expression to parse input strings, or incorporate it
1128 into a larger, more complex grammar.
1129
1130 Here is an ``infix_notation`` definition for 4-function arithmetic,
1131 taking numbers or variables as operands. The order of definition of
1132 the operators follows the standard precedence of operations for
1133 arithmetic::
1134
1135 number = pp.common.number()
1136 variable = pp.common.identifier()
1137 arithmetic_expression = pp.infix_notation(
1138 integer | variable,
1139 [
1140 ("-", 1, pp.OpAssoc.RIGHT),
1141 (pp.one_of("* /"), 2, pp.OpAssoc.LEFT),
1142 (pp.one_of("+ -"), 2, pp.OpAssoc.LEFT),
1143 ]
1144 )
1145
1146 ``infix_notation`` also supports optional arguments ``lpar`` and ``rpar``, to
1147 parse groups with symbols other than "(" and ")". They may be passed as strings
1148 (in which case they will be converted to ``Suppress`` objects, and suppressed from
1149 the parsed results), or passed as pyparsing expressions, in which case they will
1150 be kept as-is, and grouped with their contents.
1151
1152 For instance, to use "<" and ">" for grouping symbols, you could write::
1153
1154 expr = infix_notation(int_expr,
1155 [
1156 (one_of("+ -"), 2, OpAssoc.LEFT),
1157 ],
1158 lpar="<",
1159 rpar=">"
1160 )
1161 expr.parse_string("3 - <2 + 11>")
1162
1163 returning::
1164
1165 [3, '-', [2, '+', 11]]
1166
1167 If the grouping symbols are to be retained, then pass them as pyparsing ``Literals``::
1168
1169 expr = infix_notation(int_expr,
1170 [
1171 (one_of("+ -"), 2, OpAssoc.LEFT),
1172 ],
1173 lpar=Literal("<"),
1174 rpar=Literal(">")
1175 )
1176 expr.parse_string("3 - <2 + 11>")
1177
1178 returning::
1179
1180 [3, '-', ['<', [2, '+', 11], '>']]
1181
1182 - ``match_previous_literal`` and ``match_previous_expr`` - function to define an
1183 expression that matches the same content
1184 as was parsed in a previous parse expression. For instance::
1185
1186 first = Word(nums)
1187 match_expr = first + ":" + match_previous_literal(first)
1188
1189 will match "1:1", but not "1:2". Since this matches at the literal
1190 level, this will also match the leading "1:1" in "1:10".
1191
1192 In contrast::
1193
1194 first = Word(nums)
1195 match_expr = first + ":" + match_previous_expr(first)
1196
1197 will *not* match the leading "1:1" in "1:10"; the expressions are
1198 evaluated first, and then compared, so "1" is compared with "10".
1199
1200 - ``nested_expr(opener, closer, content=None, ignore_expr=quoted_string)`` - method for defining nested
1201 lists enclosed in opening and closing delimiters.
1202
1203 - ``opener`` - opening character for a nested list (default="("); can also be a pyparsing expression
1204
1205 - ``closer`` - closing character for a nested list (default=")"); can also be a pyparsing expression
1206
1207 - ``content`` - expression for items within the nested lists (default=None)
1208
1209 - ``ignore_expr`` - expression for ignoring opening and closing delimiters (default=``quoted_string``)
1210
1211 If an expression is not provided for the content argument, the nested
1212 expression will capture all whitespace-delimited content between delimiters
1213 as a list of separate values.
1214
1215 Use the ``ignore_expr`` argument to define expressions that may contain
1216 opening or closing characters that should not be treated as opening
1217 or closing characters for nesting, such as ``quoted_string`` or a comment
1218 expression. Specify multiple expressions using an Or_ or MatchFirst_.
1219 The default is ``quoted_string``, but if no expressions are to be ignored,
1220 then pass ``None`` for this argument.
1221
1222
1223 - ``IndentedBlock(statement_expr, recursive=False, grouped=True)`` -
1224 function to define an indented block of statements, similar to
1225 indentation-based blocking in Python source code:
1226
1227 - ``statement_expr`` - the expression defining a statement that
1228 will be found in the indented block; a valid ``IndentedBlock``
1229 must contain at least 1 matching ``statement_expr``
1230
1231 - ``recursive`` - flag indicating whether the IndentedBlock can
1232 itself contain nested sub-blocks of the same type of expression
1233 (default=False)
1234
1235 - ``grouped`` - flag indicating whether the tokens returned from
1236 parsing the IndentedBlock should be grouped (default=True)
1237
1238 .. _originalTextFor:
1239
1240 - ``original_text_for(expr)`` - helper function to preserve the originally parsed text, regardless of any
1241 token processing or conversion done by the contained expression. For instance, the following expression::
1242
1243 full_name = Word(alphas) + Word(alphas)
1244
1245 will return the parse of "John Smith" as ['John', 'Smith']. In some applications, the actual name as it
1246 was given in the input string is what is desired. To do this, use ``original_text_for``::
1247
1248 full_name = original_text_for(Word(alphas) + Word(alphas))
1249
1250 - ``ungroup(expr)`` - function to "ungroup" returned tokens; useful
1251 to undo the default behavior of And_ to always group the returned tokens, even
1252 if there is only one in the list.
1253
1254 - ``lineno(loc, string)`` - function to give the line number of the
1255 location within the string; the first line is line 1, newlines
1256 start new rows
1257
1258 - ``col(loc, string)`` - function to give the column number of the
1259 location within the string; the first column is column 1,
1260 newlines reset the column number to 1
1261
1262 - ``line(loc, string)`` - function to retrieve the line of text
1263 representing ``lineno(loc, string)``; useful when printing out diagnostic
1264 messages for exceptions
1265
1266 - ``srange(range_spec)`` - function to define a string of characters,
1267 given a string of the form used by regexp string ranges, such as ``"[0-9]"`` for
1268 all numeric digits, ``"[A-Z_]"`` for uppercase characters plus underscore, and
1269 so on (note that ``range_spec`` does not include support for generic regular
1270 expressions, just string range specs)
1271
1272 - ``trace_parse_action(fn)`` - decorator function to debug parse actions. Lists
1273 each call, called arguments, and return value or exception
1274
1275
1276
1277 Helper parse actions
1278 --------------------
1279
1280 - ``remove_quotes`` - removes the first and last characters of a quoted string;
1281 useful to remove the delimiting quotes from quoted strings
1282
1283 - ``replace_with(repl_string)`` - returns a parse action that simply returns the
1284 ``repl_string``; useful when using ``transform_string``, or converting HTML entities, as in::
1285
1286 nbsp = Literal("&nbsp;").set_parse_action(replace_with("<BLANK>"))
1287
1288 - ``original_text_for``- restores any internal whitespace or suppressed
1289 text within the tokens for a matched parse
1290 expression. This is especially useful when defining expressions
1291 for ``scan_string`` or ``transform_string`` applications.
1292
1293 - ``with_attribute(*args, **kwargs)`` - helper to create a validating parse action to be used with start tags created
1294 with ``make_xml_tags`` or ``make_html_tags``. Use ``with_attribute`` to qualify a starting tag
1295 with a required attribute value, to avoid false matches on common tags such as
1296 ``<TD>`` or ``<DIV>``.
1297
1298 ``with_attribute`` can be called with:
1299
1300 - keyword arguments, as in ``(class="Customer", align="right")``, or
1301
1302 - a list of name-value tuples, as in ``(("ns1:class", "Customer"), ("ns2:align", "right"))``
1303
1304 An attribute can be specified to have the special value
1305 ``with_attribute.ANY_VALUE``, which will match any value - use this to
1306 ensure that an attribute is present but any attribute value is
1307 acceptable.
1308
1309 - ``match_only_at_col(column_number)`` - a parse action that verifies that
1310 an expression was matched at a particular column, raising a
1311 ``ParseException`` if matching at a different column number; useful when parsing
1312 tabular data
1313
1314 - ``common.convert_to_integer()`` - converts all matched tokens to int
1315
1316 - ``common.convert_to_float()`` - converts all matched tokens to float
1317
1318 - ``common.convert_to_date()`` - converts matched token to a datetime.date
1319
1320 - ``common.convert_to_datetime()`` - converts matched token to a datetime.datetime
1321
1322 - ``common.strip_html_tags()`` - removes HTML tags from matched token
1323
1324 - ``common.downcase_tokens()`` - converts all matched tokens to lowercase
1325
1326 - ``common.upcase_tokens()`` - converts all matched tokens to uppercase
1327
1328
1329 Common string and token constants
1330 ---------------------------------
1331
1332 - ``alphas`` - same as ``string.ascii_letters``
1333
1334 - ``nums`` - same as ``string.digits``
1335
1336 - ``alphanums`` - a string containing ``alphas + nums``
1337
1338 - ``alphas8bit`` - a string containing alphabetic 8-bit characters::
1339
1340 ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþ
1341
1342 .. _identchars:
1343
1344 - ``identchars`` - a string containing characters that are valid as initial identifier characters::
1345
1346 ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyzª
1347 µºÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ
1348
1349 - ``identbodychars`` - a string containing characters that are valid as identifier body characters (those following a
1350 valid leading identifier character as given in identchars_)::
1351
1352 0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyzª
1353 µ·ºÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ
1354
1355 - ``printables`` - same as ``string.printable``, minus the space (``' '``) character
1356
1357 - ``empty`` - a global ``Empty()``; will always match
1358
1359 - ``sgl_quoted_string`` - a string of characters enclosed in 's; may
1360 include whitespace, but not newlines
1361
1362 - ``dbl_quoted_string`` - a string of characters enclosed in "s; may
1363 include whitespace, but not newlines
1364
1365 - ``quoted_string`` - ``sgl_quoted_string | dbl_quoted_string``
1366
1367 - ``python_quoted_string`` - ``quoted_string | multiline quoted string``
1368
1369 - ``c_style_comment`` - a comment block delimited by ``'/*'`` and ``'*/'`` sequences; can span
1370 multiple lines, but does not support nesting of comments
1371
1372 - ``html_comment`` - a comment block delimited by ``'<!--'`` and ``'-->'`` sequences; can span
1373 multiple lines, but does not support nesting of comments
1374
1375 - ``comma_separated_list`` - similar to DelimitedList_, except that the
1376 list expressions can be any text value, or a quoted string; quoted strings can
1377 safely include commas without incorrectly breaking the string into two tokens
1378
1379 - ``rest_of_line`` - all remaining printable characters up to but not including the next
1380 newline
1381
1382 - ``common.integer`` - an integer with no leading sign; parsed token is converted to int
1383
1384 - ``common.hex_integer`` - a hexadecimal integer; parsed token is converted to int
1385
1386 - ``common.signed_integer`` - an integer with optional leading sign; parsed token is converted to int
1387
1388 - ``common.fraction`` - signed_integer '/' signed_integer; parsed tokens are converted to float
1389
1390 - ``common.mixed_integer`` - signed_integer '-' fraction; parsed tokens are converted to float
1391
1392 - ``common.real`` - real number; parsed tokens are converted to float
1393
1394 - ``common.sci_real`` - real number with optional scientific notation; parsed tokens are convert to float
1395
1396 - ``common.number`` - any numeric expression; parsed tokens are returned as converted by the matched expression
1397
1398 - ``common.fnumber`` - any numeric expression; parsed tokens are converted to float
1399
1400 - ``common.ieee_float`` - any floating-point literal (int, real number, infinity, or NaN), returned as float
1401
1402 - ``common.identifier`` - a programming identifier (follows Python's syntax convention of leading alpha or "_",
1403 followed by 0 or more alpha, num, or "_")
1404
1405 - ``common.ipv4_address`` - IPv4 address
1406
1407 - ``common.ipv6_address`` - IPv6 address
1408
1409 - ``common.mac_address`` - MAC address (with ":", "-", or "." delimiters)
1410
1411 - ``common.iso8601_date`` - date in ``YYYY-MM-DD`` format
1412
1413 - ``common.iso8601_datetime`` - datetime in ``YYYY-MM-DDThh:mm:ss.s(Z|+-00:00)`` format; trailing seconds,
1414 milliseconds, and timezone optional; accepts separating ``'T'`` or ``' '``
1415
1416 - ``common.url`` - matches URL strings and returns a ParseResults with named fields like those returned
1417 by ``urllib.parse.urlparse()``
1418
1419
1420 Unicode character sets for international parsing
1421 ------------------------------------------------
1422 Pyparsing includes the ``unicode`` namespace that contains definitions for ``alphas``, ``nums``, ``alphanums``,
1423 ``identchars``, ``identbodychars``, and ``printables`` for character ranges besides 7- or 8-bit ASCII. You can
1424 access them using code like the following::
1425
1426 import pyparsing as pp
1427 ppu = pp.unicode
1428
1429 greek_word = pp.Word(ppu.Greek.alphas)
1430 greek_word[...].parse_string("Καλημέρα κόσμε")
1431
1432 The following language ranges are defined.
1433
1434 ========================== ================= ========================================================
1435 Unicode set Alternate names Description
1436 -------------------------- ----------------- --------------------------------------------------------
1437 ``Arabic`` العربية
1438 ``Chinese`` 中文
1439 ``CJK`` Union of Chinese, Japanese, and Korean sets
1440 ``Cyrillic`` кириллица
1441 ``Devanagari`` देवनागरी
1442 ``Greek`` Ελληνικά
1443 ``Hangul`` Korean, 한국어
1444 ``Hebrew`` עִברִית
1445 ``Japanese`` 日本語 Union of Kanji, Katakana, and Hiragana sets
1446 ``Japanese.Hiragana`` ひらがな
1447 ``Japanese.Kanji`` 漢字
1448 ``Japanese.Katakana`` カタカナ
1449 ``Latin1`` All Unicode characters up to code point 0x7f (255)
1450 ``LatinA`` Unicode characters for code points 0x100-0x17f (256-383)
1451 ``LatinB`` Unicode characters for code points 0x180-0x24f (384-591)
1452 ``Thai`` ไทย
1453 ``BasicMultilingualPlane`` BMP All Unicode characters up to code point 0xffff (65535)
1454 ========================== ================= ========================================================
1455
1456 The base ``unicode`` class also includes definitions based on all Unicode code points up to ``sys.maxunicode``. This
1457 set will include emojis, wingdings, and many other specialized and typographical variant characters.
1458
1459
1460 Generating Railroad Diagrams
1461 ============================
1462 Grammars are conventionally represented in what are called "railroad diagrams", which allow you to visually follow
1463 the sequence of tokens in a grammar along lines which are a bit like train tracks. You might want to generate a
1464 railroad diagram for your grammar in order to better understand it yourself, or maybe to communicate it to others.
1465
1466 Usage
1467 -----
1468 To generate a railroad diagram in pyparsing, you first have to install pyparsing with the ``diagrams`` extra.
1469 To do this, just run ``pip install pyparsing[diagrams]``, and make sure you add ``pyparsing[diagrams]`` to any
1470 ``setup.py`` or ``requirements.txt`` that specifies pyparsing as a dependency.
1471
1472 Create your parser as you normally would. Then call ``create_diagram()``, passing the name of an output HTML file.::
1473
1474 street_address = Word(nums).set_name("house_number") + Word(alphas)[1, ...].set_name("street_name")
1475 street_address.set_name("street_address")
1476 street_address.create_diagram("street_address_diagram.html")
1477
1478 This will result in the railroad diagram being written to ``street_address_diagram.html``.
1479
1480 `create_diagram` takes the following arguments:
1481
1482 - ``output_html`` (str or file-like object) - output target for generated diagram HTML
1483
1484 - ``vertical`` (int) - threshold for formatting multiple alternatives vertically instead of horizontally (default=3)
1485
1486 - ``show_results_names`` - bool flag whether diagram should show annotations for defined results names
1487
1488 - ``show_groups`` - bool flag whether groups should be highlighted with an unlabeled surrounding box
1489
1490 - ``show_hidden`` - bool flag whether internal pyparsing elements that are normally omitted in diagrams should be shown (default=False)
1491
1492 - ``embed`` - bool flag whether generated HTML should omit <HEAD>, <BODY>, and <DOCTYPE> tags to embed
1493 the resulting HTML in an enclosing HTML source (such as PyScript HTML)
1494
1495 - ``head`` - str containing additional HTML to insert into the <HEAD> section of the generated code;
1496 can be used to insert custom CSS styling
1497
1498 - ``body`` - str containing additional HTML to insert at the beginning of the <BODY> section of the
1499 generated code
1500
1501
1502 Example
1503 -------
1504 You can view an example railroad diagram generated from `a pyparsing grammar for
1505 SQL SELECT statements <_static/sql_railroad.html>`_ (generated from
1506 `examples/select_parser.py <https://github.com/pyparsing/pyparsing/blob/master/examples/select_parser.py>`_).
1507
1508 Naming tip
1509 ----------
1510 Parser elements that are separately named will be broken out as their own sub-diagrams. As a short-cut alternative
1511 to going through and adding ``.set_name()`` calls on all your sub-expressions, you can use ``autoname_elements()`` after
1512 defining your complete grammar. For example::
1513
1514 a = pp.Literal("a")
1515 b = pp.Literal("b").set_name("bbb")
1516 pp.autoname_elements()
1517
1518 `a` will get named "a", while `b` will keep its name "bbb".
1519
1520 Customization
1521 -------------
1522 You can customize the resulting diagram in a few ways.
1523 To do so, run ``pyparsing.diagrams.to_railroad`` to convert your grammar into a form understood by the
1524 `railroad-diagrams <https://github.com/tabatkins/railroad-diagrams/blob/gh-pages/README-py.md>`_ module, and
1525 then ``pyparsing.diagrams.railroad_to_html`` to convert that into an HTML document. For example::
1526
1527 from pyparsing.diagram import to_railroad, railroad_to_html
1528
1529 with open('output.html', 'w') as fp:
1530 railroad = to_railroad(my_grammar)
1531 fp.write(railroad_to_html(railroad))
1532
1533 This will result in the railroad diagram being written to ``output.html``
1534
1535 You can then pass in additional keyword arguments to ``pyparsing.diagrams.to_railroad``, which will be passed
1536 into the ``Diagram()`` constructor of the underlying library,
1537 `as explained here <https://github.com/tabatkins/railroad-diagrams/blob/gh-pages/README-py.md#diagrams>`_.
1538
1539 In addition, you can edit global options in the underlying library, by editing constants::
1540
1541 from pyparsing.diagram import to_railroad, railroad_to_html
1542 import railroad
1543
1544 railroad.DIAGRAM_CLASS = "my-custom-class"
1545 my_railroad = to_railroad(my_grammar)
1546
1547 These options `are documented here <https://github.com/tabatkins/railroad-diagrams/blob/gh-pages/README-py.md#options>`_.
1548
1549 Finally, you can edit the HTML produced by ``pyparsing.diagrams.railroad_to_html`` by passing in certain keyword
1550 arguments that will be used in the HTML template. Currently, these are:
1551
1552 - ``head``: A string containing HTML to use in the ``<head>`` tag. This might be a stylesheet or other metadata
1553
1554 - ``body``: A string containing HTML to use in the ``<body>`` tag, above the actual diagram. This might consist of a
1555 heading, description, or JavaScript.
1556
1557 If you want to provide a custom stylesheet using the ``head`` keyword, you can make use of the following CSS classes:
1558
1559 - ``railroad-group``: A group containing everything relating to a given element group (ie something with a heading)
1560
1561 - ``railroad-heading``: The title for each group
1562
1563 - ``railroad-svg``: A div containing only the diagram SVG for each group
1564
1565 - ``railroad-description``: A div containing the group description (unused)