| 1 |
# Contributing to Pyparsing |
| 2 |
|
| 3 |
Thank you for your interest in working on pyparsing! Pyparsing has become a popular module for creating simple |
| 4 |
text parsing and data scraping applications. It has been incorporated in several widely-used packages, and is |
| 5 |
often used by beginners as part of their first Python project. |
| 6 |
|
| 7 |
## Raising questions / asking for help |
| 8 |
|
| 9 |
If you have a question on using pyparsing, there are a number of resources available online. |
| 10 |
|
| 11 |
- [StackOverflow](https://stackoverflow.com/questions/tagged/pyparsing) - about 10 years of SO questions and answers |
| 12 |
can be searched on StackOverflow, tagged with the `pyparsing` tag. Note that some of the older posts will refer |
| 13 |
to features in Python 2, or to versions and coding practices for pyparsing that have been replaced by newer classes |
| 14 |
and coding idioms. |
| 15 |
|
| 16 |
- [pyparsing sub-reddit](https://www.reddit.com/r/pyparsing/) - still very lightly attended, but open to anyone |
| 17 |
wishing to post questions or links related to pyparsing. An alternative channel to StackOverflow for asking |
| 18 |
questions. |
| 19 |
|
| 20 |
- [online docs](https://pyparsing-docs.readthedocs.io/en/latest/index.html) and a separately maintained set of class |
| 21 |
library docs [here](https://pyparsing-doc.neocities.org/) - These docs are auto-generated from the docstrings |
| 22 |
embedded in the pyparsing classes, so they can also be viewed in the interactive Python console's and Jupyter |
| 23 |
Notebook's `help` commands. |
| 24 |
|
| 25 |
- [the pyparsing Wikispaces archive](https://github.com/pyparsing/wikispaces_archive) - Before hosting on GitHub, |
| 26 |
pyparsing had a separate wiki on the wikispaces.com website. In 2018 this page was discontinued. The discussion |
| 27 |
content archive has been reformatted into Markdown and can be viewed by year at the GitHub repository. Just as |
| 28 |
with some of the older questions on StackOverflow, some of these older posts may reflect out-of-date pyparsing |
| 29 |
and Python features. |
| 30 |
|
| 31 |
- [submit an issue](https://github.com/pyparsing/pyparsing/issues) - If you have a problem with pyparsing that looks |
| 32 |
like an actual bug, or have an idea for a feature to add to pyparsing please submit an issue on GitHub. Some |
| 33 |
pyparsing behavior may be counter-intuitive, so try to review some of the other resources first, or some of the |
| 34 |
other open and closed issues. Or post your question on SO or reddit. But don't wait until you are desperate and |
| 35 |
frustrated - just ask! :) |
| 36 |
|
| 37 |
## Submitting examples |
| 38 |
|
| 39 |
If you have an example you wish to submit, please follow these guidelines. |
| 40 |
|
| 41 |
- **License - Submitted example code must be available for distribution with the rest of pyparsing under the MIT |
| 42 |
open source license.** |
| 43 |
|
| 44 |
- Please follow PEP8 name and coding guidelines, and use the black formatter |
| 45 |
to auto-format code. |
| 46 |
|
| 47 |
- Examples should import pyparsing and the common namespace classes as: |
| 48 |
|
| 49 |
```python |
| 50 |
import pyparsing as pp |
| 51 |
# if necessary |
| 52 |
ppc = pp.pyparsing_common |
| 53 |
ppu = pp.pyparsing_unicode |
| 54 |
``` |
| 55 |
|
| 56 |
- Submitted examples _must_ be Python 3.9 or later compatible. |
| 57 |
(It is acceptable if examples use Python features added after 3.6) |
| 58 |
|
| 59 |
- Where possible use operators to create composite parse expressions: |
| 60 |
|
| 61 |
```python |
| 62 |
expr = expr_a + expr_b | expr_c |
| 63 |
``` |
| 64 |
|
| 65 |
instead of: |
| 66 |
|
| 67 |
```python |
| 68 |
expr = pp.MatchFirst([pp.And([expr_a, expr_b]), expr_c]) |
| 69 |
``` |
| 70 |
|
| 71 |
Exception: if using a generator to create an expression: |
| 72 |
|
| 73 |
```python |
| 74 |
import keyword |
| 75 |
python_keywords = keyword.kwlist |
| 76 |
any_keyword = pp.MatchFirst(pp.Keyword(kw) |
| 77 |
for kw in python_keywords)) |
| 78 |
``` |
| 79 |
|
| 80 |
- Learn [Common Pitfalls When Writing Parsers][pitfalls] and |
| 81 |
how to avoid them when developing new examples. |
| 82 |
|
| 83 |
- See additional notes under [Some coding points](#some-coding-points). |
| 84 |
|
| 85 |
## Submitting changes |
| 86 |
|
| 87 |
If you are considering proposing updates to pyparsing, please bear in mind the following guidelines. |
| 88 |
|
| 89 |
Please review [_The Zen of Pyparsing_ and _The Zen of Pyparsing |
| 90 |
Development_](https://github.com/pyparsing/pyparsing/wiki/Zen) |
| 91 |
article on the pyparsing wiki, to get a general feel for the historical and future approaches to pyparsing's |
| 92 |
design, and intended developer experience as an embedded DSL. |
| 93 |
|
| 94 |
If you are using new Python features or changing usage of the Python stdlib, please check that they work as |
| 95 |
intended on prior versions of Python (currently back to Python 3.6.8). |
| 96 |
|
| 97 |
## Some design points |
| 98 |
|
| 99 |
- Minimize additions to the module namespace. Over time, pyparsing's namespace has acquired a _lot_ of names. |
| 100 |
New features have been encapsulated into namespace classes to try to hold back the name flooding when importing |
| 101 |
pyparsing. |
| 102 |
|
| 103 |
- New operator overloads for ParserElement will need to show broad applicability, and should be related to |
| 104 |
parser construction. |
| 105 |
|
| 106 |
- Performance tuning should focus on parse time performance. Optimizing parser definition performance is secondary. |
| 107 |
|
| 108 |
- New external dependencies will require substantial justification, and if included, will need to be guarded for |
| 109 |
`ImportError`s raised if the external module is not installed. |
| 110 |
|
| 111 |
## Some coding points |
| 112 |
|
| 113 |
These coding styles are encouraged whether submitting code for core pyparsing or for submitting an example. |
| 114 |
|
| 115 |
- PEP8 - pyparsing has historically been very non-compliant with many PEP8 guidelines, especially those regarding |
| 116 |
name casing. I had just finished several years of Java and Smalltalk development, and camel case seemed to be the |
| 117 |
future trend in coding styles. As of version 3.0.0, pyparsing is moving over to PEP8 naming, while maintaining |
| 118 |
compatibility with existing parser code by defining synonyms using the legacy names. These names will be |
| 119 |
retained until a future release (probably 4.0), to provide a migration path for current pyparsing-dependent |
| 120 |
applications - DO NOT MODIFY OR REMOVE THESE NAMES. |
| 121 |
See more information at the [PEP8 wiki page](https://github.com/pyparsing/pyparsing/wiki/PEP-8-planning). |
| 122 |
|
| 123 |
- No backslashes for line continuations. |
| 124 |
Continuation lines for expressions in `()`'s should start with the continuing operator: |
| 125 |
|
| 126 |
```python |
| 127 |
really_long_line = ( |
| 128 |
something |
| 129 |
+ some_other_long_thing |
| 130 |
+ even_another_long_thing |
| 131 |
) |
| 132 |
``` |
| 133 |
|
| 134 |
- Maximum line length is 120 characters. (Black will override this.) |
| 135 |
|
| 136 |
- Changes to core pyparsing must be compatible back to Py3.6 without conditionalizing. Later Py3 features may be |
| 137 |
used in examples by way of illustration. |
| 138 |
|
| 139 |
- `str.format()` statements should use named format arguments (unless this proves to be a slowdown at parse time). |
| 140 |
|
| 141 |
- List, tuple, and dict literals should include a trailing comma after the last element, which reduces changeset |
| 142 |
clutter when another element gets added to the end. |
| 143 |
|
| 144 |
- New features should be accompanied by updates to `unitTests.py` and a bullet in the CHANGES file. |
| 145 |
|
| 146 |
- Do not modify `pyparsing_archive.py`. This file is kept as a reference artifact from when pyparsing was distributed |
| 147 |
as a single source file. |
| 148 |
|
| 149 |
## Some documentation points |
| 150 |
|
| 151 |
- The docstrings in pyparsing (which are generated into the package's |
| 152 |
API documentation by Sphinx) make heavy use of doctests for their |
| 153 |
example code. This allows examples to be tested and verified as |
| 154 |
working, and ensures that any changes to the code which affect |
| 155 |
output are accompanied by corresponding changes in the examples. |
| 156 |
|
| 157 |
- The codebase's docstring tests can be verified by running the |
| 158 |
command `make doctest` from the `docs/` directory. The output |
| 159 |
should ideally look something like this: |
| 160 |
|
| 161 |
```console |
| 162 |
$ make doctest |
| 163 |
[...documentation build...] |
| 164 |
running tests... |
| 165 |
|
| 166 |
Document: pyparsing |
| 167 |
------------------- |
| 168 |
1 item passed all tests: |
| 169 |
204 tests in default |
| 170 |
204 tests in 1 item. |
| 171 |
204 passed. |
| 172 |
Test passed. |
| 173 |
|
| 174 |
Document: whats_new_in_3_1 |
| 175 |
-------------------------- |
| 176 |
1 item passed all tests: |
| 177 |
15 tests in default |
| 178 |
15 tests in 1 item. |
| 179 |
15 passed. |
| 180 |
Test passed. |
| 181 |
|
| 182 |
Doctest summary |
| 183 |
=============== |
| 184 |
219 tests |
| 185 |
0 failures in tests |
| 186 |
0 failures in setup code |
| 187 |
0 failures in cleanup code |
| 188 |
``` |
| 189 |
|
| 190 |
Any failed tests will be displayed in detail. |
| 191 |
|
| 192 |
- Much more information about doctests can be found in the |
| 193 |
[Pyparsing documentation][pyparsing-docs], in the chapter titled |
| 194 |
"Writing doctest examples". Even if you have never worked with them |
| 195 |
before, it should guide you through everything you need to know in |
| 196 |
order to write Pyparsing doctest examples. If you are already familiar |
| 197 |
with doctests and with `sphinx.ext.doctest` in general, you may wish |
| 198 |
to skip over the introductory content and go straight to the section |
| 199 |
on "Doctests in Pyparsing" which covers some issues specific to the |
| 200 |
project. |
| 201 |
|
| 202 |
<!-- Named hyperlink targets, used in the preceding text --> |
| 203 |
[pitfalls]: https://github.com/pyparsing/pyparsing/wiki/Common-Pitfalls-When-Writing-Parsers |
| 204 |
[pyparsing-docs]: https://pyparsing-docs.readthedocs.io/en/latest/ |