Debian Codegraph @

doc/python-target.md

raw
1 # Python (2 and 3)
2
3 The examples from the ANTLR 4 book converted to Python are [here](https://github.com/jszheng/py3antlr4book).
4
5 There are 2 Python targets: `Python2` and `Python3`. This is because there is only limited compatibility between those 2 versions of the language. Please refer to the [Python documentation](https://wiki.python.org/moin/Python2orPython3) for full details.
6
7 How to create a Python lexer or parser?
8 This is pretty much the same as creating a Java lexer or parser, except you need to specify the language target, for example:
9
10 ```
11 $ antlr4 -Dlanguage=Python2 MyGrammar.g4
12 ```
13
14 or
15
16 ```
17 $ antlr4 -Dlanguage=Python3 MyGrammar.g4
18 ```
19
20 For a full list of antlr4 tool options, please visit the tool documentation page.
21
22 ## Where can I get the runtime?
23
24 Once you've generated the lexer and/or parser code, you need to download the runtime. The Python runtimes are available from PyPI:
25
26 * https://pypi.python.org/pypi/antlr4-python2-runtime/
27 * https://pypi.python.org/pypi/antlr4-python3-runtime/
28
29 The runtimes are provided in the form of source code, so no additional installation is required.
30
31 We will not document here how to refer to the runtime from your Python project, since this would differ a lot depending on your project type and IDE.
32
33 ## How do I run the generated lexer and/or parser?
34
35 Let's suppose that your grammar is named, as above, "MyGrammar". Let's suppose this parser comprises a rule named "startRule". The tool will have generated for you the following files:
36
37 * MyGrammarLexer.py
38 * MyGrammarParser.py
39 * MyGrammarListener.py (if you have not activated the -no-listener option)
40 * MyGrammarVisitor.py (if you have activated the -visitor option)
41
42 (Developers used to Java/C# AntLR will notice that there is no base listener or visitor generated, this is because Python having no support for interfaces, the generated listener and visitor are fully fledged classes)
43
44 Now a fully functioning script might look like the following:
45
46 ```python
47 import sys
48 from antlr4 import *
49 from MyGrammarLexer import MyGrammarLexer
50 from MyGrammarParser import MyGrammarParser
51
52 def main(argv):
53 input_stream = FileStream(argv[1])
54 lexer = MyGrammarLexer(input_stream)
55 stream = CommonTokenStream(lexer)
56 parser = MyGrammarParser(stream)
57 tree = parser.startRule()
58
59 if __name__ == '__main__':
60 main(sys.argv)
61 ```
62
63 This program will work. But it won't be useful unless you do one of the following:
64
65 * you visit the parse tree using a custom listener
66 * you visit the parse tree using a custom visitor
67 * your grammar comprises production code (like ANTLR3)
68
69 (please note that production code is target specific, so you can't have multi target grammars that include production code, except for very limited use cases, see below)
70
71 ## How do I create and run a custom listener?
72
73 Let's suppose your MyGrammar grammar comprises 2 rules: "key" and "value". The antlr4 tool will have generated the following listener:
74
75 ```python
76 class MyGrammarListener(ParseTreeListener):
77 def enterKey(self, ctx):
78 pass
79 def exitKey(self, ctx):
80 pass
81 def enterValue(self, ctx):
82 pass
83 def exitValue(self, ctx):
84 pass
85 ```
86
87 In order to provide custom behavior, you might want to create the following class:
88
89 ```python
90 class KeyPrinter(MyGrammarListener):
91 def exitKey(self, ctx):
92 print("Oh, a key!")
93 ```
94
95 In order to execute this listener, you would simply add the following lines to the above code:
96
97 ```
98 ...
99 tree = parser.startRule() - only repeated here for reference
100 printer = KeyPrinter()
101 walker = ParseTreeWalker()
102 walker.walk(printer, tree)
103 ```
104
105 Further information can be found from the ANTLR 4 definitive guide.
106
107 The Python implementation of ANTLR is as close as possible to the Java one, so you shouldn't find it difficult to adapt the examples for Python.
108
109 ## Target agnostic grammars
110
111 If your grammar is targeted to Python only, you may ignore the following. But if your goal is to get your Java parser to also run in Python, then you might find it useful.
112
113 1. Do not embed production code inside your grammar. This is not portable and will not be. Move all your code to listeners or visitors.
114 1. The only production code absolutely required to sit with the grammar should be semantic predicates, like:
115 ```
116 ID {$text.equals("test")}?
117 ```
118
119 Unfortunately, this is not portable, but you can work around it. The trick involves:
120
121 * deriving your parser from a parser you provide, such as BaseParser
122 * implementing utility methods in this BaseParser, such as "isEqualText"
123 * adding a "self" field to the Java/C# BaseParser, and initialize it with "this"
124
125 Thanks to the above, you should be able to rewrite the above semantic predicate as follows:
126
127 ```
128 ID {$self.isEqualText($text,"test")}?
129 ```