Debian Codegraph @

doc/interpreters.md

raw
1 # Parser and Lexer Interpreters
2
3 *Since ANTLR 4.2*
4
5 For small parsing tasks it is sometimes convenient to use ANTLR in interpreted mode, rather than generating a parser in a particular target, compiling it and running it as part of your application. Here's some sample code that creates lexer and parser Grammar objects and then creates interpreters. Once we have a ParserInterpreter, we can use it to parse starting in any rule we like, given a rule index (which the grammar + the parser can provide).
6
7 ## Action Code
8
9 Since interpreters don't use generated parsers + lexers they cannot execute any action code (including predicates). That means the interpreter runs as if there were no predicates at all. If your grammar requires action code in order to parse correctly you will not be able to test it using this approach.
10
11 ## Java Target Interpreter Setup
12
13 ```java
14 LexerGrammar lg = new LexerGrammar(
15 "lexer grammar L;\n" +
16 "A : 'a' ;\n" +
17 "B : 'b' ;\n" +
18 "C : 'c' ;\n");
19 Grammar g = new Grammar(
20 "parser grammar T;\n" +
21 "s : (A|B)* C ;\n",
22 lg);
23 LexerInterpreter lexEngine =
24 lg.createLexerInterpreter(new ANTLRInputStream(input));
25 CommonTokenStream tokens = new CommonTokenStream(lexEngine);
26 ParserInterpreter parser = g.createParserInterpreter(tokens);
27 ParseTree t = parser.parse(g.rules.get(startRule).index);
28 ```
29
30 You can also load combined grammars from a file:
31
32 ```java
33 public static ParseTree parse(String fileName,
34 String combinedGrammarFileName,
35 String startRule)
36 throws IOException
37 {
38 final Grammar g = Grammar.load(combinedGrammarFileName);
39 LexerInterpreter lexEngine = g.createLexerInterpreter(CharStreams.fromPath(Paths.get(fileName)));
40 CommonTokenStream tokens = new CommonTokenStream(lexEngine);
41 ParserInterpreter parser = g.createParserInterpreter(tokens);
42 ParseTree t = parser.parse(g.getRule(startRule).index);
43 System.out.println("parse tree: "+t.toStringTree(parser));
44 return t;
45 }
46 ```
47
48 Then:
49
50 ```java
51 ParseTree t = parse("T.om",
52 MantraGrammar,
53 "compilationUnit");
54 ```
55
56 To load separate lexer/parser grammars, do this:
57
58 ```java
59 public static ParseTree parse(String fileNameToParse,
60 String lexerGrammarFileName,
61 String parserGrammarFileName,
62 String startRule)
63 throws IOException
64 {
65 final LexerGrammar lg = (LexerGrammar) Grammar.load(lexerGrammarFileName);
66 final Grammar pg = Grammar.load(parserGrammarFileName, lg);
67 CharStream input = CharStreams.fromPath(Paths.get(fileNameToParse));
68 LexerInterpreter lexEngine = lg.createLexerInterpreter(input);
69 CommonTokenStream tokens = new CommonTokenStream(lexEngine);
70 ParserInterpreter parser = pg.createParserInterpreter(tokens);
71 ParseTree t = parser.parse(pg.getRule(startRule).index);
72 System.out.println("parse tree: " + t.toStringTree(parser));
73 return t;
74 }
75 ```
76
77 Then:
78
79 ```java
80 ParseTree t = parse(fileName, XMLLexerGrammar, XMLParserGrammar, "document");
81 ```
82
83 This is also how we will integrate instantaneous parsing into ANTLRWorks2 and development environment plug-ins.
84
85 See [TestParserInterpreter.java](../tool-testsuite/test/org/antlr/v4/test/tool/TestParserInterpreter.java).
86
87 ## Non-Java Target Interpreter Setup
88 The ANTLR4 runtimes do not contain any grammar parsing classes (they are in the ANTLR4 tool jar). Hence we cannot use `LexerGrammar` and `Grammar` to parse grammars for the interpreter. Instead we directly instantiate `LexerInterpreter` and `ParserInterpreter` objects. They require some data (namely symbol information and the ATNs) which only the ANTLR4 tool can give us. However, on each generation run ANTLR not only produces your parser + lexer files but also interpreter data files (*.interp) which contain all you need to feed the interpreters.
89
90 A support class (`InterpreterDataReader`) is used to load the data for your convenience, which makes this very easy to use. Btw. even the Java target go this route instead of using the non-runtime classes `Grammar` and `LexerGrammar`. Sometimes it might not be feasible to use the tool jar for whatever reason.
91
92 Here's how the setup looks like (C++ example):
93
94 ```cpp
95 /**
96 * sourceFileName - name of the file with content to parse
97 * lexerName - the name of your lexer (arbitrary, that's what is used in error messages)
98 * parserName - ditto for the parser
99 * lexerDataFileName - the lexer interpeter data file name (e.g. `<path>/ExprLexer.interp`)
100 * parserDataFileName - ditto for the parser (e.g. `<path>/Expr.interp`)
101 * startRule - the name of the rule to start parsing at
102 */
103 void parse(std::string const& sourceFileName,
104 std::string const& lexerName, std::string const& parserName,
105 std::string const& lexerDataFileName, std::string const& parserDataFileName,
106 std::string const& startRule) {
107
108 InterpreterData lexerData = InterpreterDataReader::parseFile(lexerDataFileName);
109 InterpreterData parserData = InterpreterDataReader::parseFile(parserDataFileName);
110
111 ANTLRFileStream input(sourceFileName);
112 LexerInterpreter lexEngine(lexerName, lexerData.vocabulary, lexerData.ruleNames,
113 lexerData.channels, lexerData.modes, lexerData.atn, &input);
114 CommonTokenStream tokens(&lexEngine);
115
116 /* Remove comment to print the tokens.
117 tokens.fill();
118 std::cout << "INPUT:" << std::endl;
119 for (auto token : tokens.getTokens()) {
120 std::cout << token->toString() << std::endl;
121 }
122 */
123
124 ParserInterpreter parser(parserName, parserData.vocabulary, parserData.ruleNames,
125 parserData.atn, &tokens);
126 tree::ParseTree *tree = parser.parse(parser.getRuleIndex(startRule));
127
128 std::cout << "parse tree: " << tree->toStringTree(&parser) << std::endl;
129 }
130 ```