Debian Codegraph @

doc/grammars.md

raw
1 # Grammar Structure
2
3 A grammar is essentially a grammar declaration followed by a list of rules, but has the general form:
4
5 ```
6 /** Optional javadoc style comment */
7 grammar Name; ①
8 options {...}
9 import ... ;
10
11 tokens {...}
12 channels {...} // lexer only
13 @actionName {...}
14
15 rule1 // parser and lexer rules, possibly intermingled
16 ...
17 ruleN
18 ```
19
20 The file name containing grammar `X` must be called `X.g4`. You can specify options, imports, token specifications, and actions in any order. There can be at most one each of options, imports, and token specifications. All of those elements are optional except for the header ① and at least one rule. Rules take the basic form:
21
22 ```
23 ruleName : alternative1 | ... | alternativeN ;
24 ```
25
26 Parser rule names must start with a lowercase letter and lexer rules must start with a capital letter.
27
28 Grammars defined without a prefix on the `grammar` header are combined grammars that can contain both lexical and parser rules. To make a parser grammar that only allows parser rules, use the following header.
29
30 ```
31 parser grammar Name;
32 ...
33 ```
34
35 And, naturally, a pure lexer grammar looks like this:
36
37 ```
38 lexer grammar Name;
39 ...
40 ```
41
42 Only lexer grammars can contain `mode` specifications.
43
44 Only lexer grammars can contain custom channels specifications
45
46 ```
47 channels {
48 WHITESPACE_CHANNEL,
49 COMMENTS_CHANNEL
50 }
51 ```
52
53 Those channels can then be used like enums within lexer rules:
54
55 ```
56 WS : [ \r\t\n]+ -> channel(WHITESPACE_CHANNEL) ;
57 ```
58
59 Sections 15.5, [Lexer Rules](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference) and Section 15.3, [Parser Rules](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference) contain details on rule syntax. Section 15.8, Options describes grammar options and Section 15.4, Actions and Attributes has information on grammar-level actions.
60
61 ## Grammar Imports
62
63 Grammar `imports` let you break up a grammar into logical and reusable chunks, as we saw in [Importing Grammars](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference). ANTLR treats imported grammars very much like object-oriented programming languages treat superclasses. A grammar inherits all of the rules, tokens specifications, and named actions from the imported grammar. Rules in the “main grammar” override rules from imported grammars to implement inheritance.
64
65 Think of `import` as more like a smart include statement (which does not include rules that are already defined). The result of all imports is a single combined grammar; the ANTLR code generator sees a complete grammar and has no idea there were imported grammars.
66
67 To process a main grammar, the ANTLR tool loads all of the imported grammars into subordinate grammar objects. It then merges the rules, token types, and named actions from the imported grammars into the main grammar. In the diagram below, the grammar on the right illustrates the effect of grammar `MyELang` importing grammar `ELang`.
68
69 <img src=images/combined.png width=400>
70
71 `MyELang` inherits rules `stat`, `WS`, and `ID`, but overrides rule `expr` and adds `INT`. Here’s a sample build and test run that shows `MyELang` can recognize integer expressions whereas the original `ELang` can’t. The third, erroneous input statement triggers an error message that also demonstrates the parser was looking for `MyELang`’s expr not `ELang`’s.
72
73 ```
74 $ antlr4 MyELang.g4
75 $ javac MyELang*.java
76 $ grun MyELang stat
77 => 34;
78 => a;
79 => ;
80 => EOF
81 <= line 3:0 extraneous input ';' expecting {INT, ID}
82 ```
83
84 If there are modes in the main grammar or any of the imported grammars then the import process will import those modes and merge their rules where they are not overridden. In the event any mode becomes empty as all its
85 rules have been overridden by rules outside the mode this mode will be discarded.
86
87 If there were any `tokens` specifications, the main grammar would merge the token sets. If there were any `channel` specifications, the main grammar would merge the channel sets. Any named actions such as `@members` would be merged. In general, you should avoid named actions and actions within rules in imported grammars since that limits their reuse. ANTLR also ignores any options in imported grammars.
88
89 Imported grammars can also import other grammars. ANTLR pursues all imported grammars in a depth-first fashion. If two or more imported grammars define rule `r`, ANTLR chooses the first version of `r` it finds. In the following diagram, ANTLR examines grammars in the following order `Nested`, `G1`, `G3`, `G2`.
90
91 <img src=images/nested.png width=350>
92
93 `Nested` includes the `r` rule from `G3` because it sees that version before the `r` in `G2`.
94
95 Not every kind of grammar can import every other kind of grammar:
96
97 * Lexer grammars can import lexers, including lexers containing modes.
98 * Parsers can import parsers.
99 * Combined grammars can import parsers or lexers without modes.
100
101 ANTLR adds imported rules to the end of the rule list in a main lexer grammar. That means lexer rules in the main grammar get precedence over imported rules. For example, if a main grammar defines rule `IF : ’if’ ;` and an imported grammar defines rule `ID : [a-z]+ ;` (which also recognizes `if`), the imported `ID` won’t hide the main grammar’s `IF` token definition.
102
103 ## Tokens Section
104
105 The purpose of the `tokens` section is to define token types needed by a grammar for which there is no associated lexical rule. The basic syntax is:
106
107 ```
108 tokens { Token1, ..., TokenN }
109 ```
110
111 Most of the time, the tokens section is used to define token types needed by actions in the grammar as shown in Section 10.3, [Recognizing Languages whose Keywords Aren’t Fixed](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference):
112
113 ```
114 // explicitly define keyword token types to avoid implicit definition warnings
115 tokens { BEGIN, END, IF, THEN, WHILE }
116
117 @lexer::members { // keywords map used in lexer to assign token types
118 Map<String,Integer> keywords = new HashMap<String,Integer>() {{
119 put("begin", KeywordsParser.BEGIN);
120 put("end", KeywordsParser.END);
121 ...
122 }};
123 }
124 ```
125
126 The `tokens` section really just defines a set of tokens to add to the overall set.
127
128 ```
129 $ cat Tok.g4
130 grammar Tok;
131 tokens { A, B, C }
132 a : X ;
133 $ antlr4 Tok.g4
134 warning(125): Tok.g4:3:4: implicit definition of token X in parser
135 $ cat Tok.tokens
136 A=1
137 B=2
138 C=3
139 X=4
140 ```
141
142 ## Actions at the Grammar Level
143
144 Currently there are only two defined named actions (for the Java target) used outside of grammar rules: `header` and `members`. The former injects code into the generated recognizer class file, before the recognizer class definition, and the latter injects code into the recognizer class definition, as fields and methods.
145
146 For combined grammars, ANTLR injects the actions into both the parser and the lexer. To restrict an action to the generated parser or lexer, use `@parser::name` or `@lexer::name`.
147
148 Here’s an example where the grammar specifies a package for the generated code:
149
150 ```
151 grammar Count;
152
153 @header {
154 package foo;
155 }
156
157 @members {
158 int count = 0;
159 }
160
161 list
162 @after {System.out.println(count+" ints");}
163 : INT {count++;} (',' INT {count++;} )*
164 ;
165
166 INT : [0-9]+ ;
167 WS : [ \r\t\n]+ -> skip ;
168 ```
169
170 The grammar itself then should be in directory `foo` so that ANTLR generates code in that same `foo` directory (at least when not using the `-o` ANTLR tool option):
171
172 ```
173 $ cd foo
174 $ antlr4 Count.g4 # generates code in the current directory (foo)
175 $ ls
176 Count.g4 CountLexer.java CountParser.java
177 Count.tokens CountLexer.tokens
178 CountBaseListener.java CountListener.java
179 $ javac *.java
180 $ cd ..
181 $ grun foo.Count list
182 => 9, 10, 11
183 => EOF
184 <= 3 ints
185 ```
186
187 The Java compiler expects classes in package `foo` to be in directory `foo`.