Debian Codegraph @

doc/lexer-rules.md

raw
1 # Lexer Rules
2
3 A lexer grammar is composed of lexer rules, optionally broken into multiple modes. Lexical modes allow us to split a single lexer grammar into multiple sublexers. The lexer can only return tokens matched by rules from the current mode.
4
5 Lexer rules specify token definitions and more or less follow the syntax of parser rules except that lexer rules cannot have arguments, return values, or local variables. Lexer rule names must begin with an uppercase letter, which distinguishes them from parser rule names:
6
7 ```
8 /** Optional document comment */
9 TokenName : alternative1 | ... | alternativeN ;
10 ```
11
12 You can also define rules that are not tokens but rather aid in the recognition of tokens. These fragment rules do not result in tokens visible to the parser:
13
14 ```
15 fragment
16 HelperTokenRule : alternative1 | ... | alternativeN ;
17 ```
18
19 For example, `DIGIT` is a pretty common fragment rule:
20
21 ```
22 INT : DIGIT+ ; // references the DIGIT helper rule
23 fragment DIGIT : [0-9] ; // not a token by itself
24 ```
25
26 ## Lexical Modes
27
28 Modes allow you to group lexical rules by context, such as inside and outside of XML tags. It’s like having multiple sublexers, one for each context. The lexer can only return tokens matched by entering a rule in the current mode. Lexers start out in the so-called default mode. All rules are considered to be within the default mode unless you specify a mode command. Modes are not allowed within combined grammars, just lexer grammars. (See grammar `XMLLexer` from [Tokenizing XML](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference).)
29
30 ```
31 rules in default mode
32 ...
33 mode MODE1;
34 rules in MODE1
35 ...
36 mode MODEN;
37 rules in MODEN
38 ...
39 ```
40
41 ## Lexer Rule Elements
42
43 Lexer rules allow two constructs that are unavailable to parser rules: the .. range operator and the character set notation enclosed in square brackets, [characters]. Don’t confuse character sets with arguments to parser rules. [characters] only means character set in a lexer. Here’s a summary of all lexer rule elements:
44
45 <table>
46 <tr>
47 <th>Syntax</th><th>Description</th>
48 </tr>
49 <tr>
50 <td>T</td><td>
51 Match token T at the current input position. Tokens always begin with a capital letter.</td>
52 </tr>
53
54 <tr>
55 <td>’literal’</td><td>
56 Match that character or sequence of characters. E.g., ’while’ or ’=’.</td>
57 </tr>
58
59 <tr>
60 <td>[char set]</td><td>
61 <p>Match one of the characters specified in the character set. Interpret <tt>x-y</tt> as the set of characters between range <tt>x</tt> and <tt>y</tt>, inclusively. The following escaped characters are interpreted as single special characters: <tt>\n</tt>, <tt>\r</tt>, <tt>\b</tt>, <tt>\t</tt>, <tt>\f</tt>, <tt>\uXXXX</tt>, and <tt>\u{XXXXXX}</tt>. To get <tt>]</tt> or <tt>\</tt> you must escape them with <tt>\</tt>. To get <tt>-</tt> you must escape it with <tt>\</tt> too, except for the case when <tt>-</tt> is the first or last character in the set.</p>
62
63 <p>You can also include all characters matching Unicode properties (general category, boolean, or enumerated including scripts and blocks) with <tt>\p{PropertyName}</tt> or <tt>\p{EnumProperty=Value}</tt>. (You can invert the test with <tt>\P{PropertyName}</tt> or <tt>\P{EnumProperty=Value}</tt>).</p>
64
65 <p>For a list of valid Unicode property names, see <a href="http://unicode.org/reports/tr44/#Properties">Unicode Standard Annex #44</a>. (ANTLR also supports <a href="http://unicode.org/reports/tr44/#General_Category_Values">short and long Unicode general category names and values</a> like <tt>\p{Lu}</tt>, <tt>\p{Z}</tt>, <tt>\p{Symbol}</tt>, <tt>\p{Blk=Latin_1_Sup}</tt>, and <tt>\p{Block=Latin_1_Supplement}</tt>.)</p>
66
67 <p>As a shortcut for <tt>\p{Block=Latin_1_Supplement}</tt>, you can refer to blocks using <a href="http://www.unicode.org/Public/UCD/latest/ucd/Blocks.txt">Unicode block names</a> prefixed with <tt>In</tt> and with spaces changed to <tt>_</tt>. For example: <tt>\p{InLatin_1_Supplement}</tt>, <tt>\p{InYijing_Hexagram_Symbols}</tt>, and <tt>\p{InAncient_Greek_Numbers}</tt>.</p>
68
69 <p>A few extra properties are supported:</p>
70 <ul>
71 <li><tt>\p{Extended_Pictographic}</tt> (see <a href="http://unicode.org/reports/tr35/">UTS #35</a>)</li>
72 <li><tt>\p{EmojiPresentation=EmojiDefault}</tt> (code points which have colorful emoji-style presentation by default but which can also be displayed text-style)</li>
73 <li><tt>\p{EmojiPresentation=TextDefault}</tt> (code points which have black-and-white text-style presentation by default but which can also be displayed emoji-style)</li>
74 <li><tt>\p{EmojiPresentation=Text}</tt> (code points which have only black-and-white text-style and lack a colorful emoji-style presentation)</li>
75 </ul>
76
77 <p>Property names are <b>case-insensitive</b>, and <tt>_</tt> and <tt>-</tt> are treated identically</p>
78
79 <p>Here are a few examples:</p>
80
81 <pre>
82 WS : [ \n\u000D] -> skip ; // same as [ \n\r]
83
84 UNICODE_WS : [\p{White_Space}] -> skip; // match all Unicode whitespace
85
86 ID : [a-zA-Z] [a-zA-Z0-9]* ; // match usual identifier spec
87
88 UNICODE_ID : [\p{Alpha}\p{General_Category=Other_Letter}] [\p{Alnum}\p{General_Category=Other_Letter}]* ; // match full Unicode alphabetic ids
89
90 EMOJI : [\u{1F4A9}\u{1F926}] ; // note Unicode code points > U+FFFF
91
92 DASHBRACK : [\-\]]+ ; // match - or ] one or more times
93
94 DASH : [---] ; // match a single -, i.e., "any character" between - and - (note first and last - not escaped)
95 </pre>
96 </td>
97 </tr>
98
99 <tr>
100 <td>’x’..’y’</td><td>
101 Match any single character between range x and y, inclusively. E.g., ’a’..’z’. ’a’..’z’ is identical to [a-z].</td>
102 </tr>
103
104 <tr>
105 <td>T</td><td>
106 Invoke lexer rule T; recursion is allowed in general, but not left recursion. T can be a regular token or fragment rule.
107
108 <pre>
109 ID : LETTER (LETTER|'0'..'9')* ;
110
111 fragment
112 LETTER : [a-zA-Z\u0080-\u00FF_] ;
113 </pre>
114 </td>
115 </tr>
116
117 <tr>
118 <td>.</td><td>
119 The dot is a single-character wildcard that matches any single character. Example:
120 <pre>
121 ESC : '\\' . ; // match any escaped \x character
122 </pre>
123 </td>
124 </tr>
125
126 <tr>
127 <td>{«action»}</td><td>
128 Lexer actions can appear anywhere as of 4.2, not just at the end of the outermost alternative. The lexer executes the actions at the appropriate input position, according to the placement of the action within the rule. To execute a single action for a rule that has multiple alternatives, you can enclose the alts in parentheses and put the action afterwards:
129
130 <pre>
131 END : ('endif'|'end') {System.out.println("found an end");} ;
132 </pre>
133
134 <p>The action conforms to the syntax of the target language. ANTLR copies the action’s contents into the generated code verbatim; there is no translation of expressions like $x.y as there is in parser actions.</p>
135 <p>
136 Only actions within the outermost token rule are executed. In other words, if STRING calls ESC_CHAR and ESC_CHAR has an action, that action is not executed when the lexer starts matching in STRING.</p></td>
137 </tr>
138
139 <tr>
140 <td>{«p»}?</td><td>
141 Evaluate semantic predicate «p». If «p» evaluates to false at runtime, the surrounding rule becomes “invisible” (nonviable). Expression «p» conforms to the target language syntax. While semantic predicates can appear anywhere within a lexer rule, it is most efficient to have them at the end of the rule. The one caveat is that semantic predicates must precede lexer actions. See Predicates in Lexer Rules.</td>
142 </tr>
143
144 <tr>
145 <td>~x</td><td>
146 Match any single character not in the set described by x. Set x can be a single character literal, a range, or a subrule set like ~(’x’|’y’|’z’) or ~[xyz]. Here is a rule that uses ~ to match any character other than characters using ~[\r\n]*:
147 <pre>
148 COMMENT : '#' ~[\r\n]* '\r'? '\n' -> skip ;
149 </pre>
150 </td>
151 </tr>
152 </table>
153
154 Just as with parser rules, lexer rules allow subrules in parentheses and EBNF operators: `?`, `*`, `+`. The `COMMENT` rule illustrates the `*` and `?` operators. A common use of `+` is `[0-9]+` to match integers. Lexer subrules can also use the nongreedy `?` suffix on those EBNF operators.
155
156 ## Recursive Lexer Rules
157
158 ANTLR lexer rules can be recursive, unlike most lexical grammar tools. This comes in really handy when you want to match nested tokens like nested action blocks: `{...{...}...}`.
159
160 ```
161 lexer grammar Recur;
162
163 ACTION : '{' ( ACTION | ~[{}] )* '}' ;
164
165 WS : [ \r\t\n]+ -> skip ;
166 ```
167
168 ## Redundant String Literals
169
170 Be careful that you don’t specify the same string literal on the right-hand side of multiple lexer rules. Such literals are ambiguous and could match multiple token types. ANTLR makes this literal unavailable to the parser. The same is true for rules across modes. For example, the following lexer grammar defines two tokens with the same character sequence:
171
172 ```
173 lexer grammar L;
174 AND : '&' ;
175 mode STR;
176 MASK : '&' ;
177 ```
178
179 A parser grammar cannot reference literal ’&’, but it can reference the name of the tokens:
180
181 ```
182 parser grammar P;
183 options { tokenVocab=L; }
184 a : '&' // results in a tool error: no such token
185 AND // no problem
186 MASK // no problem
187 ;
188 ```
189
190 Here’s a build and test sequence:
191
192 ```bash
193 $ antlr4 L.g4 # yields L.tokens file needed by tokenVocab option in P.g4
194 $ antlr4 P.g4
195 error(126): P.g4:3:4: cannot create implicit token for string literal '&' in non-combined grammar
196 ```
197
198 ## Lexer Rule Actions
199
200 An ANTLR lexer creates a Token object after matching a lexical rule. Each request for a token starts in `Lexer.nextToken`, which calls `emit` once it has identified a token. `emit` collects information from the current state of the lexer to build the token. It accesses fields `_type`, `_text`, `_channel`, `_tokenStartCharIndex`, `_tokenStartLine`, and `_tokenStartCharPositionInLine`. You can set the state of these with the various setter methods such as `setType`. For example, the following rule turns `enum` into an identifier if `enumIsKeyword` is false.
201
202 ```
203 ENUM : 'enum' {if (!enumIsKeyword) setType(Identifier);} ;
204 ```
205
206 ANTLR does no special `$x` attribute translations in lexer actions (unlike v3).
207
208 There can be at most a single action for a lexical rule, regardless of how many alternatives there are in that rule.
209
210 ## Lexer Commands
211
212 To avoid tying a grammar to a particular target language, ANTLR supports lexer commands. Unlike arbitrary embedded actions, these commands follow specific syntax and are limited to a few common commands. Lexer commands appear at the end of the outermost alternative of a lexer rule definition. Like arbitrary actions, there can only be one per token rule. A lexer command consists of the `->` operator followed by one or more command names that can optionally take parameters:
213
214 ```
215 TokenName : «alternative» -> command-name
216 TokenName : «alternative» -> command-name («identifier or integer»)
217 ```
218
219 An alternative can have more than one command separated by commas. Here are the valid command names:
220
221 * skip
222 * more
223 * popMode
224 * mode( x )
225 * pushMode( x )
226 * type( x )
227 * channel( x )
228
229 See the book source code for usage, some examples of which are shown here:
230
231 ### skip
232
233 A 'skip' command tells the lexer to get another token and throw out the current text.
234
235 ```
236 ID : [a-zA-Z]+ ; // match identifiers
237 INT : [0-9]+ ; // match integers
238 NEWLINE:'\r'? '\n' ; // return newlines to parser (is end-statement signal)
239 WS : [ \t]+ -> skip ; // toss out whitespace
240 ```
241
242 ### mode(), pushMode(), popMode, and more
243
244 The mode commands alter the mode stack and hence the mode of the lexer. The 'more' command forces the lexer to get another token but without throwing out the current text. The token type will be that of the "final" rule matched (i.e., the one without a more or skip command).
245
246 ```
247 // Default "mode": Everything OUTSIDE of a tag
248 COMMENT : '<!--' .*? '-->' ;
249 CDATA : '<![CDATA[' .*? ']]>' ;
250 OPEN : '<' -> pushMode(INSIDE) ;
251 ...
252 XMLDeclOpen : '<?xml' S -> pushMode(INSIDE) ;
253 SPECIAL_OPEN: '<?' Name -> more, pushMode(PROC_INSTR) ;
254 // ----------------- Everything INSIDE of a tag ---------------------
255 mode INSIDE;
256 CLOSE : '>' -> popMode ;
257 SPECIAL_CLOSE: '?>' -> popMode ; // close <?xml...?>
258 SLASH_CLOSE : '/>' -> popMode ;
259 ```
260
261 Also check out:
262
263 ```
264 lexer grammar Strings;
265 LQUOTE : '"' -> more, mode(STR) ;
266 WS : [ \r\t\n]+ -> skip ;
267 mode STR;
268 STRING : '"' -> mode(DEFAULT_MODE) ; // token we want parser to see
269 TEXT : . -> more ; // collect more text for string
270 ```
271
272 Popping the bottom layer of a mode stack will result in an exception. Switching modes with `mode` changes the current stack top. More than one `more` is the same as just one and the position does not matter.
273
274 ### type()
275
276 ```
277 lexer grammar SetType;
278 tokens { STRING }
279 DOUBLE : '"' .*? '"' -> type(STRING) ;
280 SINGLE : '\'' .*? '\'' -> type(STRING) ;
281 WS : [ \r\t\n]+ -> skip ;
282 ```
283
284 For multiple 'type()' commands, only the rightmost has an effect.
285
286 ### channel()
287
288 ```
289 BLOCK_COMMENT
290 : '/*' .*? '*/' -> channel(HIDDEN)
291 ;
292 LINE_COMMENT
293 : '//' ~[\r\n]* -> channel(HIDDEN)
294 ;
295 ...
296 // ----------
297 // Whitespace
298 //
299 // Characters and character constructs that are of no import
300 // to the parser and are used to make the grammar easier to read
301 // for humans.
302 //
303 WS : [ \t\r\n\f]+ -> channel(HIDDEN) ;
304 ```
305
306 As of 4.5, you can also define channel names like enumerations with the following construct above the lexer rules:
307
308 ```
309 channels { WSCHANNEL, MYHIDDEN }
310 ```