Debian Codegraph @

doc/lexicon.md

raw
1 # Grammar Lexicon
2
3 The lexicon of ANTLR is familiar to most programmers because it follows the syntax of C and its derivatives with some extensions for grammatical descriptions.
4
5 ## Comments
6
7 There are single-line, multiline, and Javadoc-style comments:
8
9 ```
10 /** This grammar is an example illustrating the three kinds
11 * of comments.
12 */
13 grammar T;
14 /* a multi-line
15 comment
16 */
17
18 /** This rule matches a declarator for my language */
19 decl : ID ; // match a variable name
20 ```
21
22 The Javadoc comments are hidden from the parser and are ignored at the moment. They are intended to be used only at the start of the grammar and any rule.
23
24 ## Identifiers
25
26 Token names always start with a capital letter and so do lexer rules as defined by Java’s `Character.isUpperCase` method. Parser rule names always start with a lowercase letter (those that fail `Character.isUpperCase`). The initial character can be followed by uppercase and lowercase letters, digits, and underscores. Here are some sample names:
27
28 ```
29 ID, LPAREN, RIGHT_CURLY // token names/lexer rules
30 expr, simpleDeclarator, d2, header_file // parser rule names
31 ```
32
33 Like Java, ANTLR accepts Unicode characters in ANTLR names:
34
35 <img src=images/nonascii.png width=100>
36
37 To support Unicode parser and lexer rule names, ANTLR uses the following rule:
38
39 ```
40 ID : a=NameStartChar NameChar*
41 {
42 if ( Character.isUpperCase(getText().charAt(0)) ) setType(TOKEN_REF);
43 else setType(RULE_REF);
44 }
45 ;
46 ```
47
48 Rule `NameChar` identifies the valid identifier characters:
49
50 ```
51 fragment
52 NameChar
53 : NameStartChar
54 | '0'..'9'
55 | '_'
56 | '\u00B7'
57 | '\u0300'..'\u036F'
58 | '\u203F'..'\u2040'
59 ;
60 fragment
61 NameStartChar
62 : 'A'..'Z' | 'a'..'z'
63 | '\u00C0'..'\u00D6'
64 | '\u00D8'..'\u00F6'
65 | '\u00F8'..'\u02FF'
66 | '\u0370'..'\u037D'
67 | '\u037F'..'\u1FFF'
68 | '\u200C'..'\u200D'
69 | '\u2070'..'\u218F'
70 | '\u2C00'..'\u2FEF'
71 | '\u3001'..'\uD7FF'
72 | '\uF900'..'\uFDCF'
73 | '\uFDF0'..'\uFFFD'
74 ;
75 ```
76
77 Rule `NameStartChar` is the list of characters that can start an identifier (rule, token, or label name):
78 These more or less correspond to `isJavaIdentifierPart` and `isJavaIdentifierStart` in Java’s Character class. Make sure to use the `-encoding` option on the ANTLR tool if your grammar file is not in UTF-8 format, so that ANTLR reads characters properly.
79
80 ## Literals
81
82 ANTLR does not distinguish between character and string literals as most languages do. All literal strings one or more characters in length are enclosed in single quotes such as `’;’`, `’if’`, `’>=’`, and `’\’’` (refers to the one-character string containing the single quote character). Literals never contain regular expressions.
83
84 Literals can contain Unicode escape sequences of the form `’\uXXXX’` (for Unicode code points up to `’U+FFFF’`) or `’\u{XXXXXX}’` (for all Unicode code points), where `’XXXX’` is the hexadecimal Unicode code point value.
85
86 For example, `’\u00E8’` is the French letter with a grave accent: `’è’`, and `’\u{1F4A9}’` is the famous emoji: `’💩’`.
87
88 ANTLR also understands the usual special escape sequences: `’\n’` (newline), `’\r’` (carriage return), `’\t’` (tab), `’\b’` (backspace), and `’\f’` (form feed). You can use Unicode code points directly within literals or use the Unicode escape sequences:
89
90 ```
91 grammar Foreign;
92 a : '外' ;
93 ```
94
95 The recognizers that ANTLR generates assume a character vocabulary containing all Unicode characters. The input file encoding assumed by the runtime library depends on the target language. For the Java target, the runtime library assumes files are in UTF-8. Using the factory methods in `CharStreams`, you can specify a different encoding.
96
97 ## Actions
98
99 Actions are code blocks written in the target language. You can use actions in a number of places within a grammar, but the syntax is always the same: arbitrary text surrounded by curly braces. You don’t need to escape a closing curly character if it’s in a string or comment: `"}"` or `/*}*/`. If the curlies are balanced, you also don’t need to escape }: `{...}`. Otherwise, escape extra curlies with a backslash: `\{` or `\}`. The action text should conform to the target language as specified with the language option.
100
101 Embedded code can appear in: `@header` and `@members` named actions, parser and lexer rules, exception catching specifications, attribute sections for parser rules (return values, arguments, and locals), and some rule element options (currently predicates).
102
103 The only interpretation ANTLR does inside actions relates to grammar attributes; see [Token Attributes](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference) and Chapter 10, [Attributes and Actions](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference). Actions embedded within lexer rules are emitted without any interpretation or translation into generated lexers.
104
105 ## Keywords
106
107 Here’s a list of the reserved words in ANTLR grammars:
108
109 ```
110 import, fragment, lexer, parser, grammar, returns,
111 locals, throws, catch, finally, mode, options, tokens
112 ```
113
114 Also, although it is not a keyword, do not use the word `rule` as a rule name. Further, do not use any keyword of the target language as a token, label, or rule name. For example, rule `if` would result in a generated function called `if`. That would not compile obviously.