Debian Codegraph @

doc/unicode.md

raw
1 # Lexers and Unicode text
2
3 Prior to ANTLR 4.7, generated lexers in most targets only supported part of the Unicode standard (code points up to `U+FFFF`). As of ANTLR 4.7, the lexers in all language runtimes support the full range of Unicode code points up to `U+10FFFF`.
4
5 C++, Python, Go, and Swift APIs didn't need any API changes to support Unicode code points, so we decided to leave those class interfaces as-is.
6
7 Java, C#, and JavaScript runtimes required changes and, rather than break the previous interface, we deprecated them. (The *Java-target* deprecated `ANTLRInputStream` and `ANTLRFileStream` APIs only support Unicode code points up to `U+FFFF`.) Now, those targets must create `CharStream`s from input using `CharStreams.fromPath()`, `CharStreams.fromFileName()`, etc...
8
9 A big shout out to Ben Hamilton (github bhamiltoncx) for his superhuman
10 efforts across all targets to get true support for U+10FFFF code points.
11
12 ## Example
13
14 The Java, C#, and JavaScript runtimes use the new factory style stream creation interface. For example, here is some sample Java code that uses `CharStreams.fromPath()`:
15
16 ```java
17 public static void main(String[] args) {
18 CharStream charStream = CharStreams.fromPath(Paths.get(args[0]));
19 Lexer lexer = new UnicodeLexer(charStream);
20 CommonTokenStream tokens = new CommonTokenStream(lexer);
21 tokens.fill();
22 for (Token token : tokens.getTokens()) {
23 System.out.println("Got token: " + token.toString());
24 }
25 }
26 ```
27
28 # Unicode Code Points in Lexer Grammars
29
30 To refer to Unicode [code points](https://en.wikipedia.org/wiki/Code_point)
31 in lexer grammars, use the `\u` string escape plus up to 4 hex digits. For example, to create
32 a lexer rule for a single Cyrillic character by creating a range from
33 `U+0400` to `U+04FF`:
34
35 ```ANTLR
36 CYRILLIC : '\u0400'..'\u04FF' ; // or [\u0400-\u04FF] without quotes
37 ```
38
39 Unicode literals larger than U+FFFF must use the extended `\u{12345}` syntax. For example, to create a lexer rule for a selection of smiley faces
40 from the [Emoticons Unicode block](http://www.unicode.org/charts/PDF/U1F600.pdf):
41
42 ```ANTLR
43 EMOTICONS : ('\u{1F600}' | '\u{1F602}' | '\u{1F615}') ; // or [\u{1F600}\u{1F602}\u{1F615}]
44 ```
45
46 Finally, lexer char sets can include Unicode properties. Each Unicode code point has at least one property that describes the type group to which it belongs (e.g. alpha, number, punctuation). Other properties can be the language script or special binary properties and Unicode code blocks. That means however, that a property specifies a group of code points, hence they are only allowed in lexer char sets.
47
48 ```ANTLR
49 EMOJI : [\p{Emoji}] ;
50 JAPANESE : [\p{Script=Hiragana}\p{Script=Katakana}\p{Script=Han}] ;
51 NOT_CYRILLIC : [\P{Script=Cyrillic}] ;
52 ```
53
54 See [lexer-rules.md](lexer-rules.md#lexer-rule-elements) for more detail on Unicode
55 escapes in lexer rules.
56
57 ## Migration
58
59
60 Code for **4.6** looked like this:
61
62
63 ```java
64 CharStream input = new ANTLRFileStream("myinputfile");
65 JavaLexer lexer = new JavaLexer(input);
66 CommonTokenStream tokens = new CommonTokenStream(lexer);
67 ```
68
69 (It didn't use UTF-8 by default, despite the documentation saying so previously; it actually depended on the calling environments default.)
70
71 Code for **4.7** assumes UTF-8 by default and looks like this:
72
73 ```java
74 CharStream input = CharStreams.fromFileName("inputfile");
75 JavaLexer lexer = new JavaLexer(input);
76 CommonTokenStream tokens = new CommonTokenStream(lexer);
77 ```
78
79 Or, if you'd like to specify the file encoding:
80
81 ```java
82 CharStream input = CharStreams.fromFileName("inputfile", Charset.forName("windows-1252"));
83 ```
84
85 ### Motivation
86
87 After a [lively discussion](https://github.com/antlr/antlr4/pull/1771), I (parrt) decided not to simply gut the 4.6 `ANTLRFileStream` and `ANTLRInputStream` to incorporate the new U+10FFFF functionality. I decided to *deprecate* the old interface and recommend use of the new interface to prevent confusion. My reasoning is summarized as:
88
89 * I didn't like the idea of breaking all 4.6 code. To get the previous streams to properly support > 16 bit Unicode would require a lot of changes to the method signatures.
90 * Using `int` buffer element types would double the size of memory required to hold streams in memory, given that we buffer everything (and I didn't want to change that aspect of the streams).
91 * The new factory-style interface supports creation of the smallest possible code point buffer element size according to the Unicode code points found in the input stream. This means using half as much memory
92 as the old {@link ANTLRFileStream}, which assumed 16-bit characters, for ASCII text.
93 * Through some [serious testing and performance tweaking](https://github.com/antlr/antlr4/pull/1781), the new streams perform as fast or faster than the 4.6 streams.
94
95 **WARNING**. *You should avoid using both the deprecated and the new streams* in the same application because you will see
96 a nontrivial performance degradation. This speed hit is because the
97 `Lexer`'s internal code goes from a monomorphic to megamorphic
98 dynamic dispatch to get characters from the input stream. Java's
99 on-the-fly compiler (JIT) is unable to perform the same optimizations
100 so stick with either the old or the new streams, if performance is
101 a primary concern. See the [extreme debugging and spelunking](https://github.com/antlr/antlr4/pull/1781) needed to identify this issue in our timing rig.
102
103 ### Legacy grammar using surrogate code units
104
105 Legacy grammars that did their own UTF-16 surrogate code unit matching will need to continue to use `ANTLRInputStream` (Java target) until the parser-application code can upgrade to `CharStreams` interface. Then the surrogate code unit matching should be removed from the grammar in favor of letting the new streams do the decoding.
106
107 Prior to 4.7, application code could directly pass `Token.getStartIndex()` and `Token.getStopIndex()` to Java and C# String APIs (because both used UTF-16 code units as the fundamental unit of length). With the new streams, clients will have to convert from code point indices to UTF-16 code unit indices. Here is some (Java) code to show you the necessary logic:
108
109 ```java
110 public final class CodePointCounter {
111 private final String input;
112 public int inputIndex = 0;
113 public int codePointIndex = 0;
114
115 public int advanceToIndex(int newCodePointIndex) {
116 assert newCodePointIndex >= codePointIndex;
117 while (codePointIndex < newCodePointOffset) {
118 int codePoint = Character.codePointAt(input, inputIndex);
119 inputIndex += Character.charCount(codePoint);
120 codePointIndex++;
121 }
122 return inputIndex;
123 }
124 }
125 ```
126
127 ### Character Buffering, Unbuffered streams
128
129 The ANTLR character streams still buffer all the input when you create
130 the stream, as they have done for ~20 years.
131
132 If you need unbuffered
133 access, please note that it becomes challenging to create
134 parse trees. The parse tree has to point to tokens which will either
135 point into a stale location in an unbuffered stream or you have to copy
136 the characters out of the buffer into the token. That defeats the purpose
137 of unbuffered input. See the [ANTLR 4 book](https://www.amazon.com/Definitive-ANTLR-4-Reference/dp/1934356999) "13.8 Unbuffered Character and Token Streams". Unbuffered streams are primarily
138 useful for processing infinite streams *during the parse* and require that you manually buffer characters. Use `UnbufferedCharStream` and `UnbufferedTokenStream`.
139
140 ```java
141 CharStream input = new UnbufferedCharStream(is);
142 CSVLexer lex = new CSVLexer(input); // copy text out of sliding buffer and store in tokens
143 lex.setTokenFactory(new CommonTokenFactory(true));
144 TokenStream tokens = new UnbufferedTokenStream<CommonToken>(lex);
145 CSVParser parser = new CSVParser(tokens);
146 parser.setBuildParseTree(false);
147 parser.file();
148 ```
149
150 Your grammar that needs to have embedded actions that access the tokens as they are created, but before they disappear and are garbage collected. For example,
151
152 ```
153 data : a=INT {int x = Integer.parseInt($a.text);} ;
154 ```
155
156 From the code comments of `CommonTokenFactory`:
157
158 > That `true` in `new CommonTokenFactory(true)` indicates whether `CommonToken.setText` should be called after
159 constructing tokens to explicitly set the text. This is useful for cases
160 where the input stream might not be able to provide arbitrary substrings
161 of text from the input after the lexer creates a token (e.g. the
162 implementation of `CharStream.getText` in
163 `UnbufferedCharStream` throws an
164 `UnsupportedOperationException`). Explicitly setting the token text
165 allows `Token.getText` to be called at any time regardless of the
166 input stream implementation.
167
168 *Currently, only Java, C++, and C# have these unbuffered streams implemented*.