Debian Codegraph @

www/doc/tokenmanager.html

raw
1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
2 <html xmlns="http://www.w3.org/1999/xhtml">
3 <!--
4
5 Copyright (c) 2006, Sun Microsystems, Inc.
6 All rights reserved.
7
8 Redistribution and use in source and binary forms, with or without
9 modification, are permitted provided that the following conditions are met:
10
11 * Redistributions of source code must retain the above copyright notice,
12 this list of conditions and the following disclaimer.
13 * Redistributions in binary form must reproduce the above copyright
14 notice, this list of conditions and the following disclaimer in the
15 documentation and/or other materials provided with the distribution.
16 * Neither the name of the Sun Microsystems, Inc. nor the names of its
17 contributors may be used to endorse or promote products derived from
18 this software without specific prior written permission.
19
20 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
21 AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
22 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
23 ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
24 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
25 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
26 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
27 INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
28 CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
29 ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
30 THE POSSIBILITY OF SUCH DAMAGE.
31
32 -->
33 <head>
34 <title>JavaCC: TokenManager MiniTutorial</title>
35 <!-- Changed by: Michael Van De Vanter, 14-Jan-2003 -->
36 </head>
37 <body bgcolor="#FFFFFF" >
38
39 <h1>JavaCC [tm]: TokenManager MiniTutorial</h1>
40
41 <p>
42 The JavaCC [tm] lexical specification is organized into a set of "lexical
43 states". Each lexical state is named with an identifier. There is a
44 standard lexical state called DEFAULT. The generated token manager is
45 at any moment in one of these lexical states. When the token manager
46 is initialized, it starts off in the DEFAULT state, by default. The
47 starting lexical state can also be specified as a parameter while
48 constructing a token manager object.
49 </p>
50 <p>
51 Each lexical state contains an ordered list of regular expressions;
52 the order is derived from the order of occurrence in the input file.
53 There are four kinds of regular expressions: SKIP, MORE, TOKEN, and
54 SPECIAL_TOKEN.
55 </p>
56 <p>
57 All regular expressions that occur as expansion units in the grammar
58 are considered to be in the DEFAULT lexical state and their order of
59 occurrence is determined by their position in the grammar file.
60 </p>
61 <p>
62 A token is matched as follows: All regular expressions in the current
63 lexical state are considered as potential match candidates. The
64 token manager consumes the maximum number of characters from the input
65 stream possible that match one of these regular expressions. That is,
66 the token manager prefers the longest possible match. If there are
67 multiple longest matches (of the same length), the regular expression
68 that is matched is the one with the earliest order of occurrence in
69 the grammar file.
70 </p>
71 <p>
72 As mentioned above, the token manager is in exactly one state at any
73 moment. At this moment, the token manager only considers the regular
74 expressions defined in this state for matching purposes. After a match,
75 one can specify an action to be executed as well as a new lexical
76 state to move to. If a new lexical state is not specified, the token
77 manager remains in the current state.
78 </p>
79 <p>
80 The regular expression kind specifies what to do when a regular
81 expression has been successfully matched:
82 </p>
83 <dl>
84 <dt>SKIP</dt>
85 <dd>Simply throw away the matched string (after executing any lexical action).</dd>
86 <dt>MORE</dt>
87 <dd>Continue (to whatever the next state is) taking the matched string along. T
88 his string will be a prefix of the new matched string.</dd>
89 <dt>TOKEN</dt>
90 <dd>Create a token using the matched string and send it to the parser (or any caller).</dd>
91 <dt>SPECIAL_TOKEN</dt>
92 <dd>Creates a special token that does not participate in parsing. Already described earlier.</dd>
93 </dl>
94 <p>
95 (The mechanism of accessing special tokens is at the end of this
96 page)
97 </p>
98 <p>
99 Whenever the end of file &lt;EOF&gt; is detected, it causes the creation of
100 an &lt;EOF&gt; token (regardless of the current state of the lexical
101 analyzer). However, if an &lt;EOF&gt; is detected in the middle of a match
102 for a regular expression, or immediately after a MORE regular
103 expression has been matched, an error is reported.
104 </p>
105 <p>
106 After the regular expression is matched, the lexical action is
107 executed. All the variables (and methods) declared in the
108 TOKEN_MGR_DECLS region (see below) are available here for use. In
109 addition, the variables and methods listed below are also available
110 for use.
111 </p>
112 <p>
113 Immediately after this, the token manager changes state to that
114 specified (if any).
115 </p>
116 <p>
117 After that the action specified by the kind of the regular expression
118 is taken (SKIP, MORE, ... ). If the kind is TOKEN, the matched token
119 is returned. If the kind is SPECIAL_TOKEN, the matched token is saved
120 to be returned along with the next TOKEN that is matched.
121 </p>
122 <hr />
123 <p>
124 The following variables are available for use within lexical actions:
125 </p>
126 <ol>
127 <li>
128 StringBuffer image (READ/WRITE):
129 <br />
130 <br />
131 "image" (different from the "image" field of the matched token) is a
132 StringBuffer variable that contains all the characters that have been
133 matched since the last SKIP, TOKEN, or SPECIAL_TOKEN. You are free
134 to make whatever changes you wish to it so long as you do not assign
135 it to null (since this variable is used by the generated token manager
136 also). If you make changes to "image", this change is passed on to
137 subsequent matches (if the current match is a MORE). The content of
138 "image" *does not* automatically get assigned to the "image" field
139 of the matched token. If you wish this to happen, you must explicitly
140 assign it in a lexical action of a TOKEN or SPECIAL_TOKEN regular
141 expression.
142 <br />
143 <br />
144 Example:
145 <pre>
146 &lt;DEFAULT&gt; MORE : { "a" : S1 }
147
148 &lt;S1&gt; MORE :
149 {
150 "b"
151 { int l = image.length()-1; image.setCharAt(l, image.charAt(l).toUpperCase()); }
152 ^1 ^2
153 : S2
154 }
155
156 &lt;S2&gt; TOKEN :
157 {
158 "cd" { x = image; } : DEFAULT
159 ^3
160 }
161 </pre>
162 In the above example, the value of "image" at the 3 points marked by
163 ^1, ^2, and ^3 are:
164 <pre>
165 At ^1: "ab"
166 At ^2: "aB"
167 At ^3: "aBcd"
168 </pre>
169 </li>
170 <li> int lengthOfMatch (READ ONLY):
171 <br />
172 <br />
173 This is the length of the current match (is not cumulative over MORE's).
174 See example below. You should not modify this variable.
175 <br />
176 <br />
177 Example:
178 <br />
179 <br />
180 Using the same example as above, the values of "lengthOfMatch" are:
181 <pre>
182 At ^1: 1 (the size of "b")
183 At ^2: 1 (does not change due to lexical actions)
184 At ^3: 2 (the size of "cd")
185 </pre>
186 </li>
187 <li> int curLexState (READ ONLY):
188 <br />
189 <br />
190 This is the index of the current lexical state. You should not modify
191 this variable. Integer constants whose names are those of the lexical
192 state are generated into the ...Constants file, so you can refer to
193 lexical states without worrying about their actual index value.
194 </li>
195 <li> inputStream (READ ONLY):
196 <br />
197 <br />
198 This is an input stream of the appropriate type (one of
199 ASCII_CharStream, ASCII_UCodeESC_CharStream, UCode_CharStream, or
200 UCode_UCodeESC_CharStream depending on the values of options
201 UNICODE_INPUT and JAVA_UNICODE_ESCAPE). The stream is currently at
202 the last character consumed for this match. Methods of inputStream
203 can be called. For example, getEndLine and getEndColumn can be called
204 to get the line and column number information for the current match.
205 inputStream may not be modified.
206 </li>
207 <li> Token matchedToken (READ/WRITE):
208 <br />
209 <br />
210 This variable may be used only in actions associated with TOKEN and
211 SPECIAL_TOKEN regular expressions. This is set to be the token that
212 will get returned to the parser. You may change this variable and
213 thereby cause the changed token to be returned to the parser instead
214 of the original one. It is here that you can assign the value of
215 variable "image" to "matchedToken.image". Typically that's how your
216 changes to "image" has effect outside the lexical actions.
217 <br />
218 <br />
219 Example:
220 <br />
221 <br />
222 If we modify the last regular expression specification of the
223 above example to:
224 <pre>
225 &lt;S2&gt; TOKEN :
226 {
227 "cd" { matchedToken.image = image.toString(); } : DEFAULT
228 }
229 </pre>
230 Then the token returned to the parser will have its ".image" field
231 set to "aBcd". If this assignment was not performed, then the
232 ".image" field will remain as "abcd".
233 </li>
234 <li> void SwitchTo(int):
235 <br />
236 <br />
237 Calling this method switches you to the specified lexical state. This
238 method may be called from parser actions also (in addition to being
239 called from lexical actions). However, care must be taken when using
240 this method to switch states from the parser since the lexical
241 analysis could be many tokens ahead of the parser in the presence of
242 large lookaheads. When you use this method within a lexical action,
243 you must ensure that it is the last statement executed in the action
244 (otherwise, strange things could happen). If there is a state change
245 specified using the ": state" syntax, it overrides all switchTo calls,
246 hence there is no point having a switchTo call when there is an
247 explicit state change specified. In general, calling this method
248 should be resorted to only when you cannot do it any other way. Using
249 this method of switching states also causes you to lose some of the
250 semantic checking that JavaCC does when you use the standard syntax.
251 </li>
252 </ol>
253 <hr />
254 <p>
255 Lexical actions have access to a set of class level declarations.
256 These declarations are introduced within the JavaCC file using the
257 following syntax:
258 </p>
259 <pre>
260 token_manager_decls ::=
261 "TOKEN_MGR_DECLS" ":"
262 "{" java_declarations_and_code "}"
263 </pre>
264 <p>
265 These declarations are accessible from all lexical actions.
266 </p>
267 <h2>Examples</h2>
268 <h3>Example 1: Comments</h3>
269 <pre>
270 SKIP :
271 {
272 "/*" : WithinComment
273 }
274
275 &lt;WithinComment&gt; SKIP :
276 {
277 "*/" : DEFAULT
278 }
279
280 &lt;WithinComment&gt; MORE :
281 {
282 &lt;~[]&gt;
283 }
284 </pre>
285
286 <h3>Example 2: String Literals with actions to print the length of the string</h3>
287 <pre>
288 TOKEN_MGR_DECLS :
289 {
290 int stringSize;
291 }
292
293 MORE :
294 {
295 "\"" {stringSize = 0;} : WithinString
296 }
297
298 &lt;WithinString&gt; TOKEN :
299 {
300 &lt;STRLIT: "\""&gt; {System.out.println("Size = " + stringSize);} : DEFAULT
301 }
302
303 &lt;WithinString&gt; MORE :
304 {
305 &lt;~["\n","\r"]&gt; {stringSize++;}
306 }
307 </pre>
308
309 <h2>How special tokens are sent to the parser</h2>
310 <p>
311 Special tokens are like tokens, except that they are permitted to
312 appear anywhere in the input file (between any two tokens). Special
313 tokens can be specified in the grammar input file using the reserved
314 word "SPECIAL_TOKEN" instead of "TOKEN" as in:
315 </p>
316 <pre>
317 SPECIAL_TOKEN :
318 {
319 &lt;SINGLE_LINE_COMMENT: "//" (~["\n","\r"])* ("\n"|"\r"|"\r\n")&gt;
320 }
321 </pre>
322 <p>
323 Any regular expression defined to be a SPECIAL_TOKEN may be accessed
324 in a special manner from user actions in the lexical and grammar
325 specifications. This allows these tokens to be recovered during
326 parsing while at the same time these tokens do not participate in the
327 parsing.
328 </p>
329 <p>
330 JavaCC has been bootstrapped to use this feature to automatically
331 copy relevant comments from the input grammar file into the generated
332 files.
333 </p>
334 <p>
335 Details:
336 </p>
337 <p>
338 The class Token now has an additional field:
339 </p>
340 <pre>
341 Token specialToken;
342 </pre>
343 <p>
344 This field points to the special token immediately prior to the
345 current token (special or otherwise). If the token immediately prior
346 to the current token is a regular token (and not a special token),
347 then this field is set to null. The "next" fields of regular tokens
348 continue to have the same meaning - i.e., they point to the next
349 regular token except in the case of the EOF token where the "next"
350 field is null. The "next" field of special tokens point to the
351 special token immediately following the current token. If the token
352 immediately following the current token is a regular token, the "next"
353 field is set to null.
354 </p>
355 <p>
356 This is clarified by the following example. Suppose you wish to print
357 all special tokens prior to the regular token "t" (but only those that
358 are after the regular token before "t"):
359 </p>
360 <pre>
361 if (t.specialToken == null) return;
362 // The above statement determines that there are no special tokens
363 // and returns control to the caller.
364 Token tmp_t = t.specialToken;
365 while (tmp_t.specialToken != null) tmp_t = tmp_t.specialToken;
366 // The above line walks back the special token chain until it
367 // reaches the first special token after the previous regular
368 // token.
369 while (tmp_t != null) {
370 System.out.println(tmp_t.image);
371 tmp_t = tmp_t.next;
372 }
373 // The above loop now walks the special token chain in the forward
374 // direction printing them in the process.
375 </pre>
376 </body>
377 </html>