| 1 |
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| 2 |
<html xmlns="http://www.w3.org/1999/xhtml"> |
| 3 |
<!-- |
| 4 |
|
| 5 |
Copyright (c) 2006, Sun Microsystems, Inc. |
| 6 |
All rights reserved. |
| 7 |
|
| 8 |
Redistribution and use in source and binary forms, with or without |
| 9 |
modification, are permitted provided that the following conditions are met: |
| 10 |
|
| 11 |
* Redistributions of source code must retain the above copyright notice, |
| 12 |
this list of conditions and the following disclaimer. |
| 13 |
* Redistributions in binary form must reproduce the above copyright |
| 14 |
notice, this list of conditions and the following disclaimer in the |
| 15 |
documentation and/or other materials provided with the distribution. |
| 16 |
* Neither the name of the Sun Microsystems, Inc. nor the names of its |
| 17 |
contributors may be used to endorse or promote products derived from |
| 18 |
this software without specific prior written permission. |
| 19 |
|
| 20 |
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" |
| 21 |
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE |
| 22 |
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE |
| 23 |
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE |
| 24 |
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR |
| 25 |
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF |
| 26 |
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS |
| 27 |
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN |
| 28 |
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) |
| 29 |
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF |
| 30 |
THE POSSIBILITY OF SUCH DAMAGE. |
| 31 |
|
| 32 |
--> |
| 33 |
<head> |
| 34 |
<title>JavaCC: TokenManager MiniTutorial</title> |
| 35 |
<!-- Changed by: Michael Van De Vanter, 14-Jan-2003 --> |
| 36 |
</head> |
| 37 |
<body bgcolor="#FFFFFF" > |
| 38 |
|
| 39 |
<h1>JavaCC [tm]: TokenManager MiniTutorial</h1> |
| 40 |
|
| 41 |
<p> |
| 42 |
The JavaCC [tm] lexical specification is organized into a set of "lexical |
| 43 |
states". Each lexical state is named with an identifier. There is a |
| 44 |
standard lexical state called DEFAULT. The generated token manager is |
| 45 |
at any moment in one of these lexical states. When the token manager |
| 46 |
is initialized, it starts off in the DEFAULT state, by default. The |
| 47 |
starting lexical state can also be specified as a parameter while |
| 48 |
constructing a token manager object. |
| 49 |
</p> |
| 50 |
<p> |
| 51 |
Each lexical state contains an ordered list of regular expressions; |
| 52 |
the order is derived from the order of occurrence in the input file. |
| 53 |
There are four kinds of regular expressions: SKIP, MORE, TOKEN, and |
| 54 |
SPECIAL_TOKEN. |
| 55 |
</p> |
| 56 |
<p> |
| 57 |
All regular expressions that occur as expansion units in the grammar |
| 58 |
are considered to be in the DEFAULT lexical state and their order of |
| 59 |
occurrence is determined by their position in the grammar file. |
| 60 |
</p> |
| 61 |
<p> |
| 62 |
A token is matched as follows: All regular expressions in the current |
| 63 |
lexical state are considered as potential match candidates. The |
| 64 |
token manager consumes the maximum number of characters from the input |
| 65 |
stream possible that match one of these regular expressions. That is, |
| 66 |
the token manager prefers the longest possible match. If there are |
| 67 |
multiple longest matches (of the same length), the regular expression |
| 68 |
that is matched is the one with the earliest order of occurrence in |
| 69 |
the grammar file. |
| 70 |
</p> |
| 71 |
<p> |
| 72 |
As mentioned above, the token manager is in exactly one state at any |
| 73 |
moment. At this moment, the token manager only considers the regular |
| 74 |
expressions defined in this state for matching purposes. After a match, |
| 75 |
one can specify an action to be executed as well as a new lexical |
| 76 |
state to move to. If a new lexical state is not specified, the token |
| 77 |
manager remains in the current state. |
| 78 |
</p> |
| 79 |
<p> |
| 80 |
The regular expression kind specifies what to do when a regular |
| 81 |
expression has been successfully matched: |
| 82 |
</p> |
| 83 |
<dl> |
| 84 |
<dt>SKIP</dt> |
| 85 |
<dd>Simply throw away the matched string (after executing any lexical action).</dd> |
| 86 |
<dt>MORE</dt> |
| 87 |
<dd>Continue (to whatever the next state is) taking the matched string along. T |
| 88 |
his string will be a prefix of the new matched string.</dd> |
| 89 |
<dt>TOKEN</dt> |
| 90 |
<dd>Create a token using the matched string and send it to the parser (or any caller).</dd> |
| 91 |
<dt>SPECIAL_TOKEN</dt> |
| 92 |
<dd>Creates a special token that does not participate in parsing. Already described earlier.</dd> |
| 93 |
</dl> |
| 94 |
<p> |
| 95 |
(The mechanism of accessing special tokens is at the end of this |
| 96 |
page) |
| 97 |
</p> |
| 98 |
<p> |
| 99 |
Whenever the end of file <EOF> is detected, it causes the creation of |
| 100 |
an <EOF> token (regardless of the current state of the lexical |
| 101 |
analyzer). However, if an <EOF> is detected in the middle of a match |
| 102 |
for a regular expression, or immediately after a MORE regular |
| 103 |
expression has been matched, an error is reported. |
| 104 |
</p> |
| 105 |
<p> |
| 106 |
After the regular expression is matched, the lexical action is |
| 107 |
executed. All the variables (and methods) declared in the |
| 108 |
TOKEN_MGR_DECLS region (see below) are available here for use. In |
| 109 |
addition, the variables and methods listed below are also available |
| 110 |
for use. |
| 111 |
</p> |
| 112 |
<p> |
| 113 |
Immediately after this, the token manager changes state to that |
| 114 |
specified (if any). |
| 115 |
</p> |
| 116 |
<p> |
| 117 |
After that the action specified by the kind of the regular expression |
| 118 |
is taken (SKIP, MORE, ... ). If the kind is TOKEN, the matched token |
| 119 |
is returned. If the kind is SPECIAL_TOKEN, the matched token is saved |
| 120 |
to be returned along with the next TOKEN that is matched. |
| 121 |
</p> |
| 122 |
<hr /> |
| 123 |
<p> |
| 124 |
The following variables are available for use within lexical actions: |
| 125 |
</p> |
| 126 |
<ol> |
| 127 |
<li> |
| 128 |
StringBuffer image (READ/WRITE): |
| 129 |
<br /> |
| 130 |
<br /> |
| 131 |
"image" (different from the "image" field of the matched token) is a |
| 132 |
StringBuffer variable that contains all the characters that have been |
| 133 |
matched since the last SKIP, TOKEN, or SPECIAL_TOKEN. You are free |
| 134 |
to make whatever changes you wish to it so long as you do not assign |
| 135 |
it to null (since this variable is used by the generated token manager |
| 136 |
also). If you make changes to "image", this change is passed on to |
| 137 |
subsequent matches (if the current match is a MORE). The content of |
| 138 |
"image" *does not* automatically get assigned to the "image" field |
| 139 |
of the matched token. If you wish this to happen, you must explicitly |
| 140 |
assign it in a lexical action of a TOKEN or SPECIAL_TOKEN regular |
| 141 |
expression. |
| 142 |
<br /> |
| 143 |
<br /> |
| 144 |
Example: |
| 145 |
<pre> |
| 146 |
<DEFAULT> MORE : { "a" : S1 } |
| 147 |
|
| 148 |
<S1> MORE : |
| 149 |
{ |
| 150 |
"b" |
| 151 |
{ int l = image.length()-1; image.setCharAt(l, image.charAt(l).toUpperCase()); } |
| 152 |
^1 ^2 |
| 153 |
: S2 |
| 154 |
} |
| 155 |
|
| 156 |
<S2> TOKEN : |
| 157 |
{ |
| 158 |
"cd" { x = image; } : DEFAULT |
| 159 |
^3 |
| 160 |
} |
| 161 |
</pre> |
| 162 |
In the above example, the value of "image" at the 3 points marked by |
| 163 |
^1, ^2, and ^3 are: |
| 164 |
<pre> |
| 165 |
At ^1: "ab" |
| 166 |
At ^2: "aB" |
| 167 |
At ^3: "aBcd" |
| 168 |
</pre> |
| 169 |
</li> |
| 170 |
<li> int lengthOfMatch (READ ONLY): |
| 171 |
<br /> |
| 172 |
<br /> |
| 173 |
This is the length of the current match (is not cumulative over MORE's). |
| 174 |
See example below. You should not modify this variable. |
| 175 |
<br /> |
| 176 |
<br /> |
| 177 |
Example: |
| 178 |
<br /> |
| 179 |
<br /> |
| 180 |
Using the same example as above, the values of "lengthOfMatch" are: |
| 181 |
<pre> |
| 182 |
At ^1: 1 (the size of "b") |
| 183 |
At ^2: 1 (does not change due to lexical actions) |
| 184 |
At ^3: 2 (the size of "cd") |
| 185 |
</pre> |
| 186 |
</li> |
| 187 |
<li> int curLexState (READ ONLY): |
| 188 |
<br /> |
| 189 |
<br /> |
| 190 |
This is the index of the current lexical state. You should not modify |
| 191 |
this variable. Integer constants whose names are those of the lexical |
| 192 |
state are generated into the ...Constants file, so you can refer to |
| 193 |
lexical states without worrying about their actual index value. |
| 194 |
</li> |
| 195 |
<li> inputStream (READ ONLY): |
| 196 |
<br /> |
| 197 |
<br /> |
| 198 |
This is an input stream of the appropriate type (one of |
| 199 |
ASCII_CharStream, ASCII_UCodeESC_CharStream, UCode_CharStream, or |
| 200 |
UCode_UCodeESC_CharStream depending on the values of options |
| 201 |
UNICODE_INPUT and JAVA_UNICODE_ESCAPE). The stream is currently at |
| 202 |
the last character consumed for this match. Methods of inputStream |
| 203 |
can be called. For example, getEndLine and getEndColumn can be called |
| 204 |
to get the line and column number information for the current match. |
| 205 |
inputStream may not be modified. |
| 206 |
</li> |
| 207 |
<li> Token matchedToken (READ/WRITE): |
| 208 |
<br /> |
| 209 |
<br /> |
| 210 |
This variable may be used only in actions associated with TOKEN and |
| 211 |
SPECIAL_TOKEN regular expressions. This is set to be the token that |
| 212 |
will get returned to the parser. You may change this variable and |
| 213 |
thereby cause the changed token to be returned to the parser instead |
| 214 |
of the original one. It is here that you can assign the value of |
| 215 |
variable "image" to "matchedToken.image". Typically that's how your |
| 216 |
changes to "image" has effect outside the lexical actions. |
| 217 |
<br /> |
| 218 |
<br /> |
| 219 |
Example: |
| 220 |
<br /> |
| 221 |
<br /> |
| 222 |
If we modify the last regular expression specification of the |
| 223 |
above example to: |
| 224 |
<pre> |
| 225 |
<S2> TOKEN : |
| 226 |
{ |
| 227 |
"cd" { matchedToken.image = image.toString(); } : DEFAULT |
| 228 |
} |
| 229 |
</pre> |
| 230 |
Then the token returned to the parser will have its ".image" field |
| 231 |
set to "aBcd". If this assignment was not performed, then the |
| 232 |
".image" field will remain as "abcd". |
| 233 |
</li> |
| 234 |
<li> void SwitchTo(int): |
| 235 |
<br /> |
| 236 |
<br /> |
| 237 |
Calling this method switches you to the specified lexical state. This |
| 238 |
method may be called from parser actions also (in addition to being |
| 239 |
called from lexical actions). However, care must be taken when using |
| 240 |
this method to switch states from the parser since the lexical |
| 241 |
analysis could be many tokens ahead of the parser in the presence of |
| 242 |
large lookaheads. When you use this method within a lexical action, |
| 243 |
you must ensure that it is the last statement executed in the action |
| 244 |
(otherwise, strange things could happen). If there is a state change |
| 245 |
specified using the ": state" syntax, it overrides all switchTo calls, |
| 246 |
hence there is no point having a switchTo call when there is an |
| 247 |
explicit state change specified. In general, calling this method |
| 248 |
should be resorted to only when you cannot do it any other way. Using |
| 249 |
this method of switching states also causes you to lose some of the |
| 250 |
semantic checking that JavaCC does when you use the standard syntax. |
| 251 |
</li> |
| 252 |
</ol> |
| 253 |
<hr /> |
| 254 |
<p> |
| 255 |
Lexical actions have access to a set of class level declarations. |
| 256 |
These declarations are introduced within the JavaCC file using the |
| 257 |
following syntax: |
| 258 |
</p> |
| 259 |
<pre> |
| 260 |
token_manager_decls ::= |
| 261 |
"TOKEN_MGR_DECLS" ":" |
| 262 |
"{" java_declarations_and_code "}" |
| 263 |
</pre> |
| 264 |
<p> |
| 265 |
These declarations are accessible from all lexical actions. |
| 266 |
</p> |
| 267 |
<h2>Examples</h2> |
| 268 |
<h3>Example 1: Comments</h3> |
| 269 |
<pre> |
| 270 |
SKIP : |
| 271 |
{ |
| 272 |
"/*" : WithinComment |
| 273 |
} |
| 274 |
|
| 275 |
<WithinComment> SKIP : |
| 276 |
{ |
| 277 |
"*/" : DEFAULT |
| 278 |
} |
| 279 |
|
| 280 |
<WithinComment> MORE : |
| 281 |
{ |
| 282 |
<~[]> |
| 283 |
} |
| 284 |
</pre> |
| 285 |
|
| 286 |
<h3>Example 2: String Literals with actions to print the length of the string</h3> |
| 287 |
<pre> |
| 288 |
TOKEN_MGR_DECLS : |
| 289 |
{ |
| 290 |
int stringSize; |
| 291 |
} |
| 292 |
|
| 293 |
MORE : |
| 294 |
{ |
| 295 |
"\"" {stringSize = 0;} : WithinString |
| 296 |
} |
| 297 |
|
| 298 |
<WithinString> TOKEN : |
| 299 |
{ |
| 300 |
<STRLIT: "\""> {System.out.println("Size = " + stringSize);} : DEFAULT |
| 301 |
} |
| 302 |
|
| 303 |
<WithinString> MORE : |
| 304 |
{ |
| 305 |
<~["\n","\r"]> {stringSize++;} |
| 306 |
} |
| 307 |
</pre> |
| 308 |
|
| 309 |
<h2>How special tokens are sent to the parser</h2> |
| 310 |
<p> |
| 311 |
Special tokens are like tokens, except that they are permitted to |
| 312 |
appear anywhere in the input file (between any two tokens). Special |
| 313 |
tokens can be specified in the grammar input file using the reserved |
| 314 |
word "SPECIAL_TOKEN" instead of "TOKEN" as in: |
| 315 |
</p> |
| 316 |
<pre> |
| 317 |
SPECIAL_TOKEN : |
| 318 |
{ |
| 319 |
<SINGLE_LINE_COMMENT: "//" (~["\n","\r"])* ("\n"|"\r"|"\r\n")> |
| 320 |
} |
| 321 |
</pre> |
| 322 |
<p> |
| 323 |
Any regular expression defined to be a SPECIAL_TOKEN may be accessed |
| 324 |
in a special manner from user actions in the lexical and grammar |
| 325 |
specifications. This allows these tokens to be recovered during |
| 326 |
parsing while at the same time these tokens do not participate in the |
| 327 |
parsing. |
| 328 |
</p> |
| 329 |
<p> |
| 330 |
JavaCC has been bootstrapped to use this feature to automatically |
| 331 |
copy relevant comments from the input grammar file into the generated |
| 332 |
files. |
| 333 |
</p> |
| 334 |
<p> |
| 335 |
Details: |
| 336 |
</p> |
| 337 |
<p> |
| 338 |
The class Token now has an additional field: |
| 339 |
</p> |
| 340 |
<pre> |
| 341 |
Token specialToken; |
| 342 |
</pre> |
| 343 |
<p> |
| 344 |
This field points to the special token immediately prior to the |
| 345 |
current token (special or otherwise). If the token immediately prior |
| 346 |
to the current token is a regular token (and not a special token), |
| 347 |
then this field is set to null. The "next" fields of regular tokens |
| 348 |
continue to have the same meaning - i.e., they point to the next |
| 349 |
regular token except in the case of the EOF token where the "next" |
| 350 |
field is null. The "next" field of special tokens point to the |
| 351 |
special token immediately following the current token. If the token |
| 352 |
immediately following the current token is a regular token, the "next" |
| 353 |
field is set to null. |
| 354 |
</p> |
| 355 |
<p> |
| 356 |
This is clarified by the following example. Suppose you wish to print |
| 357 |
all special tokens prior to the regular token "t" (but only those that |
| 358 |
are after the regular token before "t"): |
| 359 |
</p> |
| 360 |
<pre> |
| 361 |
if (t.specialToken == null) return; |
| 362 |
// The above statement determines that there are no special tokens |
| 363 |
// and returns control to the caller. |
| 364 |
Token tmp_t = t.specialToken; |
| 365 |
while (tmp_t.specialToken != null) tmp_t = tmp_t.specialToken; |
| 366 |
// The above line walks back the special token chain until it |
| 367 |
// reaches the first special token after the previous regular |
| 368 |
// token. |
| 369 |
while (tmp_t != null) { |
| 370 |
System.out.println(tmp_t.image); |
| 371 |
tmp_t = tmp_t.next; |
| 372 |
} |
| 373 |
// The above loop now walks the special token chain in the forward |
| 374 |
// direction printing them in the process. |
| 375 |
</pre> |
| 376 |
</body> |
| 377 |
</html> |