Debian Codegraph @

www/doc/javaccgrm.html

raw
1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
2 <html xmlns="http://www.w3.org/1999/xhtml">
3 <!--
4
5 Copyright (c) 2006, Sun Microsystems, Inc.
6 All rights reserved.
7
8 Redistribution and use in source and binary forms, with or without
9 modification, are permitted provided that the following conditions are met:
10
11 * Redistributions of source code must retain the above copyright notice,
12 this list of conditions and the following disclaimer.
13 * Redistributions in binary form must reproduce the above copyright
14 notice, this list of conditions and the following disclaimer in the
15 documentation and/or other materials provided with the distribution.
16 * Neither the name of the Sun Microsystems, Inc. nor the names of its
17 contributors may be used to endorse or promote products derived from
18 this software without specific prior written permission.
19
20 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
21 AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
22 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
23 ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
24 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
25 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
26 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
27 INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
28 CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
29 ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
30 THE POSSIBILITY OF SUCH DAMAGE.
31
32 -->
33 <head>
34 <title>JavaCC Grammar Files</title>
35 <!-- Changed by: Michael Van De Vanter, 14-Jan-2003 -->
36 </head>
37 <body bgcolor="#FFFFFF" >
38
39 <h1>JavaCC [tm]: Grammar Files</h1>
40 <p>
41 This page contains the complete syntax of Java Compiler Compiler [tm]
42 grammar files with detailed explanations of each construct.
43 </p>
44 <p>
45 Tokens in the grammar files follow the same conventions as for the Java programming language.
46 Hence identifiers, strings, characters, etc. used in the grammars are
47 the same as Java identifiers, Java strings, Java characters, etc.
48 </p>
49 <p>
50 <em>White space</em> in the grammar files also follows the same conventions as
51 for the Java programming language. This includes the syntax for comments. Most comments present in
52 the grammar files are generated into the generated parser/lexical analyzer.
53 </p>
54 <p>
55 Grammar files are preprocessed for Unicode escapes just as Java files
56 are (i.e., occurrences of strings such as <code>\uxxxx</code> - where <code>xxxx</code> is a hex value -
57 are converted to the corresponding Unicode character before lexical analysis).
58 </p>
59 <p>
60 <em>Exceptions to the above rules:</em>
61 The Java operators "<code>&lt;&lt;</code>", "<code>&gt;&gt;</code>", "<code>&gt;&gt;&gt;</code>", "<code>&lt;&lt;=</code>",
62 "<code>&gt;&gt;=</code>", and "<code>&gt;&gt;&gt;=</code>" are left out of JavaCC's input token list
63 in order to allow convenient nested use of token specifications.
64 Finally, the following are the additional reserved words in the Java Compiler
65 Compiler [tm] grammar files.
66 </p>
67
68 <table cellpadding="3">
69 <tr>
70 <td align="left"><strong>EOF</strong></td>
71 <td align="left"><strong><a href="#IGNORE_CASE">IGNORE_CASE</a></strong></td>
72 <td align="left"><strong><a href="#JAVACODE">JAVACODE</a></strong></td>
73 <td align="left"><strong><a href="#LOOKAHEAD">LOOKAHEAD</a></strong></td>
74 </tr>
75 <tr>
76 <td align="left"><strong><a href="#MORE">MORE</a></strong></td>
77 <td align="left"><strong><a href="#PARSER_BEGIN">PARSER_BEGIN</a></strong></td>
78 <td align="left"><strong><a href="#PARSER_END">PARSER_END</a></strong></td>
79 <td align="left"><strong><a href="#SKIP">SKIP</a></strong></td>
80 </tr>
81 <tr>
82 <td align="left"><strong><a href="#SPECIAL_TOKEN">SPECIAL_TOKEN</a></strong></td>
83 <td align="left"><strong><a href="#TOKEN">TOKEN</a></strong></td>
84 <td align="left"><strong><a href="#TOKEN_MGR_DECLS">TOKEN_MGR_DECLS</a></strong></td>
85 </tr>
86 </table>
87
88 <p>
89 Any Java entities used in the grammar rules that follow appear italicized
90 with the prefix <em>java_</em> (<em>e.g.</em>, <em>java_compilation_unit</em>).
91 </p>
92
93 <hr />
94
95 <a name="PARSER_BEGIN"></a><a name="PARSER_END"></a>
96 <table>
97 <tr>
98 <td align="right" valign="baseline"><a name="prod1">javacc_input</a></td>
99 <td align="center" valign="baseline">::=</td>
100 <td align="left" valign="baseline"><a href="#prod2">javacc_options</a></td>
101 </tr>
102 <tr>
103 <td></td><td></td>
104 <td align="left" valign="baseline">"PARSER_BEGIN" "(" &lt;IDENTIFIER&gt; ")"</td>
105 </tr>
106 <tr>
107 <td></td><td></td>
108 <td align="left" valign="baseline"><em>java_compilation_unit</em></td>
109 </tr>
110 <tr>
111 <td></td><td></td>
112 <td align="left" valign="baseline">"PARSER_END" "(" &lt;IDENTIFIER&gt; ")"</td>
113 </tr>
114 <tr>
115 <td></td><td></td>
116 <td align="left" valign="baseline">( <a href="#prod5">production</a> )*</td>
117 </tr>
118 <tr>
119 <td></td><td></td>
120 <td align="left" valign="baseline">&lt;EOF&gt;</td>
121 </tr>
122 </table>
123
124 <p>
125 The grammar file starts with a list of options (which is optional).
126 This is then followed by a Java compilation unit enclosed between
127 "PARSER_BEGIN(name)" and "PARSER_END(name)". After this is a list
128 of grammar productions. <a href="#prod2">Options</a> and
129 <a href="#prod5">productions</a> are described later.
130 </p>
131 <p>
132 The <em>name</em> that follows "PARSER_BEGIN" and "PARSER_END" must
133 be the same and this identifies the name of the generated parser.
134 For example, if <em>name</em> is "MyParser", then the following files
135 are generated:
136 </p>
137 <p>
138 <strong>MyParser.java:</strong>
139 The generate parser.
140 <br />
141 <strong>MyParserTokenManager.java:</strong>
142 The generated token manager (or scanner/lexical analyzer).
143 <br />
144 <strong>MyParserConstants.java:</strong>
145 A bunch of useful constants.
146 </p>
147 <p>
148 Other files such as "Token.java", "ParseException.java", etc. are also
149 generated. However, these files contain boilerplate code and are
150 the same for any grammar and may be reused across grammars (provided the
151 grammars use compatible options).
152 </p>
153 <p>
154 Between the PARSER_BEGIN and PARSER_END constructs is a regular
155 Java compilation unit (a compilation unit in Java lingo is the entire
156 contents of a Java file). This may be any arbitrary
157 Java compilation unit so long as it contains a class declaration
158 whose name is the same as the name of the generated parser ("MyParser"
159 in the above example). Hence, in general, this part of the grammar
160 file looks like:
161 </p>
162 <pre>
163 PARSER_BEGIN(parser_name)
164 . . .
165 class parser_name . . . {
166 . . .
167 }
168 . . .
169 PARSER_END(parser_name)
170 </pre>
171 <p>
172 JavaCC does not perform detailed checks on the compilation unit, so
173 it is possible for a grammar file to pass through JavaCC and generate
174 Java files that produce errors when they are compiled.
175 </p>
176 <p>
177 If the compilation unit includes a package declaration, this is
178 included in all the generated files. If the compilation unit includes
179 imports declarations, this is included in the generated parser and
180 token manager files.
181 </p>
182 <p>
183 The generated parser file contains everything in the compilation unit
184 and, in addition, contains the generated parser code that is included at
185 the end of the parser class. For the above example, the generated
186 parser will look like:
187 </p>
188 <pre>
189 . . .
190 class parser_name . . . {
191 . . .
192 // generated parser is inserted here.
193 }
194 . . .
195 </pre>
196 <p>
197 The generated parser includes a public method declaration corresponding
198 to each non-terminal (see <a href="#prod9">javacode_production</a> and
199 <a href="#prod11">bnf_production</a>) in the grammar file. Parsing with
200 respect to a non-terminal is achieved by calling the method corresponding
201 to that non-terminal. Unlike yacc, there is no single start symbol in
202 JavaCC - one can parse with respect to any non-terminal in the grammar.
203 </p>
204 <p>
205 The generated token manager provides one public method:
206 </p>
207 <pre>
208 Token getNextToken() throws ParseError;
209 </pre>
210 <p>
211 For more details on how this method may be used, please read
212 <a href="apiroutines.html">the description of the Java Compiler Compiler
213 API</a>.
214 </p>
215
216 <hr />
217
218 <table>
219 <tr>
220 <td align="right" valign="baseline"><a name="prod2">javacc_options</a></td>
221 <td align="center" valign="baseline">::=</td>
222 <td align="left" valign="baseline">[ "<a name="options">options</a>" "{" ( <a href="#prod6">option_binding</a> )* "}" ]</td>
223 </tr>
224 </table>
225
226 <p>
227 The options if present, starts with the reserved word "options" followed
228 by a list of one or more option bindings within braces. Each option
229 binding specifies the setting of one option. The same option may not be
230 set multiple times.
231 </p>
232 <p>
233 Options may be specified either here in the grammar file, or from
234 <a href="commandline.html">the command line</a>. If the option is set
235 from <a href="commandline.html">the command line</a>, that takes precedence.
236 </p>
237 <p>
238 Option names are not case-sensitive.
239 </p>
240
241 <hr />
242
243 <table>
244 <tr>
245 <td align="right" valign="baseline"><a name="prod6">option_binding</a></td>
246 <td align="center" valign="baseline">::=</td>
247 <td align="left" valign="baseline">"LOOKAHEAD" "=" <em>java_integer_literal</em> ";"</td>
248 </tr>
249 <tr>
250 <td></td><td align="center" valign="baseline">|</td>
251 <td align="left" valign="baseline">"CHOICE_AMBIGUITY_CHECK" "=" <em>java_integer_literal</em> ";"</td>
252 </tr>
253 <tr>
254 <td></td><td align="center" valign="baseline">|</td>
255 <td align="left" valign="baseline">"OTHER_AMBIGUITY_CHECK" "=" <em>java_integer_literal</em> ";"</td>
256 </tr>
257 <tr>
258 <td></td><td align="center" valign="baseline">|</td>
259 <td align="left" valign="baseline">"STATIC" "=" <em>java_boolean_literal</em> ";"</td>
260 </tr>
261 <tr>
262 <td></td><td align="center" valign="baseline">|</td>
263 <td align="left" valign="baseline">"SUPPORT_CLASS_VISIBILITY_PUBLIC" "=" <em>java_boolean_literal</em> ";"</td>
264 </tr>
265 <tr>
266 <td></td><td align="center" valign="baseline">|</td>
267 <td align="left" valign="baseline">"DEBUG_PARSER" "=" <em>java_boolean_literal</em> ";"</td>
268 </tr>
269 <tr>
270 <td></td><td align="center" valign="baseline">|</td>
271 <td align="left" valign="baseline">"DEBUG_LOOKAHEAD" "=" <em>java_boolean_literal</em> ";"</td>
272 </tr>
273 <tr>
274 <td></td><td align="center" valign="baseline">|</td>
275 <td align="left" valign="baseline">"DEBUG_TOKEN_MANAGER" "=" <em>java_boolean_literal</em> ";"</td>
276 </tr>
277 <tr>
278 <td></td><td align="center" valign="baseline">|</td>
279 <td align="left" valign="baseline">"ERROR_REPORTING" "=" <em>java_boolean_literal</em> ";"</td>
280 </tr>
281 <tr>
282 <td></td><td align="center" valign="baseline">|</td>
283 <td align="left" valign="baseline">"JAVA_UNICODE_ESCAPE" "=" <em>java_boolean_literal</em> ";"</td>
284 </tr>
285 <tr>
286 <td></td><td align="center" valign="baseline">|</td>
287 <td align="left" valign="baseline">"UNICODE_INPUT" "=" <em>java_boolean_literal</em> ";"</td>
288 </tr>
289 <tr>
290 <td></td><td align="center" valign="baseline">|</td>
291 <td align="left" valign="baseline">"IGNORE_CASE" "=" <em>java_boolean_literal</em> ";"</td>
292 </tr>
293 <tr>
294 <td></td><td align="center" valign="baseline">|</td>
295 <td align="left" valign="baseline">"USER_TOKEN_MANAGER" "=" <em>java_boolean_literal</em> ";"</td>
296 </tr>
297 <tr>
298 <td></td><td align="center" valign="baseline">|</td>
299 <td align="left" valign="baseline">"USER_CHAR_STREAM" "=" <em>java_boolean_literal</em> ";"</td>
300 </tr>
301 <tr>
302 <td></td><td align="center" valign="baseline">|</td>
303 <td align="left" valign="baseline">"BUILD_PARSER" "=" <em>java_boolean_literal</em> ";"</td>
304 </tr>
305 <tr>
306 <td></td><td align="center" valign="baseline">|</td>
307 <td align="left" valign="baseline">"BUILD_TOKEN_MANAGER" "=" <em>java_boolean_literal</em> ";"</td>
308 </tr>
309 <tr>
310 <td></td><td align="center" valign="baseline">|</td>
311 <td align="left" valign="baseline">"TOKEN_EXTENDS" "=" <em>java_string_literal</em> ";"</td>
312 </tr>
313 <tr>
314 <td></td><td align="center" valign="baseline">|</td>
315 <td align="left" valign="baseline">"TOKEN_FACTORY" "=" <em>java_string_literal</em> ";"</td>
316 </tr>
317 <tr>
318 <td></td><td align="center" valign="baseline">|</td>
319 <td align="left" valign="baseline">"TOKEN_MANAGER_USES_PARSER" "=" <em>java_boolean_literal</em> ";"</td>
320 </tr>
321 <tr>
322 <td></td><td align="center" valign="baseline">|</td>
323 <td align="left" valign="baseline">"SANITY_CHECK" "=" <em>java_boolean_literal</em> ";"</td>
324 </tr>
325 <tr>
326 <td></td><td align="center" valign="baseline">|</td>
327 <td align="left" valign="baseline">"FORCE_LA_CHECK" "=" <em>java_boolean_literal</em> ";"</td>
328 </tr>
329 <tr>
330 <td></td><td align="center" valign="baseline">|</td>
331 <td align="left" valign="baseline">"COMMON_TOKEN_ACTION" "=" <em>java_boolean_literal</em> ";"</td>
332 </tr>
333 <tr>
334 <td></td><td align="center" valign="baseline">|</td>
335 <td align="left" valign="baseline">"CACHE_TOKENS" "=" <em>java_boolean_literal</em> ";"</td>
336 </tr>
337 <tr>
338 <td></td><td align="center" valign="baseline">|</td>
339 <td align="left" valign="baseline">"OUTPUT_DIRECTORY" "=" <em>java_string_literal</em> ";"</td>
340 </tr>
341 </table>
342
343 <ul>
344 <li>
345 <strong><a name="LOOKAHEAD">LOOKAHEAD</a>:</strong>
346 The number of tokens to look ahead before making a
347 decision at a choice point during parsing. The default value is 1.
348 The smaller this number, the faster the parser. This number may be
349 overridden for specific productions within the grammar as described
350 later. See the description of
351 <a href="lookahead.html">the lookahead algorithm</a> for complete
352 details on how lookahead works.
353 </li>
354 <li>
355 <strong>CHOICE_AMBIGUITY_CHECK:</strong>
356 This is an integer option whose default value is 2.
357 This is the number of tokens considered in checking choices of the
358 form "A | B | ..." for ambiguity. For example, if there is a common
359 two token prefix for both A and B, but no common three token prefix,
360 (assume this option is set to 3) then JavaCC can tell you to use a
361 lookahead of 3 for disambiguation purposes. And if A and B have a
362 common three token prefix, then JavaCC only tell you that you need to
363 have a lookahead of 3 <em>or more</em>. Increasing this can give you more
364 comprehensive ambiguity information at the cost of more processing
365 time. For large grammars such as the Java grammar, increasing this number
366 any further causes the checking to take too much time.
367 </li>
368 <li>
369 <strong>OTHER_AMBIGUITY_CHECK:</strong>
370 This is an integer option whose default value is 1.
371 This is the number of tokens considered in checking all other kinds of
372 choices (i.e., of the forms "(A)*", "(A)+", and "(A)?") for ambiguity.
373 This takes more time to do than the choice checking, and hence the
374 default value is set to 1 rather than 2.
375 </li>
376 <li>
377 <strong>STATIC:</strong>
378 This is a boolean option whose default value is true. If
379 true, all methods and class variables are specified as static in the
380 generated parser and token manager. This allows only one parser object to be present,
381 but it improves the performance of the parser. To perform multiple
382 parses during one run of your Java program, you will have to call the
383 <a href="apiroutines.html">ReInit()</a>
384 method to reinitialize your parser if it is static.
385 If the parser is non-static, you may use the "new" operator to
386 construct as many parsers as you wish. These can all be used
387 simultaneously from different threads.
388 </li>
389 <li>
390 <strong>DEBUG_PARSER:</strong>
391 This is a boolean option whose default value is false. This
392 option is used to obtain debugging information from the generated
393 parser. Setting this option to true causes the parser to generate
394 a trace of its actions. Tracing may be disabled by
395 calling the method <a href="apiroutines.html">disable_tracing()</a>
396 in the generated parser class. Tracing may be subsequently enabled
397 by calling the method <a href="apiroutines.html">enable_tracing()</a>
398 in the generated parser class.
399 </li>
400 <li>
401 <strong>DEBUG_LOOKAHEAD:</strong>
402 This is a boolean option whose default value is false. Setting this
403 option to true causes the parser to generate all the tracing information
404 it does when the option DEBUG_PARSER is true, and in addition, also
405 causes it to generated a trace of actions performed during
406 <a href="lookahead.html">lookahead operation</a>.
407 </li>
408 <li>
409 <strong>DEBUG_TOKEN_MANAGER:</strong>
410 This is a boolean option whose default value is false. This
411 option is used to obtain debugging information from the generated
412 token manager. Setting this option to true causes the token manager to generate
413 a trace of its actions. This trace is rather large and should only
414 be used when you have a lexical error that has been reported to you
415 and you cannot understand why. Typically, in this situation, you
416 can determine the problem by looking at the last few lines of this trace.
417 </li>
418 <li>
419 <strong>ERROR_REPORTING:</strong>
420 This is a boolean option whose default value is
421 true. Setting it to false causes errors due to parse errors to be
422 reported in somewhat less detail. The only reason to set this
423 option to false is to improve performance.
424 </li>
425 <li>
426 <strong>JAVA_UNICODE_ESCAPE:</strong>
427 This is a boolean option whose default value is
428 false. When set to true, the generated parser uses
429 an input stream object that processes Java Unicode escapes
430 (\u...) before sending characters to the token manager. By
431 default, Java Unicode escapes are not processed.
432 <br />
433 This option is ignored if either of options USER_TOKEN_MANAGER,
434 USER_CHAR_STREAM is set to true.
435 </li>
436 <li>
437 <strong>UNICODE_INPUT:</strong>
438 This is a boolean option whose default value is
439 false. When set to true, the generated parser uses
440 uses an input stream object that reads Unicode files. By default,
441 ASCII files are assumed.
442 <br />
443 This option is ignored if either of
444 options USER_TOKEN_MANAGER, USER_CHAR_STREAM is set to true.
445 </li>
446 <li>
447 <strong><a name="IGNORE_CASE">IGNORE_CASE:</a></strong>
448 This is a boolean option whose default value is false.
449 Setting this option to true causes the generated token manager to ignore
450 case in the token specifications and the input files. This is useful
451 for writing grammars for languages such as HTML. It is also possible
452 to localize the effect of IGNORE_CASE by using
453 <a href="#prod10">an alternate mechanism described later</a>.
454 </li>
455 <li>
456 <strong>USER_TOKEN_MANAGER:</strong>
457 This is a boolean option whose default value is
458 false. The default action is to generate a token manager
459 that works on the specified grammar tokens. If this
460 option is set to true, then the parser is generated to accept tokens
461 from any token manager of type "TokenManager" - this interface
462 is generated into the generated parser directory.
463 </li>
464 <li>
465 <strong>SUPPORT_CLASS_VISIBILITY_PUBLIC:</strong>
466 This is a boolean option whose default value is
467 true. The default action is to generate support classes (such as
468 Token.java, ParseException.java etc) with <em>Public</em> visibility. If
469 set to false, the classes will be generated with package-private
470 visibility.
471 </li>
472 <li>
473 <strong>USER_CHAR_STREAM:</strong>
474 This is a boolean option whose default value is
475 false. The default action is to generate a character stream reader
476 as specified by the options JAVA_UNICODE_ESCAPE and UNICODE_INPUT.
477 The generated token manager receives characters
478 from this stream reader. If this option is set to true, then the
479 token manager is generated to read characters from any character
480 stream reader of type "CharStream.java". This file is generated
481 into the generated parser directory.
482 <br />
483 This option is ignored if USER_TOKEN_MANAGER is set to true.
484 </li>
485 <li>
486 <strong>BUILD_PARSER:</strong>
487 This is a boolean option whose default value is true.
488 The default action is to generate the parser file ("MyParser.java"
489 in the above example). When set to false, the parser file is
490 not generated. Typically, this option is set to false when
491 you wish to generate only the token manager and use it without
492 the associated parser.
493 </li>
494 <li>
495 <strong>BUILD_TOKEN_MANAGER:</strong>
496 This is a boolean option whose default value is true.
497 The default action is to generate the token manager file
498 ("MyParserTokenManager.java" in the above example). When set to
499 false the token manager file is not generated. The only reason
500 to set this option to false is to save some time during parser
501 generation when you fix problems in the parser part of the grammar
502 file and leave the lexical specifications untouched.
503 </li>
504 <li>
505 <strong>TOKEN_MANAGER_USES_PARSER:</strong>
506 This is a boolean option whose default value is false.
507 When set to true, the generated token manager will include a field
508 called <CODE>parser</CODE> that references the instantiating parser
509 instance (of type <CODE>MyParser</CODE> in the above example).
510 The main reason for having a parser in a token manager is using
511 some of its logic in lexical actions.
512 This option has no effect if the STATIC option is set to true.
513 </li>
514 <li>
515 <strong>TOKEN_EXTENDS:</strong>
516 This is a string option whose default value is
517 "", meaning that the generated Token class will extend
518 java.lang.Object. This option may be set to the name of a
519 class that will be used as the base class for the generated
520 <code>Token</code> class.
521 </li>
522 <li>
523 <strong>TOKEN_FACTORY:</strong>
524 This is a string option whose default value is
525 "", meaning that Tokens will be created by calling
526 <code>Token.newToken()</code>. If set the option names a
527 Token factory class containing a
528 <code>public static Token newToken(int ofKind, String image)</code>
529 method.
530 </li>
531 <li>
532 <strong>SANITY_CHECK:</strong>
533 This is a boolean option whose default value is true.
534 JavaCC performs many syntactic and semantic checks on the grammar
535 file during parser generation. Some checks such as detection of
536 left recursion, detection of ambiguity, and bad usage of empty
537 expansions may be suppressed for faster parser generation by
538 setting this option to false. Note that the presence of these
539 errors (even if they are not detected and reported by setting this
540 option to false) can cause unexpected behavior from the generated
541 parser.
542 </li>
543 <li>
544 <strong>FORCE_LA_CHECK:</strong>
545 This is a boolean option whose default value is false.
546 This option setting controls lookahead ambiguity checking performed
547 by JavaCC. By default (when this option is false), lookahead
548 ambiguity checking is performed for all choice points where the
549 default lookahead of 1 is used. Lookahead ambiguity checking is
550 not performed at choice points where there is an
551 <a href="lookahead.html">explicit lookahead specification</a>,
552 or if the option LOOKAHEAD is set to something other than 1.
553 Setting this option to true performs lookahead ambiguity checking
554 at <em>all</em> choice points regardless of the lookahead specifications
555 in the grammar file.
556 </li>
557 <li>
558 <strong>COMMON_TOKEN_ACTION:</strong>
559 This is a boolean option whose default value is false.
560 When set to true, every call to the token manager's method
561 "getNextToken" (<a href="apiroutines.html">see the description of the
562 Java Compiler Compiler API</a>) will cause a call to a used defined
563 method "CommonTokenAction" after the token has been scanned in by the
564 token manager. The user must define this method within the
565 <a href="#prod12">TOKEN_MGR_DECLS</a> section.
566 The signature of this method is:
567 <pre>
568 void CommonTokenAction(Token t)
569 </pre>
570 </li>
571 <li>
572 <strong>CACHE_TOKENS:</strong>
573 This is a boolean option whose default value is false.
574 Setting this option to true causes the generated parser to lookahead for
575 extra tokens ahead of time. This facilitates some performance improvements.
576 However, in this case (when the option is true), interactive
577 applications may not work since the parser needs to work synchronously
578 with the availability of tokens from the input stream. In such cases,
579 it's best to leave this option at its default value.
580 </li>
581 <li>
582 <strong>OUTPUT_DIRECTORY:</strong>
583 This is a string valued option whose default value is the current
584 directory. This controls where output files are generated.
585 </li>
586 </ul>
587
588 <hr />
589
590 <table>
591 <tr>
592 <td align="right" valign="baseline"><a name="prod5">production</a></td>
593 <td align="center" valign="baseline">::=</td>
594 <td align="left" valign="baseline"><a href="#prod9">javacode_production</a></td>
595 </tr>
596 <tr>
597 <td align="right" valign="baseline"></td>
598 <td align="center" valign="baseline">|</td>
599 <td align="left" valign="baseline"><a href="#prod10">regular_expr_production</a></td>
600 </tr>
601 <tr>
602 <td align="right" valign="baseline"></td>
603 <td align="center" valign="baseline">|</td>
604 <td align="left" valign="baseline"><a href="#prod11">bnf_production</a></td>
605 </tr>
606 <tr>
607 <td align="right" valign="baseline"></td>
608 <td align="center" valign="baseline">|</td>
609 <td align="left" valign="baseline"><a href="#prod12">token_manager_decls</a></td>
610 </tr>
611 </table>
612
613 <p>
614 There are four kinds of productions in JavaCC.
615 <a href="#prod9">javacode_production</a> and <a href="#prod11">bnf_production</a>
616 are used to define the grammar from which the parser is generated.
617 <a href="#prod10">regular_expr_production</a> is used to define the grammar
618 tokens - the token manager is generated from this information (as well as from
619 inline token specifications in the parser grammar).
620 <a href="#prod12">token_manager_decls</a> is used to introduce declarations
621 that get inserted into the generated token manager.
622 </p>
623
624 <hr />
625 <table>
626 <tr>
627 <td align="right" valign="baseline"><a name="prod9">javacode_production</a></td>
628 <td align="center" valign="baseline">::=</td>
629 <td align="left" valign="baseline">"<a name="JAVACODE">JAVACODE</a>"</td>
630 </tr>
631 <tr>
632 <td></td><td></td>
633 <td align="left" valign="baseline"><em>java_access_modifier</em> <em>java_return_type</em> <em>java_identifier</em> "(" <em>java_parameter_list</em> ")"</td>
634 </tr>
635 <tr>
636 <td></td><td></td>
637 <td align="left" valign="baseline"><em>java_block</em></td>
638 </tr>
639 </table>
640
641 <p>
642 The JAVACODE production is a way to write Java code for some
643 productions instead of the usual EBNF expansion. This is useful when
644 there is the need to recognize something that is not context-free
645 or for whatever reason is very difficult to write a grammar for.
646 An example of the use of JAVACODE is shown below. In this example,
647 the non-terminal "skip_to_matching_brace" consumes tokens in the input
648 stream all the way up to a matching closing brace (the opening brace
649 is assumed to have been just scanned):
650 </p>
651 <pre>
652 JAVACODE
653 void skip_to_matching_brace() {
654 <a href="apiroutines.html">Token</a> tok;
655 int nesting = 1;
656 while (true) {
657 tok = <a href="apiroutines.html">getToken</a>(1);
658 if (tok.kind == LBRACE) nesting++;
659 if (tok.kind == RBRACE) {
660 nesting--;
661 if (nesting == 0) break;
662 }
663 tok = <a href="apiroutines.html">getNextToken</a>();
664 }
665 }
666 </pre>
667 <p>
668 Care must be taken when using JAVACODE productions. While you can
669 say pretty much what you want with these productions, JavaCC simply
670 considers it a black box (that somehow performs its parsing task).
671 This becomes a problem when JAVACODE productions appear at
672 <a href="lookahead.html">choice points</a>. For example, if the
673 above JAVACODE production was referred to from the following production:
674 </p>
675 <pre>
676 void NT() :
677 {}
678 {
679 skip_to_matching_brace()
680 |
681 some_other_production()
682 }
683 </pre>
684 <p>
685 Then JavaCC would not know how to choose between the two choices.
686 On the other hand, if the JAVACODE production is used at a non-choice
687 point as in the following example, there is no problem:
688 </p>
689 <pre>
690 void NT() :
691 {}
692 {
693 "{" skip_to_matching_brace()
694 |
695 "(" parameter_list() ")"
696 }
697 </pre>
698 <p>
699 JAVACODE productions at choice points may also be preceded by syntactic or
700 semantic LOOKAHEAD, as in this example:
701 </p>
702 <pre>
703 void NT() :
704 {}
705 {
706 LOOKAHEAD( {errorOccurred} ) skip_to_matching_brace()
707 |
708 "(" parameter_list() ")"
709 }
710 </pre>
711 <!-- JavaCC *should* print a warning message, but currently doesn't (see issue 166).
712 <p>
713 When this issue is fixed ww should re-instate this paragraph.
714 When JAVACODE productions are used at choice points, JavaCC will
715 print a warning message stating this fact. You will then have to
716 insert some explicit LOOKAHEAD specifications to help JavaCC. See
717 <a href="lookahead.html">the minitutorial on LOOKAHEAD</a> for a
718 detailed guide on such issues.
719 </p>
720 -->
721 <p>
722 The default access modifier for JAVACODE productions is package private.
723 </p>
724
725 <hr />
726
727 <table>
728 <tr>
729 <td align="right" valign="baseline"><a name="prod11">bnf_production</a></td>
730 <td align="center" valign="baseline">::=</td>
731 <td align="left" valign="baseline"><em>java_access_modifier</em> <em>java_return_type</em> <em>java_identifier</em> "(" <em>java_parameter_list</em> ")" ":"</td>
732 </tr>
733 <tr>
734 <td></td><td></td>
735 <td align="left" valign="baseline"><em>java_block</em></td>
736 </tr>
737 <tr>
738 <td></td><td></td>
739 <td align="left" valign="baseline">"{" <a href="#prod16">expansion_choices</a> "}"</td>
740 </tr>
741 </table>
742
743 <p>
744 The BNF production is the standard production used
745 in specifying JavaCC grammars. Each BNF production has a left hand
746 side which is a non-terminal specification. The BNF production then
747 defines this non-terminal in terms of BNF expansions on the right hand
748 side. The non-terminal is written exactly like a declared Java method.
749 Since each non-terminal is translated into a method
750 in the generated parser, this style of writing the non-terminal makes
751 this association obvious. The name of the non-terminal is the name of
752 the method, and the parameters and return value declared are the means
753 to pass values up and down the parse tree. As will be seen later,
754 non-terminals on the right hand sides of productions are written as
755 method calls, so the passing of values up and down the tree are done
756 using exactly the same paradigm as method call and return.
757 The default access modifier for BNF productions is public.
758 </p>
759 <p>
760 There are two parts on the right hand side of an BNF production. The
761 first part is a set of arbitrary Java declarations and code (the Java
762 block). This code is generated at the beginning
763 of the method generated for the Java non-terminal. Hence, every time
764 this non-terminal is used in the parsing process, these declarations and
765 code are executed. The declarations in this part are visible to all Java
766 code in actions in the BNF expansions. JavaCC does not do any processing
767 of these declarations and code, except to skip to the matching ending
768 brace, collecting all text encountered on the way. Hence, a Java compiler
769 can detect errors in this code that has been processed by JavaCC.
770 </p>
771 <p>
772 The second part of the right hand side are the BNF expansions. This
773 is described <a href="#prod16">later</a>.
774 </p>
775
776 <hr />
777
778 <table>
779 <tr>
780 <td align="right" valign="baseline"><a name="prod10">regular_expr_production</a></td>
781 <td align="center" valign="baseline">::=</td>
782 <td align="left" valign="baseline">[ <a href="#newprod1">lexical_state_list</a> ]</td>
783 </tr>
784 <tr>
785 <td></td><td></td>
786 <td align="left" valign="baseline"><a href="#prod17">regexpr_kind</a> [ "[" "IGNORE_CASE" "]" ] ":"</td>
787 </tr>
788 <tr>
789 <td></td><td></td>
790 <td align="left" valign="baseline">"{" <a href="#prod18">regexpr_spec</a> ( "|" <a href="#prod18">regexpr_spec</a> )* "}"</td>
791 </tr>
792 </table>
793 <p>
794 A regular expression production is used to define lexical entities
795 that get processed by the generated token manager. A detailed description
796 of how the token manager works is provided in
797 <a href="tokenmanager.html">this minitutorial (click here)</a>. This
798 page describes the syntactic aspects of specifying lexical entities,
799 while <a href="tokenmanager.html">the minitutorial</a> describes how
800 these syntactic constructs tie in with how the token manager actually
801 works.
802 </p>
803 <p>
804 A regular expression production starts with a specification of the
805 lexical states for which it applies (the
806 <a href="#newprod1">lexical state list</a>).
807 There is a standard lexical state called "DEFAULT". If the
808 <a href="#newprod1">lexical state list</a> is omitted, the regular
809 expression production applies to the lexical state "DEFAULT".
810 </p>
811 <p>
812 Following this is a description of what kind of regular expression
813 production this is (<a href="#prod17">see below for what this means</a>).
814 </p>
815 <p>
816 After this is an optional "[IGNORE_CASE]". If this is present, the
817 regular expression production is case insensitive - it has the same
818 effect as the
819 <a href="#prod6">IGNORE_CASE</a>
820 option, except that in this case it applies locally to this regular
821 expression production.
822 </p>
823 <p>
824 This is then followed by a list of regular expression specifications
825 that describe in more detail the lexical entities of this regular
826 expression production.
827 </p>
828
829 <hr />
830
831 <table>
832 <tr>
833 <td align="right" valign="baseline"><a name="prod12">token_manager_decls</a></td>
834 <td align="center" valign="baseline">::=</td>
835 <td align="left" valign="baseline">"<a name="TOKEN_MGR_DECLS">TOKEN_MGR_DECLS</a>" ":" <em>java_block</em></td>
836 </tr>
837 </table>
838
839 <p>
840 The token manager declarations starts with the reserved word
841 "TOKEN_MGR_DECLS" followed by a ":" and then a set of Java declarations
842 and statements (the Java block). These declarations and statements are
843 written into the generated token manager and are accessible from within
844 <a href="#prod18">lexical actions</a>. See
845 <a href="tokenmanager.html">the minitutorial on the token manager</a>
846 for more details.
847 </p>
848 <p>
849 There can only be one token manager declaration in a JavaCC grammar file.
850 </p>
851
852 <hr />
853
854 <table>
855 <tr>
856 <td align="right" valign="baseline"><a name="newprod1">lexical_state_list</a></td>
857 <td align="center" valign="baseline">::=</td>
858 <td align="left" valign="baseline">"&lt;" "*" "&gt;"</td>
859 </tr>
860 <tr>
861 <td></td><td>|</td>
862 <td align="left" valign="baseline">"&lt;" <em>java_identifier</em> ( "," <em>java_identifier</em> )* "&gt;"</td>
863 </tr>
864 </table>
865
866 <p>
867 The lexical state list describes the set of lexical states for which
868 the corresponding <a href="#prod10">regular expression production</a>
869 applies. If this is written as "&lt;*&gt;", the regular expression production
870 applies to all lexical states. Otherwise, it applies to all the lexical
871 states in the identifier list within the angular brackets.
872 </p>
873
874
875 <hr />
876
877 <table>
878 <tr>
879 <td align="right" valign="baseline"><a name="prod17">regexpr_kind</a></td>
880 <td align="center" valign="baseline">::=</td>
881 <td align="left" valign="baseline">"TOKEN"</td>
882 </tr>
883 <tr>
884 <td align="right" valign="baseline"></td>
885 <td align="center" valign="baseline">|</td>
886 <td align="left" valign="baseline">"SPECIAL_TOKEN"</td>
887 </tr>
888 <tr>
889 <td align="right" valign="baseline"></td>
890 <td align="center" valign="baseline">|</td>
891 <td align="left" valign="baseline">"SKIP"</td>
892 </tr>
893 <tr>
894 <td align="right" valign="baseline"></td>
895 <td align="center" valign="baseline">|</td>
896 <td align="left" valign="baseline">"MORE"</td>
897 </tr>
898 </table>
899
900 <p>
901 This specifies the kind of
902 <a href="#prod10">regular expression production</a>.
903 There are four kinds:
904 </p>
905 <ul>
906 <li>
907 <strong><a name="TOKEN">TOKEN</a></strong>:
908 The regular expressions in this regular expression production describe
909 <em>tokens</em> in the grammar. The token manager creates a
910 <a href="apiroutines.html">Token</a> object for each match of such
911 a regular expression and returns it to the parser.
912 </li>
913 <li>
914 <strong><a name="SPECIAL_TOKEN">SPECIAL_TOKEN</a></strong>:
915 The regular expressions in this regular expression production describe
916 <em>special tokens</em>. Special tokens are like tokens, except that
917 they do not have significance during parsing - that is the BNF productions
918 ignore them. Special tokens are, however, still passed on to the parser
919 so that parser actions can access them. Special tokens are passed
920 to the parser by linking them to neighboring real tokens using the
921 field "specialToken" in the <a href="apiroutines.html">Token</a>
922 class. Special tokens are useful in the processing of lexical entities
923 such as comments which have no significance to parsing, but still
924 are an important part of the input file. See
925 <a href="tokenmanager.html">the minitutorial on the token manager</a>
926 for more details of special token handling.
927 </li>
928 <li>
929 <strong><a name="SKIP">SKIP</a></strong>:
930 Matches to regular expressions in this regular expression production
931 are simply skipped (ignored) by the token manager.
932 </li>
933 <li>
934 <strong><a name="MORE">MORE</a></strong>:
935 Sometimes it is useful to gradually build up a token to be passed on
936 to the parser. Matches to this kind of regular expression are stored
937 in a buffer until the next TOKEN or SPECIAL_TOKEN match. Then all
938 the matches in the buffer and the final TOKEN/SPECIAL_TOKEN match
939 are concatenated together to form one TOKEN/SPECIAL_TOKEN that is
940 passed on to the parser. If a match to a SKIP regular expression
941 follows a sequence of MORE matches, the contents of the buffer is
942 discarded.
943 </li>
944 </ul>
945
946 <hr />
947
948 <table>
949 <tr>
950 <td align="right" valign="baseline"><a name="prod18">regexpr_spec</a></td>
951 <td align="center" valign="baseline">::=</td>
952 <td align="left" valign="baseline"><a href="#prod19">regular_expression</a> [ <em>java_block</em> ] [ ":" <em>java_identifier</em> ]</td>
953 </tr>
954 </table>
955 <p>
956 The regular expression specification begins the actual description
957 of the lexical entities that are part of this
958 <a href="#prod10">regular expression production</a>.
959 Each regular expression production may contain any number of
960 regular expression specifications.
961 </p>
962 <p>
963 Each regular expression specification contains a regular expression
964 followed by a Java block (the lexical action) which is optional.
965 This is then followed by an identifier of a lexical state (which
966 is also optional). Whenever this regular expression is matched,
967 the lexical action (if any) gets executed, followed by any
968 <a href="#prod6">common token actions</a>. Then the action depending
969 on the
970 <a href="#prod17">regular expression production kind</a>
971 is taken. Finally, if a lexical state is specified, the token
972 manager moves to that lexical state for further processing (the
973 token manager starts initially in the state "DEFAULT").
974 </p>
975
976 <hr />
977
978 <table>
979 <tr>
980 <td align="right" valign="baseline"><a name="prod16">expansion_choices</a></td>
981 <td align="center" valign="baseline">::=</td>
982 <td align="left" valign="baseline"><a href="#prod20">expansion</a> ( "|" <a href="#prod20">expansion</a> )*</td>
983 </tr>
984 </table>
985
986 <p>
987 Expansion choices are written as a list of one or more expansions
988 separated by "|"s. The set of legal parses allowed by an expansion
989 choice is a legal parse of any one of the contained expansions.
990 </p>
991
992 <hr />
993
994 <table>
995 <tr>
996 <td align="right" valign="baseline"><a name="prod20">expansion</a></td>
997 <td align="center" valign="baseline">::=</td>
998 <td align="left" valign="baseline">( <a href="#prod22">expansion_unit</a> )*</td>
999 </tr>
1000 </table>
1001
1002 <p>
1003 An expansion is written as a sequence of expansion units.
1004 A concatenation of legal
1005 parses of the expansion units is a legal parse of the expansion.
1006 </p>
1007 <p>
1008 For example, the expansion "{" decls() "}" consists of three expansion
1009 units - "{", decls(), and "}". A match for the expansion is a concatenation
1010 of the matches for the individual expansion units - in this case, that would
1011 be any string that begins with a "{", ends with a "}", and contains a match
1012 for decls() in between.
1013 </p>
1014
1015 <hr />
1016
1017 <table>
1018 <tr>
1019 <td align="right" valign="baseline"><a name="prod22">expansion_unit</a></td>
1020 <td align="center" valign="baseline">::=</td>
1021 <td align="left" valign="baseline"><a href="#prod21">local_lookahead</a></td>
1022 </tr>
1023 <tr>
1024 <td align="right" valign="baseline"></td>
1025 <td align="center" valign="baseline">|</td>
1026 <td align="left" valign="baseline"><em>java_block</em></td>
1027 </tr>
1028 <tr>
1029 <td align="right" valign="baseline"></td>
1030 <td align="center" valign="baseline">|</td>
1031 <td align="left" valign="baseline">"(" <a href="#prod16">expansion_choices</a> ")" [ "+" | "*" | "?" ]</td>
1032 </tr>
1033 <tr>
1034 <td align="right" valign="baseline"></td>
1035 <td align="center" valign="baseline">|</td>
1036 <td align="left" valign="baseline">"[" <a href="#prod16">expansion_choices</a> "]"</td>
1037 </tr>
1038 <tr>
1039 <td align="right" valign="baseline"></td>
1040 <td align="center" valign="baseline">|</td>
1041 <td align="left" valign="baseline">[ <em>java_assignment_lhs</em> "=" ] <a href="#prod19">regular_expression</a></td>
1042 </tr>
1043 <tr>
1044 <td align="right" valign="baseline"></td>
1045 <td align="center" valign="baseline">|</td>
1046 <td align="left" valign="baseline">[ <em>java_assignment_lhs</em> "=" ] <em>java_identifier</em> "(" <em>java_expression_list</em> ")"</td>
1047 </tr>
1048 </table>
1049
1050 <p>
1051 An expansion unit can be a <a href="#prod21">local LOOKAHEAD specification</a>.
1052 This instructs the
1053 generated parser on how to make choices at choice points. For details
1054 on how LOOKAHEAD specifications work and how to write LOOKAHEAD specifications,
1055 <a href="lookahead.html">click here to visit the minitutorial on LOOKAHEAD</a>.
1056 </p>
1057 <p>
1058 An expansion unit can be a set of Java declarations and code enclosed
1059 within braces (the Java block). These are also called <em>parser
1060 actions</em>. This is generated into the method parsing the
1061 non-terminal at the appropriate location. This block is executed
1062 whenever the parsing process crosses this point successfully.
1063 When JavaCC processes the Java block, it does not perform any detailed
1064 syntax or semantic checking. Hence it is possible that the Java compiler
1065 will find errors in your actions that have been processed by JavaCC.
1066 <em>Actions are not executed during
1067 <a href="lookahead.html">lookahead evaluation</a>.</em>
1068 </p>
1069 <p>
1070 An expansion unit can be a parenthesized set of one or more
1071 <a href="#prod16">expansion choices</a>. In which case, a legal parse of the expansion
1072 unit is any legal parse of the nested expansion choices.
1073 The parenthesized set of expansion choices can be suffixed (optionally) by:
1074 </p>
1075 <ul>
1076 <li>
1077 <strong>"+":</strong>
1078 Then any legal parse of the expansion unit is one or more
1079 repetitions of a legal parse of the parenthesized set of
1080 expansion choices.
1081 </li>
1082 <li>
1083 <strong>"*":</strong>
1084 Then any legal parse of the expansion unit is zero or more
1085 repetitions of a legal parse of the parenthesized set of
1086 expansion choices.
1087 </li>
1088 <li>
1089 <strong>"?":</strong>
1090 Then a legal parse of the expansion unit is either the
1091 empty token sequence or any legal parse of the nested expansion choices.
1092 An alternate syntax for this construct is to enclose the
1093 expansion choices within brackets "[...]".
1094 </li>
1095 </ul>
1096 <p>
1097 An expansion unit can be a <a href="#prod19">regular expression</a>. Then a legal parse
1098 of the expansion unit is any token that matches this regular
1099 expression. When a regular expression is matched, it creates an
1100 object of type <a href="apiroutines.html">Token</a>. This object
1101 can be accessed by assigning it to a variable by prefixing the
1102 regular expression with "variable =". In general, you may have any
1103 valid Java assignment left-hand side to the left of the "=".
1104 <em>This assignment is not performed during
1105 <a href="lookahead.html">lookahead evaluation</a>.</em>
1106 </p>
1107 <p>
1108 An expansion unit can be a non-terminal (the last choice in the syntax
1109 above). In which case, it takes
1110 the form of a method call with the non-terminal name used as the
1111 name of the method. A successful parse of the non-terminal causes
1112 the parameters placed in the method call to be operated on and a
1113 value returned (in case the non-terminal was not declared to be
1114 of type "void"). The return value can be assigned (optionally) to
1115 a variable by prefixing the regular expression with "variable =".
1116 In general, you may have any
1117 valid Java assignment left-hand side to the left of the "=".
1118 <em>This assignment is not performed during
1119 <a href="lookahead.html">lookahead evaluation</a>.</em>
1120 Non-terminals may not be used in an expansion in a manner that introduces
1121 left-recursion. JavaCC checks this for you.
1122 </p>
1123
1124 <hr />
1125
1126 <table>
1127 <tr>
1128 <td align="right" valign="baseline"><a name="prod21">local_lookahead</a></td>
1129 <td align="center" valign="baseline">::=</td>
1130 <td align="left" valign="baseline">"LOOKAHEAD" "(" [ <em>java_integer_literal</em> ] [ "," ] [ <a href="#prod16">expansion_choices</a> ] [ "," ] [ "{" <em>java_expression</em> "}" ] ")"</td>
1131 </tr>
1132 </table>
1133
1134 <p>
1135 A local lookahead specification is used to influence the way the generated
1136 parser makes choices at the various
1137 <a href="lookahead.html">choice points</a>
1138 in the grammar. A local lookahead specification starts with the reserved
1139 word "LOOKAHEAD" followed by a set of lookahead constraints within parentheses.
1140 There are three different kinds of lookahead constraints - a lookahead limit
1141 (the integer literal), a syntactic lookahead (the expansion choices), and
1142 a semantic lookahead (the expression within braces). At least one lookahead
1143 constraint must be present. If more than one lookahead constraint is present,
1144 they must be separated by commas.
1145 </p>
1146 <p>
1147 For a detailed description of how lookahead works, please
1148 <a href="lookahead.html">click here to visit the minitutorial on LOOKAHEAD</a>.
1149 A brief description of each kind of lookahead constraint is given below:
1150 </p>
1151 <ul>
1152 <li>
1153 <strong>Lookahead Limit:</strong>
1154 This is the maximum number of tokens of lookahead that may be used for choice
1155 determination purposes. This overrides the default value which is specified
1156 by the <a href="#prod2">LOOKAHEAD option</a>. This lookahead limit applies
1157 only to the <a href="lookahead.html">choice point</a>
1158 at the location of the local lookahead specification.
1159 If the local lookahead specification is not at a choice point, the lookahead
1160 limit (if any) is ignored.
1161 </li>
1162 <li>
1163 <strong>Syntactic Lookahead:</strong>
1164 This is an expansion (or expansion choices) that is used for the purpose of
1165 determining whether or not the particular choice that this local lookahead
1166 specification applies to is to be taken. If this was not provided, the parser
1167 uses the expansion to be selected during lookahead determination.
1168 If the local lookahead specification is not at a
1169 <a href="lookahead.html">choice point</a>, the syntactic
1170 lookahead (if any) is ignored.
1171 </li>
1172 <li>
1173 <strong>Semantic Lookahead:</strong>
1174 This is a boolean expression that is evaluated whenever the parser crosses this
1175 point during parsing. If the expression evaluates to true, the parsing
1176 continues normally. If the expression evaluates to false and the local
1177 lookahead specification is at a <a href="lookahead.html">choice point</a>,
1178 the current choice is not taken and the next choice is considered.
1179 If the expression evaluates to false and the local lookahead specification
1180 is <em>not</em> at a choice point, then parsing aborts with a parse error.
1181 Unlike the other two lookahead constraints that are ignored at non-choice
1182 points, semantic lookahead is always evaluated. In fact, semantic lookahead
1183 is even evaluated if it is encountered during the evaluation of some other
1184 syntactic lookahead check (for more details
1185 <a href="lookahead.html">click here to visit the minitutorial on LOOKAHEAD</a>).
1186 </li>
1187 </ul>
1188 <p>
1189 <strong>Default values for lookahead constraints:</strong>
1190 If a local lookahead specification has been provided, but not all lookahead
1191 constraints have been included, then the missing ones are assigned default
1192 values as follows:
1193 </p>
1194 <ul>
1195 <li>
1196 If the lookahead limit is not provided and if the syntactic lookahead is
1197 provided, then the lookahead limit defaults to the largest integer value
1198 (2147483647). This essentially implements "infinite lookahead" - namely,
1199 look ahead as many tokens as necessary to match the syntactic lookahead that
1200 has been provided.
1201 </li>
1202 <li>
1203 If neither the lookahead limit nor the syntactic lookahead has been
1204 provided (which means the semantic lookahead is provided), the lookahead
1205 limit defaults to 0. This means that syntactic lookahead is not performed
1206 (it passes trivially), and only semantic lookahead is performed.
1207 </li>
1208 <li>
1209 If the syntactic lookahead is not provided, it defaults to the choice
1210 to which the local lookahead specification applies. If the local lookahead
1211 specification is not at a choice point, then the syntactic lookahead is
1212 ignored - hence a default value is not relevant.
1213 </li>
1214 <li>
1215 If the semantic lookahead is not provided, it defaults to the boolean
1216 expression "true". That is, it trivially passes.
1217 </li>
1218 </ul>
1219
1220 <hr />
1221
1222
1223 <table>
1224 <tr>
1225 <td align="right" valign="baseline"><a name="prod19">regular_expression</a></td>
1226 <td align="center" valign="baseline">::=</td>
1227 <td align="left" valign="baseline"><em>java_string_literal</em></td>
1228 </tr>
1229 <tr>
1230 <td align="right" valign="baseline"></td>
1231 <td align="center" valign="baseline">|</td>
1232 <td align="left" valign="baseline">"&lt;" [ [ "#" ] <em>java_identifier</em> ":" ] <a href="#prod29">complex_regular_expression_choices</a> "&gt;"</td>
1233 </tr>
1234 <tr>
1235 <td align="right" valign="baseline"></td>
1236 <td align="center" valign="baseline">|</td>
1237 <td align="left" valign="baseline">"&lt;" <em>java_identifier</em> "&gt;"</td>
1238 </tr>
1239 <tr>
1240 <td align="right" valign="baseline"></td>
1241 <td align="center" valign="baseline">|</td>
1242 <td align="left" valign="baseline">"&lt;" "EOF" "&gt;"</td>
1243 </tr>
1244 </table>
1245
1246 <p>
1247 There are two places in a grammar files where regular expressions may be
1248 written:
1249 </p>
1250 <ul>
1251 <li>
1252 Within a <a href="#prod18">regular expression specification</a>
1253 (part of a <a href="#prod10">regular expression production</a>),
1254 </li>
1255 <li>
1256 As an <a href="#prod22">expansion unit</a> with an <a href="#prod20">expansion</a>.
1257 When a regular expression is used in this manner, it is as if the regular expression
1258 were defined in the following manner at this location and then referred to by its
1259 label from the expansion unit:
1260 <pre>
1261 &lt;DEFAULT&gt; TOKEN :
1262 {
1263 regular expression
1264 }
1265 </pre>
1266 That is, this usage of regular expression can be rewritten using the other
1267 kind of usage.
1268 </li>
1269 </ul>
1270 <p>
1271 The complete details of regular expression matching by the token manager is
1272 available in
1273 <a href="tokenmanager.html">the minitutorial on the token manager</a>. The
1274 description of the syntactic constructs follows.
1275 </p>
1276 <p>
1277 The first kind of regular expression is a string literal. The input being
1278 parsed matches this regular expression if the token manager is in a
1279 <a href="#prod10">lexical state</a> for which this regular expression applies
1280 and the next set of characters in the input stream is the same (possibly with
1281 case ignored) as this string literal.
1282 </p>
1283 <p>
1284 A regular expression may also be a more <a href="#prod29">complex regular expression</a>
1285 using which more involved regular expression (than string literals can be defined).
1286 Such a regular expression is placed within angular brackets "&lt;...&gt;", and
1287 may be labeled optionally with an identifier. This label may be used to refer
1288 to this regular expression from
1289 <a href="#prod22">expansion units</a>
1290 or from within other regular expressions.
1291 If the label is preceded by a "#", then this regular expression may not be
1292 referred to from expansion units, but only from within other regular expressions.
1293 When the "#" is present, the regular expression is referred to as a
1294 "private regular expression".
1295 </p>
1296 <p>
1297 A regular expression may be a reference to some other labeled regular expression
1298 in which case it is written as the label enclosed in angular brackets "&lt;...&gt;".
1299 </p>
1300 <p>
1301 Finally, a regular expression may be a reference to the predefined regular
1302 expression "&lt;EOF&gt;" which is matched by the end of file.
1303 </p>
1304 <p>
1305 Private regular expressions are not matched as tokens by the token manager.
1306 Their purpose is solely to facilitate the definition of other more complex
1307 regular expressions.
1308 </p>
1309 <p>
1310 Consider the following example defining Java floating point literals:
1311 </p>
1312 <pre>
1313 TOKEN :
1314 {
1315 &lt; FLOATING_POINT_LITERAL:
1316 (["0"-"9"])+ "." (["0"-"9"])* (&lt;EXPONENT>)? (["f","F","d","D"])?
1317 | "." (["0"-"9"])+ (&lt;EXPONENT>)? (["f","F","d","D"])?
1318 | (["0"-"9"])+ &lt;EXPONENT> (["f","F","d","D"])?
1319 | (["0"-"9"])+ (&lt;EXPONENT>)? ["f","F","d","D"]
1320 >
1321 |
1322 &lt; #EXPONENT: ["e","E"] (["+","-"])? (["0"-"9"])+ &gt;
1323 }
1324 </pre>
1325 <p>
1326 In this example, the token FLOATING_POINT_LITERAL is defined using the
1327 definition of another token, namely, EXPONENT. The "#" before the label
1328 EXPONENT indicates that this exists solely for the purpose of defining other
1329 tokens (FLOATING_POINT_LITERAL in this case). The definition of
1330 FLOATING_POINT_LITERAL is not affected by the presence or absence of the "#".
1331 However, the token manager's behavior is. If the "#" is omitted, the
1332 token manager will
1333 erroneously recognize a string like E123 as a legal token of kind EXPONENT
1334 (instead of IDENTIFIER in the Java grammar).
1335 </p>
1336
1337 <hr />
1338
1339 <table>
1340 <tr>
1341 <td align="right" valign="baseline"><a name="prod29">complex_regular_expression_choices</a></td>
1342 <td align="center" valign="baseline">::=</td>
1343 <td align="left" valign="baseline"><a href="#prod30">complex_regular_expression</a> ( "|" <a href="#prod30">complex_regular_expression</a> )*</td>
1344 </tr>
1345 </table>
1346
1347 <p>
1348 Complex regular expression choices is made up of a list of one or more
1349 <a href="#prod30">complex regular expressions</a> separated by "|"s.
1350 A match for a complex regular expression choice is a match of any of its
1351 constituent complex regular expressions.
1352 </p>
1353
1354 <hr />
1355
1356 <table>
1357 <tr>
1358 <td align="right" valign="baseline"><a name="prod30">complex_regular_expression</a></td>
1359 <td align="center" valign="baseline">::=</td>
1360 <td align="left" valign="baseline">( <a href="#prod31">complex_regular_expression_unit</a> )*</td>
1361 </tr>
1362 </table>
1363
1364 <p>
1365 A complex regular expression is a sequence of complex regular expression units.
1366 A match for a complex regular expression is a concatenation of matches to
1367 the complex regular expression units.
1368 </p>
1369
1370 <hr />
1371
1372 <table>
1373 <tr>
1374 <td align="right" valign="baseline"><a name="prod31">complex_regular_expression_unit</a></td>
1375 <td align="center" valign="baseline">::=</td>
1376 <td align="left" valign="baseline"><em>java_string_literal</em></td>
1377 </tr>
1378 <tr>
1379 <td align="right" valign="baseline"></td>
1380 <td align="center" valign="baseline">|</td>
1381 <td align="left" valign="baseline">"&lt;" <em>java_identifier</em> "&gt;"</td>
1382 </tr>
1383 <tr>
1384 <td align="right" valign="baseline"></td>
1385 <td align="center" valign="baseline">|</td>
1386 <td align="left" valign="baseline"><a href="#prod32">character_list</a></td>
1387 </tr>
1388 <tr>
1389 <td align="right" valign="baseline"></td>
1390 <td align="center" valign="baseline">|</td>
1391 <td align="left" valign="baseline">"(" <a href="#prod29">complex_regular_expression_choices</a> ")" [ "+" | "*" | "?" ]</td>
1392 </tr>
1393 </table>
1394
1395 <p>
1396 A complex regular expression unit can be a string literal, in which case
1397 there is exactly one match for this unit, namely, the string literal itself.
1398 </p>
1399 <p>
1400 A complex regular expression unit can be a reference to another regular
1401 expression. The other regular expression has to be labeled so that it
1402 can be referenced. The matches of this unit are all the matches of this
1403 other regular expression. Such references in regular expressions cannot
1404 introduce loops in the dependency between tokens.
1405 </p>
1406 <p>
1407 A complex regular expression unit can be a <a href="#prod32">character list</a>.
1408 A character list is a way of defining a set of characters. A match for this
1409 kind of complex regular expression unit is any character that is allowed
1410 by the character list.
1411 </p>
1412 <p>
1413 A complex regular expression unit can be a parenthesized set of
1414 complex regular expression choices. In this case, a legal match of
1415 the unit is any legal match of the nested choices. The parenthesized
1416 set of choices can be suffixed (optionally) by:
1417 </p>
1418 <ul>
1419 <li>
1420 <strong>"+":</strong>
1421 Then any legal match of the unit is one or more
1422 repetitions of a legal match of the parenthesized set of
1423 choices.
1424 </li>
1425 <li>
1426 <strong>"*":</strong>
1427 Then any legal match of the unit is zero or more
1428 repetitions of a legal match of the parenthesized set of
1429 choices.
1430 </li>
1431 <li>
1432 <strong>"?":</strong>
1433 Then a legal match of the unit is either the
1434 empty string or any legal match of the nested choices.
1435 </li>
1436 </ul>
1437 <p>
1438 Note that unlike the BNF <a href="#prod20">expansions</a>,
1439 the regular expression "[...]" is not equivalent
1440 to the regular expression "(...)?". This is because the [...]
1441 construct is used to describe <a href="#prod32">character lists</a>
1442 in regular expressions.
1443 </p>
1444
1445 <hr />
1446
1447 <table>
1448 <tr>
1449 <td align="right" valign="baseline"><a name="prod32">character_list</a></td>
1450 <td align="center" valign="baseline">::=</td>
1451 <td align="left" valign="baseline">[ "~" ] "[" [ <a href="#prod33">character_descriptor</a> ( "," <a href="#prod33">character_descriptor</a> )* ] "]"</td>
1452 </tr>
1453 </table>
1454
1455 <p>
1456 A character list describes a set of characters. A legal match for a
1457 character list is any character in this set. A character list is a list
1458 of character descriptors separated by commas within square brackets.
1459 Each character descriptor describes a single character or a range of characters
1460 (see <a href="#prod33">character descriptor</a> below),
1461 and this is added to the set of characters of the character
1462 list. If the character list is prefixed by the "~" symbol, the set of
1463 characters it represents is any UNICODE character not in the specified set.
1464 </p>
1465
1466 <hr />
1467
1468 <table>
1469 <tr>
1470 <td align="right" valign="baseline"><a name="prod33">character_descriptor</a></td>
1471 <td align="center" valign="baseline">::=</td>
1472 <td align="left" valign="baseline"><em>java_string_literal</em> [ "-" <em>java_string_literal</em> ]</td>
1473 </tr>
1474 </table>
1475
1476 <p>
1477 A character descriptor can be a single character string literal, in which
1478 case it describes a singleton set containing that character; or it is
1479 two single character string literals separated by a "-", in which case, it
1480 describes the set of all characters in the range between and including these
1481 two characters.
1482 </p>
1483
1484 </body>
1485 </html>