| 1 |
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> |
| 2 |
<html xmlns="http://www.w3.org/1999/xhtml"> |
| 3 |
<!-- |
| 4 |
|
| 5 |
Copyright (c) 2006, Sun Microsystems, Inc. |
| 6 |
All rights reserved. |
| 7 |
|
| 8 |
Redistribution and use in source and binary forms, with or without |
| 9 |
modification, are permitted provided that the following conditions are met: |
| 10 |
|
| 11 |
* Redistributions of source code must retain the above copyright notice, |
| 12 |
this list of conditions and the following disclaimer. |
| 13 |
* Redistributions in binary form must reproduce the above copyright |
| 14 |
notice, this list of conditions and the following disclaimer in the |
| 15 |
documentation and/or other materials provided with the distribution. |
| 16 |
* Neither the name of the Sun Microsystems, Inc. nor the names of its |
| 17 |
contributors may be used to endorse or promote products derived from |
| 18 |
this software without specific prior written permission. |
| 19 |
|
| 20 |
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" |
| 21 |
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE |
| 22 |
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE |
| 23 |
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE |
| 24 |
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR |
| 25 |
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF |
| 26 |
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS |
| 27 |
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN |
| 28 |
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) |
| 29 |
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF |
| 30 |
THE POSSIBILITY OF SUCH DAMAGE. |
| 31 |
|
| 32 |
--> |
| 33 |
<head> |
| 34 |
<title>JavaCC: CharStream Classes MiniTutorial</title> |
| 35 |
<!-- Changed by: Michael Van De Vanter, 14-Jan-2003 --> |
| 36 |
</head> |
| 37 |
<body bgcolor="#FFFFFF" > |
| 38 |
|
| 39 |
<h1>JavaCC [tm]: CharStream Classes MiniTutorial</h1> |
| 40 |
<p> |
| 41 |
This document describes in some detail the methods of the CharStream |
| 42 |
classes. Note that some of the details may not be relevant for the |
| 43 |
CharStream interface (to be used with USER_CHAR_STREAM). |
| 44 |
</p> |
| 45 |
<p> |
| 46 |
There are 4 different kinds of char stream classes that are generated |
| 47 |
based on combinations of various options. |
| 48 |
</p> |
| 49 |
<ul> |
| 50 |
|
| 51 |
<li> <h3>ASCII_CharStream</h3> |
| 52 |
|
| 53 |
Generated when neither of the two options - <tt>UNICODE_INPUT</tt> |
| 54 |
or <tt>JAVA_UNICODE_ESCAPE</tt> is set. |
| 55 |
|
| 56 |
<p> |
| 57 |
|
| 58 |
This class treats the input as a stream of 1-byte (ISO-LATIN1) |
| 59 |
characters. Note that this class can also be used to parse |
| 60 |
binary files. It just reads a byte and returns it as a 16 bit |
| 61 |
quantity to the lexical analyzer. So any character returned by |
| 62 |
this class will be in the range <tt>'\u0000'-'\u00ff'</tt>. |
| 63 |
|
| 64 |
</p> |
| 65 |
</li> |
| 66 |
<li> <h3>ASCII_UCodeESC_CharStream</h3> |
| 67 |
|
| 68 |
Generated when the option <tt>JAVA_UNICODE_ESCAPE</tt> is set |
| 69 |
and the <tt>UNICODE_INPUT</tt> option is not set. |
| 70 |
|
| 71 |
<p> |
| 72 |
|
| 73 |
This class treats the input as a stream of 1-byte characters. |
| 74 |
However, the special escape sequence |
| 75 |
</p> |
| 76 |
<p> |
| 77 |
<tt>("\\\\")* "\\" ("u")+</tt> - (<i>odd number of backslahes followed |
| 78 |
by one or more 'u's.</i>) |
| 79 |
</p> |
| 80 |
<p> |
| 81 |
is treated as a tag indicating that the next 4 bytes following |
| 82 |
the tag will be hexadecimal digits forming a 4-digit hex number |
| 83 |
whose value will be treated as the value of the character at the |
| 84 |
position indicated by the first backslash. Note that this value |
| 85 |
can be anything in the range <tt>0x0-0xffff</tt>. |
| 86 |
</p> |
| 87 |
|
| 88 |
</li> |
| 89 |
<li> <h3>UCode_CharStream</h3> |
| 90 |
|
| 91 |
Generated when the option <tt>UNICODE_INPUT</tt> is set and |
| 92 |
the option <tt>JAVA_UNICODE_ESCAPE</tt> is not set. |
| 93 |
|
| 94 |
<p> |
| 95 |
|
| 96 |
This class treats the input as a stream of 2-byte characters. So |
| 97 |
it reads 2 bytes <tt>b1</tt> and <tt>b2</tt> and returns them as |
| 98 |
a single character using the expression <tt> b1 << 8 | b2 </tt> |
| 99 |
assuming bigendian order. So in particular all the characters in |
| 100 |
the range <tt>0x00-0xff</tt> are assumed to be stored as 2 bytes |
| 101 |
with the first (higher-order) byte being 0. |
| 102 |
|
| 103 |
</p> |
| 104 |
</li> |
| 105 |
<li> <h3>UCode_UCodeESC_CharStream</h3> |
| 106 |
|
| 107 |
Generated when both the options <tt>UNICODE_INPUT</tt> and |
| 108 |
<tt>JAVA_UNICODE_ESCAPE</tt> are set. |
| 109 |
<p> |
| 110 |
|
| 111 |
This class input is a stream of 2-byte characters (just |
| 112 |
like the UCode_CharStream class) and the special escape sequence |
| 113 |
</p> |
| 114 |
<p> |
| 115 |
<tt>("\\\\")* "\\" ("u")+</tt> - (<i>odd number of backslahes followed |
| 116 |
by one or more 'u's.</i>) |
| 117 |
</p> |
| 118 |
<p> |
| 119 |
is treated as a tag indicating that the next 4 2-byte characters |
| 120 |
following the tag will be hexadecimal digits forming a 4-digit hex |
| 121 |
number whose value will be treated as the value of the character at the |
| 122 |
position indicated by the first backslash. Note that this value |
| 123 |
can be any value in the range <tt>0x0-0xffff</tt>. Also note that |
| 124 |
the backslash(es) and u(s) are all assumed to be given as 2-byte |
| 125 |
characters (with the higher order byte value being 0). |
| 126 |
|
| 127 |
</p> |
| 128 |
</li> |
| 129 |
</ul> |
| 130 |
|
| 131 |
<h4> |
| 132 |
Note : None of the above classes can be used to read characters in a |
| 133 |
mixed mode, i.e., some characters given as 1-byte characters and others |
| 134 |
as 2-byte characters. To do this, you need to set USER_CHAR_STREAM |
| 135 |
option to true and define your own char stream. |
| 136 |
</h4> |
| 137 |
|
| 138 |
<hr /> |
| 139 |
<p> |
| 140 |
(Throughout the following, we use the notation XXXCharStream that stands |
| 141 |
for any of the above described 4 classes.) |
| 142 |
</p> |
| 143 |
<h3>Constructors</h3> |
| 144 |
<ul> |
| 145 |
|
| 146 |
<li><tt> |
| 147 |
public XXXCharStream(java.io.InputStream dstream, |
| 148 |
int startline, int startcolumn) |
| 149 |
</tt> |
| 150 |
|
| 151 |
<p> |
| 152 |
Takes an input stream, starting line and column numbers and constructs a |
| 153 |
CharStream object. It also creates buffers of initial size 4K for buffering the |
| 154 |
characters and also for line and column numbers for each of those characters. |
| 155 |
|
| 156 |
</p> |
| 157 |
</li> |
| 158 |
|
| 159 |
<li><tt> public XXXCharStream(java.io.InputStream dstream, |
| 160 |
int startline, int startcolumn, int buffersize) |
| 161 |
</tt> |
| 162 |
|
| 163 |
<p> |
| 164 |
Takes an input stream, starting line and column numbers and constructs a |
| 165 |
CharStream object. It also creates buffers of initial size <tt>buffsize</tt> for buffering the |
| 166 |
characters and also for line and column numbers for each of those characters. |
| 167 |
</p> |
| 168 |
<p> |
| 169 |
So when you have an estimate on the maximum size of any token that can occur, |
| 170 |
you can use that size to optimize the buffer sizes. Note, however, that |
| 171 |
these sizes are only initial sizes and they will be expanded as and when |
| 172 |
needed (in 2K steps). |
| 173 |
</p> |
| 174 |
</li> |
| 175 |
</ul> |
| 176 |
|
| 177 |
<h3>Methods</h3> |
| 178 |
<p> |
| 179 |
All the following methods will be static or nonstatic depending on |
| 180 |
whether the STATIC option is true or false at the generation time. Also only |
| 181 |
those methods that users can use in their lexical actions (using the |
| 182 |
<tt>input_stream</tt> variable of the lexical analyzer) are documented |
| 183 |
here. Rest of the (public) methods are very tightly coupled with the |
| 184 |
implementation of the lexical analyzer and thus <b> should not </b> be |
| 185 |
used in lexical actions. In the future when we adopt version 1.1 of the Java [tm] programming language, we will |
| 186 |
streamline this by making that part of the interface an innerclass to |
| 187 |
the lexical analyzer. |
| 188 |
</p> |
| 189 |
<ul> |
| 190 |
|
| 191 |
<li> <tt>public final char readChar() throws java.io.IOException</tt> |
| 192 |
<p> |
| 193 |
|
| 194 |
This method returns the next "character" in the input according |
| 195 |
to the rules of the CharStream class as described above. It will |
| 196 |
throw <tt>java.io.IOException</tt> if it reaches EOF during the |
| 197 |
process of "constructing" the character. It also updates the line |
| 198 |
and column number and buffers the character for any possible |
| 199 |
backtracking that may be required later. It also stores the line |
| 200 |
and column numbers for the same purpose. |
| 201 |
</p> |
| 202 |
</li> |
| 203 |
<li> <tt> public final int getBeginLine() </tt> |
| 204 |
|
| 205 |
<p> |
| 206 |
This method returns the line number for the beginning of the |
| 207 |
current match. |
| 208 |
</p> |
| 209 |
</li> |
| 210 |
<li> <tt> public final int getBeginColumn() </tt> |
| 211 |
|
| 212 |
<p> |
| 213 |
This method returns the column number for the beginning of the |
| 214 |
current match. |
| 215 |
</p> |
| 216 |
</li> |
| 217 |
<li> <tt> public final int getEndLine() </tt> |
| 218 |
|
| 219 |
<p> |
| 220 |
This method returns the line number for the ending of the |
| 221 |
current match. |
| 222 |
|
| 223 |
</p> |
| 224 |
</li> |
| 225 |
<li> <tt> public final int getEndColumn() </tt> |
| 226 |
|
| 227 |
<p> |
| 228 |
This method returns the column number for the ending of the |
| 229 |
current match. |
| 230 |
</p> |
| 231 |
</li> |
| 232 |
<li> <tt> public final void backup(int amount) </tt> |
| 233 |
|
| 234 |
<p> |
| 235 |
This method puts back <tt>amount</tt> number of characters |
| 236 |
into the stream. Note that the amount indicates the number |
| 237 |
of characters as constructed by <tt>readChar</tt>. Since the |
| 238 |
buffers used are circular buffers, it cannot check for |
| 239 |
illegal <tt>amount</tt> values, it just wraps around. So it |
| 240 |
is the user's responsibility to make sure that those many |
| 241 |
characters are really produced before a call to this method. |
| 242 |
</p> |
| 243 |
</li> |
| 244 |
<li><tt> public final String GetImage()</tt> |
| 245 |
|
| 246 |
<p> |
| 247 |
Returns the image of the current match. As far as the XXXCharStream |
| 248 |
is concerned, all characters after the last call to the private method |
| 249 |
<tt>BeginToken</tt> are considered a part of the current match. |
| 250 |
</p> |
| 251 |
</li> |
| 252 |
<li><tt> public void ReInit(java.io.InputStream dstream, |
| 253 |
int startline, int startcolumn)</tt> |
| 254 |
|
| 255 |
<p> |
| 256 |
|
| 257 |
This method reinitializes the XXXCharStream classes with a (possibly |
| 258 |
new) input stream and starting line and column numbers. |
| 259 |
</p> |
| 260 |
</li> |
| 261 |
<li><tt> public void ReInit(java.io.InputStream dstream, |
| 262 |
int startline, int startcolumn, int buffersize)</tt> |
| 263 |
|
| 264 |
<p> |
| 265 |
|
| 266 |
This method reinitializes the XXXCharStream classes with a (possibly |
| 267 |
new) input stream and starting line and column numbers and adjusts |
| 268 |
the size of the buffers to <tt>buffersize</tt>, by extending them. |
| 269 |
Note that if the value of <tt>buffersize</tt> is less than the current |
| 270 |
buffer sizes, they remain unchanged. |
| 271 |
</p> |
| 272 |
</li> |
| 273 |
<li><tt> public void adjustBeginLineColumn(int newLine, int newCol)</tt> |
| 274 |
<p> |
| 275 |
|
| 276 |
This method adjusts the line and column number of the beginning of |
| 277 |
the current match to <tt>newLine</tt> and <tt>newCol</tt> and |
| 278 |
also adjusts the line and column numbers for all the characters |
| 279 |
in the lookahead buffer. |
| 280 |
</p> |
| 281 |
</li> |
| 282 |
</ul> |
| 283 |
|
| 284 |
</body> |
| 285 |
</html> |