Debian Codegraph @

www/doc/CharStream.html

raw
1 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
2 <html xmlns="http://www.w3.org/1999/xhtml">
3 <!--
4
5 Copyright (c) 2006, Sun Microsystems, Inc.
6 All rights reserved.
7
8 Redistribution and use in source and binary forms, with or without
9 modification, are permitted provided that the following conditions are met:
10
11 * Redistributions of source code must retain the above copyright notice,
12 this list of conditions and the following disclaimer.
13 * Redistributions in binary form must reproduce the above copyright
14 notice, this list of conditions and the following disclaimer in the
15 documentation and/or other materials provided with the distribution.
16 * Neither the name of the Sun Microsystems, Inc. nor the names of its
17 contributors may be used to endorse or promote products derived from
18 this software without specific prior written permission.
19
20 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
21 AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
22 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
23 ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
24 LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
25 CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
26 SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
27 INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
28 CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
29 ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
30 THE POSSIBILITY OF SUCH DAMAGE.
31
32 -->
33 <head>
34 <title>JavaCC: CharStream Classes MiniTutorial</title>
35 <!-- Changed by: Michael Van De Vanter, 14-Jan-2003 -->
36 </head>
37 <body bgcolor="#FFFFFF" >
38
39 <h1>JavaCC [tm]: CharStream Classes MiniTutorial</h1>
40 <p>
41 This document describes in some detail the methods of the CharStream
42 classes. Note that some of the details may not be relevant for the
43 CharStream interface (to be used with USER_CHAR_STREAM).
44 </p>
45 <p>
46 There are 4 different kinds of char stream classes that are generated
47 based on combinations of various options.
48 </p>
49 <ul>
50
51 <li> <h3>ASCII_CharStream</h3>
52
53 Generated when neither of the two options - <tt>UNICODE_INPUT</tt>
54 or <tt>JAVA_UNICODE_ESCAPE</tt> is set.
55
56 <p>
57
58 This class treats the input as a stream of 1-byte (ISO-LATIN1)
59 characters. Note that this class can also be used to parse
60 binary files. It just reads a byte and returns it as a 16 bit
61 quantity to the lexical analyzer. So any character returned by
62 this class will be in the range <tt>'\u0000'-'\u00ff'</tt>.
63
64 </p>
65 </li>
66 <li> <h3>ASCII_UCodeESC_CharStream</h3>
67
68 Generated when the option <tt>JAVA_UNICODE_ESCAPE</tt> is set
69 and the <tt>UNICODE_INPUT</tt> option is not set.
70
71 <p>
72
73 This class treats the input as a stream of 1-byte characters.
74 However, the special escape sequence
75 </p>
76 <p>
77 <tt>("\\\\")* "\\" ("u")+</tt> - (<i>odd number of backslahes followed
78 by one or more 'u's.</i>)
79 </p>
80 <p>
81 is treated as a tag indicating that the next 4 bytes following
82 the tag will be hexadecimal digits forming a 4-digit hex number
83 whose value will be treated as the value of the character at the
84 position indicated by the first backslash. Note that this value
85 can be anything in the range <tt>0x0-0xffff</tt>.
86 </p>
87
88 </li>
89 <li> <h3>UCode_CharStream</h3>
90
91 Generated when the option <tt>UNICODE_INPUT</tt> is set and
92 the option <tt>JAVA_UNICODE_ESCAPE</tt> is not set.
93
94 <p>
95
96 This class treats the input as a stream of 2-byte characters. So
97 it reads 2 bytes <tt>b1</tt> and <tt>b2</tt> and returns them as
98 a single character using the expression <tt> b1 &lt;&lt; 8 | b2 </tt>
99 assuming bigendian order. So in particular all the characters in
100 the range <tt>0x00-0xff</tt> are assumed to be stored as 2 bytes
101 with the first (higher-order) byte being 0.
102
103 </p>
104 </li>
105 <li> <h3>UCode_UCodeESC_CharStream</h3>
106
107 Generated when both the options <tt>UNICODE_INPUT</tt> and
108 <tt>JAVA_UNICODE_ESCAPE</tt> are set.
109 <p>
110
111 This class input is a stream of 2-byte characters (just
112 like the UCode_CharStream class) and the special escape sequence
113 </p>
114 <p>
115 <tt>("\\\\")* "\\" ("u")+</tt> - (<i>odd number of backslahes followed
116 by one or more 'u's.</i>)
117 </p>
118 <p>
119 is treated as a tag indicating that the next 4 2-byte characters
120 following the tag will be hexadecimal digits forming a 4-digit hex
121 number whose value will be treated as the value of the character at the
122 position indicated by the first backslash. Note that this value
123 can be any value in the range <tt>0x0-0xffff</tt>. Also note that
124 the backslash(es) and u(s) are all assumed to be given as 2-byte
125 characters (with the higher order byte value being 0).
126
127 </p>
128 </li>
129 </ul>
130
131 <h4>
132 Note : None of the above classes can be used to read characters in a
133 mixed mode, i.e., some characters given as 1-byte characters and others
134 as 2-byte characters. To do this, you need to set USER_CHAR_STREAM
135 option to true and define your own char stream.
136 </h4>
137
138 <hr />
139 <p>
140 (Throughout the following, we use the notation XXXCharStream that stands
141 for any of the above described 4 classes.)
142 </p>
143 <h3>Constructors</h3>
144 <ul>
145
146 <li><tt>
147 public XXXCharStream(java.io.InputStream dstream,
148 int startline, int startcolumn)
149 </tt>
150
151 <p>
152 Takes an input stream, starting line and column numbers and constructs a
153 CharStream object. It also creates buffers of initial size 4K for buffering the
154 characters and also for line and column numbers for each of those characters.
155
156 </p>
157 </li>
158
159 <li><tt> public XXXCharStream(java.io.InputStream dstream,
160 int startline, int startcolumn, int buffersize)
161 </tt>
162
163 <p>
164 Takes an input stream, starting line and column numbers and constructs a
165 CharStream object. It also creates buffers of initial size <tt>buffsize</tt> for buffering the
166 characters and also for line and column numbers for each of those characters.
167 </p>
168 <p>
169 So when you have an estimate on the maximum size of any token that can occur,
170 you can use that size to optimize the buffer sizes. Note, however, that
171 these sizes are only initial sizes and they will be expanded as and when
172 needed (in 2K steps).
173 </p>
174 </li>
175 </ul>
176
177 <h3>Methods</h3>
178 <p>
179 All the following methods will be static or nonstatic depending on
180 whether the STATIC option is true or false at the generation time. Also only
181 those methods that users can use in their lexical actions (using the
182 <tt>input_stream</tt> variable of the lexical analyzer) are documented
183 here. Rest of the (public) methods are very tightly coupled with the
184 implementation of the lexical analyzer and thus <b> should not </b> be
185 used in lexical actions. In the future when we adopt version 1.1 of the Java [tm] programming language, we will
186 streamline this by making that part of the interface an innerclass to
187 the lexical analyzer.
188 </p>
189 <ul>
190
191 <li> <tt>public final char readChar() throws java.io.IOException</tt>
192 <p>
193
194 This method returns the next "character" in the input according
195 to the rules of the CharStream class as described above. It will
196 throw <tt>java.io.IOException</tt> if it reaches EOF during the
197 process of "constructing" the character. It also updates the line
198 and column number and buffers the character for any possible
199 backtracking that may be required later. It also stores the line
200 and column numbers for the same purpose.
201 </p>
202 </li>
203 <li> <tt> public final int getBeginLine() </tt>
204
205 <p>
206 This method returns the line number for the beginning of the
207 current match.
208 </p>
209 </li>
210 <li> <tt> public final int getBeginColumn() </tt>
211
212 <p>
213 This method returns the column number for the beginning of the
214 current match.
215 </p>
216 </li>
217 <li> <tt> public final int getEndLine() </tt>
218
219 <p>
220 This method returns the line number for the ending of the
221 current match.
222
223 </p>
224 </li>
225 <li> <tt> public final int getEndColumn() </tt>
226
227 <p>
228 This method returns the column number for the ending of the
229 current match.
230 </p>
231 </li>
232 <li> <tt> public final void backup(int amount) </tt>
233
234 <p>
235 This method puts back <tt>amount</tt> number of characters
236 into the stream. Note that the amount indicates the number
237 of characters as constructed by <tt>readChar</tt>. Since the
238 buffers used are circular buffers, it cannot check for
239 illegal <tt>amount</tt> values, it just wraps around. So it
240 is the user's responsibility to make sure that those many
241 characters are really produced before a call to this method.
242 </p>
243 </li>
244 <li><tt> public final String GetImage()</tt>
245
246 <p>
247 Returns the image of the current match. As far as the XXXCharStream
248 is concerned, all characters after the last call to the private method
249 <tt>BeginToken</tt> are considered a part of the current match.
250 </p>
251 </li>
252 <li><tt> public void ReInit(java.io.InputStream dstream,
253 int startline, int startcolumn)</tt>
254
255 <p>
256
257 This method reinitializes the XXXCharStream classes with a (possibly
258 new) input stream and starting line and column numbers.
259 </p>
260 </li>
261 <li><tt> public void ReInit(java.io.InputStream dstream,
262 int startline, int startcolumn, int buffersize)</tt>
263
264 <p>
265
266 This method reinitializes the XXXCharStream classes with a (possibly
267 new) input stream and starting line and column numbers and adjusts
268 the size of the buffers to <tt>buffersize</tt>, by extending them.
269 Note that if the value of <tt>buffersize</tt> is less than the current
270 buffer sizes, they remain unchanged.
271 </p>
272 </li>
273 <li><tt> public void adjustBeginLineColumn(int newLine, int newCol)</tt>
274 <p>
275
276 This method adjusts the line and column number of the beginning of
277 the current match to <tt>newLine</tt> and <tt>newCol</tt> and
278 also adjusts the line and column numbers for all the characters
279 in the lookahead buffer.
280 </p>
281 </li>
282 </ul>
283
284 </body>
285 </html>