| 1 |
The lexical analyzer's job is to convert characters or sequences of characters |
| 2 |
read from the input stream into tokens. B()'s parser obtains its tokens by |
| 3 |
calling its lexical analyzer member (tt(lex)), which is a predefined member |
| 4 |
of the parser class. See section ref(LEX). |
| 5 |
|
| 6 |
Our calculator only needs a simple lexical analyzer. It skips blanks and tabs, |
| 7 |
then reads numbers, returning them as tt(NUM) tokens. Any other character that |
| 8 |
isn't part of a number is a separate token. Note that the token-value for such |
| 9 |
a single-character token is the character itself. |
| 10 |
|
| 11 |
The lexical analyzer function's return value is a numeric value representing a |
| 12 |
token. If the token is a literal character, then its numeric code is the |
| 13 |
character's tt(ASCII) code; if the token is the name of a token defined by b() |
| 14 |
in a grammar specification file, then such named tokens are defined by b() in |
| 15 |
a bf(C++) enumeration. In the current example, therefore, tt(NUM) becomes an |
| 16 |
enumeration value, returned by the tt(lex) member as the `value' tt(NUM). |
| 17 |
|
| 18 |
Semantic values of nonterminals are stored in the parser's data member |
| 19 |
tt(d_val_) (comparable to the variable tt(yylval) used by, e.g., Bison). This |
| 20 |
data member has tt(int) as its default type, but by specifying tt(%stype) in |
| 21 |
the grammar file's directive section this default type is modified (to, e.g., |
| 22 |
tt(double)). |
| 23 |
|
| 24 |
A token value of zero is returned once end-of-file is encountered (the parsing |
| 25 |
function produced by b() interprets any nonpositive token value as |
| 26 |
end-of-input). |
| 27 |
|
| 28 |
Here is the lexical scanner's implementation: |
| 29 |
verbinclude(rpn/parser/lex.cc) |