Debian Codegraph @

documentation/manual/grammar/symbols.yo

raw
1 em(Symbols) are the building blocks of b() grammars.
2
3 A em(terminal symbol) (also known as a em(token)) represents a class of
4 syntactically equivalent symbols. Tokens are represented in b()'s parser class
5 by a number, defined in an enum. The parser's tt(lex) member function returns
6 a token value indicating what kind of token has been read. You don't need to
7 know what the code value is; instead, its symbol should always be used. By
8 convention, it contains uppercase characters.
9
10 em(Nonterminal symbol) define concepts of grammars. The symbol name is used in
11 writing grammar rules. By convention, it contains lowercase characters.
12
13 Symbol names consist of letters, digits (not at the beginning), and
14 underscores. B() does not support periods in symbol names
15 (users familiar with Bison may observe that Bison em(does) support periods in
16 symbol names, but the Bison's user guide states that `periods make sense only
17 in nonterminals'. Even so, it appears that periods in symbols are hardly ever
18 used).
19
20 There are two ways terminal symbols can be referred to:
21 itemization(
22 it() A em(named token) is an identifier, like an identifier in
23 bf(C++). Each token name must be defined with a b() directive such as
24 tt(%token). See section ref(TOKTYPENAMES).
25 it() A tt(character token) (or tt(literal character token)) is written in
26 the grammar using bf(C++)'s character constants syntax; for example, 'tt(+)'
27 is a character token. A character token doesn't need to be declared unless you
28 need to specify its semantic value data type (cf. section ref(SEMANTICS)),
29 associativity, or precedence (cf. section ref(PRECEDENCE)).
30 )
31
32 By convention, a character token is only used to represent that
33 particular character. Thus, the token 'tt(+)' is represents the character
34 `tt(+)' as a token.
35
36 All common escape sequences that can be used in bf(C++)'s character
37 constants can be used in b() as well. Be careful not to use the tt(0)
38 character as a character literal because its ASCII code, zero, is the code
39 tt(lex) returns to indicate end-of-input (see section ref(LEX)). If your
40 program em(must) be able to return 0-byte characters, define a special token
41 (e.g., tt(ZERO_BYTE)) and return that token instead.
42
43 Note that em(literal string tokens), formally supported in Bison, are
44 em(not) supported by b(). Again, such tokens are hardly ever encountered, and
45 lexical scanner generators (like bf(flex)(1) and bf(flexc++)(1)) do not
46 support them. Common practice is to define a symbolic name for a literal
47 string token. So, a token like tt(EQ) may be defined in the grammar file, with
48 the lexical scanner returning tt(EQ) when it matches tt(==).
49
50 The value returned by the parser's tt(lex) member is always a terminal
51 token. The numeric code for a character token is simply the ASCII code of that
52 character, and tt(lex) can simply return that character constant as a token
53 value. Each named token becomes a bf(C++) enumeration value in the parser base
54 class header file, and tt(lex) can return such enumeration identifiers as
55 well. When using an externally defined lexical scanner, that lexical scanner
56 should include the parser's base class header file, and it should return
57 either character constants or the token identifiers defined in that header
58 file. So, if (%token NUM) was defined in the parser class tt(Parser), then the
59 lexical scanner may return tt(Parser::NUM).
60
61 The symbol `tt(error)' is a reserved em(terminal) symbol reserved by b() for
62 error recovery purposes (see chapter ref(RECOVERY)). The tt(error) symbol
63 should not be used for other purposes. In particular, the parser's member
64 function tt(lex) should never return tt(error). Several more identifiers
65 should not be used as terminal symbols either. See section ref(IMPROPER) for
66 an overview.
67
68