Debian Codegraph @

doc/source/lexer.rst

raw
1
2 Lexical analysis
3 ================
4
5 .. py:decorator:: token(regex, types=None)
6
7 Decorator for token definitions in classes derived from :py:class:`LexerBase`.
8
9 :param str rx: A regular expression defining the possible token values
10 :param types: A list of token types that this method can recognize. If omitted, the token type is assumed to be the method's name.
11 :type types: List of strings
12
13 Basic usage:
14
15 .. code-block:: python
16
17 from ptk.lexer import ReLexer, token
18
19 class MyLexer(ReLexer):
20 @token(r'[a-zA-Z_][a-zA-Z0-9_]*')
21 def identifier(self, tok):
22 pass
23
24 This will define an *identifier* token type, which value is the
25 recognized string. The *tok* parameter holds two attributes,
26 *type* and *value*. You can modify the value in place:
27
28 .. code-block:: python
29
30 from ptk.lexer import ReLexer, token
31
32 class MyLexer(ReLexer):
33 @token(r'[1-9][0-9]*')
34 def number(self, tok):
35 tok.value = int(tok.value)
36
37 In some cases it may be necessary to change the token's type as
38 well; for instance to disambiguate between identifiers that
39 are builtins and other ones. In order for the lexer to know which
40 token types can be generated, you should pass a list of strings as the
41 *types* parameter:
42
43 .. code-block:: python
44
45 from ptk.lexer import ReLexer, token
46
47 class MyLexer(ReLexer):
48 @token(r'[a-zA-Z_][a-zA-Z0-9_]*', types=['builtin', 'identifier'])
49 def identifier_or_builtin(self, tok):
50 tok.type = 'builtin' if tok.value in ['len', 'id'] else 'identifier'
51
52 In this case the default value of the *type* attribute is *None*
53 and you **must** set it. Letting None as token type (or setting it
54 to None) will cause the token to be ignored.
55
56 .. note::
57
58 The type of token values depends on the type of the strings
59 used to define the regular expressions. Unicode expressions
60 will hold Unicode values, and bytes expressions will hold
61 bytes values.
62
63 .. note::
64
65 Disambiguation is done the regular way: if several regular
66 expressions match the input, the longest match is choosen. If
67 the matches are of equal length, the first (in source code
68 order) declaration wins.
69
70 .. automodule:: ptk.lexer
71 :members:
72 :member-order: bysource
73
74 .. py:data:: EOF
75
76 This is a singleton used to indicate end of stream. It may be used
77 as a token, a token type and a token value. In the first case it is
78 its own type and value.