| 1 |
|
| 2 |
Lexical analysis |
| 3 |
================ |
| 4 |
|
| 5 |
.. py:decorator:: token(regex, types=None) |
| 6 |
|
| 7 |
Decorator for token definitions in classes derived from :py:class:`LexerBase`. |
| 8 |
|
| 9 |
:param str rx: A regular expression defining the possible token values |
| 10 |
:param types: A list of token types that this method can recognize. If omitted, the token type is assumed to be the method's name. |
| 11 |
:type types: List of strings |
| 12 |
|
| 13 |
Basic usage: |
| 14 |
|
| 15 |
.. code-block:: python |
| 16 |
|
| 17 |
from ptk.lexer import ReLexer, token |
| 18 |
|
| 19 |
class MyLexer(ReLexer): |
| 20 |
@token(r'[a-zA-Z_][a-zA-Z0-9_]*') |
| 21 |
def identifier(self, tok): |
| 22 |
pass |
| 23 |
|
| 24 |
This will define an *identifier* token type, which value is the |
| 25 |
recognized string. The *tok* parameter holds two attributes, |
| 26 |
*type* and *value*. You can modify the value in place: |
| 27 |
|
| 28 |
.. code-block:: python |
| 29 |
|
| 30 |
from ptk.lexer import ReLexer, token |
| 31 |
|
| 32 |
class MyLexer(ReLexer): |
| 33 |
@token(r'[1-9][0-9]*') |
| 34 |
def number(self, tok): |
| 35 |
tok.value = int(tok.value) |
| 36 |
|
| 37 |
In some cases it may be necessary to change the token's type as |
| 38 |
well; for instance to disambiguate between identifiers that |
| 39 |
are builtins and other ones. In order for the lexer to know which |
| 40 |
token types can be generated, you should pass a list of strings as the |
| 41 |
*types* parameter: |
| 42 |
|
| 43 |
.. code-block:: python |
| 44 |
|
| 45 |
from ptk.lexer import ReLexer, token |
| 46 |
|
| 47 |
class MyLexer(ReLexer): |
| 48 |
@token(r'[a-zA-Z_][a-zA-Z0-9_]*', types=['builtin', 'identifier']) |
| 49 |
def identifier_or_builtin(self, tok): |
| 50 |
tok.type = 'builtin' if tok.value in ['len', 'id'] else 'identifier' |
| 51 |
|
| 52 |
In this case the default value of the *type* attribute is *None* |
| 53 |
and you **must** set it. Letting None as token type (or setting it |
| 54 |
to None) will cause the token to be ignored. |
| 55 |
|
| 56 |
.. note:: |
| 57 |
|
| 58 |
The type of token values depends on the type of the strings |
| 59 |
used to define the regular expressions. Unicode expressions |
| 60 |
will hold Unicode values, and bytes expressions will hold |
| 61 |
bytes values. |
| 62 |
|
| 63 |
.. note:: |
| 64 |
|
| 65 |
Disambiguation is done the regular way: if several regular |
| 66 |
expressions match the input, the longest match is choosen. If |
| 67 |
the matches are of equal length, the first (in source code |
| 68 |
order) declaration wins. |
| 69 |
|
| 70 |
.. automodule:: ptk.lexer |
| 71 |
:members: |
| 72 |
:member-order: bysource |
| 73 |
|
| 74 |
.. py:data:: EOF |
| 75 |
|
| 76 |
This is a singleton used to indicate end of stream. It may be used |
| 77 |
as a token, a token type and a token value. In the first case it is |
| 78 |
its own type and value. |