| 1 |
:mod:`!token` --- Constants used with Python parse trees |
| 2 |
======================================================== |
| 3 |
|
| 4 |
.. module:: token |
| 5 |
:synopsis: Constants representing terminal nodes of the parse tree. |
| 6 |
|
| 7 |
.. sectionauthor:: Fred L. Drake, Jr. <[email protected]> |
| 8 |
|
| 9 |
**Source code:** :source:`Lib/token.py` |
| 10 |
|
| 11 |
-------------- |
| 12 |
|
| 13 |
This module provides constants which represent the numeric values of leaf nodes |
| 14 |
of the parse tree (terminal tokens). Refer to the file :file:`Grammar/Tokens` |
| 15 |
in the Python distribution for the definitions of the names in the context of |
| 16 |
the language grammar. The specific numeric values which the names map to may |
| 17 |
change between Python versions. |
| 18 |
|
| 19 |
The module also provides a mapping from numeric codes to names and some |
| 20 |
functions. The functions mirror definitions in the Python C header files. |
| 21 |
|
| 22 |
Note that a token's value may depend on tokenizer options. For example, a |
| 23 |
``"+"`` token may be reported as either :data:`PLUS` or :data:`OP`, or |
| 24 |
a ``"match"`` token may be either :data:`NAME` or :data:`SOFT_KEYWORD`. |
| 25 |
|
| 26 |
|
| 27 |
.. data:: tok_name |
| 28 |
|
| 29 |
Dictionary mapping the numeric values of the constants defined in this module |
| 30 |
back to name strings, allowing more human-readable representation of parse trees |
| 31 |
to be generated. |
| 32 |
|
| 33 |
|
| 34 |
.. function:: ISTERMINAL(x) |
| 35 |
|
| 36 |
Return ``True`` for terminal token values. |
| 37 |
|
| 38 |
|
| 39 |
.. function:: ISNONTERMINAL(x) |
| 40 |
|
| 41 |
Return ``True`` for non-terminal token values. |
| 42 |
|
| 43 |
|
| 44 |
.. function:: ISEOF(x) |
| 45 |
|
| 46 |
Return ``True`` if *x* is the marker indicating the end of input. |
| 47 |
|
| 48 |
|
| 49 |
The token constants are: |
| 50 |
|
| 51 |
.. data:: NAME |
| 52 |
|
| 53 |
Token value that indicates an :ref:`identifier <identifiers>`. |
| 54 |
Note that keywords are also initially tokenized as ``NAME`` tokens. |
| 55 |
|
| 56 |
.. data:: NUMBER |
| 57 |
|
| 58 |
Token value that indicates a :ref:`numeric literal <numbers>` |
| 59 |
|
| 60 |
.. data:: STRING |
| 61 |
|
| 62 |
Token value that indicates a :ref:`string or byte literal <strings>`, |
| 63 |
excluding :ref:`formatted string literals <f-strings>`. |
| 64 |
The token string is not interpreted: |
| 65 |
it includes the surrounding quotation marks and the prefix (if given); |
| 66 |
backslashes are included literally, without processing escape sequences. |
| 67 |
|
| 68 |
.. data:: OP |
| 69 |
|
| 70 |
A generic token value that indicates an |
| 71 |
:ref:`operator <operators>` or :ref:`delimiter <delimiters>`. |
| 72 |
|
| 73 |
.. impl-detail:: |
| 74 |
|
| 75 |
This value is only reported by the :mod:`tokenize` module. |
| 76 |
Internally, the tokenizer uses |
| 77 |
:ref:`exact token types <token_operators_delimiters>` instead. |
| 78 |
|
| 79 |
.. data:: COMMENT |
| 80 |
|
| 81 |
Token value used to indicate a comment. |
| 82 |
The parser ignores :data:`!COMMENT` tokens. |
| 83 |
|
| 84 |
.. data:: NEWLINE |
| 85 |
|
| 86 |
Token value that indicates the end of a :ref:`logical line <logical-lines>`. |
| 87 |
|
| 88 |
.. data:: NL |
| 89 |
|
| 90 |
Token value used to indicate a non-terminating newline. |
| 91 |
:data:`!NL` tokens are generated when a logical line of code is continued |
| 92 |
over multiple physical lines. The parser ignores :data:`!NL` tokens. |
| 93 |
|
| 94 |
.. data:: INDENT |
| 95 |
|
| 96 |
Token value used at the beginning of a :ref:`logical line <logical-lines>` |
| 97 |
to indicate the start of an :ref:`indented block <indentation>`. |
| 98 |
|
| 99 |
.. data:: DEDENT |
| 100 |
|
| 101 |
Token value used at the beginning of a :ref:`logical line <logical-lines>` |
| 102 |
to indicate the end of an :ref:`indented block <indentation>`. |
| 103 |
|
| 104 |
.. data:: FSTRING_START |
| 105 |
|
| 106 |
Token value used to indicate the beginning of an |
| 107 |
:ref:`f-string literal <f-strings>`. |
| 108 |
|
| 109 |
.. impl-detail:: |
| 110 |
|
| 111 |
The token string includes the prefix and the opening quote(s), but none |
| 112 |
of the contents of the literal. |
| 113 |
|
| 114 |
.. data:: FSTRING_MIDDLE |
| 115 |
|
| 116 |
Token value used for literal text inside an :ref:`f-string literal <f-strings>`, |
| 117 |
including format specifications. |
| 118 |
|
| 119 |
.. impl-detail:: |
| 120 |
|
| 121 |
Replacement fields (that is, the non-literal parts of f-strings) use |
| 122 |
the same tokens as other expressions, and are delimited by |
| 123 |
:data:`LBRACE`, :data:`RBRACE`, :data:`EXCLAMATION` and :data:`COLON` |
| 124 |
tokens. |
| 125 |
|
| 126 |
.. data:: FSTRING_END |
| 127 |
|
| 128 |
Token value used to indicate the end of a :ref:`f-string <f-strings>`. |
| 129 |
|
| 130 |
.. impl-detail:: |
| 131 |
|
| 132 |
The token string contains the closing quote(s). |
| 133 |
|
| 134 |
.. data:: ENDMARKER |
| 135 |
|
| 136 |
Token value that indicates the end of input. |
| 137 |
|
| 138 |
.. data:: ENCODING |
| 139 |
|
| 140 |
Token value that indicates the encoding used to decode the source bytes |
| 141 |
into text. The first token returned by :func:`tokenize.tokenize` will |
| 142 |
always be an ``ENCODING`` token. |
| 143 |
|
| 144 |
.. impl-detail:: |
| 145 |
|
| 146 |
This token type isn't used by the C tokenizer but is needed for |
| 147 |
the :mod:`tokenize` module. |
| 148 |
|
| 149 |
|
| 150 |
The following token types are not produced by the :mod:`tokenize` module, |
| 151 |
and are defined for special uses in the tokenizer or parser: |
| 152 |
|
| 153 |
.. data:: TYPE_IGNORE |
| 154 |
|
| 155 |
Token value indicating that a ``type: ignore`` comment was recognized. |
| 156 |
Such tokens are produced instead of regular :data:`COMMENT` tokens only |
| 157 |
with the :data:`~ast.PyCF_TYPE_COMMENTS` flag. |
| 158 |
|
| 159 |
.. data:: TYPE_COMMENT |
| 160 |
|
| 161 |
Token value indicating that a type comment was recognized. |
| 162 |
Such tokens are produced instead of regular :data:`COMMENT` tokens only |
| 163 |
with the :data:`~ast.PyCF_TYPE_COMMENTS` flag. |
| 164 |
|
| 165 |
.. data:: SOFT_KEYWORD |
| 166 |
|
| 167 |
Token value indicating a :ref:`soft keyword <soft-keywords>`. |
| 168 |
|
| 169 |
The tokenizer never produces this value. |
| 170 |
To check for a soft keyword, pass a :data:`NAME` token's string to |
| 171 |
:func:`keyword.issoftkeyword`. |
| 172 |
|
| 173 |
.. data:: ERRORTOKEN |
| 174 |
|
| 175 |
Token value used to indicate wrong input. |
| 176 |
|
| 177 |
The :mod:`tokenize` module generally indicates errors by |
| 178 |
raising exceptions instead of emitting this token. |
| 179 |
It can also emit tokens such as :data:`OP` or :data:`NAME` with strings that |
| 180 |
are later rejected by the parser. |
| 181 |
|
| 182 |
|
| 183 |
.. _token_operators_delimiters: |
| 184 |
|
| 185 |
The remaining tokens represent specific :ref:`operators <operators>` and |
| 186 |
:ref:`delimiters <delimiters>`. |
| 187 |
(The :mod:`tokenize` module reports these as :data:`OP`; see ``exact_type`` |
| 188 |
in the :mod:`tokenize` documentation for details.) |
| 189 |
|
| 190 |
.. include:: token-list.inc |
| 191 |
|
| 192 |
|
| 193 |
The following non-token constants are provided: |
| 194 |
|
| 195 |
.. data:: N_TOKENS |
| 196 |
|
| 197 |
The number of token types defined in this module. |
| 198 |
|
| 199 |
.. NT_OFFSET is deliberately undocumented; if you need it you should be |
| 200 |
reading the source |
| 201 |
|
| 202 |
.. data:: EXACT_TOKEN_TYPES |
| 203 |
|
| 204 |
A dictionary mapping the string representation of a token to its numeric code. |
| 205 |
|
| 206 |
.. versionadded:: 3.8 |
| 207 |
|
| 208 |
|
| 209 |
.. versionchanged:: 3.5 |
| 210 |
Added :data:`!AWAIT` and :data:`!ASYNC` tokens. |
| 211 |
|
| 212 |
.. versionchanged:: 3.7 |
| 213 |
Added :data:`COMMENT`, :data:`NL` and :data:`ENCODING` tokens. |
| 214 |
|
| 215 |
.. versionchanged:: 3.7 |
| 216 |
Removed :data:`!AWAIT` and :data:`!ASYNC` tokens. "async" and "await" are |
| 217 |
now tokenized as :data:`NAME` tokens. |
| 218 |
|
| 219 |
.. versionchanged:: 3.8 |
| 220 |
Added :data:`TYPE_COMMENT`, :data:`TYPE_IGNORE`, :data:`COLONEQUAL`. |
| 221 |
Added :data:`!AWAIT` and :data:`!ASYNC` tokens back (they're needed |
| 222 |
to support parsing older Python versions for :func:`ast.parse` with |
| 223 |
``feature_version`` set to 6 or lower). |
| 224 |
|
| 225 |
.. versionchanged:: 3.12 |
| 226 |
Added :data:`EXCLAMATION`. |
| 227 |
|
| 228 |
.. versionchanged:: 3.13 |
| 229 |
Removed :data:`!AWAIT` and :data:`!ASYNC` tokens again. |
| 230 |
|