| 1 |
:mod:`!token` --- Constants used with Python parse trees |
| 2 |
======================================================== |
| 3 |
|
| 4 |
.. module:: token |
| 5 |
:synopsis: Constants representing terminal nodes of the parse tree. |
| 6 |
|
| 7 |
.. sectionauthor:: Fred L. Drake, Jr. <[email protected]> |
| 8 |
|
| 9 |
**Source code:** :source:`Lib/token.py` |
| 10 |
|
| 11 |
-------------- |
| 12 |
|
| 13 |
This module provides constants which represent the numeric values of leaf nodes |
| 14 |
of the parse tree (terminal tokens). Refer to the file :file:`Grammar/Tokens` |
| 15 |
in the Python distribution for the definitions of the names in the context of |
| 16 |
the language grammar. The specific numeric values which the names map to may |
| 17 |
change between Python versions. |
| 18 |
|
| 19 |
The module also provides a mapping from numeric codes to names and some |
| 20 |
functions. The functions mirror definitions in the Python C header files. |
| 21 |
|
| 22 |
Note that a token's value may depend on tokenizer options. For example, a |
| 23 |
``"+"`` token may be reported as either :data:`PLUS` or :data:`OP`, or |
| 24 |
a ``"match"`` token may be either :data:`NAME` or :data:`SOFT_KEYWORD`. |
| 25 |
|
| 26 |
|
| 27 |
.. data:: tok_name |
| 28 |
|
| 29 |
Dictionary mapping the numeric values of the constants defined in this module |
| 30 |
back to name strings, allowing more human-readable representation of parse trees |
| 31 |
to be generated. |
| 32 |
|
| 33 |
|
| 34 |
.. function:: ISTERMINAL(x) |
| 35 |
|
| 36 |
Return ``True`` for terminal token values. |
| 37 |
|
| 38 |
|
| 39 |
.. function:: ISNONTERMINAL(x) |
| 40 |
|
| 41 |
Return ``True`` for non-terminal token values. |
| 42 |
|
| 43 |
|
| 44 |
.. function:: ISEOF(x) |
| 45 |
|
| 46 |
Return ``True`` if *x* is the marker indicating the end of input. |
| 47 |
|
| 48 |
|
| 49 |
The token constants are: |
| 50 |
|
| 51 |
.. data:: NAME |
| 52 |
|
| 53 |
Token value that indicates an :ref:`identifier or keyword <identifiers>`. |
| 54 |
|
| 55 |
.. data:: NUMBER |
| 56 |
|
| 57 |
Token value that indicates a :ref:`numeric literal <numbers>` |
| 58 |
|
| 59 |
.. data:: STRING |
| 60 |
|
| 61 |
Token value that indicates a :ref:`string or byte literal <strings>`, |
| 62 |
excluding :ref:`formatted string literals <f-strings>`. |
| 63 |
The token string is not interpreted: |
| 64 |
it includes the surrounding quotation marks and the prefix (if given); |
| 65 |
backslashes are included literally, without processing escape sequences. |
| 66 |
|
| 67 |
.. data:: OP |
| 68 |
|
| 69 |
A generic token value that indicates an |
| 70 |
:ref:`operator <operators>` or :ref:`delimiter <delimiters>`. |
| 71 |
|
| 72 |
.. impl-detail:: |
| 73 |
|
| 74 |
This value is only reported by the :mod:`tokenize` module. |
| 75 |
Internally, the tokenizer uses |
| 76 |
:ref:`exact token types <token_operators_delimiters>` instead. |
| 77 |
|
| 78 |
.. data:: COMMENT |
| 79 |
|
| 80 |
Token value used to indicate a comment. |
| 81 |
The parser ignores :data:`!COMMENT` tokens. |
| 82 |
|
| 83 |
.. data:: NEWLINE |
| 84 |
|
| 85 |
Token value that indicates the end of a :ref:`logical line <logical-lines>`. |
| 86 |
|
| 87 |
.. data:: NL |
| 88 |
|
| 89 |
Token value used to indicate a non-terminating newline. |
| 90 |
:data:`!NL` tokens are generated when a logical line of code is continued |
| 91 |
over multiple physical lines. The parser ignores :data:`!NL` tokens. |
| 92 |
|
| 93 |
.. data:: INDENT |
| 94 |
|
| 95 |
Token value used at the beginning of a :ref:`logical line <logical-lines>` |
| 96 |
to indicate the start of an :ref:`indented block <indentation>`. |
| 97 |
|
| 98 |
.. data:: DEDENT |
| 99 |
|
| 100 |
Token value used at the beginning of a :ref:`logical line <logical-lines>` |
| 101 |
to indicate the end of an :ref:`indented block <indentation>`. |
| 102 |
|
| 103 |
.. data:: FSTRING_START |
| 104 |
|
| 105 |
Token value used to indicate the beginning of an |
| 106 |
:ref:`f-string literal <f-strings>`. |
| 107 |
|
| 108 |
.. impl-detail:: |
| 109 |
|
| 110 |
The token string includes the prefix and the opening quote(s), but none |
| 111 |
of the contents of the literal. |
| 112 |
|
| 113 |
.. data:: FSTRING_MIDDLE |
| 114 |
|
| 115 |
Token value used for literal text inside an :ref:`f-string literal <f-strings>`, |
| 116 |
including format specifications. |
| 117 |
|
| 118 |
.. impl-detail:: |
| 119 |
|
| 120 |
Replacement fields (that is, the non-literal parts of f-strings) use |
| 121 |
the same tokens as other expressions, and are delimited by |
| 122 |
:data:`LBRACE`, :data:`RBRACE`, :data:`EXCLAMATION` and :data:`COLON` |
| 123 |
tokens. |
| 124 |
|
| 125 |
.. data:: FSTRING_END |
| 126 |
|
| 127 |
Token value used to indicate the end of a :ref:`f-string <f-strings>`. |
| 128 |
|
| 129 |
.. impl-detail:: |
| 130 |
|
| 131 |
The token string contains the closing quote(s). |
| 132 |
|
| 133 |
.. data:: TSTRING_START |
| 134 |
|
| 135 |
Token value used to indicate the beginning of a template string literal. |
| 136 |
|
| 137 |
.. impl-detail:: |
| 138 |
|
| 139 |
The token string includes the prefix and the opening quote(s), but none |
| 140 |
of the contents of the literal. |
| 141 |
|
| 142 |
.. versionadded:: 3.14 |
| 143 |
|
| 144 |
.. data:: TSTRING_MIDDLE |
| 145 |
|
| 146 |
Token value used for literal text inside a template string literal |
| 147 |
including format specifications. |
| 148 |
|
| 149 |
.. impl-detail:: |
| 150 |
|
| 151 |
Replacement fields (that is, the non-literal parts of t-strings) use |
| 152 |
the same tokens as other expressions, and are delimited by |
| 153 |
:data:`LBRACE`, :data:`RBRACE`, :data:`EXCLAMATION` and :data:`COLON` |
| 154 |
tokens. |
| 155 |
|
| 156 |
.. versionadded:: 3.14 |
| 157 |
|
| 158 |
.. data:: TSTRING_END |
| 159 |
|
| 160 |
Token value used to indicate the end of a template string literal. |
| 161 |
|
| 162 |
.. impl-detail:: |
| 163 |
|
| 164 |
The token string contains the closing quote(s). |
| 165 |
|
| 166 |
.. versionadded:: 3.14 |
| 167 |
|
| 168 |
.. data:: ENDMARKER |
| 169 |
|
| 170 |
Token value that indicates the end of input. |
| 171 |
Used in :ref:`top-level grammar rules <top-level>`. |
| 172 |
|
| 173 |
.. data:: ENCODING |
| 174 |
|
| 175 |
Token value that indicates the encoding used to decode the source bytes |
| 176 |
into text. The first token returned by :func:`tokenize.tokenize` will |
| 177 |
always be an ``ENCODING`` token. |
| 178 |
|
| 179 |
.. impl-detail:: |
| 180 |
|
| 181 |
This token type isn't used by the C tokenizer but is needed for |
| 182 |
the :mod:`tokenize` module. |
| 183 |
|
| 184 |
|
| 185 |
The following token types are not produced by the :mod:`tokenize` module, |
| 186 |
and are defined for special uses in the tokenizer or parser: |
| 187 |
|
| 188 |
.. data:: TYPE_IGNORE |
| 189 |
|
| 190 |
Token value indicating that a ``type: ignore`` comment was recognized. |
| 191 |
Such tokens are produced instead of regular :data:`COMMENT` tokens only |
| 192 |
with the :data:`~ast.PyCF_TYPE_COMMENTS` flag. |
| 193 |
|
| 194 |
.. data:: TYPE_COMMENT |
| 195 |
|
| 196 |
Token value indicating that a type comment was recognized. |
| 197 |
Such tokens are produced instead of regular :data:`COMMENT` tokens only |
| 198 |
with the :data:`~ast.PyCF_TYPE_COMMENTS` flag. |
| 199 |
|
| 200 |
.. data:: SOFT_KEYWORD |
| 201 |
|
| 202 |
Token value indicating a :ref:`soft keyword <soft-keywords>`. |
| 203 |
|
| 204 |
The tokenizer never produces this value. |
| 205 |
To check for a soft keyword, pass a :data:`NAME` token's string to |
| 206 |
:func:`keyword.issoftkeyword`. |
| 207 |
|
| 208 |
.. data:: ERRORTOKEN |
| 209 |
|
| 210 |
Token value used to indicate wrong input. |
| 211 |
|
| 212 |
The :mod:`tokenize` module generally indicates errors by |
| 213 |
raising exceptions instead of emitting this token. |
| 214 |
It can also emit tokens such as :data:`OP` or :data:`NAME` with strings that |
| 215 |
are later rejected by the parser. |
| 216 |
|
| 217 |
|
| 218 |
.. _token_operators_delimiters: |
| 219 |
|
| 220 |
The remaining tokens represent specific :ref:`operators <operators>` and |
| 221 |
:ref:`delimiters <delimiters>`. |
| 222 |
(The :mod:`tokenize` module reports these as :data:`OP`; see ``exact_type`` |
| 223 |
in the :mod:`tokenize` documentation for details.) |
| 224 |
|
| 225 |
.. include:: token-list.inc |
| 226 |
|
| 227 |
|
| 228 |
The following non-token constants are provided: |
| 229 |
|
| 230 |
.. data:: N_TOKENS |
| 231 |
|
| 232 |
The number of token types defined in this module. |
| 233 |
|
| 234 |
.. NT_OFFSET is deliberately undocumented; if you need it you should be |
| 235 |
reading the source |
| 236 |
|
| 237 |
.. data:: EXACT_TOKEN_TYPES |
| 238 |
|
| 239 |
A dictionary mapping the string representation of a token to its numeric code. |
| 240 |
|
| 241 |
.. versionadded:: 3.8 |
| 242 |
|
| 243 |
|
| 244 |
.. versionchanged:: 3.5 |
| 245 |
Added :data:`!AWAIT` and :data:`!ASYNC` tokens. |
| 246 |
|
| 247 |
.. versionchanged:: 3.7 |
| 248 |
Added :data:`COMMENT`, :data:`NL` and :data:`ENCODING` tokens. |
| 249 |
|
| 250 |
.. versionchanged:: 3.7 |
| 251 |
Removed :data:`!AWAIT` and :data:`!ASYNC` tokens. "async" and "await" are |
| 252 |
now tokenized as :data:`NAME` tokens. |
| 253 |
|
| 254 |
.. versionchanged:: 3.8 |
| 255 |
Added :data:`TYPE_COMMENT`, :data:`TYPE_IGNORE`, :data:`COLONEQUAL`. |
| 256 |
Added :data:`!AWAIT` and :data:`!ASYNC` tokens back (they're needed |
| 257 |
to support parsing older Python versions for :func:`ast.parse` with |
| 258 |
``feature_version`` set to 6 or lower). |
| 259 |
|
| 260 |
.. versionchanged:: 3.12 |
| 261 |
Added :data:`EXCLAMATION`. |
| 262 |
|
| 263 |
.. versionchanged:: 3.13 |
| 264 |
Removed :data:`!AWAIT` and :data:`!ASYNC` tokens again. |
| 265 |
|