Debian Codegraph @

Doc/library/token.rst

raw
1 :mod:`!token` --- Constants used with Python parse trees
2 ========================================================
3
4 .. module:: token
5 :synopsis: Constants representing terminal nodes of the parse tree.
6
7 .. sectionauthor:: Fred L. Drake, Jr. <[email protected]>
8
9 **Source code:** :source:`Lib/token.py`
10
11 --------------
12
13 This module provides constants which represent the numeric values of leaf nodes
14 of the parse tree (terminal tokens). Refer to the file :file:`Grammar/Tokens`
15 in the Python distribution for the definitions of the names in the context of
16 the language grammar. The specific numeric values which the names map to may
17 change between Python versions.
18
19 The module also provides a mapping from numeric codes to names and some
20 functions. The functions mirror definitions in the Python C header files.
21
22 Note that a token's value may depend on tokenizer options. For example, a
23 ``"+"`` token may be reported as either :data:`PLUS` or :data:`OP`, or
24 a ``"match"`` token may be either :data:`NAME` or :data:`SOFT_KEYWORD`.
25
26
27 .. data:: tok_name
28
29 Dictionary mapping the numeric values of the constants defined in this module
30 back to name strings, allowing more human-readable representation of parse trees
31 to be generated.
32
33
34 .. function:: ISTERMINAL(x)
35
36 Return ``True`` for terminal token values.
37
38
39 .. function:: ISNONTERMINAL(x)
40
41 Return ``True`` for non-terminal token values.
42
43
44 .. function:: ISEOF(x)
45
46 Return ``True`` if *x* is the marker indicating the end of input.
47
48
49 The token constants are:
50
51 .. data:: NAME
52
53 Token value that indicates an :ref:`identifier or keyword <identifiers>`.
54
55 .. data:: NUMBER
56
57 Token value that indicates a :ref:`numeric literal <numbers>`
58
59 .. data:: STRING
60
61 Token value that indicates a :ref:`string or byte literal <strings>`,
62 excluding :ref:`formatted string literals <f-strings>`.
63 The token string is not interpreted:
64 it includes the surrounding quotation marks and the prefix (if given);
65 backslashes are included literally, without processing escape sequences.
66
67 .. data:: OP
68
69 A generic token value that indicates an
70 :ref:`operator <operators>` or :ref:`delimiter <delimiters>`.
71
72 .. impl-detail::
73
74 This value is only reported by the :mod:`tokenize` module.
75 Internally, the tokenizer uses
76 :ref:`exact token types <token_operators_delimiters>` instead.
77
78 .. data:: COMMENT
79
80 Token value used to indicate a comment.
81 The parser ignores :data:`!COMMENT` tokens.
82
83 .. data:: NEWLINE
84
85 Token value that indicates the end of a :ref:`logical line <logical-lines>`.
86
87 .. data:: NL
88
89 Token value used to indicate a non-terminating newline.
90 :data:`!NL` tokens are generated when a logical line of code is continued
91 over multiple physical lines. The parser ignores :data:`!NL` tokens.
92
93 .. data:: INDENT
94
95 Token value used at the beginning of a :ref:`logical line <logical-lines>`
96 to indicate the start of an :ref:`indented block <indentation>`.
97
98 .. data:: DEDENT
99
100 Token value used at the beginning of a :ref:`logical line <logical-lines>`
101 to indicate the end of an :ref:`indented block <indentation>`.
102
103 .. data:: FSTRING_START
104
105 Token value used to indicate the beginning of an
106 :ref:`f-string literal <f-strings>`.
107
108 .. impl-detail::
109
110 The token string includes the prefix and the opening quote(s), but none
111 of the contents of the literal.
112
113 .. data:: FSTRING_MIDDLE
114
115 Token value used for literal text inside an :ref:`f-string literal <f-strings>`,
116 including format specifications.
117
118 .. impl-detail::
119
120 Replacement fields (that is, the non-literal parts of f-strings) use
121 the same tokens as other expressions, and are delimited by
122 :data:`LBRACE`, :data:`RBRACE`, :data:`EXCLAMATION` and :data:`COLON`
123 tokens.
124
125 .. data:: FSTRING_END
126
127 Token value used to indicate the end of a :ref:`f-string <f-strings>`.
128
129 .. impl-detail::
130
131 The token string contains the closing quote(s).
132
133 .. data:: TSTRING_START
134
135 Token value used to indicate the beginning of a template string literal.
136
137 .. impl-detail::
138
139 The token string includes the prefix and the opening quote(s), but none
140 of the contents of the literal.
141
142 .. versionadded:: 3.14
143
144 .. data:: TSTRING_MIDDLE
145
146 Token value used for literal text inside a template string literal
147 including format specifications.
148
149 .. impl-detail::
150
151 Replacement fields (that is, the non-literal parts of t-strings) use
152 the same tokens as other expressions, and are delimited by
153 :data:`LBRACE`, :data:`RBRACE`, :data:`EXCLAMATION` and :data:`COLON`
154 tokens.
155
156 .. versionadded:: 3.14
157
158 .. data:: TSTRING_END
159
160 Token value used to indicate the end of a template string literal.
161
162 .. impl-detail::
163
164 The token string contains the closing quote(s).
165
166 .. versionadded:: 3.14
167
168 .. data:: ENDMARKER
169
170 Token value that indicates the end of input.
171 Used in :ref:`top-level grammar rules <top-level>`.
172
173 .. data:: ENCODING
174
175 Token value that indicates the encoding used to decode the source bytes
176 into text. The first token returned by :func:`tokenize.tokenize` will
177 always be an ``ENCODING`` token.
178
179 .. impl-detail::
180
181 This token type isn't used by the C tokenizer but is needed for
182 the :mod:`tokenize` module.
183
184
185 The following token types are not produced by the :mod:`tokenize` module,
186 and are defined for special uses in the tokenizer or parser:
187
188 .. data:: TYPE_IGNORE
189
190 Token value indicating that a ``type: ignore`` comment was recognized.
191 Such tokens are produced instead of regular :data:`COMMENT` tokens only
192 with the :data:`~ast.PyCF_TYPE_COMMENTS` flag.
193
194 .. data:: TYPE_COMMENT
195
196 Token value indicating that a type comment was recognized.
197 Such tokens are produced instead of regular :data:`COMMENT` tokens only
198 with the :data:`~ast.PyCF_TYPE_COMMENTS` flag.
199
200 .. data:: SOFT_KEYWORD
201
202 Token value indicating a :ref:`soft keyword <soft-keywords>`.
203
204 The tokenizer never produces this value.
205 To check for a soft keyword, pass a :data:`NAME` token's string to
206 :func:`keyword.issoftkeyword`.
207
208 .. data:: ERRORTOKEN
209
210 Token value used to indicate wrong input.
211
212 The :mod:`tokenize` module generally indicates errors by
213 raising exceptions instead of emitting this token.
214 It can also emit tokens such as :data:`OP` or :data:`NAME` with strings that
215 are later rejected by the parser.
216
217
218 .. _token_operators_delimiters:
219
220 The remaining tokens represent specific :ref:`operators <operators>` and
221 :ref:`delimiters <delimiters>`.
222 (The :mod:`tokenize` module reports these as :data:`OP`; see ``exact_type``
223 in the :mod:`tokenize` documentation for details.)
224
225 .. include:: token-list.inc
226
227
228 The following non-token constants are provided:
229
230 .. data:: N_TOKENS
231
232 The number of token types defined in this module.
233
234 .. NT_OFFSET is deliberately undocumented; if you need it you should be
235 reading the source
236
237 .. data:: EXACT_TOKEN_TYPES
238
239 A dictionary mapping the string representation of a token to its numeric code.
240
241 .. versionadded:: 3.8
242
243
244 .. versionchanged:: 3.5
245 Added :data:`!AWAIT` and :data:`!ASYNC` tokens.
246
247 .. versionchanged:: 3.7
248 Added :data:`COMMENT`, :data:`NL` and :data:`ENCODING` tokens.
249
250 .. versionchanged:: 3.7
251 Removed :data:`!AWAIT` and :data:`!ASYNC` tokens. "async" and "await" are
252 now tokenized as :data:`NAME` tokens.
253
254 .. versionchanged:: 3.8
255 Added :data:`TYPE_COMMENT`, :data:`TYPE_IGNORE`, :data:`COLONEQUAL`.
256 Added :data:`!AWAIT` and :data:`!ASYNC` tokens back (they're needed
257 to support parsing older Python versions for :func:`ast.parse` with
258 ``feature_version`` set to 6 or lower).
259
260 .. versionchanged:: 3.12
261 Added :data:`EXCLAMATION`.
262
263 .. versionchanged:: 3.13
264 Removed :data:`!AWAIT` and :data:`!ASYNC` tokens again.
265