Debian Codegraph @

Doc/library/token.rst

raw
1 :mod:`!token` --- Constants used with Python parse trees
2 ========================================================
3
4 .. module:: token
5 :synopsis: Constants representing terminal nodes of the parse tree.
6
7 .. sectionauthor:: Fred L. Drake, Jr. <[email protected]>
8
9 **Source code:** :source:`Lib/token.py`
10
11 --------------
12
13 This module provides constants which represent the numeric values of leaf nodes
14 of the parse tree (terminal tokens). Refer to the file :file:`Grammar/Tokens`
15 in the Python distribution for the definitions of the names in the context of
16 the language grammar. The specific numeric values which the names map to may
17 change between Python versions.
18
19 The module also provides a mapping from numeric codes to names and some
20 functions. The functions mirror definitions in the Python C header files.
21
22 Note that a token's value may depend on tokenizer options. For example, a
23 ``"+"`` token may be reported as either :data:`PLUS` or :data:`OP`, or
24 a ``"match"`` token may be either :data:`NAME` or :data:`SOFT_KEYWORD`.
25
26
27 .. data:: tok_name
28
29 Dictionary mapping the numeric values of the constants defined in this module
30 back to name strings, allowing more human-readable representation of parse trees
31 to be generated.
32
33
34 .. function:: ISTERMINAL(x)
35
36 Return ``True`` for terminal token values.
37
38
39 .. function:: ISNONTERMINAL(x)
40
41 Return ``True`` for non-terminal token values.
42
43
44 .. function:: ISEOF(x)
45
46 Return ``True`` if *x* is the marker indicating the end of input.
47
48
49 The token constants are:
50
51 .. data:: NAME
52
53 Token value that indicates an :ref:`identifier <identifiers>`.
54 Note that keywords are also initially tokenized as ``NAME`` tokens.
55
56 .. data:: NUMBER
57
58 Token value that indicates a :ref:`numeric literal <numbers>`
59
60 .. data:: STRING
61
62 Token value that indicates a :ref:`string or byte literal <strings>`,
63 excluding :ref:`formatted string literals <f-strings>`.
64 The token string is not interpreted:
65 it includes the surrounding quotation marks and the prefix (if given);
66 backslashes are included literally, without processing escape sequences.
67
68 .. data:: OP
69
70 A generic token value that indicates an
71 :ref:`operator <operators>` or :ref:`delimiter <delimiters>`.
72
73 .. impl-detail::
74
75 This value is only reported by the :mod:`tokenize` module.
76 Internally, the tokenizer uses
77 :ref:`exact token types <token_operators_delimiters>` instead.
78
79 .. data:: COMMENT
80
81 Token value used to indicate a comment.
82 The parser ignores :data:`!COMMENT` tokens.
83
84 .. data:: NEWLINE
85
86 Token value that indicates the end of a :ref:`logical line <logical-lines>`.
87
88 .. data:: NL
89
90 Token value used to indicate a non-terminating newline.
91 :data:`!NL` tokens are generated when a logical line of code is continued
92 over multiple physical lines. The parser ignores :data:`!NL` tokens.
93
94 .. data:: INDENT
95
96 Token value used at the beginning of a :ref:`logical line <logical-lines>`
97 to indicate the start of an :ref:`indented block <indentation>`.
98
99 .. data:: DEDENT
100
101 Token value used at the beginning of a :ref:`logical line <logical-lines>`
102 to indicate the end of an :ref:`indented block <indentation>`.
103
104 .. data:: FSTRING_START
105
106 Token value used to indicate the beginning of an
107 :ref:`f-string literal <f-strings>`.
108
109 .. impl-detail::
110
111 The token string includes the prefix and the opening quote(s), but none
112 of the contents of the literal.
113
114 .. data:: FSTRING_MIDDLE
115
116 Token value used for literal text inside an :ref:`f-string literal <f-strings>`,
117 including format specifications.
118
119 .. impl-detail::
120
121 Replacement fields (that is, the non-literal parts of f-strings) use
122 the same tokens as other expressions, and are delimited by
123 :data:`LBRACE`, :data:`RBRACE`, :data:`EXCLAMATION` and :data:`COLON`
124 tokens.
125
126 .. data:: FSTRING_END
127
128 Token value used to indicate the end of a :ref:`f-string <f-strings>`.
129
130 .. impl-detail::
131
132 The token string contains the closing quote(s).
133
134 .. data:: ENDMARKER
135
136 Token value that indicates the end of input.
137
138 .. data:: ENCODING
139
140 Token value that indicates the encoding used to decode the source bytes
141 into text. The first token returned by :func:`tokenize.tokenize` will
142 always be an ``ENCODING`` token.
143
144 .. impl-detail::
145
146 This token type isn't used by the C tokenizer but is needed for
147 the :mod:`tokenize` module.
148
149
150 The following token types are not produced by the :mod:`tokenize` module,
151 and are defined for special uses in the tokenizer or parser:
152
153 .. data:: TYPE_IGNORE
154
155 Token value indicating that a ``type: ignore`` comment was recognized.
156 Such tokens are produced instead of regular :data:`COMMENT` tokens only
157 with the :data:`~ast.PyCF_TYPE_COMMENTS` flag.
158
159 .. data:: TYPE_COMMENT
160
161 Token value indicating that a type comment was recognized.
162 Such tokens are produced instead of regular :data:`COMMENT` tokens only
163 with the :data:`~ast.PyCF_TYPE_COMMENTS` flag.
164
165 .. data:: SOFT_KEYWORD
166
167 Token value indicating a :ref:`soft keyword <soft-keywords>`.
168
169 The tokenizer never produces this value.
170 To check for a soft keyword, pass a :data:`NAME` token's string to
171 :func:`keyword.issoftkeyword`.
172
173 .. data:: ERRORTOKEN
174
175 Token value used to indicate wrong input.
176
177 The :mod:`tokenize` module generally indicates errors by
178 raising exceptions instead of emitting this token.
179 It can also emit tokens such as :data:`OP` or :data:`NAME` with strings that
180 are later rejected by the parser.
181
182
183 .. _token_operators_delimiters:
184
185 The remaining tokens represent specific :ref:`operators <operators>` and
186 :ref:`delimiters <delimiters>`.
187 (The :mod:`tokenize` module reports these as :data:`OP`; see ``exact_type``
188 in the :mod:`tokenize` documentation for details.)
189
190 .. include:: token-list.inc
191
192
193 The following non-token constants are provided:
194
195 .. data:: N_TOKENS
196
197 The number of token types defined in this module.
198
199 .. NT_OFFSET is deliberately undocumented; if you need it you should be
200 reading the source
201
202 .. data:: EXACT_TOKEN_TYPES
203
204 A dictionary mapping the string representation of a token to its numeric code.
205
206 .. versionadded:: 3.8
207
208
209 .. versionchanged:: 3.5
210 Added :data:`!AWAIT` and :data:`!ASYNC` tokens.
211
212 .. versionchanged:: 3.7
213 Added :data:`COMMENT`, :data:`NL` and :data:`ENCODING` tokens.
214
215 .. versionchanged:: 3.7
216 Removed :data:`!AWAIT` and :data:`!ASYNC` tokens. "async" and "await" are
217 now tokenized as :data:`NAME` tokens.
218
219 .. versionchanged:: 3.8
220 Added :data:`TYPE_COMMENT`, :data:`TYPE_IGNORE`, :data:`COLONEQUAL`.
221 Added :data:`!AWAIT` and :data:`!ASYNC` tokens back (they're needed
222 to support parsing older Python versions for :func:`ast.parse` with
223 ``feature_version`` set to 6 or lower).
224
225 .. versionchanged:: 3.12
226 Added :data:`EXCLAMATION`.
227
228 .. versionchanged:: 3.13
229 Removed :data:`!AWAIT` and :data:`!ASYNC` tokens again.
230