Debian Codegraph @

Doc/library/tokenize.rst

raw
1 :mod:`!tokenize` --- Tokenizer for Python source
2 ================================================
3
4 .. module:: tokenize
5 :synopsis: Lexical scanner for Python source code.
6
7 .. moduleauthor:: Ka Ping Yee
8 .. sectionauthor:: Fred L. Drake, Jr. <[email protected]>
9
10 **Source code:** :source:`Lib/tokenize.py`
11
12 --------------
13
14 The :mod:`!tokenize` module provides a lexical scanner for Python source code,
15 implemented in Python. The scanner in this module returns comments as tokens
16 as well, making it useful for implementing "pretty-printers", including
17 colorizers for on-screen displays.
18
19 To simplify token stream handling, all :ref:`operator <operators>` and
20 :ref:`delimiter <delimiters>` tokens and :data:`Ellipsis` are returned using
21 the generic :data:`~token.OP` token type. The exact
22 type can be determined by checking the ``exact_type`` property on the
23 :term:`named tuple` returned from :func:`tokenize.tokenize`.
24
25
26 .. warning::
27
28 Note that the functions in this module are only designed to parse
29 syntactically valid Python code (code that does not raise when parsed
30 using :func:`ast.parse`). The behavior of the functions in this module is
31 **undefined** when providing invalid Python code and it can change at any
32 point.
33
34 Tokenizing Input
35 ----------------
36
37 The primary entry point is a :term:`generator`:
38
39 .. function:: tokenize(readline)
40
41 The :func:`.tokenize` generator requires one argument, *readline*, which
42 must be a callable object which provides the same interface as the
43 :meth:`io.IOBase.readline` method of file objects. Each call to the
44 function should return one line of input as bytes.
45
46 The generator produces 5-tuples with these members: the token type; the
47 token string; a 2-tuple ``(srow, scol)`` of ints specifying the row and
48 column where the token begins in the source; a 2-tuple ``(erow, ecol)`` of
49 ints specifying the row and column where the token ends in the source; and
50 the line on which the token was found. The line passed (the last tuple item)
51 is the *physical* line. The 5 tuple is returned as a :term:`named tuple`
52 with the field names:
53 ``type string start end line``.
54
55 The returned :term:`named tuple` has an additional property named
56 ``exact_type`` that contains the exact operator type for
57 :data:`~token.OP` tokens. For all other token types ``exact_type``
58 equals the named tuple ``type`` field.
59
60 .. versionchanged:: 3.1
61 Added support for named tuples.
62
63 .. versionchanged:: 3.3
64 Added support for ``exact_type``.
65
66 :func:`.tokenize` determines the source encoding of the file by looking for a
67 UTF-8 BOM or encoding cookie, according to :pep:`263`.
68
69 .. function:: generate_tokens(readline)
70
71 Tokenize a source reading unicode strings instead of bytes.
72
73 Like :func:`.tokenize`, the *readline* argument is a callable returning
74 a single line of input. However, :func:`generate_tokens` expects *readline*
75 to return a str object rather than bytes.
76
77 The result is an iterator yielding named tuples, exactly like
78 :func:`.tokenize`. It does not yield an :data:`~token.ENCODING` token.
79
80 All constants from the :mod:`token` module are also exported from
81 :mod:`!tokenize`.
82
83 Another function is provided to reverse the tokenization process. This is
84 useful for creating tools that tokenize a script, modify the token stream, and
85 write back the modified script.
86
87
88 .. function:: untokenize(iterable)
89
90 Converts tokens back into Python source code. The *iterable* must return
91 sequences with at least two elements, the token type and the token string.
92 Any additional sequence elements are ignored.
93
94 The result is guaranteed to tokenize back to match the input so that the
95 conversion is lossless and round-trips are assured. The guarantee applies
96 only to the token type and token string as the spacing between tokens
97 (column positions) may change.
98
99 It returns bytes, encoded using the :data:`~token.ENCODING` token, which
100 is the first token sequence output by :func:`.tokenize`. If there is no
101 encoding token in the input, it returns a str instead.
102
103
104 :func:`.tokenize` needs to detect the encoding of source files it tokenizes. The
105 function it uses to do this is available:
106
107 .. function:: detect_encoding(readline)
108
109 The :func:`detect_encoding` function is used to detect the encoding that
110 should be used to decode a Python source file. It requires one argument,
111 readline, in the same way as the :func:`.tokenize` generator.
112
113 It will call readline a maximum of twice, and return the encoding used
114 (as a string) and a list of any lines (not decoded from bytes) it has read
115 in.
116
117 It detects the encoding from the presence of a UTF-8 BOM or an encoding
118 cookie as specified in :pep:`263`. If both a BOM and a cookie are present,
119 but disagree, a :exc:`SyntaxError` will be raised. Note that if the BOM is found,
120 ``'utf-8-sig'`` will be returned as an encoding.
121
122 If no encoding is specified, then the default of ``'utf-8'`` will be
123 returned.
124
125 Use :func:`.open` to open Python source files: it uses
126 :func:`detect_encoding` to detect the file encoding.
127
128
129 .. function:: open(filename)
130
131 Open a file in read only mode using the encoding detected by
132 :func:`detect_encoding`.
133
134 .. versionadded:: 3.2
135
136 .. exception:: TokenError
137
138 Raised when either a docstring or expression that may be split over several
139 lines is not completed anywhere in the file, for example::
140
141 """Beginning of
142 docstring
143
144 or::
145
146 [1,
147 2,
148 3
149
150 .. _tokenize-cli:
151
152 Command-Line Usage
153 ------------------
154
155 .. versionadded:: 3.3
156
157 The :mod:`!tokenize` module can be executed as a script from the command line.
158 It is as simple as:
159
160 .. code-block:: sh
161
162 python -m tokenize [-e] [filename.py]
163
164 The following options are accepted:
165
166 .. program:: tokenize
167
168 .. option:: -h, --help
169
170 show this help message and exit
171
172 .. option:: -e, --exact
173
174 display token names using the exact type
175
176 If :file:`filename.py` is specified its contents are tokenized to stdout.
177 Otherwise, tokenization is performed on stdin.
178
179 Examples
180 ------------------
181
182 Example of a script rewriter that transforms float literals into Decimal
183 objects::
184
185 from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP
186 from io import BytesIO
187
188 def decistmt(s):
189 """Substitute Decimals for floats in a string of statements.
190
191 >>> from decimal import Decimal
192 >>> s = 'print(+21.3e-5*-.1234/81.7)'
193 >>> decistmt(s)
194 "print (+Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7'))"
195
196 The format of the exponent is inherited from the platform C library.
197 Known cases are "e-007" (Windows) and "e-07" (not Windows). Since
198 we're only showing 12 digits, and the 13th isn't close to 5, the
199 rest of the output should be platform-independent.
200
201 >>> exec(s) #doctest: +ELLIPSIS
202 -3.21716034272e-0...7
203
204 Output from calculations with Decimal should be identical across all
205 platforms.
206
207 >>> exec(decistmt(s))
208 -3.217160342717258261933904529E-7
209 """
210 result = []
211 g = tokenize(BytesIO(s.encode('utf-8')).readline) # tokenize the string
212 for toknum, tokval, _, _, _ in g:
213 if toknum == NUMBER and '.' in tokval: # replace NUMBER tokens
214 result.extend([
215 (NAME, 'Decimal'),
216 (OP, '('),
217 (STRING, repr(tokval)),
218 (OP, ')')
219 ])
220 else:
221 result.append((toknum, tokval))
222 return untokenize(result).decode('utf-8')
223
224 Example of tokenizing from the command line. The script::
225
226 def say_hello():
227 print("Hello, World!")
228
229 say_hello()
230
231 will be tokenized to the following output where the first column is the range
232 of the line/column coordinates where the token is found, the second column is
233 the name of the token, and the final column is the value of the token (if any)
234
235 .. code-block:: shell-session
236
237 $ python -m tokenize hello.py
238 0,0-0,0: ENCODING 'utf-8'
239 1,0-1,3: NAME 'def'
240 1,4-1,13: NAME 'say_hello'
241 1,13-1,14: OP '('
242 1,14-1,15: OP ')'
243 1,15-1,16: OP ':'
244 1,16-1,17: NEWLINE '\n'
245 2,0-2,4: INDENT ' '
246 2,4-2,9: NAME 'print'
247 2,9-2,10: OP '('
248 2,10-2,25: STRING '"Hello, World!"'
249 2,25-2,26: OP ')'
250 2,26-2,27: NEWLINE '\n'
251 3,0-3,1: NL '\n'
252 4,0-4,0: DEDENT ''
253 4,0-4,9: NAME 'say_hello'
254 4,9-4,10: OP '('
255 4,10-4,11: OP ')'
256 4,11-4,12: NEWLINE '\n'
257 5,0-5,0: ENDMARKER ''
258
259 The exact token type names can be displayed using the :option:`-e` option:
260
261 .. code-block:: shell-session
262
263 $ python -m tokenize -e hello.py
264 0,0-0,0: ENCODING 'utf-8'
265 1,0-1,3: NAME 'def'
266 1,4-1,13: NAME 'say_hello'
267 1,13-1,14: LPAR '('
268 1,14-1,15: RPAR ')'
269 1,15-1,16: COLON ':'
270 1,16-1,17: NEWLINE '\n'
271 2,0-2,4: INDENT ' '
272 2,4-2,9: NAME 'print'
273 2,9-2,10: LPAR '('
274 2,10-2,25: STRING '"Hello, World!"'
275 2,25-2,26: RPAR ')'
276 2,26-2,27: NEWLINE '\n'
277 3,0-3,1: NL '\n'
278 4,0-4,0: DEDENT ''
279 4,0-4,9: NAME 'say_hello'
280 4,9-4,10: LPAR '('
281 4,10-4,11: RPAR ')'
282 4,11-4,12: NEWLINE '\n'
283 5,0-5,0: ENDMARKER ''
284
285 Example of tokenizing a file programmatically, reading unicode
286 strings instead of bytes with :func:`generate_tokens`::
287
288 import tokenize
289
290 with tokenize.open('hello.py') as f:
291 tokens = tokenize.generate_tokens(f.readline)
292 for token in tokens:
293 print(token)
294
295 Or reading bytes directly with :func:`.tokenize`::
296
297 import tokenize
298
299 with open('hello.py', 'rb') as f:
300 tokens = tokenize.tokenize(f.readline)
301 for token in tokens:
302 print(token)