| 1 |
:mod:`!tokenize` --- Tokenizer for Python source |
| 2 |
================================================ |
| 3 |
|
| 4 |
.. module:: tokenize |
| 5 |
:synopsis: Lexical scanner for Python source code. |
| 6 |
|
| 7 |
.. moduleauthor:: Ka Ping Yee |
| 8 |
.. sectionauthor:: Fred L. Drake, Jr. <[email protected]> |
| 9 |
|
| 10 |
**Source code:** :source:`Lib/tokenize.py` |
| 11 |
|
| 12 |
-------------- |
| 13 |
|
| 14 |
The :mod:`!tokenize` module provides a lexical scanner for Python source code, |
| 15 |
implemented in Python. The scanner in this module returns comments as tokens |
| 16 |
as well, making it useful for implementing "pretty-printers", including |
| 17 |
colorizers for on-screen displays. |
| 18 |
|
| 19 |
To simplify token stream handling, all :ref:`operator <operators>` and |
| 20 |
:ref:`delimiter <delimiters>` tokens and :data:`Ellipsis` are returned using |
| 21 |
the generic :data:`~token.OP` token type. The exact |
| 22 |
type can be determined by checking the ``exact_type`` property on the |
| 23 |
:term:`named tuple` returned from :func:`tokenize.tokenize`. |
| 24 |
|
| 25 |
|
| 26 |
.. warning:: |
| 27 |
|
| 28 |
Note that the functions in this module are only designed to parse |
| 29 |
syntactically valid Python code (code that does not raise when parsed |
| 30 |
using :func:`ast.parse`). The behavior of the functions in this module is |
| 31 |
**undefined** when providing invalid Python code and it can change at any |
| 32 |
point. |
| 33 |
|
| 34 |
Tokenizing Input |
| 35 |
---------------- |
| 36 |
|
| 37 |
The primary entry point is a :term:`generator`: |
| 38 |
|
| 39 |
.. function:: tokenize(readline) |
| 40 |
|
| 41 |
The :func:`.tokenize` generator requires one argument, *readline*, which |
| 42 |
must be a callable object which provides the same interface as the |
| 43 |
:meth:`io.IOBase.readline` method of file objects. Each call to the |
| 44 |
function should return one line of input as bytes. |
| 45 |
|
| 46 |
The generator produces 5-tuples with these members: the token type; the |
| 47 |
token string; a 2-tuple ``(srow, scol)`` of ints specifying the row and |
| 48 |
column where the token begins in the source; a 2-tuple ``(erow, ecol)`` of |
| 49 |
ints specifying the row and column where the token ends in the source; and |
| 50 |
the line on which the token was found. The line passed (the last tuple item) |
| 51 |
is the *physical* line. The 5 tuple is returned as a :term:`named tuple` |
| 52 |
with the field names: |
| 53 |
``type string start end line``. |
| 54 |
|
| 55 |
The returned :term:`named tuple` has an additional property named |
| 56 |
``exact_type`` that contains the exact operator type for |
| 57 |
:data:`~token.OP` tokens. For all other token types ``exact_type`` |
| 58 |
equals the named tuple ``type`` field. |
| 59 |
|
| 60 |
.. versionchanged:: 3.1 |
| 61 |
Added support for named tuples. |
| 62 |
|
| 63 |
.. versionchanged:: 3.3 |
| 64 |
Added support for ``exact_type``. |
| 65 |
|
| 66 |
:func:`.tokenize` determines the source encoding of the file by looking for a |
| 67 |
UTF-8 BOM or encoding cookie, according to :pep:`263`. |
| 68 |
|
| 69 |
.. function:: generate_tokens(readline) |
| 70 |
|
| 71 |
Tokenize a source reading unicode strings instead of bytes. |
| 72 |
|
| 73 |
Like :func:`.tokenize`, the *readline* argument is a callable returning |
| 74 |
a single line of input. However, :func:`generate_tokens` expects *readline* |
| 75 |
to return a str object rather than bytes. |
| 76 |
|
| 77 |
The result is an iterator yielding named tuples, exactly like |
| 78 |
:func:`.tokenize`. It does not yield an :data:`~token.ENCODING` token. |
| 79 |
|
| 80 |
All constants from the :mod:`token` module are also exported from |
| 81 |
:mod:`!tokenize`. |
| 82 |
|
| 83 |
Another function is provided to reverse the tokenization process. This is |
| 84 |
useful for creating tools that tokenize a script, modify the token stream, and |
| 85 |
write back the modified script. |
| 86 |
|
| 87 |
|
| 88 |
.. function:: untokenize(iterable) |
| 89 |
|
| 90 |
Converts tokens back into Python source code. The *iterable* must return |
| 91 |
sequences with at least two elements, the token type and the token string. |
| 92 |
Any additional sequence elements are ignored. |
| 93 |
|
| 94 |
The result is guaranteed to tokenize back to match the input so that the |
| 95 |
conversion is lossless and round-trips are assured. The guarantee applies |
| 96 |
only to the token type and token string as the spacing between tokens |
| 97 |
(column positions) may change. |
| 98 |
|
| 99 |
It returns bytes, encoded using the :data:`~token.ENCODING` token, which |
| 100 |
is the first token sequence output by :func:`.tokenize`. If there is no |
| 101 |
encoding token in the input, it returns a str instead. |
| 102 |
|
| 103 |
|
| 104 |
:func:`.tokenize` needs to detect the encoding of source files it tokenizes. The |
| 105 |
function it uses to do this is available: |
| 106 |
|
| 107 |
.. function:: detect_encoding(readline) |
| 108 |
|
| 109 |
The :func:`detect_encoding` function is used to detect the encoding that |
| 110 |
should be used to decode a Python source file. It requires one argument, |
| 111 |
readline, in the same way as the :func:`.tokenize` generator. |
| 112 |
|
| 113 |
It will call readline a maximum of twice, and return the encoding used |
| 114 |
(as a string) and a list of any lines (not decoded from bytes) it has read |
| 115 |
in. |
| 116 |
|
| 117 |
It detects the encoding from the presence of a UTF-8 BOM or an encoding |
| 118 |
cookie as specified in :pep:`263`. If both a BOM and a cookie are present, |
| 119 |
but disagree, a :exc:`SyntaxError` will be raised. Note that if the BOM is found, |
| 120 |
``'utf-8-sig'`` will be returned as an encoding. |
| 121 |
|
| 122 |
If no encoding is specified, then the default of ``'utf-8'`` will be |
| 123 |
returned. |
| 124 |
|
| 125 |
Use :func:`.open` to open Python source files: it uses |
| 126 |
:func:`detect_encoding` to detect the file encoding. |
| 127 |
|
| 128 |
|
| 129 |
.. function:: open(filename) |
| 130 |
|
| 131 |
Open a file in read only mode using the encoding detected by |
| 132 |
:func:`detect_encoding`. |
| 133 |
|
| 134 |
.. versionadded:: 3.2 |
| 135 |
|
| 136 |
.. exception:: TokenError |
| 137 |
|
| 138 |
Raised when either a docstring or expression that may be split over several |
| 139 |
lines is not completed anywhere in the file, for example:: |
| 140 |
|
| 141 |
"""Beginning of |
| 142 |
docstring |
| 143 |
|
| 144 |
or:: |
| 145 |
|
| 146 |
[1, |
| 147 |
2, |
| 148 |
3 |
| 149 |
|
| 150 |
.. _tokenize-cli: |
| 151 |
|
| 152 |
Command-Line Usage |
| 153 |
------------------ |
| 154 |
|
| 155 |
.. versionadded:: 3.3 |
| 156 |
|
| 157 |
The :mod:`!tokenize` module can be executed as a script from the command line. |
| 158 |
It is as simple as: |
| 159 |
|
| 160 |
.. code-block:: sh |
| 161 |
|
| 162 |
python -m tokenize [-e] [filename.py] |
| 163 |
|
| 164 |
The following options are accepted: |
| 165 |
|
| 166 |
.. program:: tokenize |
| 167 |
|
| 168 |
.. option:: -h, --help |
| 169 |
|
| 170 |
show this help message and exit |
| 171 |
|
| 172 |
.. option:: -e, --exact |
| 173 |
|
| 174 |
display token names using the exact type |
| 175 |
|
| 176 |
If :file:`filename.py` is specified its contents are tokenized to stdout. |
| 177 |
Otherwise, tokenization is performed on stdin. |
| 178 |
|
| 179 |
Examples |
| 180 |
------------------ |
| 181 |
|
| 182 |
Example of a script rewriter that transforms float literals into Decimal |
| 183 |
objects:: |
| 184 |
|
| 185 |
from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP |
| 186 |
from io import BytesIO |
| 187 |
|
| 188 |
def decistmt(s): |
| 189 |
"""Substitute Decimals for floats in a string of statements. |
| 190 |
|
| 191 |
>>> from decimal import Decimal |
| 192 |
>>> s = 'print(+21.3e-5*-.1234/81.7)' |
| 193 |
>>> decistmt(s) |
| 194 |
"print (+Decimal ('21.3e-5')*-Decimal ('.1234')/Decimal ('81.7'))" |
| 195 |
|
| 196 |
The format of the exponent is inherited from the platform C library. |
| 197 |
Known cases are "e-007" (Windows) and "e-07" (not Windows). Since |
| 198 |
we're only showing 12 digits, and the 13th isn't close to 5, the |
| 199 |
rest of the output should be platform-independent. |
| 200 |
|
| 201 |
>>> exec(s) #doctest: +ELLIPSIS |
| 202 |
-3.21716034272e-0...7 |
| 203 |
|
| 204 |
Output from calculations with Decimal should be identical across all |
| 205 |
platforms. |
| 206 |
|
| 207 |
>>> exec(decistmt(s)) |
| 208 |
-3.217160342717258261933904529E-7 |
| 209 |
""" |
| 210 |
result = [] |
| 211 |
g = tokenize(BytesIO(s.encode('utf-8')).readline) # tokenize the string |
| 212 |
for toknum, tokval, _, _, _ in g: |
| 213 |
if toknum == NUMBER and '.' in tokval: # replace NUMBER tokens |
| 214 |
result.extend([ |
| 215 |
(NAME, 'Decimal'), |
| 216 |
(OP, '('), |
| 217 |
(STRING, repr(tokval)), |
| 218 |
(OP, ')') |
| 219 |
]) |
| 220 |
else: |
| 221 |
result.append((toknum, tokval)) |
| 222 |
return untokenize(result).decode('utf-8') |
| 223 |
|
| 224 |
Example of tokenizing from the command line. The script:: |
| 225 |
|
| 226 |
def say_hello(): |
| 227 |
print("Hello, World!") |
| 228 |
|
| 229 |
say_hello() |
| 230 |
|
| 231 |
will be tokenized to the following output where the first column is the range |
| 232 |
of the line/column coordinates where the token is found, the second column is |
| 233 |
the name of the token, and the final column is the value of the token (if any) |
| 234 |
|
| 235 |
.. code-block:: shell-session |
| 236 |
|
| 237 |
$ python -m tokenize hello.py |
| 238 |
0,0-0,0: ENCODING 'utf-8' |
| 239 |
1,0-1,3: NAME 'def' |
| 240 |
1,4-1,13: NAME 'say_hello' |
| 241 |
1,13-1,14: OP '(' |
| 242 |
1,14-1,15: OP ')' |
| 243 |
1,15-1,16: OP ':' |
| 244 |
1,16-1,17: NEWLINE '\n' |
| 245 |
2,0-2,4: INDENT ' ' |
| 246 |
2,4-2,9: NAME 'print' |
| 247 |
2,9-2,10: OP '(' |
| 248 |
2,10-2,25: STRING '"Hello, World!"' |
| 249 |
2,25-2,26: OP ')' |
| 250 |
2,26-2,27: NEWLINE '\n' |
| 251 |
3,0-3,1: NL '\n' |
| 252 |
4,0-4,0: DEDENT '' |
| 253 |
4,0-4,9: NAME 'say_hello' |
| 254 |
4,9-4,10: OP '(' |
| 255 |
4,10-4,11: OP ')' |
| 256 |
4,11-4,12: NEWLINE '\n' |
| 257 |
5,0-5,0: ENDMARKER '' |
| 258 |
|
| 259 |
The exact token type names can be displayed using the :option:`-e` option: |
| 260 |
|
| 261 |
.. code-block:: shell-session |
| 262 |
|
| 263 |
$ python -m tokenize -e hello.py |
| 264 |
0,0-0,0: ENCODING 'utf-8' |
| 265 |
1,0-1,3: NAME 'def' |
| 266 |
1,4-1,13: NAME 'say_hello' |
| 267 |
1,13-1,14: LPAR '(' |
| 268 |
1,14-1,15: RPAR ')' |
| 269 |
1,15-1,16: COLON ':' |
| 270 |
1,16-1,17: NEWLINE '\n' |
| 271 |
2,0-2,4: INDENT ' ' |
| 272 |
2,4-2,9: NAME 'print' |
| 273 |
2,9-2,10: LPAR '(' |
| 274 |
2,10-2,25: STRING '"Hello, World!"' |
| 275 |
2,25-2,26: RPAR ')' |
| 276 |
2,26-2,27: NEWLINE '\n' |
| 277 |
3,0-3,1: NL '\n' |
| 278 |
4,0-4,0: DEDENT '' |
| 279 |
4,0-4,9: NAME 'say_hello' |
| 280 |
4,9-4,10: LPAR '(' |
| 281 |
4,10-4,11: RPAR ')' |
| 282 |
4,11-4,12: NEWLINE '\n' |
| 283 |
5,0-5,0: ENDMARKER '' |
| 284 |
|
| 285 |
Example of tokenizing a file programmatically, reading unicode |
| 286 |
strings instead of bytes with :func:`generate_tokens`:: |
| 287 |
|
| 288 |
import tokenize |
| 289 |
|
| 290 |
with tokenize.open('hello.py') as f: |
| 291 |
tokens = tokenize.generate_tokens(f.readline) |
| 292 |
for token in tokens: |
| 293 |
print(token) |
| 294 |
|
| 295 |
Or reading bytes directly with :func:`.tokenize`:: |
| 296 |
|
| 297 |
import tokenize |
| 298 |
|
| 299 |
with open('hello.py', 'rb') as f: |
| 300 |
tokens = tokenize.tokenize(f.readline) |
| 301 |
for token in tokens: |
| 302 |
print(token) |