Debian Codegraph @

Doc/reference/lexical_analysis.rst

raw
1
2 .. _lexical:
3
4 ****************
5 Lexical analysis
6 ****************
7
8 .. index:: lexical analysis, parser, token
9
10 A Python program is read by a *parser*. Input to the parser is a stream of
11 :term:`tokens <token>`, generated by the *lexical analyzer* (also known as
12 the *tokenizer*).
13 This chapter describes how the lexical analyzer produces these tokens.
14
15 The lexical analyzer determines the program text's :ref:`encoding <encodings>`
16 (UTF-8 by default), and decodes the text into
17 :ref:`source characters <lexical-source-character>`.
18 If the text cannot be decoded, a :exc:`SyntaxError` is raised.
19
20 Next, the lexical analyzer uses the source characters to generate a stream of tokens.
21 The type of a generated token generally depends on the next source character to
22 be processed. Similarly, other special behavior of the analyzer depends on
23 the first source character that hasn't yet been processed.
24 The following table gives a quick summary of these source characters,
25 with links to sections that contain more information.
26
27 .. list-table::
28 :header-rows: 1
29
30 * - Character
31 - Next token (or other relevant documentation)
32
33 * - * space
34 * tab
35 * formfeed
36 - * :ref:`Whitespace <whitespace>`
37
38 * - * CR, LF
39 - * :ref:`New line <line-structure>`
40 * :ref:`Indentation <indentation>`
41
42 * - * backslash (``\``)
43 - * :ref:`Explicit line joining <explicit-joining>`
44 * (Also significant in :ref:`string escape sequences <escape-sequences>`)
45
46 * - * hash (``#``)
47 - * :ref:`Comment <comments>`
48
49 * - * quote (``'``, ``"``)
50 - * :ref:`String literal <strings>`
51
52 * - * ASCII letter (``a``-``z``, ``A``-``Z``)
53 * non-ASCII character
54 - * :ref:`Name <identifiers>`
55 * Prefixed :ref:`string or bytes literal <strings>`
56
57 * - * underscore (``_``)
58 - * :ref:`Name <identifiers>`
59 * (Can also be part of :ref:`numeric literals <numbers>`)
60
61 * - * number (``0``-``9``)
62 - * :ref:`Numeric literal <numbers>`
63
64 * - * dot (``.``)
65 - * :ref:`Numeric literal <numbers>`
66 * :ref:`Operator <operators>`
67
68 * - * question mark (``?``)
69 * dollar (``$``)
70 *
71 .. (the following uses zero-width space characters to render
72 .. a literal backquote)
73
74 backquote (``​`​``)
75 * control character
76 - * Error (outside string literals and comments)
77
78 * - * other printing character
79 - * :ref:`Operator or delimiter <operators>`
80
81 * - * end of file
82 - * :ref:`End marker <endmarker-token>`
83
84
85 .. _line-structure:
86
87 Line structure
88 ==============
89
90 .. index:: line structure
91
92 A Python program is divided into a number of *logical lines*.
93
94
95 .. _logical-lines:
96
97 Logical lines
98 -------------
99
100 .. index:: logical line, physical line, line joining, NEWLINE token
101
102 The end of a logical line is represented by the token :data:`~token.NEWLINE`.
103 Statements cannot cross logical line boundaries except where :data:`!NEWLINE`
104 is allowed by the syntax (e.g., between statements in compound statements).
105 A logical line is constructed from one or more *physical lines* by following
106 the :ref:`explicit <explicit-joining>` or :ref:`implicit <implicit-joining>`
107 *line joining* rules.
108
109
110 .. _physical-lines:
111
112 Physical lines
113 --------------
114
115 A physical line is a sequence of characters terminated by one the following
116 end-of-line sequences:
117
118 * the Unix form using ASCII LF (linefeed),
119 * the Windows form using the ASCII sequence CR LF (return followed by linefeed),
120 * the '`Classic Mac OS`__' form using the ASCII CR (return) character.
121
122 __ https://en.wikipedia.org/wiki/Classic_Mac_OS
123
124 Regardless of platform, each of these sequences is replaced by a single
125 ASCII LF (linefeed) character.
126 (This is done even inside :ref:`string literals <strings>`.)
127 Each line can use any of the sequences; they do not need to be consistent
128 within a file.
129
130 The end of input also serves as an implicit terminator for the final
131 physical line.
132
133 Formally:
134
135 .. grammar-snippet::
136 :group: python-grammar
137
138 newline: <ASCII LF> | <ASCII CR> <ASCII LF> | <ASCII CR>
139
140
141 .. _comments:
142
143 Comments
144 --------
145
146 .. index:: comment, hash character
147 single: # (hash); comment
148
149 A comment starts with a hash character (``#``) that is not part of a string
150 literal, and ends at the end of the physical line. A comment signifies the end
151 of the logical line unless the implicit line joining rules are invoked. Comments
152 are ignored by the syntax.
153
154
155 .. _encodings:
156
157 Encoding declarations
158 ---------------------
159
160 .. index:: source character set, encoding declarations (source file)
161 single: # (hash); source encoding declaration
162
163 If a comment in the first or second line of the Python script matches the
164 regular expression ``coding[=:]\s*([-\w.]+)``, this comment is processed as an
165 encoding declaration; the first group of this expression names the encoding of
166 the source code file. The encoding declaration must appear on a line of its
167 own. If it is the second line, the first line must also be a comment-only line.
168 The recommended forms of an encoding expression are ::
169
170 # -*- coding: <encoding-name> -*-
171
172 which is recognized also by GNU Emacs, and ::
173
174 # vim:fileencoding=<encoding-name>
175
176 which is recognized by Bram Moolenaar's VIM.
177
178 If no encoding declaration is found, the default encoding is UTF-8. If the
179 implicit or explicit encoding of a file is UTF-8, an initial UTF-8 byte-order
180 mark (``b'\xef\xbb\xbf'``) is ignored rather than being a syntax error.
181
182 If an encoding is declared, the encoding name must be recognized by Python
183 (see :ref:`standard-encodings`). The
184 encoding is used for all lexical analysis, including string literals, comments
185 and identifiers.
186
187 .. _lexical-source-character:
188
189 All lexical analysis, including string literals, comments
190 and identifiers, works on Unicode text decoded using the source encoding.
191 Any Unicode code point, except the NUL control character, can appear in
192 Python source.
193
194 .. grammar-snippet::
195 :group: python-grammar
196
197 source_character: <any Unicode code point, except NUL>
198
199
200 .. _explicit-joining:
201
202 Explicit line joining
203 ---------------------
204
205 .. index:: physical line, line joining, line continuation, backslash character
206
207 Two or more physical lines may be joined into logical lines using backslash
208 characters (``\``), as follows: when a physical line ends in a backslash that is
209 not part of a string literal or comment, it is joined with the following forming
210 a single logical line, deleting the backslash and the following end-of-line
211 character. For example::
212
213 if 1900 < year < 2100 and 1 <= month <= 12 \
214 and 1 <= day <= 31 and 0 <= hour < 24 \
215 and 0 <= minute < 60 and 0 <= second < 60: # Looks like a valid date
216 return 1
217
218 A line ending in a backslash cannot carry a comment. A backslash does not
219 continue a comment. A backslash does not continue a token except for string
220 literals (i.e., tokens other than string literals cannot be split across
221 physical lines using a backslash). A backslash is illegal elsewhere on a line
222 outside a string literal.
223
224
225 .. _implicit-joining:
226
227 Implicit line joining
228 ---------------------
229
230 Expressions in parentheses, square brackets or curly braces can be split over
231 more than one physical line without using backslashes. For example::
232
233 month_names = ['Januari', 'Februari', 'Maart', # These are the
234 'April', 'Mei', 'Juni', # Dutch names
235 'Juli', 'Augustus', 'September', # for the months
236 'Oktober', 'November', 'December'] # of the year
237
238 Implicitly continued lines can carry comments. The indentation of the
239 continuation lines is not important. Blank continuation lines are allowed.
240 There is no NEWLINE token between implicit continuation lines. Implicitly
241 continued lines can also occur within triple-quoted strings (see below); in that
242 case they cannot carry comments.
243
244
245 .. _blank-lines:
246
247 Blank lines
248 -----------
249
250 .. index:: single: blank line
251
252 A logical line that contains only spaces, tabs, formfeeds and possibly a
253 comment, is ignored (i.e., no :data:`~token.NEWLINE` token is generated).
254 During interactive input of statements, handling of a blank line may differ
255 depending on the implementation of the read-eval-print loop.
256 In the standard interactive interpreter, an entirely blank logical line (that
257 is, one containing not even whitespace or a comment) terminates a multi-line
258 statement.
259
260
261 .. _indentation:
262
263 Indentation
264 -----------
265
266 .. index:: indentation, leading whitespace, space, tab, grouping, statement grouping
267
268 Leading whitespace (spaces and tabs) at the beginning of a logical line is used
269 to compute the indentation level of the line, which in turn is used to determine
270 the grouping of statements.
271
272 Tabs are replaced (from left to right) by one to eight spaces such that the
273 total number of characters up to and including the replacement is a multiple of
274 eight (this is intended to be the same rule as used by Unix). The total number
275 of spaces preceding the first non-blank character then determines the line's
276 indentation. Indentation cannot be split over multiple physical lines using
277 backslashes; the whitespace up to the first backslash determines the
278 indentation.
279
280 Indentation is rejected as inconsistent if a source file mixes tabs and spaces
281 in a way that makes the meaning dependent on the worth of a tab in spaces; a
282 :exc:`TabError` is raised in that case.
283
284 **Cross-platform compatibility note:** because of the nature of text editors on
285 non-UNIX platforms, it is unwise to use a mixture of spaces and tabs for the
286 indentation in a single source file. It should also be noted that different
287 platforms may explicitly limit the maximum indentation level.
288
289 A formfeed character may be present at the start of the line; it will be ignored
290 for the indentation calculations above. Formfeed characters occurring elsewhere
291 in the leading whitespace have an undefined effect (for instance, they may reset
292 the space count to zero).
293
294 .. index:: INDENT token, DEDENT token
295
296 The indentation levels of consecutive lines are used to generate
297 :data:`~token.INDENT` and :data:`~token.DEDENT` tokens, using a stack,
298 as follows.
299
300 Before the first line of the file is read, a single zero is pushed on the stack;
301 this will never be popped off again. The numbers pushed on the stack will
302 always be strictly increasing from bottom to top. At the beginning of each
303 logical line, the line's indentation level is compared to the top of the stack.
304 If it is equal, nothing happens. If it is larger, it is pushed on the stack, and
305 one :data:`!INDENT` token is generated. If it is smaller, it *must* be one of the
306 numbers occurring on the stack; all numbers on the stack that are larger are
307 popped off, and for each number popped off a :data:`!DEDENT` token is generated.
308 At the end of the file, a :data:`!DEDENT` token is generated for each number
309 remaining on the stack that is larger than zero.
310
311 Here is an example of a correctly (though confusingly) indented piece of Python
312 code::
313
314 def perm(l):
315 # Compute the list of all permutations of l
316 if len(l) <= 1:
317 return [l]
318 r = []
319 for i in range(len(l)):
320 s = l[:i] + l[i+1:]
321 p = perm(s)
322 for x in p:
323 r.append(l[i:i+1] + x)
324 return r
325
326 The following example shows various indentation errors::
327
328 def perm(l): # error: first line indented
329 for i in range(len(l)): # error: not indented
330 s = l[:i] + l[i+1:]
331 p = perm(l[:i] + l[i+1:]) # error: unexpected indent
332 for x in p:
333 r.append(l[i:i+1] + x)
334 return r # error: inconsistent dedent
335
336 (Actually, the first three errors are detected by the parser; only the last
337 error is found by the lexical analyzer --- the indentation of ``return r`` does
338 not match a level popped off the stack.)
339
340
341 .. _whitespace:
342
343 Whitespace between tokens
344 -------------------------
345
346 Except at the beginning of a logical line or in string literals, the whitespace
347 characters space, tab and formfeed can be used interchangeably to separate
348 tokens:
349
350 .. grammar-snippet::
351 :group: python-grammar
352
353 whitespace: ' ' | tab | formfeed
354
355
356 Whitespace is needed between two tokens only if their concatenation
357 could otherwise be interpreted as a different token. For example, ``ab`` is one
358 token, but ``a b`` is two tokens. However, ``+a`` and ``+ a`` both produce
359 two tokens, ``+`` and ``a``, as ``+a`` is not a valid token.
360
361
362 .. _endmarker-token:
363
364 End marker
365 ----------
366
367 At the end of non-interactive input, the lexical analyzer generates an
368 :data:`~token.ENDMARKER` token.
369
370
371 .. _other-tokens:
372
373 Other tokens
374 ============
375
376 Besides :data:`~token.NEWLINE`, :data:`~token.INDENT` and :data:`~token.DEDENT`,
377 the following categories of tokens exist:
378 *identifiers* and *keywords* (:data:`~token.NAME`), *literals* (such as
379 :data:`~token.NUMBER` and :data:`~token.STRING`), and other symbols
380 (*operators* and *delimiters*, :data:`~token.OP`).
381 Whitespace characters (other than logical line terminators, discussed earlier)
382 are not tokens, but serve to delimit tokens.
383 Where ambiguity exists, a token comprises the longest possible string that
384 forms a legal token, when read from left to right.
385
386
387 .. _identifiers:
388
389 Names (identifiers and keywords)
390 ================================
391
392 .. index:: identifier, name
393
394 :data:`~token.NAME` tokens represent *identifiers*, *keywords*, and
395 *soft keywords*.
396
397 Names are composed of the following characters:
398
399 * uppercase and lowercase letters (``A-Z`` and ``a-z``),
400 * the underscore (``_``),
401 * digits (``0`` through ``9``), which cannot appear as the first character, and
402 * non-ASCII characters. Valid names may only contain "letter-like" and
403 "digit-like" characters; see :ref:`lexical-names-nonascii` for details.
404
405 Names must contain at least one character, but have no upper length limit.
406 Case is significant.
407
408 Formally, names are described by the following lexical definitions:
409
410 .. grammar-snippet::
411 :group: python-grammar
412
413 NAME: `name_start` `name_continue`*
414 name_start: "a"..."z" | "A"..."Z" | "_" | <non-ASCII character>
415 name_continue: name_start | "0"..."9"
416 identifier: <`NAME`, except keywords>
417
418 Note that not all names matched by this grammar are valid; see
419 :ref:`lexical-names-nonascii` for details.
420
421
422 .. _keywords:
423
424 Keywords
425 --------
426
427 .. index::
428 single: keyword
429 single: reserved word
430
431 The following names are used as reserved words, or *keywords* of the
432 language, and cannot be used as ordinary identifiers. They must be spelled
433 exactly as written here:
434
435 .. sourcecode:: text
436
437 False await else import pass
438 None break except in raise
439 True class finally is return
440 and continue for lambda try
441 as def from nonlocal while
442 assert del global not with
443 async elif if or yield
444
445
446 .. _soft-keywords:
447
448 Soft Keywords
449 -------------
450
451 .. index:: soft keyword, keyword
452
453 .. versionadded:: 3.10
454
455 Some names are only reserved under specific contexts. These are known as
456 *soft keywords*:
457
458 - ``match``, ``case``, and ``_``, when used in the :keyword:`match` statement.
459 - ``type``, when used in the :keyword:`type` statement.
460
461 These syntactically act as keywords in their specific contexts,
462 but this distinction is done at the parser level, not when tokenizing.
463
464 As soft keywords, their use in the grammar is possible while still
465 preserving compatibility with existing code that uses these names as
466 identifier names.
467
468 .. versionchanged:: 3.12
469 ``type`` is now a soft keyword.
470
471 .. index::
472 single: _, identifiers
473 single: __, identifiers
474 .. _id-classes:
475
476 Reserved classes of identifiers
477 -------------------------------
478
479 Certain classes of identifiers (besides keywords) have special meanings. These
480 classes are identified by the patterns of leading and trailing underscore
481 characters:
482
483 ``_*``
484 Not imported by ``from module import *``.
485
486 ``_``
487 In a ``case`` pattern within a :keyword:`match` statement, ``_`` is a
488 :ref:`soft keyword <soft-keywords>` that denotes a
489 :ref:`wildcard <wildcard-patterns>`.
490
491 Separately, the interactive interpreter makes the result of the last evaluation
492 available in the variable ``_``.
493 (It is stored in the :mod:`builtins` module, alongside built-in
494 functions like ``print``.)
495
496 Elsewhere, ``_`` is a regular identifier. It is often used to name
497 "special" items, but it is not special to Python itself.
498
499 .. note::
500
501 The name ``_`` is often used in conjunction with internationalization;
502 refer to the documentation for the :mod:`gettext` module for more
503 information on this convention.
504
505 It is also commonly used for unused variables.
506
507 ``__*__``
508 System-defined names, informally known as "dunder" names. These names are
509 defined by the interpreter and its implementation (including the standard library).
510 Current system names are discussed in the :ref:`specialnames` section and elsewhere.
511 More will likely be defined in future versions of Python. *Any* use of ``__*__`` names,
512 in any context, that does not follow explicitly documented use, is subject to
513 breakage without warning.
514
515 ``__*``
516 Class-private names. Names in this category, when used within the context of a
517 class definition, are re-written to use a mangled form to help avoid name
518 clashes between "private" attributes of base and derived classes. See section
519 :ref:`atom-identifiers`.
520
521
522 .. _lexical-names-nonascii:
523
524 Non-ASCII characters in names
525 -----------------------------
526
527 Names that contain non-ASCII characters need additional normalization
528 and validation beyond the rules and grammar explained
529 :ref:`above <identifiers>`.
530 For example, ``ř_1``, ``蛇``, or ``साँप`` are valid names, but ``r〰2``,
531 ``€``, or ``🐍`` are not.
532
533 This section explains the exact rules.
534
535 All names are converted into the `normalization form`_ NFKC while parsing.
536 This means that, for example, some typographic variants of characters are
537 converted to their "basic" form. For example, ``fiⁿₐˡᵢᶻₐᵗᵢᵒₙ`` normalizes to
538 ``finalization``, so Python treats them as the same name::
539
540 >>> fiⁿₐˡᵢᶻₐᵗᵢᵒₙ = 3
541 >>> finalization
542 3
543
544 .. note::
545
546 Normalization is done at the lexical level only.
547 Run-time functions that take names as *strings* generally do not normalize
548 their arguments.
549 For example, the variable defined above is accessible at run time in the
550 :func:`globals` dictionary as ``globals()["finalization"]`` but not
551 ``globals()["fiⁿₐˡᵢᶻₐᵗᵢᵒₙ"]``.
552
553 Similarly to how ASCII-only names must contain only letters, digits and
554 the underscore, and cannot start with a digit, a valid name must
555 start with a character in the "letter-like" set ``xid_start``,
556 and the remaining characters must be in the "letter- and digit-like" set
557 ``xid_continue``.
558
559 These sets are based on the *XID_Start* and *XID_Continue* sets as defined by the
560 Unicode standard annex `UAX-31`_.
561 Python's ``xid_start`` additionally includes the underscore (``_``).
562 Note that Python does not necessarily conform to `UAX-31`_.
563
564 A non-normative listing of characters in the *XID_Start* and *XID_Continue*
565 sets as defined by Unicode is available in the `DerivedCoreProperties.txt`_
566 file in the Unicode Character Database.
567 For reference, the construction rules for the ``xid_*`` sets are given below.
568
569 The set ``id_start`` is defined as the union of:
570
571 * Unicode category ``<Lu>`` - uppercase letters (includes ``A`` to ``Z``)
572 * Unicode category ``<Ll>`` - lowercase letters (includes ``a`` to ``z``)
573 * Unicode category ``<Lt>`` - titlecase letters
574 * Unicode category ``<Lm>`` - modifier letters
575 * Unicode category ``<Lo>`` - other letters
576 * Unicode category ``<Nl>`` - letter numbers
577 * {``"_"``} - the underscore
578 * ``<Other_ID_Start>`` - an explicit set of characters in `PropList.txt`_
579 to support backwards compatibility
580
581 The set ``xid_start`` then closes this set under NFKC normalization, by
582 removing all characters whose normalization is not of the form
583 ``id_start id_continue*``.
584
585 The set ``id_continue`` is defined as the union of:
586
587 * ``id_start`` (see above)
588 * Unicode category ``<Nd>`` - decimal numbers (includes ``0`` to ``9``)
589 * Unicode category ``<Pc>`` - connector punctuations
590 * Unicode category ``<Mn>`` - nonspacing marks
591 * Unicode category ``<Mc>`` - spacing combining marks
592 * ``<Other_ID_Continue>`` - another explicit set of characters in
593 `PropList.txt`_ to support backwards compatibility
594
595 Again, ``xid_continue`` closes this set under NFKC normalization.
596
597 Unicode categories use the version of the Unicode Character Database as
598 included in the :mod:`unicodedata` module.
599
600 .. _UAX-31: https://www.unicode.org/reports/tr31/
601 .. _PropList.txt: https://www.unicode.org/Public/16.0.0/ucd/PropList.txt
602 .. _DerivedCoreProperties.txt: https://www.unicode.org/Public/16.0.0/ucd/DerivedCoreProperties.txt
603 .. _normalization form: https://www.unicode.org/reports/tr15/#Norm_Forms
604
605 .. seealso::
606
607 * :pep:`3131` -- Supporting Non-ASCII Identifiers
608 * :pep:`672` -- Unicode-related Security Considerations for Python
609
610
611 .. _literals:
612
613 Literals
614 ========
615
616 .. index:: literal, constant
617
618 Literals are notations for constant values of some built-in types.
619
620 In terms of lexical analysis, Python has :ref:`string, bytes <strings>`
621 and :ref:`numeric <numbers>` literals.
622
623 Other "literals" are lexically denoted using :ref:`keywords <keywords>`
624 (``None``, ``True``, ``False``) and the special
625 :ref:`ellipsis token <lexical-ellipsis>` (``...``).
626
627
628 .. index:: string literal, bytes literal, ASCII
629 single: ' (single quote); string literal
630 single: " (double quote); string literal
631 .. _strings:
632
633 String and Bytes literals
634 =========================
635
636 String literals are text enclosed in single quotes (``'``) or double
637 quotes (``"``). For example:
638
639 .. code-block:: python
640
641 "spam"
642 'eggs'
643
644 The quote used to start the literal also terminates it, so a string literal
645 can only contain the other quote (except with escape sequences, see below).
646 For example:
647
648 .. code-block:: python
649
650 'Say "Hello", please.'
651 "Don't do that!"
652
653 Except for this limitation, the choice of quote character (``'`` or ``"``)
654 does not affect how the literal is parsed.
655
656 Inside a string literal, the backslash (``\``) character introduces an
657 :dfn:`escape sequence`, which has special meaning depending on the character
658 after the backslash.
659 For example, ``\"`` denotes the double quote character, and does *not* end
660 the string:
661
662 .. code-block:: pycon
663
664 >>> print("Say \"Hello\" to everyone!")
665 Say "Hello" to everyone!
666
667 See :ref:`escape sequences <escape-sequences>` below for a full list of such
668 sequences, and more details.
669
670
671 .. index:: triple-quoted string
672 single: """; string literal
673 single: '''; string literal
674
675 Triple-quoted strings
676 ---------------------
677
678 Strings can also be enclosed in matching groups of three single or double
679 quotes.
680 These are generally referred to as :dfn:`triple-quoted strings`::
681
682 """This is a triple-quoted string."""
683
684 In triple-quoted literals, unescaped quotes are allowed (and are
685 retained), except that three unescaped quotes in a row terminate the literal,
686 if they are of the same kind (``'`` or ``"``) used at the start::
687
688 """This string has "quotes" inside."""
689
690 Unescaped newlines are also allowed and retained::
691
692 '''This triple-quoted string
693 continues on the next line.'''
694
695
696 .. index::
697 single: u'; string literal
698 single: u"; string literal
699
700 String prefixes
701 ---------------
702
703 String literals can have an optional :dfn:`prefix` that influences how the
704 content of the literal is parsed, for example:
705
706 .. code-block:: python
707
708 b"data"
709 f'{result=}'
710
711 The allowed prefixes are:
712
713 * ``b``: :ref:`Bytes literal <bytes-literal>`
714 * ``r``: :ref:`Raw string <raw-strings>`
715 * ``f``: :ref:`Formatted string literal <f-strings>` ("f-string")
716 * ``t``: :ref:`Template string literal <t-strings>` ("t-string")
717 * ``u``: No effect (allowed for backwards compatibility)
718
719 See the linked sections for details on each type.
720
721 Prefixes are case-insensitive (for example, '``B``' works the same as '``b``').
722 The '``r``' prefix can be combined with '``f``', '``t``' or '``b``', so '``fr``',
723 '``rf``', '``tr``', '``rt``', '``br``', and '``rb``' are also valid prefixes.
724
725 .. versionadded:: 3.3
726 The ``'rb'`` prefix of raw bytes literals has been added as a synonym
727 of ``'br'``.
728
729 Support for the unicode legacy literal (``u'value'``) was reintroduced
730 to simplify the maintenance of dual Python 2.x and 3.x codebases.
731 See :pep:`414` for more information.
732
733
734 Formal grammar
735 --------------
736
737 String literals, except :ref:`"f-strings" <f-strings>` and
738 :ref:`"t-strings" <t-strings>`, are described by the
739 following lexical definitions.
740
741 These definitions use :ref:`negative lookaheads <lexical-lookaheads>` (``!``)
742 to indicate that an ending quote ends the literal.
743
744 .. grammar-snippet::
745 :group: python-grammar
746
747 STRING: [`stringprefix`] (`stringcontent`)
748 stringprefix: <("r" | "u" | "b" | "br" | "rb"), case-insensitive>
749 stringcontent:
750 | "'''" ( !"'''" `longstringitem`)* "'''"
751 | '"""' ( !'"""' `longstringitem`)* '"""'
752 | "'" ( !"'" `stringitem`)* "'"
753 | '"' ( !'"' `stringitem`)* '"'
754 stringitem: `stringchar` | `stringescapeseq`
755 stringchar: <any `source_character`, except backslash and newline>
756 longstringitem: `stringitem` | newline
757 stringescapeseq: "\" <any `source_character`>
758
759 Note that as in all lexical definitions, whitespace is significant.
760 In particular, the prefix (if any) must be immediately followed by the starting
761 quote.
762
763 .. index:: physical line, escape sequence, Standard C, C
764 single: \ (backslash); escape sequence
765 single: \\; escape sequence
766 single: \a; escape sequence
767 single: \b; escape sequence
768 single: \f; escape sequence
769 single: \n; escape sequence
770 single: \r; escape sequence
771 single: \t; escape sequence
772 single: \v; escape sequence
773 single: \x; escape sequence
774 single: \N; escape sequence
775 single: \u; escape sequence
776 single: \U; escape sequence
777
778 .. _escape-sequences:
779
780 Escape sequences
781 ----------------
782
783 Unless an '``r``' or '``R``' prefix is present, escape sequences in string and
784 bytes literals are interpreted according to rules similar to those used by
785 Standard C. The recognized escape sequences are:
786
787 .. list-table::
788 :widths: auto
789 :header-rows: 1
790
791 * * Escape Sequence
792 * Meaning
793 * * ``\``\ <newline>
794 * :ref:`string-escape-ignore`
795 * * ``\\``
796 * :ref:`Backslash <string-escape-escaped-char>`
797 * * ``\'``
798 * :ref:`Single quote <string-escape-escaped-char>`
799 * * ``\"``
800 * :ref:`Double quote <string-escape-escaped-char>`
801 * * ``\a``
802 * ASCII Bell (BEL)
803 * * ``\b``
804 * ASCII Backspace (BS)
805 * * ``\f``
806 * ASCII Formfeed (FF)
807 * * ``\n``
808 * ASCII Linefeed (LF)
809 * * ``\r``
810 * ASCII Carriage Return (CR)
811 * * ``\t``
812 * ASCII Horizontal Tab (TAB)
813 * * ``\v``
814 * ASCII Vertical Tab (VT)
815 * * :samp:`\\\\{ooo}`
816 * :ref:`string-escape-oct`
817 * * :samp:`\\x{hh}`
818 * :ref:`string-escape-hex`
819 * * :samp:`\\N\\{{name}\\}`
820 * :ref:`string-escape-named`
821 * * :samp:`\\u{xxxx}`
822 * :ref:`Hexadecimal Unicode character <string-escape-long-hex>`
823 * * :samp:`\\U{xxxxxxxx}`
824 * :ref:`Hexadecimal Unicode character <string-escape-long-hex>`
825
826 .. _string-escape-ignore:
827
828 Ignored end of line
829 ^^^^^^^^^^^^^^^^^^^
830
831 A backslash can be added at the end of a line to ignore the newline::
832
833 >>> 'This string will not include \
834 ... backslashes or newline characters.'
835 'This string will not include backslashes or newline characters.'
836
837 The same result can be achieved using :ref:`triple-quoted strings <strings>`,
838 or parentheses and :ref:`string literal concatenation <string-concatenation>`.
839
840 .. _string-escape-escaped-char:
841
842 Escaped characters
843 ^^^^^^^^^^^^^^^^^^
844
845 To include a backslash in a non-:ref:`raw <raw-strings>` Python string
846 literal, it must be doubled. The ``\\`` escape sequence denotes a single
847 backslash character::
848
849 >>> print('C:\\Program Files')
850 C:\Program Files
851
852 Similarly, the ``\'`` and ``\"`` sequences denote the single and double
853 quote character, respectively::
854
855 >>> print('\' and \"')
856 ' and "
857
858 .. _string-escape-oct:
859
860 Octal character
861 ^^^^^^^^^^^^^^^
862
863 The sequence :samp:`\\\\{ooo}` denotes a *character* with the octal (base 8)
864 value *ooo*::
865
866 >>> '\120'
867 'P'
868
869 Up to three octal digits (0 through 7) are accepted.
870
871 In a bytes literal, *character* means a *byte* with the given value.
872 In a string literal, it means a Unicode character with the given value.
873
874 .. versionchanged:: 3.11
875 Octal escapes with value larger than ``0o377`` (255) produce a
876 :exc:`DeprecationWarning`.
877
878 .. versionchanged:: 3.12
879 Octal escapes with value larger than ``0o377`` (255) produce a
880 :exc:`SyntaxWarning`.
881 In a future Python version they will raise a :exc:`SyntaxError`.
882
883 .. _string-escape-hex:
884
885 Hexadecimal character
886 ^^^^^^^^^^^^^^^^^^^^^
887
888 The sequence :samp:`\\x{hh}` denotes a *character* with the hex (base 16)
889 value *hh*::
890
891 >>> '\x50'
892 'P'
893
894 Unlike in Standard C, exactly two hex digits are required.
895
896 In a bytes literal, *character* means a *byte* with the given value.
897 In a string literal, it means a Unicode character with the given value.
898
899 .. _string-escape-named:
900
901 Named Unicode character
902 ^^^^^^^^^^^^^^^^^^^^^^^
903
904 The sequence :samp:`\\N\\{{name}\\}` denotes a Unicode character
905 with the given *name*::
906
907 >>> '\N{LATIN CAPITAL LETTER P}'
908 'P'
909 >>> '\N{SNAKE}'
910 '🐍'
911
912 This sequence cannot appear in :ref:`bytes literals <bytes-literal>`.
913
914 .. versionchanged:: 3.3
915 Support for `name aliases <https://www.unicode.org/Public/16.0.0/ucd/NameAliases.txt>`__
916 has been added.
917
918 .. _string-escape-long-hex:
919
920 Hexadecimal Unicode characters
921 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
922
923 These sequences :samp:`\\u{xxxx}` and :samp:`\\U{xxxxxxxx}` denote the
924 Unicode character with the given hex (base 16) value.
925 Exactly four digits are required for ``\u``; exactly eight digits are
926 required for ``\U``.
927 The latter can encode any Unicode character.
928
929 .. code-block:: pycon
930
931 >>> '\u1234'
932 'ሴ'
933 >>> '\U0001f40d'
934 '🐍'
935
936 These sequences cannot appear in :ref:`bytes literals <bytes-literal>`.
937
938
939 .. index:: unrecognized escape sequence
940
941 Unrecognized escape sequences
942 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
943
944 Unlike in Standard C, all unrecognized escape sequences are left in the string
945 unchanged, that is, *the backslash is left in the result*::
946
947 >>> print('\q')
948 \q
949 >>> list('\q')
950 ['\\', 'q']
951
952 Note that for bytes literals, the escape sequences only recognized in string
953 literals (``\N...``, ``\u...``, ``\U...``) fall into the category of
954 unrecognized escapes.
955
956 .. versionchanged:: 3.6
957 Unrecognized escape sequences produce a :exc:`DeprecationWarning`.
958
959 .. versionchanged:: 3.12
960 Unrecognized escape sequences produce a :exc:`SyntaxWarning`.
961 In a future Python version they will raise a :exc:`SyntaxError`.
962
963
964 .. index::
965 single: b'; bytes literal
966 single: b"; bytes literal
967
968
969 .. _bytes-literal:
970
971 Bytes literals
972 --------------
973
974 :dfn:`Bytes literals` are always prefixed with '``b``' or '``B``'; they produce an
975 instance of the :class:`bytes` type instead of the :class:`str` type.
976 They may only contain ASCII characters; bytes with a numeric value of 128
977 or greater must be expressed with escape sequences (typically
978 :ref:`string-escape-hex` or :ref:`string-escape-oct`):
979
980 .. code-block:: pycon
981
982 >>> b'\x89PNG\r\n\x1a\n'
983 b'\x89PNG\r\n\x1a\n'
984 >>> list(b'\x89PNG\r\n\x1a\n')
985 [137, 80, 78, 71, 13, 10, 26, 10]
986
987 Similarly, a zero byte must be expressed using an escape sequence (typically
988 ``\0`` or ``\x00``).
989
990
991 .. index::
992 single: r'; raw string literal
993 single: r"; raw string literal
994
995 .. _raw-strings:
996
997 Raw string literals
998 -------------------
999
1000 Both string and bytes literals may optionally be prefixed with a letter '``r``'
1001 or '``R``'; such constructs are called :dfn:`raw string literals`
1002 and :dfn:`raw bytes literals` respectively and treat backslashes as
1003 literal characters.
1004 As a result, in raw string literals, :ref:`escape sequences <escape-sequences>`
1005 are not treated specially:
1006
1007 .. code-block:: pycon
1008
1009 >>> r'\d{4}-\d{2}-\d{2}'
1010 '\\d{4}-\\d{2}-\\d{2}'
1011
1012 Even in a raw literal, quotes can be escaped with a backslash, but the
1013 backslash remains in the result; for example, ``r"\""`` is a valid string
1014 literal consisting of two characters: a backslash and a double quote; ``r"\"``
1015 is not a valid string literal (even a raw string cannot end in an odd number of
1016 backslashes). Specifically, *a raw literal cannot end in a single backslash*
1017 (since the backslash would escape the following quote character). Note also
1018 that a single backslash followed by a newline is interpreted as those two
1019 characters as part of the literal, *not* as a line continuation.
1020
1021
1022 .. index::
1023 single: formatted string literal
1024 single: interpolated string literal
1025 single: string; formatted literal
1026 single: string; interpolated literal
1027 single: f-string
1028 single: fstring
1029 single: f'; formatted string literal
1030 single: f"; formatted string literal
1031 single: {} (curly brackets); in formatted string literal
1032 single: ! (exclamation); in formatted string literal
1033 single: : (colon); in formatted string literal
1034 single: = (equals); for help in debugging using string literals
1035
1036 .. _f-strings:
1037 .. _formatted-string-literals:
1038
1039 f-strings
1040 ---------
1041
1042 .. versionadded:: 3.6
1043 .. versionchanged:: 3.7
1044 The :keyword:`await` and :keyword:`async for` can be used in expressions
1045 within f-strings.
1046 .. versionchanged:: 3.8
1047 Added the debug specifier (``=``)
1048 .. versionchanged:: 3.12
1049 Many restrictions on expressions within f-strings have been removed.
1050 Notably, nested strings, comments, and backslashes are now permitted.
1051
1052 A :dfn:`formatted string literal` or :dfn:`f-string` is a string literal
1053 that is prefixed with '``f``' or '``F``'.
1054 Unlike other string literals, f-strings do not have a constant value.
1055 They may contain *replacement fields* delimited by curly braces ``{}``.
1056 Replacement fields contain expressions which are evaluated at run time.
1057 For example::
1058
1059 >>> who = 'nobody'
1060 >>> nationality = 'Spanish'
1061 >>> f'{who.title()} expects the {nationality} Inquisition!'
1062 'Nobody expects the Spanish Inquisition!'
1063
1064 Any doubled curly braces (``{{`` or ``}}``) outside replacement fields
1065 are replaced with the corresponding single curly brace::
1066
1067 >>> print(f'{{...}}')
1068 {...}
1069
1070 Other characters outside replacement fields are treated like in ordinary
1071 string literals.
1072 This means that escape sequences are decoded (except when a literal is
1073 also marked as a raw string), and newlines are possible in triple-quoted
1074 f-strings::
1075
1076 >>> name = 'Galahad'
1077 >>> favorite_color = 'blue'
1078 >>> print(f'{name}:\t{favorite_color}')
1079 Galahad: blue
1080 >>> print(rf"C:\Users\{name}")
1081 C:\Users\Galahad
1082 >>> print(f'''Three shall be the number of the counting
1083 ... and the number of the counting shall be three.''')
1084 Three shall be the number of the counting
1085 and the number of the counting shall be three.
1086
1087 Expressions in formatted string literals are treated like regular
1088 Python expressions.
1089 Each expression is evaluated in the context where the formatted string literal
1090 appears, in order from left to right.
1091 An empty expression is not allowed, and both :keyword:`lambda` and
1092 assignment expressions ``:=`` must be surrounded by explicit parentheses::
1093
1094 >>> f'{(half := 1/2)}, {half * 42}'
1095 '0.5, 21.0'
1096
1097 Reusing the outer f-string quoting type inside a replacement field is
1098 permitted::
1099
1100 >>> a = dict(x=2)
1101 >>> f"abc {a["x"]} def"
1102 'abc 2 def'
1103
1104 Backslashes are also allowed in replacement fields and are evaluated the same
1105 way as in any other context::
1106
1107 >>> a = ["a", "b", "c"]
1108 >>> print(f"List a contains:\n{"\n".join(a)}")
1109 List a contains:
1110 a
1111 b
1112 c
1113
1114 It is possible to nest f-strings::
1115
1116 >>> name = 'world'
1117 >>> f'Repeated:{f' hello {name}' * 3}'
1118 'Repeated: hello world hello world hello world'
1119
1120 Portable Python programs should not use more than 5 levels of nesting.
1121
1122 .. impl-detail::
1123
1124 CPython does not limit nesting of f-strings.
1125
1126 Replacement expressions can contain newlines in both single-quoted and
1127 triple-quoted f-strings and they can contain comments.
1128 Everything that comes after a ``#`` inside a replacement field
1129 is a comment (even closing braces and quotes).
1130 This means that replacement fields with comments must be closed in a
1131 different line:
1132
1133 .. code-block:: text
1134
1135 >>> a = 2
1136 >>> f"abc{a # This comment }" continues until the end of the line
1137 ... + 3}"
1138 'abc5'
1139
1140 After the expression, replacement fields may optionally contain:
1141
1142 * a *debug specifier* -- an equal sign (``=``), optionally surrounded by
1143 whitespace on one or both sides;
1144 * a *conversion specifier* -- ``!s``, ``!r`` or ``!a``; and/or
1145 * a *format specifier* prefixed with a colon (``:``).
1146
1147 See the :ref:`Standard Library section on f-strings <stdtypes-fstrings>`
1148 for details on how these fields are evaluated.
1149
1150 As that section explains, *format specifiers* are passed as the second argument
1151 to the :func:`format` function to format a replacement field value.
1152 For example, they can be used to specify a field width and padding characters
1153 using the :ref:`Format Specification Mini-Language <formatspec>`::
1154
1155 >>> number = 14.3
1156 >>> f'{number:20.7f}'
1157 ' 14.3000000'
1158
1159 Top-level format specifiers may include nested replacement fields::
1160
1161 >>> field_size = 20
1162 >>> precision = 7
1163 >>> f'{number:{field_size}.{precision}f}'
1164 ' 14.3000000'
1165
1166 These nested fields may include their own conversion fields and
1167 :ref:`format specifiers <formatspec>`::
1168
1169 >>> number = 3
1170 >>> f'{number:{field_size}}'
1171 ' 3'
1172 >>> f'{number:{field_size:05}}'
1173 '00000000000000000003'
1174
1175 However, these nested fields may not include more deeply nested replacement
1176 fields.
1177
1178 Formatted string literals cannot be used as :term:`docstrings <docstring>`,
1179 even if they do not include expressions::
1180
1181 >>> def foo():
1182 ... f"Not a docstring"
1183 ...
1184 >>> print(foo.__doc__)
1185 None
1186
1187 .. seealso::
1188
1189 * :pep:`498` -- Literal String Interpolation
1190 * :pep:`701` -- Syntactic formalization of f-strings
1191 * :meth:`str.format`, which uses a related format string mechanism.
1192
1193
1194 .. _t-strings:
1195 .. _template-string-literals:
1196
1197 t-strings
1198 ---------
1199
1200 .. versionadded:: 3.14
1201
1202 A :dfn:`template string literal` or :dfn:`t-string` is a string literal
1203 that is prefixed with '``t``' or '``T``'.
1204 These strings follow the same syntax rules as
1205 :ref:`formatted string literals <f-strings>`.
1206 For differences in evaluation rules, see the
1207 :ref:`Standard Library section on t-strings <stdtypes-tstrings>`
1208
1209
1210 Formal grammar for f-strings
1211 ----------------------------
1212
1213 F-strings are handled partly by the :term:`lexical analyzer`, which produces the
1214 tokens :py:data:`~token.FSTRING_START`, :py:data:`~token.FSTRING_MIDDLE`
1215 and :py:data:`~token.FSTRING_END`, and partly by the parser, which handles
1216 expressions in the replacement field.
1217 The exact way the work is split is a CPython implementation detail.
1218
1219 Correspondingly, the f-string grammar is a mix of
1220 :ref:`lexical and syntactic definitions <notation-lexical-vs-syntactic>`.
1221
1222 Whitespace is significant in these situations:
1223
1224 * There may be no whitespace in :py:data:`~token.FSTRING_START` (between
1225 the prefix and quote).
1226 * Whitespace in :py:data:`~token.FSTRING_MIDDLE` is part of the literal
1227 string contents.
1228 * In ``fstring_replacement_field``, if ``f_debug_specifier`` is present,
1229 all whitespace after the opening brace until the ``f_debug_specifier``,
1230 as well as whitespace immediately following ``f_debug_specifier``,
1231 is retained as part of the expression.
1232
1233 .. impl-detail::
1234
1235 The expression is not handled in the tokenization phase; it is
1236 retrieved from the source code using locations of the ``{`` token
1237 and the token after ``=``.
1238
1239
1240 The ``FSTRING_MIDDLE`` definition uses
1241 :ref:`negative lookaheads <lexical-lookaheads>` (``!``)
1242 to indicate special characters (backslash, newline, ``{``, ``}``) and
1243 sequences (``f_quote``).
1244
1245 .. grammar-snippet::
1246 :group: python-grammar
1247
1248 fstring: `FSTRING_START` `fstring_middle`* `FSTRING_END`
1249
1250 FSTRING_START: `fstringprefix` ("'" | '"' | "'''" | '"""')
1251 FSTRING_END: `f_quote`
1252 fstringprefix: <("f" | "fr" | "rf"), case-insensitive>
1253 f_debug_specifier: '='
1254 f_quote: <the quote character(s) used in FSTRING_START>
1255
1256 fstring_middle:
1257 | `fstring_replacement_field`
1258 | `FSTRING_MIDDLE`
1259 FSTRING_MIDDLE:
1260 | (!"\" !`newline` !'{' !'}' !`f_quote`) `source_character`
1261 | `stringescapeseq`
1262 | "{{"
1263 | "}}"
1264 | <newline, in triple-quoted f-strings only>
1265 fstring_replacement_field:
1266 | '{' `f_expression` [`f_debug_specifier`] [`fstring_conversion`]
1267 [`fstring_full_format_spec`] '}'
1268 fstring_conversion:
1269 | "!" ("s" | "r" | "a")
1270 fstring_full_format_spec:
1271 | ':' `fstring_format_spec`*
1272 fstring_format_spec:
1273 | `FSTRING_MIDDLE`
1274 | `fstring_replacement_field`
1275 f_expression:
1276 | ','.(`conditional_expression` | "*" `or_expr`)+ [","]
1277 | `yield_expression`
1278
1279 .. note::
1280
1281 In the above grammar snippet, the ``f_quote`` and ``FSTRING_MIDDLE`` rules
1282 are context-sensitive -- they depend on the contents of ``FSTRING_START``
1283 of the nearest enclosing ``fstring``.
1284
1285 Constructing a more traditional formal grammar from this template is left
1286 as an exercise for the reader.
1287
1288 The grammar for t-strings is identical to the one for f-strings, with *t*
1289 instead of *f* at the beginning of rule and token names and in the prefix.
1290
1291 .. grammar-snippet::
1292 :group: python-grammar
1293
1294 tstring: TSTRING_START tstring_middle* TSTRING_END
1295
1296 <rest of the t-string grammar is omitted; see above>
1297
1298
1299 .. _numbers:
1300
1301 Numeric literals
1302 ================
1303
1304 .. index:: number, numeric literal, integer literal
1305 floating-point literal, hexadecimal literal
1306 octal literal, binary literal, decimal literal, imaginary literal, complex literal
1307
1308 :data:`~token.NUMBER` tokens represent numeric literals, of which there are
1309 three types: integers, floating-point numbers, and imaginary numbers.
1310
1311 .. grammar-snippet::
1312 :group: python-grammar
1313
1314 NUMBER: `integer` | `floatnumber` | `imagnumber`
1315
1316 The numeric value of a numeric literal is the same as if it were passed as a
1317 string to the :class:`int`, :class:`float` or :class:`complex` class
1318 constructor, respectively.
1319 Note that not all valid inputs for those constructors are also valid literals.
1320
1321 Numeric literals do not include a sign; a phrase like ``-1`` is
1322 actually an expression composed of the unary operator '``-``' and the literal
1323 ``1``.
1324
1325
1326 .. index::
1327 single: 0b; integer literal
1328 single: 0o; integer literal
1329 single: 0x; integer literal
1330 single: _ (underscore); in numeric literal
1331
1332 .. _integers:
1333
1334 Integer literals
1335 ----------------
1336
1337 Integer literals denote whole numbers. For example::
1338
1339 7
1340 3
1341 2147483647
1342
1343 There is no limit for the length of integer literals apart from what can be
1344 stored in available memory::
1345
1346 7922816251426433759354395033679228162514264337593543950336
1347
1348 Underscores can be used to group digits for enhanced readability,
1349 and are ignored for determining the numeric value of the literal.
1350 For example, the following literals are equivalent::
1351
1352 100_000_000_000
1353 100000000000
1354 1_00_00_00_00_000
1355
1356 Underscores can only occur between digits.
1357 For example, ``_123``, ``321_``, and ``123__321`` are *not* valid literals.
1358
1359 Integers can be specified in binary (base 2), octal (base 8), or hexadecimal
1360 (base 16) using the prefixes ``0b``, ``0o`` and ``0x``, respectively.
1361 Hexadecimal digits 10 through 15 are represented by letters ``A``-``F``,
1362 case-insensitive. For example::
1363
1364 0b100110111
1365 0b_1110_0101
1366 0o177
1367 0o377
1368 0xdeadbeef
1369 0xDead_Beef
1370
1371 An underscore can follow the base specifier.
1372 For example, ``0x_1f`` is a valid literal, but ``0_x1f`` and ``0x__1f`` are
1373 not.
1374
1375 Leading zeros in a non-zero decimal number are not allowed.
1376 For example, ``0123`` is not a valid literal.
1377 This is for disambiguation with C-style octal literals, which Python used
1378 before version 3.0.
1379
1380 Formally, integer literals are described by the following lexical definitions:
1381
1382 .. grammar-snippet::
1383 :group: python-grammar
1384
1385 integer: `decinteger` | `bininteger` | `octinteger` | `hexinteger` | `zerointeger`
1386 decinteger: `nonzerodigit` (["_"] `digit`)*
1387 bininteger: "0" ("b" | "B") (["_"] `bindigit`)+
1388 octinteger: "0" ("o" | "O") (["_"] `octdigit`)+
1389 hexinteger: "0" ("x" | "X") (["_"] `hexdigit`)+
1390 zerointeger: "0"+ (["_"] "0")*
1391 nonzerodigit: "1"..."9"
1392 digit: "0"..."9"
1393 bindigit: "0" | "1"
1394 octdigit: "0"..."7"
1395 hexdigit: `digit` | "a"..."f" | "A"..."F"
1396
1397 .. versionchanged:: 3.6
1398 Underscores are now allowed for grouping purposes in literals.
1399
1400
1401 .. index::
1402 single: . (dot); in numeric literal
1403 single: e; in numeric literal
1404 single: _ (underscore); in numeric literal
1405 .. _floating:
1406
1407 Floating-point literals
1408 -----------------------
1409
1410 Floating-point (float) literals, such as ``3.14`` or ``1.5``, denote
1411 :ref:`approximations of real numbers <datamodel-float>`.
1412
1413 They consist of *integer* and *fraction* parts, each composed of decimal digits.
1414 The parts are separated by a decimal point, ``.``::
1415
1416 2.71828
1417 4.0
1418
1419 Unlike in integer literals, leading zeros are allowed.
1420 For example, ``077.010`` is legal, and denotes the same number as ``77.01``.
1421
1422 As in integer literals, single underscores may occur between digits to help
1423 readability::
1424
1425 96_485.332_123
1426 3.14_15_93
1427
1428 Either of these parts, but not both, can be empty. For example::
1429
1430 10. # (equivalent to 10.0)
1431 .001 # (equivalent to 0.001)
1432
1433 Optionally, the integer and fraction may be followed by an *exponent*:
1434 the letter ``e`` or ``E``, followed by an optional sign, ``+`` or ``-``,
1435 and a number in the same format as the integer and fraction parts.
1436 The ``e`` or ``E`` represents "times ten raised to the power of"::
1437
1438 1.0e3 # (represents 1.0×10³, or 1000.0)
1439 1.166e-5 # (represents 1.166×10⁻⁵, or 0.00001166)
1440 6.02214076e+23 # (represents 6.02214076×10²³, or 602214076000000000000000.)
1441
1442 In floats with only integer and exponent parts, the decimal point may be
1443 omitted::
1444
1445 1e3 # (equivalent to 1.e3 and 1.0e3)
1446 0e0 # (equivalent to 0.)
1447
1448 Formally, floating-point literals are described by the following
1449 lexical definitions:
1450
1451 .. grammar-snippet::
1452 :group: python-grammar
1453
1454 floatnumber:
1455 | `digitpart` "." [`digitpart`] [`exponent`]
1456 | "." `digitpart` [`exponent`]
1457 | `digitpart` `exponent`
1458 digitpart: `digit` (["_"] `digit`)*
1459 exponent: ("e" | "E") ["+" | "-"] `digitpart`
1460
1461 .. versionchanged:: 3.6
1462 Underscores are now allowed for grouping purposes in literals.
1463
1464
1465 .. index::
1466 single: j; in numeric literal
1467 .. _imaginary:
1468
1469 Imaginary literals
1470 ------------------
1471
1472 Python has :ref:`complex number <typesnumeric>` objects, but no complex
1473 literals.
1474 Instead, *imaginary literals* denote complex numbers with a zero
1475 real part.
1476
1477 For example, in math, the complex number 3+4.2\ *i* is written
1478 as the real number 3 added to the imaginary number 4.2\ *i*.
1479 Python uses a similar syntax, except the imaginary unit is written as ``j``
1480 rather than *i*::
1481
1482 3+4.2j
1483
1484 This is an expression composed
1485 of the :ref:`integer literal <integers>` ``3``,
1486 the :ref:`operator <operators>` '``+``',
1487 and the :ref:`imaginary literal <imaginary>` ``4.2j``.
1488 Since these are three separate tokens, whitespace is allowed between them::
1489
1490 3 + 4.2j
1491
1492 No whitespace is allowed *within* each token.
1493 In particular, the ``j`` suffix, may not be separated from the number
1494 before it.
1495
1496 The number before the ``j`` has the same syntax as a floating-point literal.
1497 Thus, the following are valid imaginary literals::
1498
1499 4.2j
1500 3.14j
1501 10.j
1502 .001j
1503 1e100j
1504 3.14e-10j
1505 3.14_15_93j
1506
1507 Unlike in a floating-point literal the decimal point can be omitted if the
1508 imaginary number only has an integer part.
1509 The number is still evaluated as a floating-point number, not an integer::
1510
1511 10j
1512 0j
1513 1000000000000000000000000j # equivalent to 1e+24j
1514
1515 The ``j`` suffix is case-insensitive.
1516 That means you can use ``J`` instead::
1517
1518 3.14J # equivalent to 3.14j
1519
1520 Formally, imaginary literals are described by the following lexical definition:
1521
1522 .. grammar-snippet::
1523 :group: python-grammar
1524
1525 imagnumber: (`floatnumber` | `digitpart`) ("j" | "J")
1526
1527
1528 .. _delimiters:
1529 .. _operators:
1530 .. _lexical-ellipsis:
1531
1532 Operators and delimiters
1533 ========================
1534
1535 .. index::
1536 single: operators
1537 single: delimiters
1538
1539 The following grammar defines :dfn:`operator` and :dfn:`delimiter` tokens,
1540 that is, the generic :data:`~token.OP` token type.
1541 A :ref:`list of these tokens and their names <token_operators_delimiters>`
1542 is also available in the :mod:`!token` module documentation.
1543
1544 .. grammar-snippet::
1545 :group: python-grammar
1546
1547 OP:
1548 | assignment_operator
1549 | bitwise_operator
1550 | comparison_operator
1551 | enclosing_delimiter
1552 | other_delimiter
1553 | arithmetic_operator
1554 | "..."
1555 | other_op
1556
1557 assignment_operator: "+=" | "-=" | "*=" | "**=" | "/=" | "//=" | "%=" |
1558 "&=" | "|=" | "^=" | "<<=" | ">>=" | "@=" | ":="
1559 bitwise_operator: "&" | "|" | "^" | "~" | "<<" | ">>"
1560 comparison_operator: "<=" | ">=" | "<" | ">" | "==" | "!="
1561 enclosing_delimiter: "(" | ")" | "[" | "]" | "{" | "}"
1562 other_delimiter: "," | ":" | "!" | ";" | "=" | "->"
1563 arithmetic_operator: "+" | "-" | "**" | "*" | "//" | "/" | "%"
1564 other_op: "." | "@"
1565
1566 .. note::
1567
1568 Generally, *operators* are used to combine :ref:`expressions <expressions>`,
1569 while *delimiters* serve other purposes.
1570 However, there is no clear, formal distinction between the two categories.
1571
1572 Some tokens can serve as either operators or delimiters, depending on usage.
1573 For example, ``*`` is both the multiplication operator and a delimiter used
1574 for sequence unpacking, and ``@`` is both the matrix multiplication and
1575 a delimiter that introduces decorators.
1576
1577 For some tokens, the distinction is unclear.
1578 For example, some people consider ``.``, ``(``, and ``)`` to be delimiters, while others
1579 see the :py:func:`getattr` operator and the function call operator(s).
1580
1581 Some of Python's operators, like ``and``, ``or``, and ``not in``, use
1582 :ref:`keyword <keywords>` tokens rather than "symbols" (operator tokens).
1583
1584 A sequence of three consecutive periods (``...``) has a special
1585 meaning as an :py:data:`Ellipsis` literal.
1586