Debian Codegraph @

Doc/reference/lexical_analysis.rst

raw
1
2 .. _lexical:
3
4 ****************
5 Lexical analysis
6 ****************
7
8 .. index:: lexical analysis, parser, token
9
10 A Python program is read by a *parser*. Input to the parser is a stream of
11 :term:`tokens <token>`, generated by the *lexical analyzer* (also known as
12 the *tokenizer*).
13 This chapter describes how the lexical analyzer breaks a file into tokens.
14
15 Python reads program text as Unicode code points; the encoding of a source file
16 can be given by an encoding declaration and defaults to UTF-8, see :pep:`3120`
17 for details. If the source file cannot be decoded, a :exc:`SyntaxError` is
18 raised.
19
20
21 .. _line-structure:
22
23 Line structure
24 ==============
25
26 .. index:: line structure
27
28 A Python program is divided into a number of *logical lines*.
29
30
31 .. _logical-lines:
32
33 Logical lines
34 -------------
35
36 .. index:: logical line, physical line, line joining, NEWLINE token
37
38 The end of a logical line is represented by the token NEWLINE. Statements
39 cannot cross logical line boundaries except where NEWLINE is allowed by the
40 syntax (e.g., between statements in compound statements). A logical line is
41 constructed from one or more *physical lines* by following the explicit or
42 implicit *line joining* rules.
43
44
45 .. _physical-lines:
46
47 Physical lines
48 --------------
49
50 A physical line is a sequence of characters terminated by an end-of-line
51 sequence. In source files and strings, any of the standard platform line
52 termination sequences can be used - the Unix form using ASCII LF (linefeed),
53 the Windows form using the ASCII sequence CR LF (return followed by linefeed),
54 or the old Macintosh form using the ASCII CR (return) character. All of these
55 forms can be used equally, regardless of platform. The end of input also serves
56 as an implicit terminator for the final physical line.
57
58 When embedding Python, source code strings should be passed to Python APIs using
59 the standard C conventions for newline characters (the ``\n`` character,
60 representing ASCII LF, is the line terminator).
61
62
63 .. _comments:
64
65 Comments
66 --------
67
68 .. index:: comment, hash character
69 single: # (hash); comment
70
71 A comment starts with a hash character (``#``) that is not part of a string
72 literal, and ends at the end of the physical line. A comment signifies the end
73 of the logical line unless the implicit line joining rules are invoked. Comments
74 are ignored by the syntax.
75
76
77 .. _encodings:
78
79 Encoding declarations
80 ---------------------
81
82 .. index:: source character set, encoding declarations (source file)
83 single: # (hash); source encoding declaration
84
85 If a comment in the first or second line of the Python script matches the
86 regular expression ``coding[=:]\s*([-\w.]+)``, this comment is processed as an
87 encoding declaration; the first group of this expression names the encoding of
88 the source code file. The encoding declaration must appear on a line of its
89 own. If it is the second line, the first line must also be a comment-only line.
90 The recommended forms of an encoding expression are ::
91
92 # -*- coding: <encoding-name> -*-
93
94 which is recognized also by GNU Emacs, and ::
95
96 # vim:fileencoding=<encoding-name>
97
98 which is recognized by Bram Moolenaar's VIM.
99
100 If no encoding declaration is found, the default encoding is UTF-8. If the
101 implicit or explicit encoding of a file is UTF-8, an initial UTF-8 byte-order
102 mark (``b'\xef\xbb\xbf'``) is ignored rather than being a syntax error.
103
104 If an encoding is declared, the encoding name must be recognized by Python
105 (see :ref:`standard-encodings`). The
106 encoding is used for all lexical analysis, including string literals, comments
107 and identifiers.
108
109
110 .. _explicit-joining:
111
112 Explicit line joining
113 ---------------------
114
115 .. index:: physical line, line joining, line continuation, backslash character
116
117 Two or more physical lines may be joined into logical lines using backslash
118 characters (``\``), as follows: when a physical line ends in a backslash that is
119 not part of a string literal or comment, it is joined with the following forming
120 a single logical line, deleting the backslash and the following end-of-line
121 character. For example::
122
123 if 1900 < year < 2100 and 1 <= month <= 12 \
124 and 1 <= day <= 31 and 0 <= hour < 24 \
125 and 0 <= minute < 60 and 0 <= second < 60: # Looks like a valid date
126 return 1
127
128 A line ending in a backslash cannot carry a comment. A backslash does not
129 continue a comment. A backslash does not continue a token except for string
130 literals (i.e., tokens other than string literals cannot be split across
131 physical lines using a backslash). A backslash is illegal elsewhere on a line
132 outside a string literal.
133
134
135 .. _implicit-joining:
136
137 Implicit line joining
138 ---------------------
139
140 Expressions in parentheses, square brackets or curly braces can be split over
141 more than one physical line without using backslashes. For example::
142
143 month_names = ['Januari', 'Februari', 'Maart', # These are the
144 'April', 'Mei', 'Juni', # Dutch names
145 'Juli', 'Augustus', 'September', # for the months
146 'Oktober', 'November', 'December'] # of the year
147
148 Implicitly continued lines can carry comments. The indentation of the
149 continuation lines is not important. Blank continuation lines are allowed.
150 There is no NEWLINE token between implicit continuation lines. Implicitly
151 continued lines can also occur within triple-quoted strings (see below); in that
152 case they cannot carry comments.
153
154
155 .. _blank-lines:
156
157 Blank lines
158 -----------
159
160 .. index:: single: blank line
161
162 A logical line that contains only spaces, tabs, formfeeds and possibly a
163 comment, is ignored (i.e., no NEWLINE token is generated). During interactive
164 input of statements, handling of a blank line may differ depending on the
165 implementation of the read-eval-print loop. In the standard interactive
166 interpreter, an entirely blank logical line (i.e. one containing not even
167 whitespace or a comment) terminates a multi-line statement.
168
169
170 .. _indentation:
171
172 Indentation
173 -----------
174
175 .. index:: indentation, leading whitespace, space, tab, grouping, statement grouping
176
177 Leading whitespace (spaces and tabs) at the beginning of a logical line is used
178 to compute the indentation level of the line, which in turn is used to determine
179 the grouping of statements.
180
181 Tabs are replaced (from left to right) by one to eight spaces such that the
182 total number of characters up to and including the replacement is a multiple of
183 eight (this is intended to be the same rule as used by Unix). The total number
184 of spaces preceding the first non-blank character then determines the line's
185 indentation. Indentation cannot be split over multiple physical lines using
186 backslashes; the whitespace up to the first backslash determines the
187 indentation.
188
189 Indentation is rejected as inconsistent if a source file mixes tabs and spaces
190 in a way that makes the meaning dependent on the worth of a tab in spaces; a
191 :exc:`TabError` is raised in that case.
192
193 **Cross-platform compatibility note:** because of the nature of text editors on
194 non-UNIX platforms, it is unwise to use a mixture of spaces and tabs for the
195 indentation in a single source file. It should also be noted that different
196 platforms may explicitly limit the maximum indentation level.
197
198 A formfeed character may be present at the start of the line; it will be ignored
199 for the indentation calculations above. Formfeed characters occurring elsewhere
200 in the leading whitespace have an undefined effect (for instance, they may reset
201 the space count to zero).
202
203 .. index:: INDENT token, DEDENT token
204
205 The indentation levels of consecutive lines are used to generate INDENT and
206 DEDENT tokens, using a stack, as follows.
207
208 Before the first line of the file is read, a single zero is pushed on the stack;
209 this will never be popped off again. The numbers pushed on the stack will
210 always be strictly increasing from bottom to top. At the beginning of each
211 logical line, the line's indentation level is compared to the top of the stack.
212 If it is equal, nothing happens. If it is larger, it is pushed on the stack, and
213 one INDENT token is generated. If it is smaller, it *must* be one of the
214 numbers occurring on the stack; all numbers on the stack that are larger are
215 popped off, and for each number popped off a DEDENT token is generated. At the
216 end of the file, a DEDENT token is generated for each number remaining on the
217 stack that is larger than zero.
218
219 Here is an example of a correctly (though confusingly) indented piece of Python
220 code::
221
222 def perm(l):
223 # Compute the list of all permutations of l
224 if len(l) <= 1:
225 return [l]
226 r = []
227 for i in range(len(l)):
228 s = l[:i] + l[i+1:]
229 p = perm(s)
230 for x in p:
231 r.append(l[i:i+1] + x)
232 return r
233
234 The following example shows various indentation errors::
235
236 def perm(l): # error: first line indented
237 for i in range(len(l)): # error: not indented
238 s = l[:i] + l[i+1:]
239 p = perm(l[:i] + l[i+1:]) # error: unexpected indent
240 for x in p:
241 r.append(l[i:i+1] + x)
242 return r # error: inconsistent dedent
243
244 (Actually, the first three errors are detected by the parser; only the last
245 error is found by the lexical analyzer --- the indentation of ``return r`` does
246 not match a level popped off the stack.)
247
248
249 .. _whitespace:
250
251 Whitespace between tokens
252 -------------------------
253
254 Except at the beginning of a logical line or in string literals, the whitespace
255 characters space, tab and formfeed can be used interchangeably to separate
256 tokens. Whitespace is needed between two tokens only if their concatenation
257 could otherwise be interpreted as a different token (e.g., ab is one token, but
258 a b is two tokens).
259
260
261 .. _other-tokens:
262
263 Other tokens
264 ============
265
266 Besides NEWLINE, INDENT and DEDENT, the following categories of tokens exist:
267 *identifiers*, *keywords*, *literals*, *operators*, and *delimiters*. Whitespace
268 characters (other than line terminators, discussed earlier) are not tokens, but
269 serve to delimit tokens. Where ambiguity exists, a token comprises the longest
270 possible string that forms a legal token, when read from left to right.
271
272
273 .. _identifiers:
274
275 Identifiers and keywords
276 ========================
277
278 .. index:: identifier, name
279
280 Identifiers (also referred to as *names*) are described by the following lexical
281 definitions.
282
283 The syntax of identifiers in Python is based on the Unicode standard annex
284 UAX-31, with elaboration and changes as defined below; see also :pep:`3131` for
285 further details.
286
287 Within the ASCII range (U+0001..U+007F), the valid characters for identifiers
288 include the uppercase and lowercase letters ``A`` through
289 ``Z``, the underscore ``_`` and, except for the first character, the digits
290 ``0`` through ``9``.
291 Python 3.0 introduced additional characters from outside the ASCII range (see
292 :pep:`3131`). For these characters, the classification uses the version of the
293 Unicode Character Database as included in the :mod:`unicodedata` module.
294
295 Identifiers are unlimited in length. Case is significant.
296
297 .. productionlist:: python-grammar
298 identifier: `xid_start` `xid_continue`*
299 id_start: <all characters in general categories Lu, Ll, Lt, Lm, Lo, Nl, the underscore, and characters with the Other_ID_Start property>
300 id_continue: <all characters in `id_start`, plus characters in the categories Mn, Mc, Nd, Pc and others with the Other_ID_Continue property>
301 xid_start: <all characters in `id_start` whose NFKC normalization is in "id_start xid_continue*">
302 xid_continue: <all characters in `id_continue` whose NFKC normalization is in "id_continue*">
303
304 The Unicode category codes mentioned above stand for:
305
306 * *Lu* - uppercase letters
307 * *Ll* - lowercase letters
308 * *Lt* - titlecase letters
309 * *Lm* - modifier letters
310 * *Lo* - other letters
311 * *Nl* - letter numbers
312 * *Mn* - nonspacing marks
313 * *Mc* - spacing combining marks
314 * *Nd* - decimal numbers
315 * *Pc* - connector punctuations
316 * *Other_ID_Start* - explicit list of characters in `PropList.txt
317 <https://www.unicode.org/Public/15.1.0/ucd/PropList.txt>`_ to support backwards
318 compatibility
319 * *Other_ID_Continue* - likewise
320
321 All identifiers are converted into the normal form NFKC while parsing; comparison
322 of identifiers is based on NFKC.
323
324 A non-normative HTML file listing all valid identifier characters for Unicode
325 15.1.0 can be found at
326 https://www.unicode.org/Public/15.1.0/ucd/DerivedCoreProperties.txt
327
328
329 .. _keywords:
330
331 Keywords
332 --------
333
334 .. index::
335 single: keyword
336 single: reserved word
337
338 The following identifiers are used as reserved words, or *keywords* of the
339 language, and cannot be used as ordinary identifiers. They must be spelled
340 exactly as written here:
341
342 .. sourcecode:: text
343
344 False await else import pass
345 None break except in raise
346 True class finally is return
347 and continue for lambda try
348 as def from nonlocal while
349 assert del global not with
350 async elif if or yield
351
352
353 .. _soft-keywords:
354
355 Soft Keywords
356 -------------
357
358 .. index:: soft keyword, keyword
359
360 .. versionadded:: 3.10
361
362 Some identifiers are only reserved under specific contexts. These are known as
363 *soft keywords*. The identifiers ``match``, ``case``, ``type`` and ``_`` can
364 syntactically act as keywords in certain contexts,
365 but this distinction is done at the parser level, not when tokenizing.
366
367 As soft keywords, their use in the grammar is possible while still
368 preserving compatibility with existing code that uses these names as
369 identifier names.
370
371 ``match``, ``case``, and ``_`` are used in the :keyword:`match` statement.
372 ``type`` is used in the :keyword:`type` statement.
373
374 .. versionchanged:: 3.12
375 ``type`` is now a soft keyword.
376
377 .. index::
378 single: _, identifiers
379 single: __, identifiers
380 .. _id-classes:
381
382 Reserved classes of identifiers
383 -------------------------------
384
385 Certain classes of identifiers (besides keywords) have special meanings. These
386 classes are identified by the patterns of leading and trailing underscore
387 characters:
388
389 ``_*``
390 Not imported by ``from module import *``.
391
392 ``_``
393 In a ``case`` pattern within a :keyword:`match` statement, ``_`` is a
394 :ref:`soft keyword <soft-keywords>` that denotes a
395 :ref:`wildcard <wildcard-patterns>`.
396
397 Separately, the interactive interpreter makes the result of the last evaluation
398 available in the variable ``_``.
399 (It is stored in the :mod:`builtins` module, alongside built-in
400 functions like ``print``.)
401
402 Elsewhere, ``_`` is a regular identifier. It is often used to name
403 "special" items, but it is not special to Python itself.
404
405 .. note::
406
407 The name ``_`` is often used in conjunction with internationalization;
408 refer to the documentation for the :mod:`gettext` module for more
409 information on this convention.
410
411 It is also commonly used for unused variables.
412
413 ``__*__``
414 System-defined names, informally known as "dunder" names. These names are
415 defined by the interpreter and its implementation (including the standard library).
416 Current system names are discussed in the :ref:`specialnames` section and elsewhere.
417 More will likely be defined in future versions of Python. *Any* use of ``__*__`` names,
418 in any context, that does not follow explicitly documented use, is subject to
419 breakage without warning.
420
421 ``__*``
422 Class-private names. Names in this category, when used within the context of a
423 class definition, are re-written to use a mangled form to help avoid name
424 clashes between "private" attributes of base and derived classes. See section
425 :ref:`atom-identifiers`.
426
427
428 .. _literals:
429
430 Literals
431 ========
432
433 .. index:: literal, constant
434
435 Literals are notations for constant values of some built-in types.
436
437
438 .. index:: string literal, bytes literal, ASCII
439 single: ' (single quote); string literal
440 single: " (double quote); string literal
441 single: u'; string literal
442 single: u"; string literal
443 .. _strings:
444
445 String and Bytes literals
446 -------------------------
447
448 String literals are described by the following lexical definitions:
449
450 .. productionlist:: python-grammar
451 stringliteral: [`stringprefix`](`shortstring` | `longstring`)
452 stringprefix: "r" | "u" | "R" | "U" | "f" | "F"
453 : | "fr" | "Fr" | "fR" | "FR" | "rf" | "rF" | "Rf" | "RF"
454 shortstring: "'" `shortstringitem`* "'" | '"' `shortstringitem`* '"'
455 longstring: "'''" `longstringitem`* "'''" | '"""' `longstringitem`* '"""'
456 shortstringitem: `shortstringchar` | `stringescapeseq`
457 longstringitem: `longstringchar` | `stringescapeseq`
458 shortstringchar: <any source character except "\" or newline or the quote>
459 longstringchar: <any source character except "\">
460 stringescapeseq: "\" <any source character>
461
462 .. productionlist:: python-grammar
463 bytesliteral: `bytesprefix`(`shortbytes` | `longbytes`)
464 bytesprefix: "b" | "B" | "br" | "Br" | "bR" | "BR" | "rb" | "rB" | "Rb" | "RB"
465 shortbytes: "'" `shortbytesitem`* "'" | '"' `shortbytesitem`* '"'
466 longbytes: "'''" `longbytesitem`* "'''" | '"""' `longbytesitem`* '"""'
467 shortbytesitem: `shortbyteschar` | `bytesescapeseq`
468 longbytesitem: `longbyteschar` | `bytesescapeseq`
469 shortbyteschar: <any ASCII character except "\" or newline or the quote>
470 longbyteschar: <any ASCII character except "\">
471 bytesescapeseq: "\" <any ASCII character>
472
473 One syntactic restriction not indicated by these productions is that whitespace
474 is not allowed between the :token:`~python-grammar:stringprefix` or
475 :token:`~python-grammar:bytesprefix` and the rest of the literal. The source
476 character set is defined by the encoding declaration; it is UTF-8 if no encoding
477 declaration is given in the source file; see section :ref:`encodings`.
478
479 .. index:: triple-quoted string, Unicode Consortium, raw string
480 single: """; string literal
481 single: '''; string literal
482
483 In plain English: Both types of literals can be enclosed in matching single quotes
484 (``'``) or double quotes (``"``). They can also be enclosed in matching groups
485 of three single or double quotes (these are generally referred to as
486 *triple-quoted strings*). The backslash (``\``) character is used to give special
487 meaning to otherwise ordinary characters like ``n``, which means 'newline' when
488 escaped (``\n``). It can also be used to escape characters that otherwise have a
489 special meaning, such as newline, backslash itself, or the quote character.
490 See :ref:`escape sequences <escape-sequences>` below for examples.
491
492 .. index::
493 single: b'; bytes literal
494 single: b"; bytes literal
495
496 Bytes literals are always prefixed with ``'b'`` or ``'B'``; they produce an
497 instance of the :class:`bytes` type instead of the :class:`str` type. They
498 may only contain ASCII characters; bytes with a numeric value of 128 or greater
499 must be expressed with escapes.
500
501 .. index::
502 single: r'; raw string literal
503 single: r"; raw string literal
504
505 Both string and bytes literals may optionally be prefixed with a letter ``'r'``
506 or ``'R'``; such constructs are called :dfn:`raw string literals`
507 and :dfn:`raw bytes literals` respectively and treat backslashes as
508 literal characters. As a result, in raw string literals, ``'\U'`` and ``'\u'``
509 escapes are not treated specially.
510
511 .. versionadded:: 3.3
512 The ``'rb'`` prefix of raw bytes literals has been added as a synonym
513 of ``'br'``.
514
515 Support for the unicode legacy literal (``u'value'``) was reintroduced
516 to simplify the maintenance of dual Python 2.x and 3.x codebases.
517 See :pep:`414` for more information.
518
519 .. index::
520 single: f'; formatted string literal
521 single: f"; formatted string literal
522
523 A string literal with ``'f'`` or ``'F'`` in its prefix is a
524 :dfn:`formatted string literal`; see :ref:`f-strings`. The ``'f'`` may be
525 combined with ``'r'``, but not with ``'b'`` or ``'u'``, therefore raw
526 formatted strings are possible, but formatted bytes literals are not.
527
528 In triple-quoted literals, unescaped newlines and quotes are allowed (and are
529 retained), except that three unescaped quotes in a row terminate the literal. (A
530 "quote" is the character used to open the literal, i.e. either ``'`` or ``"``.)
531
532 .. index:: physical line, escape sequence, Standard C, C
533 single: \ (backslash); escape sequence
534 single: \\; escape sequence
535 single: \a; escape sequence
536 single: \b; escape sequence
537 single: \f; escape sequence
538 single: \n; escape sequence
539 single: \r; escape sequence
540 single: \t; escape sequence
541 single: \v; escape sequence
542 single: \x; escape sequence
543 single: \N; escape sequence
544 single: \u; escape sequence
545 single: \U; escape sequence
546
547 .. _escape-sequences:
548
549
550 Escape sequences
551 ^^^^^^^^^^^^^^^^
552
553 Unless an ``'r'`` or ``'R'`` prefix is present, escape sequences in string and
554 bytes literals are interpreted according to rules similar to those used by
555 Standard C. The recognized escape sequences are:
556
557 +-------------------------+---------------------------------+-------+
558 | Escape Sequence | Meaning | Notes |
559 +=========================+=================================+=======+
560 | ``\``\ <newline> | Backslash and newline ignored | \(1) |
561 +-------------------------+---------------------------------+-------+
562 | ``\\`` | Backslash (``\``) | |
563 +-------------------------+---------------------------------+-------+
564 | ``\'`` | Single quote (``'``) | |
565 +-------------------------+---------------------------------+-------+
566 | ``\"`` | Double quote (``"``) | |
567 +-------------------------+---------------------------------+-------+
568 | ``\a`` | ASCII Bell (BEL) | |
569 +-------------------------+---------------------------------+-------+
570 | ``\b`` | ASCII Backspace (BS) | |
571 +-------------------------+---------------------------------+-------+
572 | ``\f`` | ASCII Formfeed (FF) | |
573 +-------------------------+---------------------------------+-------+
574 | ``\n`` | ASCII Linefeed (LF) | |
575 +-------------------------+---------------------------------+-------+
576 | ``\r`` | ASCII Carriage Return (CR) | |
577 +-------------------------+---------------------------------+-------+
578 | ``\t`` | ASCII Horizontal Tab (TAB) | |
579 +-------------------------+---------------------------------+-------+
580 | ``\v`` | ASCII Vertical Tab (VT) | |
581 +-------------------------+---------------------------------+-------+
582 | :samp:`\\\\{ooo}` | Character with octal value | (2,4) |
583 | | *ooo* | |
584 +-------------------------+---------------------------------+-------+
585 | :samp:`\\x{hh}` | Character with hex value *hh* | (3,4) |
586 +-------------------------+---------------------------------+-------+
587
588 Escape sequences only recognized in string literals are:
589
590 +-------------------------+---------------------------------+-------+
591 | Escape Sequence | Meaning | Notes |
592 +=========================+=================================+=======+
593 | :samp:`\\N\\{{name}\\}` | Character named *name* in the | \(5) |
594 | | Unicode database | |
595 +-------------------------+---------------------------------+-------+
596 | :samp:`\\u{xxxx}` | Character with 16-bit hex value | \(6) |
597 | | *xxxx* | |
598 +-------------------------+---------------------------------+-------+
599 | :samp:`\\U{xxxxxxxx}` | Character with 32-bit hex value | \(7) |
600 | | *xxxxxxxx* | |
601 +-------------------------+---------------------------------+-------+
602
603 Notes:
604
605 (1)
606 A backslash can be added at the end of a line to ignore the newline::
607
608 >>> 'This string will not include \
609 ... backslashes or newline characters.'
610 'This string will not include backslashes or newline characters.'
611
612 The same result can be achieved using :ref:`triple-quoted strings <strings>`,
613 or parentheses and :ref:`string literal concatenation <string-concatenation>`.
614
615
616 (2)
617 As in Standard C, up to three octal digits are accepted.
618
619 .. versionchanged:: 3.11
620 Octal escapes with value larger than ``0o377`` produce a
621 :exc:`DeprecationWarning`.
622
623 .. versionchanged:: 3.12
624 Octal escapes with value larger than ``0o377`` produce a
625 :exc:`SyntaxWarning`. In a future Python version they will be eventually
626 a :exc:`SyntaxError`.
627
628 (3)
629 Unlike in Standard C, exactly two hex digits are required.
630
631 (4)
632 In a bytes literal, hexadecimal and octal escapes denote the byte with the
633 given value. In a string literal, these escapes denote a Unicode character
634 with the given value.
635
636 (5)
637 .. versionchanged:: 3.3
638 Support for name aliases [#]_ has been added.
639
640 (6)
641 Exactly four hex digits are required.
642
643 (7)
644 Any Unicode character can be encoded this way. Exactly eight hex digits
645 are required.
646
647
648 .. index:: unrecognized escape sequence
649
650 Unlike Standard C, all unrecognized escape sequences are left in the string
651 unchanged, i.e., *the backslash is left in the result*. (This behavior is
652 useful when debugging: if an escape sequence is mistyped, the resulting output
653 is more easily recognized as broken.) It is also important to note that the
654 escape sequences only recognized in string literals fall into the category of
655 unrecognized escapes for bytes literals.
656
657 .. versionchanged:: 3.6
658 Unrecognized escape sequences produce a :exc:`DeprecationWarning`.
659
660 .. versionchanged:: 3.12
661 Unrecognized escape sequences produce a :exc:`SyntaxWarning`. In a future
662 Python version they will be eventually a :exc:`SyntaxError`.
663
664 Even in a raw literal, quotes can be escaped with a backslash, but the
665 backslash remains in the result; for example, ``r"\""`` is a valid string
666 literal consisting of two characters: a backslash and a double quote; ``r"\"``
667 is not a valid string literal (even a raw string cannot end in an odd number of
668 backslashes). Specifically, *a raw literal cannot end in a single backslash*
669 (since the backslash would escape the following quote character). Note also
670 that a single backslash followed by a newline is interpreted as those two
671 characters as part of the literal, *not* as a line continuation.
672
673
674 .. _string-concatenation:
675
676 String literal concatenation
677 ----------------------------
678
679 Multiple adjacent string or bytes literals (delimited by whitespace), possibly
680 using different quoting conventions, are allowed, and their meaning is the same
681 as their concatenation. Thus, ``"hello" 'world'`` is equivalent to
682 ``"helloworld"``. This feature can be used to reduce the number of backslashes
683 needed, to split long strings conveniently across long lines, or even to add
684 comments to parts of strings, for example::
685
686 re.compile("[A-Za-z_]" # letter or underscore
687 "[A-Za-z0-9_]*" # letter, digit or underscore
688 )
689
690 Note that this feature is defined at the syntactical level, but implemented at
691 compile time. The '+' operator must be used to concatenate string expressions
692 at run time. Also note that literal concatenation can use different quoting
693 styles for each component (even mixing raw strings and triple quoted strings),
694 and formatted string literals may be concatenated with plain string literals.
695
696
697 .. index::
698 single: formatted string literal
699 single: interpolated string literal
700 single: string; formatted literal
701 single: string; interpolated literal
702 single: f-string
703 single: fstring
704 single: {} (curly brackets); in formatted string literal
705 single: ! (exclamation); in formatted string literal
706 single: : (colon); in formatted string literal
707 single: = (equals); for help in debugging using string literals
708
709 .. _f-strings:
710 .. _formatted-string-literals:
711
712 f-strings
713 ---------
714
715 .. versionadded:: 3.6
716
717 A :dfn:`formatted string literal` or :dfn:`f-string` is a string literal
718 that is prefixed with ``'f'`` or ``'F'``. These strings may contain
719 replacement fields, which are expressions delimited by curly braces ``{}``.
720 While other string literals always have a constant value, formatted strings
721 are really expressions evaluated at run time.
722
723 Escape sequences are decoded like in ordinary string literals (except when
724 a literal is also marked as a raw string). After decoding, the grammar
725 for the contents of the string is:
726
727 .. productionlist:: python-grammar
728 f_string: (`literal_char` | "{{" | "}}" | `replacement_field`)*
729 replacement_field: "{" `f_expression` ["="] ["!" `conversion`] [":" `format_spec`] "}"
730 f_expression: (`conditional_expression` | "*" `or_expr`)
731 : ("," `conditional_expression` | "," "*" `or_expr`)* [","]
732 : | `yield_expression`
733 conversion: "s" | "r" | "a"
734 format_spec: (`literal_char` | `replacement_field`)*
735 literal_char: <any code point except "{", "}" or NULL>
736
737 The parts of the string outside curly braces are treated literally,
738 except that any doubled curly braces ``'{{'`` or ``'}}'`` are replaced
739 with the corresponding single curly brace. A single opening curly
740 bracket ``'{'`` marks a replacement field, which starts with a
741 Python expression. To display both the expression text and its value after
742 evaluation, (useful in debugging), an equal sign ``'='`` may be added after the
743 expression. A conversion field, introduced by an exclamation point ``'!'`` may
744 follow. A format specifier may also be appended, introduced by a colon ``':'``.
745 A replacement field ends with a closing curly bracket ``'}'``.
746
747 Expressions in formatted string literals are treated like regular
748 Python expressions surrounded by parentheses, with a few exceptions.
749 An empty expression is not allowed, and both :keyword:`lambda` and
750 assignment expressions ``:=`` must be surrounded by explicit parentheses.
751 Each expression is evaluated in the context where the formatted string literal
752 appears, in order from left to right. Replacement expressions can contain
753 newlines in both single-quoted and triple-quoted f-strings and they can contain
754 comments. Everything that comes after a ``#`` inside a replacement field
755 is a comment (even closing braces and quotes). In that case, replacement fields
756 must be closed in a different line.
757
758 .. code-block:: text
759
760 >>> f"abc{a # This is a comment }"
761 ... + 3}"
762 'abc5'
763
764 .. versionchanged:: 3.7
765 Prior to Python 3.7, an :keyword:`await` expression and comprehensions
766 containing an :keyword:`async for` clause were illegal in the expressions
767 in formatted string literals due to a problem with the implementation.
768
769 .. versionchanged:: 3.12
770 Prior to Python 3.12, comments were not allowed inside f-string replacement
771 fields.
772
773 When the equal sign ``'='`` is provided, the output will have the expression
774 text, the ``'='`` and the evaluated value. Spaces after the opening brace
775 ``'{'``, within the expression and after the ``'='`` are all retained in the
776 output. By default, the ``'='`` causes the :func:`repr` of the expression to be
777 provided, unless there is a format specified. When a format is specified it
778 defaults to the :func:`str` of the expression unless a conversion ``'!r'`` is
779 declared.
780
781 .. versionadded:: 3.8
782 The equal sign ``'='``.
783
784 If a conversion is specified, the result of evaluating the expression
785 is converted before formatting. Conversion ``'!s'`` calls :func:`str` on
786 the result, ``'!r'`` calls :func:`repr`, and ``'!a'`` calls :func:`ascii`.
787
788 The result is then formatted using the :func:`format` protocol. The
789 format specifier is passed to the :meth:`~object.__format__` method of the
790 expression or conversion result. An empty string is passed when the
791 format specifier is omitted. The formatted result is then included in
792 the final value of the whole string.
793
794 Top-level format specifiers may include nested replacement fields. These nested
795 fields may include their own conversion fields and :ref:`format specifiers
796 <formatspec>`, but may not include more deeply nested replacement fields. The
797 :ref:`format specifier mini-language <formatspec>` is the same as that used by
798 the :meth:`str.format` method.
799
800 Formatted string literals may be concatenated, but replacement fields
801 cannot be split across literals.
802
803 Some examples of formatted string literals::
804
805 >>> name = "Fred"
806 >>> f"He said his name is {name!r}."
807 "He said his name is 'Fred'."
808 >>> f"He said his name is {repr(name)}." # repr() is equivalent to !r
809 "He said his name is 'Fred'."
810 >>> width = 10
811 >>> precision = 4
812 >>> value = decimal.Decimal("12.34567")
813 >>> f"result: {value:{width}.{precision}}" # nested fields
814 'result: 12.35'
815 >>> today = datetime(year=2017, month=1, day=27)
816 >>> f"{today:%B %d, %Y}" # using date format specifier
817 'January 27, 2017'
818 >>> f"{today=:%B %d, %Y}" # using date format specifier and debugging
819 'today=January 27, 2017'
820 >>> number = 1024
821 >>> f"{number:#0x}" # using integer format specifier
822 '0x400'
823 >>> foo = "bar"
824 >>> f"{ foo = }" # preserves whitespace
825 " foo = 'bar'"
826 >>> line = "The mill's closed"
827 >>> f"{line = }"
828 'line = "The mill\'s closed"'
829 >>> f"{line = :20}"
830 "line = The mill's closed "
831 >>> f"{line = !r:20}"
832 'line = "The mill\'s closed" '
833
834
835 Reusing the outer f-string quoting type inside a replacement field is
836 permitted::
837
838 >>> a = dict(x=2)
839 >>> f"abc {a["x"]} def"
840 'abc 2 def'
841
842 .. versionchanged:: 3.12
843 Prior to Python 3.12, reuse of the same quoting type of the outer f-string
844 inside a replacement field was not possible.
845
846 Backslashes are also allowed in replacement fields and are evaluated the same
847 way as in any other context::
848
849 >>> a = ["a", "b", "c"]
850 >>> print(f"List a contains:\n{"\n".join(a)}")
851 List a contains:
852 a
853 b
854 c
855
856 .. versionchanged:: 3.12
857 Prior to Python 3.12, backslashes were not permitted inside an f-string
858 replacement field.
859
860 Formatted string literals cannot be used as docstrings, even if they do not
861 include expressions.
862
863 ::
864
865 >>> def foo():
866 ... f"Not a docstring"
867 ...
868 >>> foo.__doc__ is None
869 True
870
871 See also :pep:`498` for the proposal that added formatted string literals,
872 and :meth:`str.format`, which uses a related format string mechanism.
873
874
875 .. _numbers:
876
877 Numeric literals
878 ----------------
879
880 .. index:: number, numeric literal, integer literal
881 floating-point literal, hexadecimal literal
882 octal literal, binary literal, decimal literal, imaginary literal, complex literal
883
884 There are three types of numeric literals: integers, floating-point numbers, and
885 imaginary numbers. There are no complex literals (complex numbers can be formed
886 by adding a real number and an imaginary number).
887
888 Note that numeric literals do not include a sign; a phrase like ``-1`` is
889 actually an expression composed of the unary operator '``-``' and the literal
890 ``1``.
891
892
893 .. index::
894 single: 0b; integer literal
895 single: 0o; integer literal
896 single: 0x; integer literal
897 single: _ (underscore); in numeric literal
898
899 .. _integers:
900
901 Integer literals
902 ----------------
903
904 Integer literals are described by the following lexical definitions:
905
906 .. productionlist:: python-grammar
907 integer: `decinteger` | `bininteger` | `octinteger` | `hexinteger`
908 decinteger: `nonzerodigit` (["_"] `digit`)* | "0"+ (["_"] "0")*
909 bininteger: "0" ("b" | "B") (["_"] `bindigit`)+
910 octinteger: "0" ("o" | "O") (["_"] `octdigit`)+
911 hexinteger: "0" ("x" | "X") (["_"] `hexdigit`)+
912 nonzerodigit: "1"..."9"
913 digit: "0"..."9"
914 bindigit: "0" | "1"
915 octdigit: "0"..."7"
916 hexdigit: `digit` | "a"..."f" | "A"..."F"
917
918 There is no limit for the length of integer literals apart from what can be
919 stored in available memory.
920
921 Underscores are ignored for determining the numeric value of the literal. They
922 can be used to group digits for enhanced readability. One underscore can occur
923 between digits, and after base specifiers like ``0x``.
924
925 Note that leading zeros in a non-zero decimal number are not allowed. This is
926 for disambiguation with C-style octal literals, which Python used before version
927 3.0.
928
929 Some examples of integer literals::
930
931 7 2147483647 0o177 0b100110111
932 3 79228162514264337593543950336 0o377 0xdeadbeef
933 100_000_000_000 0b_1110_0101
934
935 .. versionchanged:: 3.6
936 Underscores are now allowed for grouping purposes in literals.
937
938
939 .. index::
940 single: . (dot); in numeric literal
941 single: e; in numeric literal
942 single: _ (underscore); in numeric literal
943 .. _floating:
944
945 Floating-point literals
946 -----------------------
947
948 Floating-point literals are described by the following lexical definitions:
949
950 .. productionlist:: python-grammar
951 floatnumber: `pointfloat` | `exponentfloat`
952 pointfloat: [`digitpart`] `fraction` | `digitpart` "."
953 exponentfloat: (`digitpart` | `pointfloat`) `exponent`
954 digitpart: `digit` (["_"] `digit`)*
955 fraction: "." `digitpart`
956 exponent: ("e" | "E") ["+" | "-"] `digitpart`
957
958 Note that the integer and exponent parts are always interpreted using radix 10.
959 For example, ``077e010`` is legal, and denotes the same number as ``77e10``. The
960 allowed range of floating-point literals is implementation-dependent. As in
961 integer literals, underscores are supported for digit grouping.
962
963 Some examples of floating-point literals::
964
965 3.14 10. .001 1e100 3.14e-10 0e0 3.14_15_93
966
967 .. versionchanged:: 3.6
968 Underscores are now allowed for grouping purposes in literals.
969
970
971 .. index::
972 single: j; in numeric literal
973 .. _imaginary:
974
975 Imaginary literals
976 ------------------
977
978 Imaginary literals are described by the following lexical definitions:
979
980 .. productionlist:: python-grammar
981 imagnumber: (`floatnumber` | `digitpart`) ("j" | "J")
982
983 An imaginary literal yields a complex number with a real part of 0.0. Complex
984 numbers are represented as a pair of floating-point numbers and have the same
985 restrictions on their range. To create a complex number with a nonzero real
986 part, add a floating-point number to it, e.g., ``(3+4j)``. Some examples of
987 imaginary literals::
988
989 3.14j 10.j 10j .001j 1e100j 3.14e-10j 3.14_15_93j
990
991
992 .. _operators:
993
994 Operators
995 =========
996
997 .. index:: single: operators
998
999 The following tokens are operators:
1000
1001 .. code-block:: none
1002
1003
1004 + - * ** / // % @
1005 << >> & | ^ ~ :=
1006 < > <= >= == !=
1007
1008
1009 .. _delimiters:
1010
1011 Delimiters
1012 ==========
1013
1014 .. index:: single: delimiters
1015
1016 The following tokens serve as delimiters in the grammar:
1017
1018 .. code-block:: none
1019
1020 ( ) [ ] { }
1021 , : ! . ; @ =
1022 -> += -= *= /= //= %=
1023 @= &= |= ^= >>= <<= **=
1024
1025 The period can also occur in floating-point and imaginary literals. A sequence
1026 of three periods has a special meaning as an ellipsis literal. The second half
1027 of the list, the augmented assignment operators, serve lexically as delimiters,
1028 but also perform an operation.
1029
1030 The following printing ASCII characters have special meaning as part of other
1031 tokens or are otherwise significant to the lexical analyzer:
1032
1033 .. code-block:: none
1034
1035 ' " # \
1036
1037 The following printing ASCII characters are not used in Python. Their
1038 occurrence outside string literals and comments is an unconditional error:
1039
1040 .. code-block:: none
1041
1042 $ ? `
1043
1044
1045 .. rubric:: Footnotes
1046
1047 .. [#] https://www.unicode.org/Public/15.1.0/ucd/NameAliases.txt