Debian Codegraph @

Doc/library/re.rst

raw
1 :mod:`!re` --- Regular expression operations
2 ============================================
3
4 .. module:: re
5 :synopsis: Regular expression operations.
6
7 .. moduleauthor:: Fredrik Lundh <[email protected]>
8 .. sectionauthor:: Andrew M. Kuchling <[email protected]>
9
10 **Source code:** :source:`Lib/re/`
11
12 --------------
13
14 This module provides regular expression matching operations similar to
15 those found in Perl.
16
17 Both patterns and strings to be searched can be Unicode strings (:class:`str`)
18 as well as 8-bit strings (:class:`bytes`).
19 However, Unicode strings and 8-bit strings cannot be mixed:
20 that is, you cannot match a Unicode string with a bytes pattern or
21 vice-versa; similarly, when asking for a substitution, the replacement
22 string must be of the same type as both the pattern and the search string.
23
24 Regular expressions use the backslash character (``'\'``) to indicate
25 special forms or to allow special characters to be used without invoking
26 their special meaning. This collides with Python's usage of the same
27 character for the same purpose in string literals; for example, to match
28 a literal backslash, one might have to write ``'\\\\'`` as the pattern
29 string, because the regular expression must be ``\\``, and each
30 backslash must be expressed as ``\\`` inside a regular Python string
31 literal. Also, please note that any invalid escape sequences in Python's
32 usage of the backslash in string literals now generate a :exc:`SyntaxWarning`
33 and in the future this will become a :exc:`SyntaxError`. This behaviour
34 will happen even if it is a valid escape sequence for a regular expression.
35
36 The solution is to use Python's raw string notation for regular expression
37 patterns; backslashes are not handled in any special way in a string literal
38 prefixed with ``'r'``. So ``r"\n"`` is a two-character string containing
39 ``'\'`` and ``'n'``, while ``"\n"`` is a one-character string containing a
40 newline. Usually patterns will be expressed in Python code using this raw
41 string notation.
42
43 It is important to note that most regular expression operations are available as
44 module-level functions and methods on
45 :ref:`compiled regular expressions <re-objects>`. The functions are shortcuts
46 that don't require you to compile a regex object first, but miss some
47 fine-tuning parameters.
48
49 .. seealso::
50
51 The third-party :pypi:`regex` module,
52 which has an API compatible with the standard library :mod:`re` module,
53 but offers additional functionality and a more thorough Unicode support.
54
55
56 .. _re-syntax:
57
58 Regular Expression Syntax
59 -------------------------
60
61 A regular expression (or RE) specifies a set of strings that matches it; the
62 functions in this module let you check if a particular string matches a given
63 regular expression (or if a given regular expression matches a particular
64 string, which comes down to the same thing).
65
66 Regular expressions can be concatenated to form new regular expressions; if *A*
67 and *B* are both regular expressions, then *AB* is also a regular expression.
68 In general, if a string *p* matches *A* and another string *q* matches *B*, the
69 string *pq* will match AB. This holds unless *A* or *B* contain low precedence
70 operations; boundary conditions between *A* and *B*; or have numbered group
71 references. Thus, complex expressions can easily be constructed from simpler
72 primitive expressions like the ones described here. For details of the theory
73 and implementation of regular expressions, consult the Friedl book [Frie09]_,
74 or almost any textbook about compiler construction.
75
76 A brief explanation of the format of regular expressions follows. For further
77 information and a gentler presentation, consult the :ref:`regex-howto`.
78
79 Regular expressions can contain both special and ordinary characters. Most
80 ordinary characters, like ``'A'``, ``'a'``, or ``'0'``, are the simplest regular
81 expressions; they simply match themselves. You can concatenate ordinary
82 characters, so ``last`` matches the string ``'last'``. (In the rest of this
83 section, we'll write RE's in ``this special style``, usually without quotes, and
84 strings to be matched ``'in single quotes'``.)
85
86 Some characters, like ``'|'`` or ``'('``, are special. Special
87 characters either stand for classes of ordinary characters, or affect
88 how the regular expressions around them are interpreted.
89
90 Repetition operators or quantifiers (``*``, ``+``, ``?``, ``{m,n}``, etc) cannot be
91 directly nested. This avoids ambiguity with the non-greedy modifier suffix
92 ``?``, and with other modifiers in other implementations. To apply a second
93 repetition to an inner repetition, parentheses may be used. For example,
94 the expression ``(?:a{6})*`` matches any multiple of six ``'a'`` characters.
95
96
97 The special characters are:
98
99 .. index:: single: . (dot); in regular expressions
100
101 ``.``
102 (Dot.) In the default mode, this matches any character except a newline. If
103 the :const:`DOTALL` flag has been specified, this matches any character
104 including a newline. ``(?s:.)`` matches any character regardless of flags.
105
106 .. index:: single: ^ (caret); in regular expressions
107
108 ``^``
109 (Caret.) Matches the start of the string, and in :const:`MULTILINE` mode also
110 matches immediately after each newline.
111
112 .. index:: single: $ (dollar); in regular expressions
113
114 ``$``
115 Matches the end of the string or just before the newline at the end of the
116 string, and in :const:`MULTILINE` mode also matches before a newline. ``foo``
117 matches both 'foo' and 'foobar', while the regular expression ``foo$`` matches
118 only 'foo'. More interestingly, searching for ``foo.$`` in ``'foo1\nfoo2\n'``
119 matches 'foo2' normally, but 'foo1' in :const:`MULTILINE` mode; searching for
120 a single ``$`` in ``'foo\n'`` will find two (empty) matches: one just before
121 the newline, and one at the end of the string.
122
123 .. index:: single: * (asterisk); in regular expressions
124
125 ``*``
126 Causes the resulting RE to match 0 or more repetitions of the preceding RE, as
127 many repetitions as are possible. ``ab*`` will match 'a', 'ab', or 'a' followed
128 by any number of 'b's.
129
130 .. index:: single: + (plus); in regular expressions
131
132 ``+``
133 Causes the resulting RE to match 1 or more repetitions of the preceding RE.
134 ``ab+`` will match 'a' followed by any non-zero number of 'b's; it will not
135 match just 'a'.
136
137 .. index:: single: ? (question mark); in regular expressions
138
139 ``?``
140 Causes the resulting RE to match 0 or 1 repetitions of the preceding RE.
141 ``ab?`` will match either 'a' or 'ab'.
142
143 .. index::
144 single: *?; in regular expressions
145 single: +?; in regular expressions
146 single: ??; in regular expressions
147
148 ``*?``, ``+?``, ``??``
149 The ``'*'``, ``'+'``, and ``'?'`` quantifiers are all :dfn:`greedy`; they match
150 as much text as possible. Sometimes this behaviour isn't desired; if the RE
151 ``<.*>`` is matched against ``'<a> b <c>'``, it will match the entire
152 string, and not just ``'<a>'``. Adding ``?`` after the quantifier makes it
153 perform the match in :dfn:`non-greedy` or :dfn:`minimal` fashion; as *few*
154 characters as possible will be matched. Using the RE ``<.*?>`` will match
155 only ``'<a>'``.
156
157 .. index::
158 single: *+; in regular expressions
159 single: ++; in regular expressions
160 single: ?+; in regular expressions
161
162 ``*+``, ``++``, ``?+``
163 Like the ``'*'``, ``'+'``, and ``'?'`` quantifiers, those where ``'+'`` is
164 appended also match as many times as possible.
165 However, unlike the true greedy quantifiers, these do not allow
166 back-tracking when the expression following it fails to match.
167 These are known as :dfn:`possessive` quantifiers.
168 For example, ``a*a`` will match ``'aaaa'`` because the ``a*`` will match
169 all 4 ``'a'``\ s, but, when the final ``'a'`` is encountered, the
170 expression is backtracked so that in the end the ``a*`` ends up matching
171 3 ``'a'``\ s total, and the fourth ``'a'`` is matched by the final ``'a'``.
172 However, when ``a*+a`` is used to match ``'aaaa'``, the ``a*+`` will
173 match all 4 ``'a'``, but when the final ``'a'`` fails to find any more
174 characters to match, the expression cannot be backtracked and will thus
175 fail to match.
176 ``x*+``, ``x++`` and ``x?+`` are equivalent to ``(?>x*)``, ``(?>x+)``
177 and ``(?>x?)`` correspondingly.
178
179 .. versionadded:: 3.11
180
181 .. index::
182 single: {} (curly brackets); in regular expressions
183
184 ``{m}``
185 Specifies that exactly *m* copies of the previous RE should be matched; fewer
186 matches cause the entire RE not to match. For example, ``a{6}`` will match
187 exactly six ``'a'`` characters, but not five.
188
189 ``{m,n}``
190 Causes the resulting RE to match from *m* to *n* repetitions of the preceding
191 RE, attempting to match as many repetitions as possible. For example,
192 ``a{3,5}`` will match from 3 to 5 ``'a'`` characters. Omitting *m* specifies a
193 lower bound of zero, and omitting *n* specifies an infinite upper bound. As an
194 example, ``a{4,}b`` will match ``'aaaab'`` or a thousand ``'a'`` characters
195 followed by a ``'b'``, but not ``'aaab'``. The comma may not be omitted or the
196 modifier would be confused with the previously described form.
197
198 ``{m,n}?``
199 Causes the resulting RE to match from *m* to *n* repetitions of the preceding
200 RE, attempting to match as *few* repetitions as possible. This is the
201 non-greedy version of the previous quantifier. For example, on the
202 6-character string ``'aaaaaa'``, ``a{3,5}`` will match 5 ``'a'`` characters,
203 while ``a{3,5}?`` will only match 3 characters.
204
205 ``{m,n}+``
206 Causes the resulting RE to match from *m* to *n* repetitions of the
207 preceding RE, attempting to match as many repetitions as possible
208 *without* establishing any backtracking points.
209 This is the possessive version of the quantifier above.
210 For example, on the 6-character string ``'aaaaaa'``, ``a{3,5}+aa``
211 attempt to match 5 ``'a'`` characters, then, requiring 2 more ``'a'``\ s,
212 will need more characters than available and thus fail, while
213 ``a{3,5}aa`` will match with ``a{3,5}`` capturing 5, then 4 ``'a'``\ s
214 by backtracking and then the final 2 ``'a'``\ s are matched by the final
215 ``aa`` in the pattern.
216 ``x{m,n}+`` is equivalent to ``(?>x{m,n})``.
217
218 .. versionadded:: 3.11
219
220 .. index:: single: \ (backslash); in regular expressions
221
222 ``\``
223 Either escapes special characters (permitting you to match characters like
224 ``'*'``, ``'?'``, and so forth), or signals a special sequence; special
225 sequences are discussed below.
226
227 If you're not using a raw string to express the pattern, remember that Python
228 also uses the backslash as an escape sequence in string literals; if the escape
229 sequence isn't recognized by Python's parser, the backslash and subsequent
230 character are included in the resulting string. However, if Python would
231 recognize the resulting sequence, the backslash should be repeated twice. This
232 is complicated and hard to understand, so it's highly recommended that you use
233 raw strings for all but the simplest expressions.
234
235 .. index::
236 single: [] (square brackets); in regular expressions
237
238 ``[]``
239 Used to indicate a set of characters. In a set:
240
241 * Characters can be listed individually, e.g. ``[amk]`` will match ``'a'``,
242 ``'m'``, or ``'k'``.
243
244 .. index:: single: - (minus); in regular expressions
245
246 * Ranges of characters can be indicated by giving two characters and separating
247 them by a ``'-'``, for example ``[a-z]`` will match any lowercase ASCII letter,
248 ``[0-5][0-9]`` will match all the two-digits numbers from ``00`` to ``59``, and
249 ``[0-9A-Fa-f]`` will match any hexadecimal digit. If ``-`` is escaped (e.g.
250 ``[a\-z]``) or if it's placed as the first or last character
251 (e.g. ``[-a]`` or ``[a-]``), it will match a literal ``'-'``.
252
253 * Special characters except backslash lose their special meaning inside sets.
254 For example,
255 ``[(+*)]`` will match any of the literal characters ``'('``, ``'+'``,
256 ``'*'``, or ``')'``.
257
258 .. index:: single: \ (backslash); in regular expressions
259
260 * Backslash either escapes characters which have special meaning in a set
261 such as ``'-'``, ``']'``, ``'^'`` and ``'\\'`` itself or signals
262 a special sequence which represents a single character such as
263 ``\xa0`` or ``\n`` or a character class such as ``\w`` or ``\S``
264 (defined below).
265 Note that ``\b`` represents a single "backspace" character,
266 not a word boundary as outside a set, and numeric escapes
267 such as ``\1`` are always octal escapes, not group references.
268 Special sequences which do not match a single character such as ``\A``
269 and ``\Z`` are not allowed.
270
271 .. index:: single: ^ (caret); in regular expressions
272
273 * Characters that are not within a range can be matched by :dfn:`complementing`
274 the set. If the first character of the set is ``'^'``, all the characters
275 that are *not* in the set will be matched. For example, ``[^5]`` will match
276 any character except ``'5'``, and ``[^^]`` will match any character except
277 ``'^'``. ``^`` has no special meaning if it's not the first character in
278 the set.
279
280 * To match a literal ``']'`` inside a set, precede it with a backslash, or
281 place it at the beginning of the set. For example, both ``[()[\]{}]`` and
282 ``[]()[{}]`` will match a right bracket, as well as left bracket, braces,
283 and parentheses.
284
285 .. .. index:: single: --; in regular expressions
286 .. .. index:: single: &&; in regular expressions
287 .. .. index:: single: ~~; in regular expressions
288 .. .. index:: single: ||; in regular expressions
289
290 * Support of nested sets and set operations as in `Unicode Technical
291 Standard #18`_ might be added in the future. This would change the
292 syntax, so to facilitate this change a :exc:`FutureWarning` will be raised
293 in ambiguous cases for the time being.
294 That includes sets starting with a literal ``'['`` or containing literal
295 character sequences ``'--'``, ``'&&'``, ``'~~'``, and ``'||'``. To
296 avoid a warning escape them with a backslash.
297
298 .. _Unicode Technical Standard #18: https://unicode.org/reports/tr18/
299
300 .. versionchanged:: 3.7
301 :exc:`FutureWarning` is raised if a character set contains constructs
302 that will change semantically in the future.
303
304 .. index:: single: | (vertical bar); in regular expressions
305
306 ``|``
307 ``A|B``, where *A* and *B* can be arbitrary REs, creates a regular expression that
308 will match either *A* or *B*. An arbitrary number of REs can be separated by the
309 ``'|'`` in this way. This can be used inside groups (see below) as well. As
310 the target string is scanned, REs separated by ``'|'`` are tried from left to
311 right. When one pattern completely matches, that branch is accepted. This means
312 that once *A* matches, *B* will not be tested further, even if it would
313 produce a longer overall match. In other words, the ``'|'`` operator is never
314 greedy. To match a literal ``'|'``, use ``\|``, or enclose it inside a
315 character class, as in ``[|]``.
316
317 .. index::
318 single: () (parentheses); in regular expressions
319
320 ``(...)``
321 Matches whatever regular expression is inside the parentheses, and indicates the
322 start and end of a group; the contents of a group can be retrieved after a match
323 has been performed, and can be matched later in the string with the ``\number``
324 special sequence, described below. To match the literals ``'('`` or ``')'``,
325 use ``\(`` or ``\)``, or enclose them inside a character class: ``[(]``, ``[)]``.
326
327 .. index:: single: (?; in regular expressions
328
329 ``(?...)``
330 This is an extension notation (a ``'?'`` following a ``'('`` is not meaningful
331 otherwise). The first character after the ``'?'`` determines what the meaning
332 and further syntax of the construct is. Extensions usually do not create a new
333 group; ``(?P<name>...)`` is the only exception to this rule. Following are the
334 currently supported extensions.
335
336 ``(?aiLmsux)``
337 (One or more letters from the set
338 ``'a'``, ``'i'``, ``'L'``, ``'m'``, ``'s'``, ``'u'``, ``'x'``.)
339 The group matches the empty string;
340 the letters set the corresponding flags for the entire regular expression:
341
342 * :const:`re.A` (ASCII-only matching)
343 * :const:`re.I` (ignore case)
344 * :const:`re.L` (locale dependent)
345 * :const:`re.M` (multi-line)
346 * :const:`re.S` (dot matches all)
347 * :const:`re.U` (Unicode matching)
348 * :const:`re.X` (verbose)
349
350 (The flags are described in :ref:`contents-of-module-re`.)
351 This is useful if you wish to include the flags as part of the
352 regular expression, instead of passing a *flag* argument to the
353 :func:`re.compile` function.
354 Flags should be used first in the expression string.
355
356 .. versionchanged:: 3.11
357 This construction can only be used at the start of the expression.
358
359 .. index:: single: (?:; in regular expressions
360
361 ``(?:...)``
362 A non-capturing version of regular parentheses. Matches whatever regular
363 expression is inside the parentheses, but the substring matched by the group
364 *cannot* be retrieved after performing a match or referenced later in the
365 pattern.
366
367 ``(?aiLmsux-imsx:...)``
368 (Zero or more letters from the set
369 ``'a'``, ``'i'``, ``'L'``, ``'m'``, ``'s'``, ``'u'``, ``'x'``,
370 optionally followed by ``'-'`` followed by
371 one or more letters from the ``'i'``, ``'m'``, ``'s'``, ``'x'``.)
372 The letters set or remove the corresponding flags for the part of the expression:
373
374 * :const:`re.A` (ASCII-only matching)
375 * :const:`re.I` (ignore case)
376 * :const:`re.L` (locale dependent)
377 * :const:`re.M` (multi-line)
378 * :const:`re.S` (dot matches all)
379 * :const:`re.U` (Unicode matching)
380 * :const:`re.X` (verbose)
381
382 (The flags are described in :ref:`contents-of-module-re`.)
383
384 The letters ``'a'``, ``'L'`` and ``'u'`` are mutually exclusive when used
385 as inline flags, so they can't be combined or follow ``'-'``. Instead,
386 when one of them appears in an inline group, it overrides the matching mode
387 in the enclosing group. In Unicode patterns ``(?a:...)`` switches to
388 ASCII-only matching, and ``(?u:...)`` switches to Unicode matching
389 (default). In bytes patterns ``(?L:...)`` switches to locale dependent
390 matching, and ``(?a:...)`` switches to ASCII-only matching (default).
391 This override is only in effect for the narrow inline group, and the
392 original matching mode is restored outside of the group.
393
394 .. versionadded:: 3.6
395
396 .. versionchanged:: 3.7
397 The letters ``'a'``, ``'L'`` and ``'u'`` also can be used in a group.
398
399 ``(?>...)``
400 Attempts to match ``...`` as if it was a separate regular expression, and
401 if successful, continues to match the rest of the pattern following it.
402 If the subsequent pattern fails to match, the stack can only be unwound
403 to a point *before* the ``(?>...)`` because once exited, the expression,
404 known as an :dfn:`atomic group`, has thrown away all stack points within
405 itself.
406 Thus, ``(?>.*).`` would never match anything because first the ``.*``
407 would match all characters possible, then, having nothing left to match,
408 the final ``.`` would fail to match.
409 Since there are no stack points saved in the Atomic Group, and there is
410 no stack point before it, the entire expression would thus fail to match.
411
412 .. versionadded:: 3.11
413
414 .. index:: single: (?P<; in regular expressions
415
416 ``(?P<name>...)``
417 Similar to regular parentheses, but the substring matched by the group is
418 accessible via the symbolic group name *name*. Group names must be valid
419 Python identifiers, and in :class:`bytes` patterns they can only contain
420 bytes in the ASCII range. Each group name must be defined only once within
421 a regular expression. A symbolic group is also a numbered group, just as if
422 the group were not named.
423
424 Named groups can be referenced in three contexts. If the pattern is
425 ``(?P<quote>['"]).*?(?P=quote)`` (i.e. matching a string quoted with either
426 single or double quotes):
427
428 +---------------------------------------+----------------------------------+
429 | Context of reference to group "quote" | Ways to reference it |
430 +=======================================+==================================+
431 | in the same pattern itself | * ``(?P=quote)`` (as shown) |
432 | | * ``\1`` |
433 +---------------------------------------+----------------------------------+
434 | when processing match object *m* | * ``m.group('quote')`` |
435 | | * ``m.end('quote')`` (etc.) |
436 +---------------------------------------+----------------------------------+
437 | in a string passed to the *repl* | * ``\g<quote>`` |
438 | argument of ``re.sub()`` | * ``\g<1>`` |
439 | | * ``\1`` |
440 +---------------------------------------+----------------------------------+
441
442 .. versionchanged:: 3.12
443 In :class:`bytes` patterns, group *name* can only contain bytes
444 in the ASCII range (``b'\x00'``-``b'\x7f'``).
445
446 .. index:: single: (?P=; in regular expressions
447
448 ``(?P=name)``
449 A backreference to a named group; it matches whatever text was matched by the
450 earlier group named *name*.
451
452 .. index:: single: (?#; in regular expressions
453
454 ``(?#...)``
455 A comment; the contents of the parentheses are simply ignored.
456
457 .. index:: single: (?=; in regular expressions
458
459 ``(?=...)``
460 Matches if ``...`` matches next, but doesn't consume any of the string. This is
461 called a :dfn:`lookahead assertion`. For example, ``Isaac (?=Asimov)`` will match
462 ``'Isaac '`` only if it's followed by ``'Asimov'``.
463
464 .. index:: single: (?!; in regular expressions
465
466 ``(?!...)``
467 Matches if ``...`` doesn't match next. This is a :dfn:`negative lookahead assertion`.
468 For example, ``Isaac (?!Asimov)`` will match ``'Isaac '`` only if it's *not*
469 followed by ``'Asimov'``.
470
471 .. index:: single: (?<=; in regular expressions
472
473 ``(?<=...)``
474 Matches if the current position in the string is preceded by a match for ``...``
475 that ends at the current position. This is called a :dfn:`positive lookbehind
476 assertion`. ``(?<=abc)def`` will find a match in ``'abcdef'``, since the
477 lookbehind will back up 3 characters and check if the contained pattern matches.
478 The contained pattern must only match strings of some fixed length, meaning that
479 ``abc`` or ``a|b`` are allowed, but ``a*`` and ``a{3,4}`` are not. Note that
480 patterns which start with positive lookbehind assertions will not match at the
481 beginning of the string being searched; you will most likely want to use the
482 :func:`search` function rather than the :func:`match` function:
483
484 >>> import re
485 >>> m = re.search('(?<=abc)def', 'abcdef')
486 >>> m.group(0)
487 'def'
488
489 This example looks for a word following a hyphen:
490
491 >>> m = re.search(r'(?<=-)\w+', 'spam-egg')
492 >>> m.group(0)
493 'egg'
494
495 .. versionchanged:: 3.5
496 Added support for group references of fixed length.
497
498 .. index:: single: (?<!; in regular expressions
499
500 ``(?<!...)``
501 Matches if the current position in the string is not preceded by a match for
502 ``...``. This is called a :dfn:`negative lookbehind assertion`. Similar to
503 positive lookbehind assertions, the contained pattern must only match strings of
504 some fixed length. Patterns which start with negative lookbehind assertions may
505 match at the beginning of the string being searched.
506
507 .. _re-conditional-expression:
508 .. index:: single: (?(; in regular expressions
509
510 ``(?(id/name)yes-pattern|no-pattern)``
511 Will try to match with ``yes-pattern`` if the group with given *id* or
512 *name* exists, and with ``no-pattern`` if it doesn't. ``no-pattern`` is
513 optional and can be omitted. For example,
514 ``(<)?(\w+@\w+(?:\.\w+)+)(?(1)>|$)`` is a poor email matching pattern, which
515 matches ``'<[email protected]>'`` as well as ``'[email protected]'``, but does not
516 match ``'<[email protected]'`` nor ``'[email protected]>'`` in their entirety
517 (:func:`re.search` finds only ``'[email protected]'`` in the former).
518
519 .. versionchanged:: 3.12
520 Group *id* can only contain ASCII digits.
521 In :class:`bytes` patterns, group *name* can only contain bytes
522 in the ASCII range (``b'\x00'``-``b'\x7f'``).
523
524
525 .. _re-special-sequences:
526
527 The special sequences consist of ``'\'`` and a character from the list below.
528 If the ordinary character is not an ASCII digit or an ASCII letter, then the
529 resulting RE will match the second character. For example, ``\$`` matches the
530 character ``'$'``.
531
532 .. index:: single: \ (backslash); in regular expressions
533
534 ``\number``
535 Matches the contents of the group of the same number. Groups are numbered
536 starting from 1. For example, ``(.+) \1`` matches ``'the the'`` or ``'55 55'``,
537 but not ``'thethe'`` (note the space after the group). This special sequence
538 can only be used to match one of the first 99 groups. If the first digit of
539 *number* is 0, or *number* is 3 octal digits long, it will not be interpreted as
540 a group match, but as the character with octal value *number*. Inside the
541 ``'['`` and ``']'`` of a character class, all numeric escapes are treated as
542 characters.
543
544 .. index:: single: \A; in regular expressions
545
546 ``\A``
547 Matches only at the start of the string.
548
549 .. index:: single: \b; in regular expressions
550
551 ``\b``
552 Matches the empty string, but only at the beginning or end of a word.
553 A word is defined as a sequence of word characters.
554 Note that formally, ``\b`` is defined as the boundary
555 between a ``\w`` and a ``\W`` character (or vice versa),
556 or between ``\w`` and the beginning or end of the string.
557 This means that ``r'\bat\b'`` matches ``'at'``, ``'at.'``, ``'(at)'``,
558 and ``'as at ay'`` but not ``'attempt'`` or ``'atlas'``.
559
560 The default word characters in Unicode (str) patterns
561 are Unicode alphanumerics and the underscore,
562 but this can be changed by using the :py:const:`~re.ASCII` flag.
563 Word boundaries are determined by the current locale
564 if the :py:const:`~re.LOCALE` flag is used.
565
566 .. note::
567
568 Inside a character range, ``\b`` represents the backspace character,
569 for compatibility with Python's string literals.
570
571 .. index:: single: \B; in regular expressions
572
573 ``\B``
574 Matches the empty string,
575 but only when it is *not* at the beginning or end of a word.
576 This means that ``r'at\B'`` matches ``'athens'``, ``'atom'``,
577 ``'attorney'``, but not ``'at'``, ``'at.'``, or ``'at!'``.
578 ``\B`` is the opposite of ``\b``,
579 so word characters in Unicode (str) patterns
580 are Unicode alphanumerics or the underscore,
581 although this can be changed by using the :py:const:`~re.ASCII` flag.
582 Word boundaries are determined by the current locale
583 if the :py:const:`~re.LOCALE` flag is used.
584
585 .. note::
586
587 Note that ``\B`` does not match an empty string, which differs from
588 RE implementations in other programming languages such as Perl.
589 This behavior is kept for compatibility reasons.
590
591 .. index:: single: \d; in regular expressions
592
593 ``\d``
594 For Unicode (str) patterns:
595 Matches any Unicode decimal digit
596 (that is, any character in Unicode character category `[Nd]`__).
597 This includes ``[0-9]``, and also many other digit characters.
598
599 Matches ``[0-9]`` if the :py:const:`~re.ASCII` flag is used.
600
601 __ https://www.unicode.org/versions/Unicode15.0.0/ch04.pdf#G134153
602
603 For 8-bit (bytes) patterns:
604 Matches any decimal digit in the ASCII character set;
605 this is equivalent to ``[0-9]``.
606
607 .. index:: single: \D; in regular expressions
608
609 ``\D``
610 Matches any character which is not a decimal digit.
611 This is the opposite of ``\d``.
612
613 Matches ``[^0-9]`` if the :py:const:`~re.ASCII` flag is used.
614
615 .. index:: single: \s; in regular expressions
616
617 ``\s``
618 For Unicode (str) patterns:
619 Matches Unicode whitespace characters (as defined by :py:meth:`str.isspace`).
620 This includes ``[ \t\n\r\f\v]``, and also many other characters, for example the
621 non-breaking spaces mandated by typography rules in many languages.
622
623 Matches ``[ \t\n\r\f\v]`` if the :py:const:`~re.ASCII` flag is used.
624
625 For 8-bit (bytes) patterns:
626 Matches characters considered whitespace in the ASCII character set;
627 this is equivalent to ``[ \t\n\r\f\v]``.
628
629 .. index:: single: \S; in regular expressions
630
631 ``\S``
632 Matches any character which is not a whitespace character. This is
633 the opposite of ``\s``.
634
635 Matches ``[^ \t\n\r\f\v]`` if the :py:const:`~re.ASCII` flag is used.
636
637 .. index:: single: \w; in regular expressions
638
639 ``\w``
640 For Unicode (str) patterns:
641 Matches Unicode word characters;
642 this includes all Unicode alphanumeric characters
643 (as defined by :py:meth:`str.isalnum`),
644 as well as the underscore (``_``).
645
646 Matches ``[a-zA-Z0-9_]`` if the :py:const:`~re.ASCII` flag is used.
647
648 For 8-bit (bytes) patterns:
649 Matches characters considered alphanumeric in the ASCII character set;
650 this is equivalent to ``[a-zA-Z0-9_]``.
651 If the :py:const:`~re.LOCALE` flag is used,
652 matches characters considered alphanumeric in the current locale and the underscore.
653
654 .. index:: single: \W; in regular expressions
655
656 ``\W``
657 Matches any character which is not a word character.
658 This is the opposite of ``\w``.
659 By default, matches non-underscore (``_``) characters
660 for which :py:meth:`str.isalnum` returns ``False``.
661
662 Matches ``[^a-zA-Z0-9_]`` if the :py:const:`~re.ASCII` flag is used.
663
664 If the :py:const:`~re.LOCALE` flag is used,
665 matches characters which are neither alphanumeric in the current locale
666 nor the underscore.
667
668 .. index:: single: \Z; in regular expressions
669
670 ``\Z``
671 Matches only at the end of the string.
672
673 .. index::
674 single: \a; in regular expressions
675 single: \b; in regular expressions
676 single: \f; in regular expressions
677 single: \n; in regular expressions
678 single: \N; in regular expressions
679 single: \r; in regular expressions
680 single: \t; in regular expressions
681 single: \u; in regular expressions
682 single: \U; in regular expressions
683 single: \v; in regular expressions
684 single: \x; in regular expressions
685 single: \\; in regular expressions
686
687 Most of the :ref:`escape sequences <escape-sequences>` supported by Python
688 string literals are also accepted by the regular expression parser::
689
690 \a \b \f \n
691 \N \r \t \u
692 \U \v \x \\
693
694 (Note that ``\b`` is used to represent word boundaries, and means "backspace"
695 only inside character classes.)
696
697 ``'\u'``, ``'\U'``, and ``'\N'`` escape sequences are
698 only recognized in Unicode (str) patterns.
699 In bytes patterns they are errors.
700 Unknown escapes of ASCII letters are reserved
701 for future use and treated as errors.
702
703 Octal escapes are included in a limited form. If the first digit is a 0, or if
704 there are three octal digits, it is considered an octal escape. Otherwise, it is
705 a group reference. As for string literals, octal escapes are always at most
706 three digits in length.
707
708 .. versionchanged:: 3.3
709 The ``'\u'`` and ``'\U'`` escape sequences have been added.
710
711 .. versionchanged:: 3.6
712 Unknown escapes consisting of ``'\'`` and an ASCII letter now are errors.
713
714 .. versionchanged:: 3.8
715 The :samp:`'\\N\\{{name}\\}'` escape sequence has been added. As in string literals,
716 it expands to the named Unicode character (e.g. ``'\N{EM DASH}'``).
717
718
719 .. _contents-of-module-re:
720
721 Module Contents
722 ---------------
723
724 The module defines several functions, constants, and an exception. Some of the
725 functions are simplified versions of the full featured methods for compiled
726 regular expressions. Most non-trivial applications always use the compiled
727 form.
728
729
730 Flags
731 ^^^^^
732
733 .. versionchanged:: 3.6
734 Flag constants are now instances of :class:`RegexFlag`, which is a subclass of
735 :class:`enum.IntFlag`.
736
737
738 .. class:: RegexFlag
739
740 An :class:`enum.IntFlag` class containing the regex options listed below.
741
742 .. versionadded:: 3.11 - added to ``__all__``
743
744 .. data:: A
745 ASCII
746
747 Make ``\w``, ``\W``, ``\b``, ``\B``, ``\d``, ``\D``, ``\s`` and ``\S``
748 perform ASCII-only matching instead of full Unicode matching. This is only
749 meaningful for Unicode (str) patterns, and is ignored for bytes patterns.
750
751 Corresponds to the inline flag ``(?a)``.
752
753 .. note::
754
755 The :py:const:`~re.U` flag still exists for backward compatibility,
756 but is redundant in Python 3 since
757 matches are Unicode by default for ``str`` patterns,
758 and Unicode matching isn't allowed for bytes patterns.
759 :py:const:`~re.UNICODE` and the inline flag ``(?u)`` are similarly redundant.
760
761
762 .. data:: DEBUG
763
764 Display debug information about compiled expression.
765
766 No corresponding inline flag.
767
768
769 .. data:: I
770 IGNORECASE
771
772 Perform case-insensitive matching;
773 expressions like ``[A-Z]`` will also match lowercase letters.
774 Full Unicode matching (such as ``Ü`` matching ``ü``)
775 also works unless the :py:const:`~re.ASCII` flag
776 is used to disable non-ASCII matches.
777 The current locale does not change the effect of this flag
778 unless the :py:const:`~re.LOCALE` flag is also used.
779
780 Corresponds to the inline flag ``(?i)``.
781
782 Note that when the Unicode patterns ``[a-z]`` or ``[A-Z]`` are used in
783 combination with the :const:`IGNORECASE` flag, they will match the 52 ASCII
784 letters and 4 additional non-ASCII letters: 'İ' (U+0130, Latin capital
785 letter I with dot above), 'ı' (U+0131, Latin small letter dotless i),
786 'ſ' (U+017F, Latin small letter long s) and 'K' (U+212A, Kelvin sign).
787 If the :py:const:`~re.ASCII` flag is used, only letters 'a' to 'z'
788 and 'A' to 'Z' are matched.
789
790 .. data:: L
791 LOCALE
792
793 Make ``\w``, ``\W``, ``\b``, ``\B`` and case-insensitive matching
794 dependent on the current locale.
795 This flag can be used only with bytes patterns.
796
797 Corresponds to the inline flag ``(?L)``.
798
799 .. warning::
800
801 This flag is discouraged; consider Unicode matching instead.
802 The locale mechanism is very unreliable
803 as it only handles one "culture" at a time
804 and only works with 8-bit locales.
805 Unicode matching is enabled by default for Unicode (str) patterns
806 and it is able to handle different locales and languages.
807
808 .. versionchanged:: 3.6
809 :py:const:`~re.LOCALE` can be used only with bytes patterns
810 and is not compatible with :py:const:`~re.ASCII`.
811
812 .. versionchanged:: 3.7
813 Compiled regular expression objects with the :py:const:`~re.LOCALE` flag
814 no longer depend on the locale at compile time.
815 Only the locale at matching time affects the result of matching.
816
817
818 .. data:: M
819 MULTILINE
820
821 When specified, the pattern character ``'^'`` matches at the beginning of the
822 string and at the beginning of each line (immediately following each newline);
823 and the pattern character ``'$'`` matches at the end of the string and at the
824 end of each line (immediately preceding each newline). By default, ``'^'``
825 matches only at the beginning of the string, and ``'$'`` only at the end of the
826 string and immediately before the newline (if any) at the end of the string.
827
828 Corresponds to the inline flag ``(?m)``.
829
830 .. data:: NOFLAG
831
832 Indicates no flag being applied, the value is ``0``. This flag may be used
833 as a default value for a function keyword argument or as a base value that
834 will be conditionally ORed with other flags. Example of use as a default
835 value::
836
837 def myfunc(text, flag=re.NOFLAG):
838 return re.match(text, flag)
839
840 .. versionadded:: 3.11
841
842 .. data:: S
843 DOTALL
844
845 Make the ``'.'`` special character match any character at all, including a
846 newline; without this flag, ``'.'`` will match anything *except* a newline.
847
848 Corresponds to the inline flag ``(?s)``.
849
850
851 .. data:: U
852 UNICODE
853
854 In Python 3, Unicode characters are matched by default
855 for ``str`` patterns.
856 This flag is therefore redundant with **no effect**
857 and is only kept for backward compatibility.
858
859 See :py:const:`~re.ASCII` to restrict matching to ASCII characters instead.
860
861 .. data:: X
862 VERBOSE
863
864 .. index:: single: # (hash); in regular expressions
865
866 This flag allows you to write regular expressions that look nicer and are
867 more readable by allowing you to visually separate logical sections of the
868 pattern and add comments. Whitespace within the pattern is ignored, except
869 when in a character class, or when preceded by an unescaped backslash,
870 or within tokens like ``*?``, ``(?:`` or ``(?P<...>``. For example, ``(? :``
871 and ``* ?`` are not allowed.
872 When a line contains a ``#`` that is not in a character class and is not
873 preceded by an unescaped backslash, all characters from the leftmost such
874 ``#`` through the end of the line are ignored.
875
876 This means that the two following regular expression objects that match a
877 decimal number are functionally equal::
878
879 a = re.compile(r"""\d + # the integral part
880 \. # the decimal point
881 \d * # some fractional digits""", re.X)
882 b = re.compile(r"\d+\.\d*")
883
884 Corresponds to the inline flag ``(?x)``.
885
886
887 Functions
888 ^^^^^^^^^
889
890 .. function:: compile(pattern, flags=0)
891
892 Compile a regular expression pattern into a :ref:`regular expression object
893 <re-objects>`, which can be used for matching using its
894 :func:`~Pattern.match`, :func:`~Pattern.search` and other methods, described
895 below.
896
897 The expression's behaviour can be modified by specifying a *flags* value.
898 Values can be any of the `flags`_ variables, combined using bitwise OR
899 (the ``|`` operator).
900
901 The sequence ::
902
903 prog = re.compile(pattern)
904 result = prog.match(string)
905
906 is equivalent to ::
907
908 result = re.match(pattern, string)
909
910 but using :func:`re.compile` and saving the resulting regular expression
911 object for reuse is more efficient when the expression will be used several
912 times in a single program.
913
914 .. note::
915
916 The compiled versions of the most recent patterns passed to
917 :func:`re.compile` and the module-level matching functions are cached, so
918 programs that use only a few regular expressions at a time needn't worry
919 about compiling regular expressions.
920
921
922 .. function:: search(pattern, string, flags=0)
923
924 Scan through *string* looking for the first location where the regular expression
925 *pattern* produces a match, and return a corresponding :class:`~re.Match`. Return
926 ``None`` if no position in the string matches the pattern; note that this is
927 different from finding a zero-length match at some point in the string.
928
929 The expression's behaviour can be modified by specifying a *flags* value.
930 Values can be any of the `flags`_ variables, combined using bitwise OR
931 (the ``|`` operator).
932
933
934 .. function:: match(pattern, string, flags=0)
935
936 If zero or more characters at the beginning of *string* match the regular
937 expression *pattern*, return a corresponding :class:`~re.Match`. Return
938 ``None`` if the string does not match the pattern; note that this is
939 different from a zero-length match.
940
941 Note that even in :const:`MULTILINE` mode, :func:`re.match` will only match
942 at the beginning of the string and not at the beginning of each line.
943
944 If you want to locate a match anywhere in *string*, use :func:`search`
945 instead (see also :ref:`search-vs-match`).
946
947 The expression's behaviour can be modified by specifying a *flags* value.
948 Values can be any of the `flags`_ variables, combined using bitwise OR
949 (the ``|`` operator).
950
951
952 .. function:: fullmatch(pattern, string, flags=0)
953
954 If the whole *string* matches the regular expression *pattern*, return a
955 corresponding :class:`~re.Match`. Return ``None`` if the string does not match
956 the pattern; note that this is different from a zero-length match.
957
958 The expression's behaviour can be modified by specifying a *flags* value.
959 Values can be any of the `flags`_ variables, combined using bitwise OR
960 (the ``|`` operator).
961
962 .. versionadded:: 3.4
963
964
965 .. function:: split(pattern, string, maxsplit=0, flags=0)
966
967 Split *string* by the occurrences of *pattern*. If capturing parentheses are
968 used in *pattern*, then the text of all groups in the pattern are also returned
969 as part of the resulting list. If *maxsplit* is nonzero, at most *maxsplit*
970 splits occur, and the remainder of the string is returned as the final element
971 of the list. ::
972
973 >>> re.split(r'\W+', 'Words, words, words.')
974 ['Words', 'words', 'words', '']
975 >>> re.split(r'(\W+)', 'Words, words, words.')
976 ['Words', ', ', 'words', ', ', 'words', '.', '']
977 >>> re.split(r'\W+', 'Words, words, words.', maxsplit=1)
978 ['Words', 'words, words.']
979 >>> re.split('[a-f]+', '0a3B9', flags=re.IGNORECASE)
980 ['0', '3', '9']
981
982 If there are capturing groups in the separator and it matches at the start of
983 the string, the result will start with an empty string. The same holds for
984 the end of the string::
985
986 >>> re.split(r'(\W+)', '...words, words...')
987 ['', '...', 'words', ', ', 'words', '...', '']
988
989 That way, separator components are always found at the same relative
990 indices within the result list.
991
992 Adjacent empty matches are not possible, but an empty match can occur
993 immediately after a non-empty match.
994
995 .. code:: pycon
996
997 >>> re.split(r'\b', 'Words, words, words.')
998 ['', 'Words', ', ', 'words', ', ', 'words', '.']
999 >>> re.split(r'\W*', '...words...')
1000 ['', '', 'w', 'o', 'r', 'd', 's', '', '']
1001 >>> re.split(r'(\W*)', '...words...')
1002 ['', '...', '', '', 'w', '', 'o', '', 'r', '', 'd', '', 's', '...', '', '', '']
1003
1004 The expression's behaviour can be modified by specifying a *flags* value.
1005 Values can be any of the `flags`_ variables, combined using bitwise OR
1006 (the ``|`` operator).
1007
1008 .. versionchanged:: 3.1
1009 Added the optional flags argument.
1010
1011 .. versionchanged:: 3.7
1012 Added support of splitting on a pattern that could match an empty string.
1013
1014 .. deprecated:: 3.13
1015 Passing *maxsplit* and *flags* as positional arguments is deprecated.
1016 In future Python versions they will be
1017 :ref:`keyword-only parameters <keyword-only_parameter>`.
1018
1019
1020 .. function:: findall(pattern, string, flags=0)
1021
1022 Return all non-overlapping matches of *pattern* in *string*, as a list of
1023 strings or tuples. The *string* is scanned left-to-right, and matches
1024 are returned in the order found. Empty matches are included in the result.
1025
1026 The result depends on the number of capturing groups in the pattern.
1027 If there are no groups, return a list of strings matching the whole
1028 pattern. If there is exactly one group, return a list of strings
1029 matching that group. If multiple groups are present, return a list
1030 of tuples of strings matching the groups. Non-capturing groups do not
1031 affect the form of the result.
1032
1033 >>> re.findall(r'\bf[a-z]*', 'which foot or hand fell fastest')
1034 ['foot', 'fell', 'fastest']
1035 >>> re.findall(r'(\w+)=(\d+)', 'set width=20 and height=10')
1036 [('width', '20'), ('height', '10')]
1037
1038 The expression's behaviour can be modified by specifying a *flags* value.
1039 Values can be any of the `flags`_ variables, combined using bitwise OR
1040 (the ``|`` operator).
1041
1042 .. versionchanged:: 3.7
1043 Non-empty matches can now start just after a previous empty match.
1044
1045
1046 .. function:: finditer(pattern, string, flags=0)
1047
1048 Return an :term:`iterator` yielding :class:`~re.Match` objects over
1049 all non-overlapping matches for the RE *pattern* in *string*. The *string*
1050 is scanned left-to-right, and matches are returned in the order found. Empty
1051 matches are included in the result.
1052
1053 The expression's behaviour can be modified by specifying a *flags* value.
1054 Values can be any of the `flags`_ variables, combined using bitwise OR
1055 (the ``|`` operator).
1056
1057 .. versionchanged:: 3.7
1058 Non-empty matches can now start just after a previous empty match.
1059
1060
1061 .. function:: sub(pattern, repl, string, count=0, flags=0)
1062
1063 Return the string obtained by replacing the leftmost non-overlapping occurrences
1064 of *pattern* in *string* by the replacement *repl*. If the pattern isn't found,
1065 *string* is returned unchanged. *repl* can be a string or a function; if it is
1066 a string, any backslash escapes in it are processed. That is, ``\n`` is
1067 converted to a single newline character, ``\r`` is converted to a carriage return, and
1068 so forth. Unknown escapes of ASCII letters are reserved for future use and
1069 treated as errors. Other unknown escapes such as ``\&`` are left alone.
1070 Backreferences, such
1071 as ``\6``, are replaced with the substring matched by group 6 in the pattern.
1072 For example::
1073
1074 >>> re.sub(r'def\s+([a-zA-Z_][a-zA-Z_0-9]*)\s*\(\s*\):',
1075 ... r'static PyObject*\npy_\1(void)\n{',
1076 ... 'def myfunc():')
1077 'static PyObject*\npy_myfunc(void)\n{'
1078
1079 If *repl* is a function, it is called for every non-overlapping occurrence of
1080 *pattern*. The function takes a single :class:`~re.Match` argument, and returns
1081 the replacement string. For example::
1082
1083 >>> def dashrepl(matchobj):
1084 ... if matchobj.group(0) == '-': return ' '
1085 ... else: return '-'
1086 ...
1087 >>> re.sub('-{1,2}', dashrepl, 'pro----gram-files')
1088 'pro--gram files'
1089 >>> re.sub(r'\sAND\s', ' & ', 'Baked Beans And Spam', flags=re.IGNORECASE)
1090 'Baked Beans & Spam'
1091
1092 The pattern may be a string or a :class:`~re.Pattern`.
1093
1094 The optional argument *count* is the maximum number of pattern occurrences to be
1095 replaced; *count* must be a non-negative integer. If omitted or zero, all
1096 occurrences will be replaced.
1097
1098 Adjacent empty matches are not possible, but an empty match can occur
1099 immediately after a non-empty match.
1100 As a result, ``sub('x*', '-', 'abxd')`` returns ``'-a-b--d-'``
1101 instead of ``'-a-b-d-'``.
1102
1103 .. index:: single: \g; in regular expressions
1104
1105 In string-type *repl* arguments, in addition to the character escapes and
1106 backreferences described above,
1107 ``\g<name>`` will use the substring matched by the group named ``name``, as
1108 defined by the ``(?P<name>...)`` syntax. ``\g<number>`` uses the corresponding
1109 group number; ``\g<2>`` is therefore equivalent to ``\2``, but isn't ambiguous
1110 in a replacement such as ``\g<2>0``. ``\20`` would be interpreted as a
1111 reference to group 20, not a reference to group 2 followed by the literal
1112 character ``'0'``. The backreference ``\g<0>`` substitutes in the entire
1113 substring matched by the RE.
1114
1115 The expression's behaviour can be modified by specifying a *flags* value.
1116 Values can be any of the `flags`_ variables, combined using bitwise OR
1117 (the ``|`` operator).
1118
1119 .. versionchanged:: 3.1
1120 Added the optional flags argument.
1121
1122 .. versionchanged:: 3.5
1123 Unmatched groups are replaced with an empty string.
1124
1125 .. versionchanged:: 3.6
1126 Unknown escapes in *pattern* consisting of ``'\'`` and an ASCII letter
1127 now are errors.
1128
1129 .. versionchanged:: 3.7
1130 Unknown escapes in *repl* consisting of ``'\'`` and an ASCII letter
1131 now are errors.
1132 An empty match can occur immediately after a non-empty match.
1133
1134 .. versionchanged:: 3.12
1135 Group *id* can only contain ASCII digits.
1136 In :class:`bytes` replacement strings, group *name* can only contain bytes
1137 in the ASCII range (``b'\x00'``-``b'\x7f'``).
1138
1139 .. deprecated:: 3.13
1140 Passing *count* and *flags* as positional arguments is deprecated.
1141 In future Python versions they will be
1142 :ref:`keyword-only parameters <keyword-only_parameter>`.
1143
1144
1145 .. function:: subn(pattern, repl, string, count=0, flags=0)
1146
1147 Perform the same operation as :func:`sub`, but return a tuple ``(new_string,
1148 number_of_subs_made)``.
1149
1150 The expression's behaviour can be modified by specifying a *flags* value.
1151 Values can be any of the `flags`_ variables, combined using bitwise OR
1152 (the ``|`` operator).
1153
1154
1155 .. function:: escape(pattern)
1156
1157 Escape special characters in *pattern*.
1158 This is useful if you want to match an arbitrary literal string that may
1159 have regular expression metacharacters in it. For example::
1160
1161 >>> print(re.escape('https://www.python.org'))
1162 https://www\.python\.org
1163
1164 >>> legal_chars = string.ascii_lowercase + string.digits + "!#$%&'*+-.^_`|~:"
1165 >>> print('[%s]+' % re.escape(legal_chars))
1166 [abcdefghijklmnopqrstuvwxyz0123456789!\#\$%\&'\*\+\-\.\^_`\|\~:]+
1167
1168 >>> operators = ['+', '-', '*', '/', '**']
1169 >>> print('|'.join(map(re.escape, sorted(operators, reverse=True))))
1170 /|\-|\+|\*\*|\*
1171
1172 This function must not be used for the replacement string in :func:`sub`
1173 and :func:`subn`, only backslashes should be escaped. For example::
1174
1175 >>> digits_re = r'\d+'
1176 >>> sample = '/usr/sbin/sendmail - 0 errors, 12 warnings'
1177 >>> print(re.sub(digits_re, digits_re.replace('\\', r'\\'), sample))
1178 /usr/sbin/sendmail - \d+ errors, \d+ warnings
1179
1180 .. versionchanged:: 3.3
1181 The ``'_'`` character is no longer escaped.
1182
1183 .. versionchanged:: 3.7
1184 Only characters that can have special meaning in a regular expression
1185 are escaped. As a result, ``'!'``, ``'"'``, ``'%'``, ``"'"``, ``','``,
1186 ``'/'``, ``':'``, ``';'``, ``'<'``, ``'='``, ``'>'``, ``'@'``, and
1187 ``"`"`` are no longer escaped.
1188
1189
1190 .. function:: purge()
1191
1192 Clear the regular expression cache.
1193
1194
1195 Exceptions
1196 ^^^^^^^^^^
1197
1198 .. exception:: PatternError(msg, pattern=None, pos=None)
1199
1200 Exception raised when a string passed to one of the functions here is not a
1201 valid regular expression (for example, it might contain unmatched parentheses)
1202 or when some other error occurs during compilation or matching. It is never an
1203 error if a string contains no match for a pattern. The ``PatternError`` instance has
1204 the following additional attributes:
1205
1206 .. attribute:: msg
1207
1208 The unformatted error message.
1209
1210 .. attribute:: pattern
1211
1212 The regular expression pattern.
1213
1214 .. attribute:: pos
1215
1216 The index in *pattern* where compilation failed (may be ``None``).
1217
1218 .. attribute:: lineno
1219
1220 The line corresponding to *pos* (may be ``None``).
1221
1222 .. attribute:: colno
1223
1224 The column corresponding to *pos* (may be ``None``).
1225
1226 .. versionchanged:: 3.5
1227 Added additional attributes.
1228
1229 .. versionchanged:: 3.13
1230 ``PatternError`` was originally named ``error``; the latter is kept as an alias for
1231 backward compatibility.
1232
1233 .. _re-objects:
1234
1235 Regular Expression Objects
1236 --------------------------
1237
1238 .. class:: Pattern
1239
1240 Compiled regular expression object returned by :func:`re.compile`.
1241
1242 Patterns are :ref:`generic <generics>` over the type of string they handle
1243 (:class:`str` or :class:`bytes`).
1244
1245 .. versionchanged:: 3.9
1246 :py:class:`re.Pattern` supports ``[]`` to indicate a Unicode (str) or bytes pattern.
1247 See :ref:`types-genericalias`.
1248
1249 .. method:: Pattern.search(string[, pos[, endpos]])
1250
1251 Scan through *string* looking for the first location where this regular
1252 expression produces a match, and return a corresponding :class:`~re.Match`.
1253 Return ``None`` if no position in the string matches the pattern; note that
1254 this is different from finding a zero-length match at some point in the string.
1255
1256 The optional second parameter *pos* gives an index in the string where the
1257 search is to start; it defaults to ``0``. This is not completely equivalent to
1258 slicing the string; the ``'^'`` pattern character matches at the real beginning
1259 of the string and at positions just after a newline, but not necessarily at the
1260 index where the search is to start.
1261
1262 The optional parameter *endpos* limits how far the string will be searched; it
1263 will be as if the string is *endpos* characters long, so only the characters
1264 from *pos* to ``endpos - 1`` will be searched for a match. If *endpos* is less
1265 than *pos*, no match will be found; otherwise, if *rx* is a compiled regular
1266 expression object, ``rx.search(string, 0, 50)`` is equivalent to
1267 ``rx.search(string[:50], 0)``. ::
1268
1269 >>> pattern = re.compile("d")
1270 >>> pattern.search("dog") # Match at index 0
1271 <re.Match object; span=(0, 1), match='d'>
1272 >>> pattern.search("dog", 1) # No match; search doesn't include the "d"
1273
1274
1275 .. method:: Pattern.match(string[, pos[, endpos]])
1276
1277 If zero or more characters at the *beginning* of *string* match this regular
1278 expression, return a corresponding :class:`~re.Match`. Return ``None`` if the
1279 string does not match the pattern; note that this is different from a
1280 zero-length match.
1281
1282 The optional *pos* and *endpos* parameters have the same meaning as for the
1283 :meth:`~Pattern.search` method. ::
1284
1285 >>> pattern = re.compile("o")
1286 >>> pattern.match("dog") # No match as "o" is not at the start of "dog".
1287 >>> pattern.match("dog", 1) # Match as "o" is the 2nd character of "dog".
1288 <re.Match object; span=(1, 2), match='o'>
1289
1290 If you want to locate a match anywhere in *string*, use
1291 :meth:`~Pattern.search` instead (see also :ref:`search-vs-match`).
1292
1293
1294 .. method:: Pattern.fullmatch(string[, pos[, endpos]])
1295
1296 If the whole *string* matches this regular expression, return a corresponding
1297 :class:`~re.Match`. Return ``None`` if the string does not match the pattern;
1298 note that this is different from a zero-length match.
1299
1300 The optional *pos* and *endpos* parameters have the same meaning as for the
1301 :meth:`~Pattern.search` method. ::
1302
1303 >>> pattern = re.compile("o[gh]")
1304 >>> pattern.fullmatch("dog") # No match as "o" is not at the start of "dog".
1305 >>> pattern.fullmatch("ogre") # No match as not the full string matches.
1306 >>> pattern.fullmatch("doggie", 1, 3) # Matches within given limits.
1307 <re.Match object; span=(1, 3), match='og'>
1308
1309 .. versionadded:: 3.4
1310
1311
1312 .. method:: Pattern.split(string, maxsplit=0)
1313
1314 Identical to the :func:`split` function, using the compiled pattern.
1315
1316
1317 .. method:: Pattern.findall(string[, pos[, endpos]])
1318
1319 Similar to the :func:`findall` function, using the compiled pattern, but
1320 also accepts optional *pos* and *endpos* parameters that limit the search
1321 region like for :meth:`search`.
1322
1323
1324 .. method:: Pattern.finditer(string[, pos[, endpos]])
1325
1326 Similar to the :func:`finditer` function, using the compiled pattern, but
1327 also accepts optional *pos* and *endpos* parameters that limit the search
1328 region like for :meth:`search`.
1329
1330
1331 .. method:: Pattern.sub(repl, string, count=0)
1332
1333 Identical to the :func:`sub` function, using the compiled pattern.
1334
1335
1336 .. method:: Pattern.subn(repl, string, count=0)
1337
1338 Identical to the :func:`subn` function, using the compiled pattern.
1339
1340
1341 .. attribute:: Pattern.flags
1342
1343 The regex matching flags. This is a combination of the flags given to
1344 :func:`.compile`, any ``(?...)`` inline flags in the pattern, and implicit
1345 flags such as :py:const:`~re.UNICODE` if the pattern is a Unicode string.
1346
1347
1348 .. attribute:: Pattern.groups
1349
1350 The number of capturing groups in the pattern.
1351
1352
1353 .. attribute:: Pattern.groupindex
1354
1355 A dictionary mapping any symbolic group names defined by ``(?P<id>)`` to group
1356 numbers. The dictionary is empty if no symbolic groups were used in the
1357 pattern.
1358
1359
1360 .. attribute:: Pattern.pattern
1361
1362 The pattern string from which the pattern object was compiled.
1363
1364
1365 .. versionchanged:: 3.7
1366 Added support of :func:`copy.copy` and :func:`copy.deepcopy`. Compiled
1367 regular expression objects are considered atomic.
1368
1369
1370 .. _match-objects:
1371
1372 Match Objects
1373 -------------
1374
1375 Match objects always have a boolean value of ``True``.
1376 Since :meth:`~Pattern.match` and :meth:`~Pattern.search` return ``None``
1377 when there is no match, you can test whether there was a match with a simple
1378 ``if`` statement::
1379
1380 match = re.search(pattern, string)
1381 if match:
1382 process(match)
1383
1384 .. class:: Match
1385
1386 Match object returned by successful ``match``\ es and ``search``\ es.
1387
1388 Matches are :ref:`generic <generics>` over the type of string which was
1389 matched (:class:`str` or :class:`bytes`).
1390
1391 .. versionchanged:: 3.9
1392 :py:class:`re.Match` supports ``[]`` to indicate a Unicode (str) or bytes match.
1393 See :ref:`types-genericalias`.
1394
1395 .. method:: Match.expand(template)
1396
1397 Return the string obtained by doing backslash substitution on the template
1398 string *template*, as done by the :meth:`~Pattern.sub` method.
1399 Escapes such as ``\n`` are converted to the appropriate characters,
1400 and numeric backreferences (``\1``, ``\2``) and named backreferences
1401 (``\g<1>``, ``\g<name>``) are replaced by the contents of the
1402 corresponding group. The backreference ``\g<0>`` will be
1403 replaced by the entire match.
1404
1405 .. versionchanged:: 3.5
1406 Unmatched groups are replaced with an empty string.
1407
1408 .. method:: Match.group([group1, ...])
1409
1410 Returns one or more subgroups of the match. If there is a single argument, the
1411 result is a single string; if there are multiple arguments, the result is a
1412 tuple with one item per argument. Without arguments, *group1* defaults to zero
1413 (the whole match is returned). If a *groupN* argument is zero, the corresponding
1414 return value is the entire matching string; if it is a positive integer, it is
1415 the string matching the corresponding parenthesized group. If a group number is
1416 negative or larger than the number of groups defined in the pattern, an
1417 :exc:`IndexError` exception is raised. If a group is contained in a
1418 part of the pattern that did not match, the corresponding result is ``None``.
1419 If a group is contained in a part of the pattern that matched multiple times,
1420 the last match is returned. ::
1421
1422 >>> m = re.match(r"(\w+) (\w+)", "Isaac Newton, physicist")
1423 >>> m.group(0) # The entire match
1424 'Isaac Newton'
1425 >>> m.group(1) # The first parenthesized subgroup.
1426 'Isaac'
1427 >>> m.group(2) # The second parenthesized subgroup.
1428 'Newton'
1429 >>> m.group(1, 2) # Multiple arguments give us a tuple.
1430 ('Isaac', 'Newton')
1431
1432 If the regular expression uses the ``(?P<name>...)`` syntax, the *groupN*
1433 arguments may also be strings identifying groups by their group name. If a
1434 string argument is not used as a group name in the pattern, an :exc:`IndexError`
1435 exception is raised.
1436
1437 A moderately complicated example::
1438
1439 >>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds")
1440 >>> m.group('first_name')
1441 'Malcolm'
1442 >>> m.group('last_name')
1443 'Reynolds'
1444
1445 Named groups can also be referred to by their index::
1446
1447 >>> m.group(1)
1448 'Malcolm'
1449 >>> m.group(2)
1450 'Reynolds'
1451
1452 If a group matches multiple times, only the last match is accessible::
1453
1454 >>> m = re.match(r"(..)+", "a1b2c3") # Matches 3 times.
1455 >>> m.group(1) # Returns only the last match.
1456 'c3'
1457
1458
1459 .. method:: Match.__getitem__(g)
1460
1461 This is identical to ``m.group(g)``. This allows easier access to
1462 an individual group from a match::
1463
1464 >>> m = re.match(r"(\w+) (\w+)", "Isaac Newton, physicist")
1465 >>> m[0] # The entire match
1466 'Isaac Newton'
1467 >>> m[1] # The first parenthesized subgroup.
1468 'Isaac'
1469 >>> m[2] # The second parenthesized subgroup.
1470 'Newton'
1471
1472 Named groups are supported as well::
1473
1474 >>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Isaac Newton")
1475 >>> m['first_name']
1476 'Isaac'
1477 >>> m['last_name']
1478 'Newton'
1479
1480 .. versionadded:: 3.6
1481
1482
1483 .. method:: Match.groups(default=None)
1484
1485 Return a tuple containing all the subgroups of the match, from 1 up to however
1486 many groups are in the pattern. The *default* argument is used for groups that
1487 did not participate in the match; it defaults to ``None``.
1488
1489 For example::
1490
1491 >>> m = re.match(r"(\d+)\.(\d+)", "24.1632")
1492 >>> m.groups()
1493 ('24', '1632')
1494
1495 If we make the decimal place and everything after it optional, not all groups
1496 might participate in the match. These groups will default to ``None`` unless
1497 the *default* argument is given::
1498
1499 >>> m = re.match(r"(\d+)\.?(\d+)?", "24")
1500 >>> m.groups() # Second group defaults to None.
1501 ('24', None)
1502 >>> m.groups('0') # Now, the second group defaults to '0'.
1503 ('24', '0')
1504
1505
1506 .. method:: Match.groupdict(default=None)
1507
1508 Return a dictionary containing all the *named* subgroups of the match, keyed by
1509 the subgroup name. The *default* argument is used for groups that did not
1510 participate in the match; it defaults to ``None``. For example::
1511
1512 >>> m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds")
1513 >>> m.groupdict()
1514 {'first_name': 'Malcolm', 'last_name': 'Reynolds'}
1515
1516
1517 .. method:: Match.start([group])
1518 Match.end([group])
1519
1520 Return the indices of the start and end of the substring matched by *group*;
1521 *group* defaults to zero (meaning the whole matched substring). Return ``-1`` if
1522 *group* exists but did not contribute to the match. For a match object *m*, and
1523 a group *g* that did contribute to the match, the substring matched by group *g*
1524 (equivalent to ``m.group(g)``) is ::
1525
1526 m.string[m.start(g):m.end(g)]
1527
1528 Note that ``m.start(group)`` will equal ``m.end(group)`` if *group* matched a
1529 null string. For example, after ``m = re.search('b(c?)', 'cba')``,
1530 ``m.start(0)`` is 1, ``m.end(0)`` is 2, ``m.start(1)`` and ``m.end(1)`` are both
1531 2, and ``m.start(2)`` raises an :exc:`IndexError` exception.
1532
1533 An example that will remove *remove_this* from email addresses::
1534
1535 >>> email = "tony@tiremove_thisger.net"
1536 >>> m = re.search("remove_this", email)
1537 >>> email[:m.start()] + email[m.end():]
1538 '[email protected]'
1539
1540
1541 .. method:: Match.span([group])
1542
1543 For a match *m*, return the 2-tuple ``(m.start(group), m.end(group))``. Note
1544 that if *group* did not contribute to the match, this is ``(-1, -1)``.
1545 *group* defaults to zero, the entire match.
1546
1547
1548 .. attribute:: Match.pos
1549
1550 The value of *pos* which was passed to the :meth:`~Pattern.search` or
1551 :meth:`~Pattern.match` method of a :ref:`regex object <re-objects>`. This is
1552 the index into the string at which the RE engine started looking for a match.
1553
1554
1555 .. attribute:: Match.endpos
1556
1557 The value of *endpos* which was passed to the :meth:`~Pattern.search` or
1558 :meth:`~Pattern.match` method of a :ref:`regex object <re-objects>`. This is
1559 the index into the string beyond which the RE engine will not go.
1560
1561
1562 .. attribute:: Match.lastindex
1563
1564 The integer index of the last matched capturing group, or ``None`` if no group
1565 was matched at all. For example, the expressions ``(a)b``, ``((a)(b))``, and
1566 ``((ab))`` will have ``lastindex == 1`` if applied to the string ``'ab'``, while
1567 the expression ``(a)(b)`` will have ``lastindex == 2``, if applied to the same
1568 string.
1569
1570
1571 .. attribute:: Match.lastgroup
1572
1573 The name of the last matched capturing group, or ``None`` if the group didn't
1574 have a name, or if no group was matched at all.
1575
1576
1577 .. attribute:: Match.re
1578
1579 The :ref:`regular expression object <re-objects>` whose :meth:`~Pattern.match` or
1580 :meth:`~Pattern.search` method produced this match instance.
1581
1582
1583 .. attribute:: Match.string
1584
1585 The string passed to :meth:`~Pattern.match` or :meth:`~Pattern.search`.
1586
1587
1588 .. versionchanged:: 3.7
1589 Added support of :func:`copy.copy` and :func:`copy.deepcopy`. Match objects
1590 are considered atomic.
1591
1592
1593 .. _re-examples:
1594
1595 Regular Expression Examples
1596 ---------------------------
1597
1598
1599 Checking for a Pair
1600 ^^^^^^^^^^^^^^^^^^^
1601
1602 In this example, we'll use the following helper function to display match
1603 objects a little more gracefully::
1604
1605 def displaymatch(match):
1606 if match is None:
1607 return None
1608 return '<Match: %r, groups=%r>' % (match.group(), match.groups())
1609
1610 Suppose you are writing a poker program where a player's hand is represented as
1611 a 5-character string with each character representing a card, "a" for ace, "k"
1612 for king, "q" for queen, "j" for jack, "t" for 10, and "2" through "9"
1613 representing the card with that value.
1614
1615 To see if a given string is a valid hand, one could do the following::
1616
1617 >>> valid = re.compile(r"^[a2-9tjqk]{5}$")
1618 >>> displaymatch(valid.match("akt5q")) # Valid.
1619 "<Match: 'akt5q', groups=()>"
1620 >>> displaymatch(valid.match("akt5e")) # Invalid.
1621 >>> displaymatch(valid.match("akt")) # Invalid.
1622 >>> displaymatch(valid.match("727ak")) # Valid.
1623 "<Match: '727ak', groups=()>"
1624
1625 That last hand, ``"727ak"``, contained a pair, or two of the same valued cards.
1626 To match this with a regular expression, one could use backreferences as such::
1627
1628 >>> pair = re.compile(r".*(.).*\1")
1629 >>> displaymatch(pair.match("717ak")) # Pair of 7s.
1630 "<Match: '717', groups=('7',)>"
1631 >>> displaymatch(pair.match("718ak")) # No pairs.
1632 >>> displaymatch(pair.match("354aa")) # Pair of aces.
1633 "<Match: '354aa', groups=('a',)>"
1634
1635 To find out what card the pair consists of, one could use the
1636 :meth:`~Match.group` method of the match object in the following manner::
1637
1638 >>> pair = re.compile(r".*(.).*\1")
1639 >>> pair.match("717ak").group(1)
1640 '7'
1641
1642 # Error because re.match() returns None, which doesn't have a group() method:
1643 >>> pair.match("718ak").group(1)
1644 Traceback (most recent call last):
1645 File "<pyshell#23>", line 1, in <module>
1646 re.match(r".*(.).*\1", "718ak").group(1)
1647 AttributeError: 'NoneType' object has no attribute 'group'
1648
1649 >>> pair.match("354aa").group(1)
1650 'a'
1651
1652
1653 Simulating scanf()
1654 ^^^^^^^^^^^^^^^^^^
1655
1656 .. index:: single: scanf (C function)
1657
1658 Python does not currently have an equivalent to :c:func:`!scanf`. Regular
1659 expressions are generally more powerful, though also more verbose, than
1660 :c:func:`!scanf` format strings. The table below offers some more-or-less
1661 equivalent mappings between :c:func:`!scanf` format tokens and regular
1662 expressions.
1663
1664 +--------------------------------+---------------------------------------------+
1665 | :c:func:`!scanf` Token | Regular Expression |
1666 +================================+=============================================+
1667 | ``%c`` | ``.`` |
1668 +--------------------------------+---------------------------------------------+
1669 | ``%5c`` | ``.{5}`` |
1670 +--------------------------------+---------------------------------------------+
1671 | ``%d`` | ``[-+]?\d+`` |
1672 +--------------------------------+---------------------------------------------+
1673 | ``%e``, ``%E``, ``%f``, ``%g`` | ``[-+]?(\d+(\.\d*)?|\.\d+)([eE][-+]?\d+)?`` |
1674 +--------------------------------+---------------------------------------------+
1675 | ``%i`` | ``[-+]?(0[xX][\dA-Fa-f]+|0[0-7]*|\d+)`` |
1676 +--------------------------------+---------------------------------------------+
1677 | ``%o`` | ``[-+]?[0-7]+`` |
1678 +--------------------------------+---------------------------------------------+
1679 | ``%s`` | ``\S+`` |
1680 +--------------------------------+---------------------------------------------+
1681 | ``%u`` | ``\d+`` |
1682 +--------------------------------+---------------------------------------------+
1683 | ``%x``, ``%X`` | ``[-+]?(0[xX])?[\dA-Fa-f]+`` |
1684 +--------------------------------+---------------------------------------------+
1685
1686 To extract the filename and numbers from a string like ::
1687
1688 /usr/sbin/sendmail - 0 errors, 4 warnings
1689
1690 you would use a :c:func:`!scanf` format like ::
1691
1692 %s - %d errors, %d warnings
1693
1694 The equivalent regular expression would be ::
1695
1696 (\S+) - (\d+) errors, (\d+) warnings
1697
1698
1699 .. _search-vs-match:
1700
1701 search() vs. match()
1702 ^^^^^^^^^^^^^^^^^^^^
1703
1704 .. sectionauthor:: Fred L. Drake, Jr. <[email protected]>
1705
1706 Python offers different primitive operations based on regular expressions:
1707
1708 + :func:`re.match` checks for a match only at the beginning of the string
1709 + :func:`re.search` checks for a match anywhere in the string
1710 (this is what Perl does by default)
1711 + :func:`re.fullmatch` checks for entire string to be a match
1712
1713
1714 For example::
1715
1716 >>> re.match("c", "abcdef") # No match
1717 >>> re.search("c", "abcdef") # Match
1718 <re.Match object; span=(2, 3), match='c'>
1719 >>> re.fullmatch("p.*n", "python") # Match
1720 <re.Match object; span=(0, 6), match='python'>
1721 >>> re.fullmatch("r.*n", "python") # No match
1722
1723 Regular expressions beginning with ``'^'`` can be used with :func:`search` to
1724 restrict the match at the beginning of the string::
1725
1726 >>> re.match("c", "abcdef") # No match
1727 >>> re.search("^c", "abcdef") # No match
1728 >>> re.search("^a", "abcdef") # Match
1729 <re.Match object; span=(0, 1), match='a'>
1730
1731 Note however that in :const:`MULTILINE` mode :func:`match` only matches at the
1732 beginning of the string, whereas using :func:`search` with a regular expression
1733 beginning with ``'^'`` will match at the beginning of each line. ::
1734
1735 >>> re.match("X", "A\nB\nX", re.MULTILINE) # No match
1736 >>> re.search("^X", "A\nB\nX", re.MULTILINE) # Match
1737 <re.Match object; span=(4, 5), match='X'>
1738
1739
1740 Making a Phonebook
1741 ^^^^^^^^^^^^^^^^^^
1742
1743 :func:`split` splits a string into a list delimited by the passed pattern. The
1744 method is invaluable for converting textual data into data structures that can be
1745 easily read and modified by Python as demonstrated in the following example that
1746 creates a phonebook.
1747
1748 First, here is the input. Normally it may come from a file, here we are using
1749 triple-quoted string syntax
1750
1751 .. doctest::
1752
1753 >>> text = """Ross McFluff: 834.345.1254 155 Elm Street
1754 ...
1755 ... Ronald Heathmore: 892.345.3428 436 Finley Avenue
1756 ... Frank Burger: 925.541.7625 662 South Dogwood Way
1757 ...
1758 ...
1759 ... Heather Albrecht: 548.326.4584 919 Park Place"""
1760
1761 The entries are separated by one or more newlines. Now we convert the string
1762 into a list with each nonempty line having its own entry:
1763
1764 .. doctest::
1765 :options: +NORMALIZE_WHITESPACE
1766
1767 >>> entries = re.split("\n+", text)
1768 >>> entries
1769 ['Ross McFluff: 834.345.1254 155 Elm Street',
1770 'Ronald Heathmore: 892.345.3428 436 Finley Avenue',
1771 'Frank Burger: 925.541.7625 662 South Dogwood Way',
1772 'Heather Albrecht: 548.326.4584 919 Park Place']
1773
1774 Finally, split each entry into a list with first name, last name, telephone
1775 number, and address. We use the ``maxsplit`` parameter of :func:`split`
1776 because the address has spaces, our splitting pattern, in it:
1777
1778 .. doctest::
1779 :options: +NORMALIZE_WHITESPACE
1780
1781 >>> [re.split(":? ", entry, maxsplit=3) for entry in entries]
1782 [['Ross', 'McFluff', '834.345.1254', '155 Elm Street'],
1783 ['Ronald', 'Heathmore', '892.345.3428', '436 Finley Avenue'],
1784 ['Frank', 'Burger', '925.541.7625', '662 South Dogwood Way'],
1785 ['Heather', 'Albrecht', '548.326.4584', '919 Park Place']]
1786
1787 The ``:?`` pattern matches the colon after the last name, so that it does not
1788 occur in the result list. With a ``maxsplit`` of ``4``, we could separate the
1789 house number from the street name:
1790
1791 .. doctest::
1792 :options: +NORMALIZE_WHITESPACE
1793
1794 >>> [re.split(":? ", entry, maxsplit=4) for entry in entries]
1795 [['Ross', 'McFluff', '834.345.1254', '155', 'Elm Street'],
1796 ['Ronald', 'Heathmore', '892.345.3428', '436', 'Finley Avenue'],
1797 ['Frank', 'Burger', '925.541.7625', '662', 'South Dogwood Way'],
1798 ['Heather', 'Albrecht', '548.326.4584', '919', 'Park Place']]
1799
1800
1801 Text Munging
1802 ^^^^^^^^^^^^
1803
1804 :func:`sub` replaces every occurrence of a pattern with a string or the
1805 result of a function. This example demonstrates using :func:`sub` with
1806 a function to "munge" text, or randomize the order of all the characters
1807 in each word of a sentence except for the first and last characters::
1808
1809 >>> def repl(m):
1810 ... inner_word = list(m.group(2))
1811 ... random.shuffle(inner_word)
1812 ... return m.group(1) + "".join(inner_word) + m.group(3)
1813 ...
1814 >>> text = "Professor Abdolmalek, please report your absences promptly."
1815 >>> re.sub(r"(\w)(\w+)(\w)", repl, text)
1816 'Poefsrosr Aealmlobdk, pslaee reorpt your abnseces plmrptoy.'
1817 >>> re.sub(r"(\w)(\w+)(\w)", repl, text)
1818 'Pofsroser Aodlambelk, plasee reoprt yuor asnebces potlmrpy.'
1819
1820
1821 Finding all Adverbs
1822 ^^^^^^^^^^^^^^^^^^^
1823
1824 :func:`findall` matches *all* occurrences of a pattern, not just the first
1825 one as :func:`search` does. For example, if a writer wanted to
1826 find all of the adverbs in some text, they might use :func:`findall` in
1827 the following manner::
1828
1829 >>> text = "He was carefully disguised but captured quickly by police."
1830 >>> re.findall(r"\w+ly\b", text)
1831 ['carefully', 'quickly']
1832
1833
1834 Finding all Adverbs and their Positions
1835 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1836
1837 If one wants more information about all matches of a pattern than the matched
1838 text, :func:`finditer` is useful as it provides :class:`~re.Match` objects
1839 instead of strings. Continuing with the previous example, if a writer wanted
1840 to find all of the adverbs *and their positions* in some text, they would use
1841 :func:`finditer` in the following manner::
1842
1843 >>> text = "He was carefully disguised but captured quickly by police."
1844 >>> for m in re.finditer(r"\w+ly\b", text):
1845 ... print('%02d-%02d: %s' % (m.start(), m.end(), m.group(0)))
1846 07-16: carefully
1847 40-47: quickly
1848
1849
1850 Raw String Notation
1851 ^^^^^^^^^^^^^^^^^^^
1852
1853 Raw string notation (``r"text"``) keeps regular expressions sane. Without it,
1854 every backslash (``'\'``) in a regular expression would have to be prefixed with
1855 another one to escape it. For example, the two following lines of code are
1856 functionally identical::
1857
1858 >>> re.match(r"\W(.)\1\W", " ff ")
1859 <re.Match object; span=(0, 4), match=' ff '>
1860 >>> re.match("\\W(.)\\1\\W", " ff ")
1861 <re.Match object; span=(0, 4), match=' ff '>
1862
1863 When one wants to match a literal backslash, it must be escaped in the regular
1864 expression. With raw string notation, this means ``r"\\"``. Without raw string
1865 notation, one must use ``"\\\\"``, making the following lines of code
1866 functionally identical::
1867
1868 >>> re.match(r"\\", r"\\")
1869 <re.Match object; span=(0, 1), match='\\'>
1870 >>> re.match("\\\\", r"\\")
1871 <re.Match object; span=(0, 1), match='\\'>
1872
1873
1874 Writing a Tokenizer
1875 ^^^^^^^^^^^^^^^^^^^
1876
1877 A `tokenizer or scanner <https://en.wikipedia.org/wiki/Lexical_analysis>`_
1878 analyzes a string to categorize groups of characters. This is a useful first
1879 step in writing a compiler or interpreter.
1880
1881 The text categories are specified with regular expressions. The technique is
1882 to combine those into a single master regular expression and to loop over
1883 successive matches::
1884
1885 from typing import NamedTuple
1886 import re
1887
1888 class Token(NamedTuple):
1889 type: str
1890 value: int | float | str
1891 line: int
1892 column: int
1893
1894 def tokenize(code):
1895 keywords = {'IF', 'THEN', 'ENDIF', 'FOR', 'NEXT', 'GOSUB', 'RETURN'}
1896 token_specification = [
1897 ('NUMBER', r'\d+(\.\d*)?'), # Integer or decimal number
1898 ('ASSIGN', r':='), # Assignment operator
1899 ('END', r';'), # Statement terminator
1900 ('ID', r'[A-Za-z]+'), # Identifiers
1901 ('OP', r'[+\-*/]'), # Arithmetic operators
1902 ('NEWLINE', r'\n'), # Line endings
1903 ('SKIP', r'[ \t]+'), # Skip over spaces and tabs
1904 ('MISMATCH', r'.'), # Any other character
1905 ]
1906 tok_regex = '|'.join('(?P<%s>%s)' % pair for pair in token_specification)
1907 line_num = 1
1908 line_start = 0
1909 for mo in re.finditer(tok_regex, code):
1910 kind = mo.lastgroup
1911 value = mo.group()
1912 column = mo.start() - line_start
1913 if kind == 'NUMBER':
1914 value = float(value) if '.' in value else int(value)
1915 elif kind == 'ID' and value in keywords:
1916 kind = value
1917 elif kind == 'NEWLINE':
1918 line_start = mo.end()
1919 line_num += 1
1920 continue
1921 elif kind == 'SKIP':
1922 continue
1923 elif kind == 'MISMATCH':
1924 raise RuntimeError(f'{value!r} unexpected on line {line_num}')
1925 yield Token(kind, value, line_num, column)
1926
1927 statements = '''
1928 IF quantity THEN
1929 total := total + price * quantity;
1930 tax := price * 0.05;
1931 ENDIF;
1932 '''
1933
1934 for token in tokenize(statements):
1935 print(token)
1936
1937 The tokenizer produces the following output::
1938
1939 Token(type='IF', value='IF', line=2, column=4)
1940 Token(type='ID', value='quantity', line=2, column=7)
1941 Token(type='THEN', value='THEN', line=2, column=16)
1942 Token(type='ID', value='total', line=3, column=8)
1943 Token(type='ASSIGN', value=':=', line=3, column=14)
1944 Token(type='ID', value='total', line=3, column=17)
1945 Token(type='OP', value='+', line=3, column=23)
1946 Token(type='ID', value='price', line=3, column=25)
1947 Token(type='OP', value='*', line=3, column=31)
1948 Token(type='ID', value='quantity', line=3, column=33)
1949 Token(type='END', value=';', line=3, column=41)
1950 Token(type='ID', value='tax', line=4, column=8)
1951 Token(type='ASSIGN', value=':=', line=4, column=12)
1952 Token(type='ID', value='price', line=4, column=15)
1953 Token(type='OP', value='*', line=4, column=21)
1954 Token(type='NUMBER', value=0.05, line=4, column=23)
1955 Token(type='END', value=';', line=4, column=27)
1956 Token(type='ENDIF', value='ENDIF', line=5, column=4)
1957 Token(type='END', value=';', line=5, column=9)
1958
1959
1960 .. [Frie09] Friedl, Jeffrey. Mastering Regular Expressions. 3rd ed., O'Reilly
1961 Media, 2009. The third edition of the book no longer covers Python at all,
1962 but the first edition covered writing good regular expression patterns in
1963 great detail.