| 1 |
.. _regex-howto: |
| 2 |
|
| 3 |
**************************** |
| 4 |
Regular expression HOWTO |
| 5 |
**************************** |
| 6 |
|
| 7 |
:Author: A.M. Kuchling <[email protected]> |
| 8 |
|
| 9 |
.. TODO: |
| 10 |
Document lookbehind assertions |
| 11 |
Better way of displaying a RE, a string, and what it matches |
| 12 |
Mention optional argument to match.groups() |
| 13 |
Unicode (at least a reference) |
| 14 |
|
| 15 |
|
| 16 |
.. topic:: Abstract |
| 17 |
|
| 18 |
This document is an introductory tutorial to using regular expressions in Python |
| 19 |
with the :mod:`re` module. It provides a gentler introduction than the |
| 20 |
corresponding section in the Library Reference. |
| 21 |
|
| 22 |
|
| 23 |
Introduction |
| 24 |
============ |
| 25 |
|
| 26 |
Regular expressions (called REs, or regexes, or regex patterns) are essentially |
| 27 |
a tiny, highly specialized programming language embedded inside Python and made |
| 28 |
available through the :mod:`re` module. Using this little language, you specify |
| 29 |
the rules for the set of possible strings that you want to match; this set might |
| 30 |
contain English sentences, or e-mail addresses, or TeX commands, or anything you |
| 31 |
like. You can then ask questions such as "Does this string match the pattern?", |
| 32 |
or "Is there a match for the pattern anywhere in this string?". You can also |
| 33 |
use REs to modify a string or to split it apart in various ways. |
| 34 |
|
| 35 |
Regular expression patterns are compiled into a series of bytecodes which are |
| 36 |
then executed by a matching engine written in C. For advanced use, it may be |
| 37 |
necessary to pay careful attention to how the engine will execute a given RE, |
| 38 |
and write the RE in a certain way in order to produce bytecode that runs faster. |
| 39 |
Optimization isn't covered in this document, because it requires that you have a |
| 40 |
good understanding of the matching engine's internals. |
| 41 |
|
| 42 |
The regular expression language is relatively small and restricted, so not all |
| 43 |
possible string processing tasks can be done using regular expressions. There |
| 44 |
are also tasks that *can* be done with regular expressions, but the expressions |
| 45 |
turn out to be very complicated. In these cases, you may be better off writing |
| 46 |
Python code to do the processing; while Python code will be slower than an |
| 47 |
elaborate regular expression, it will also probably be more understandable. |
| 48 |
|
| 49 |
|
| 50 |
Simple patterns |
| 51 |
=============== |
| 52 |
|
| 53 |
We'll start by learning about the simplest possible regular expressions. Since |
| 54 |
regular expressions are used to operate on strings, we'll begin with the most |
| 55 |
common task: matching characters. |
| 56 |
|
| 57 |
For a detailed explanation of the computer science underlying regular |
| 58 |
expressions (deterministic and non-deterministic finite automata), you can refer |
| 59 |
to almost any textbook on writing compilers. |
| 60 |
|
| 61 |
|
| 62 |
Matching characters |
| 63 |
------------------- |
| 64 |
|
| 65 |
Most letters and characters will simply match themselves. For example, the |
| 66 |
regular expression ``test`` will match the string ``test`` exactly. (You can |
| 67 |
enable a case-insensitive mode that would let this RE match ``Test`` or ``TEST`` |
| 68 |
as well; more about this later.) |
| 69 |
|
| 70 |
There are exceptions to this rule; some characters are special |
| 71 |
:dfn:`metacharacters`, and don't match themselves. Instead, they signal that |
| 72 |
some out-of-the-ordinary thing should be matched, or they affect other portions |
| 73 |
of the RE by repeating them or changing their meaning. Much of this document is |
| 74 |
devoted to discussing various metacharacters and what they do. |
| 75 |
|
| 76 |
Here's a complete list of the metacharacters; their meanings will be discussed |
| 77 |
in the rest of this HOWTO. |
| 78 |
|
| 79 |
.. code-block:: none |
| 80 |
|
| 81 |
. ^ $ * + ? { } [ ] \ | ( ) |
| 82 |
|
| 83 |
The first metacharacters we'll look at are ``[`` and ``]``. They're used for |
| 84 |
specifying a character class, which is a set of characters that you wish to |
| 85 |
match. Characters can be listed individually, or a range of characters can be |
| 86 |
indicated by giving two characters and separating them by a ``'-'``. For |
| 87 |
example, ``[abc]`` will match any of the characters ``a``, ``b``, or ``c``; this |
| 88 |
is the same as ``[a-c]``, which uses a range to express the same set of |
| 89 |
characters. If you wanted to match only lowercase letters, your RE would be |
| 90 |
``[a-z]``. |
| 91 |
|
| 92 |
Metacharacters (except ``\``) are not active inside classes. For example, ``[akm$]`` will |
| 93 |
match any of the characters ``'a'``, ``'k'``, ``'m'``, or ``'$'``; ``'$'`` is |
| 94 |
usually a metacharacter, but inside a character class it's stripped of its |
| 95 |
special nature. |
| 96 |
|
| 97 |
You can match the characters not listed within the class by :dfn:`complementing` |
| 98 |
the set. This is indicated by including a ``'^'`` as the first character of the |
| 99 |
class. For example, ``[^5]`` will match any character except ``'5'``. If the |
| 100 |
caret appears elsewhere in a character class, it does not have special meaning. |
| 101 |
For example: ``[5^]`` will match either a ``'5'`` or a ``'^'``. |
| 102 |
|
| 103 |
Perhaps the most important metacharacter is the backslash, ``\``. As in Python |
| 104 |
string literals, the backslash can be followed by various characters to signal |
| 105 |
various special sequences. It's also used to escape all the metacharacters so |
| 106 |
you can still match them in patterns; for example, if you need to match a ``[`` |
| 107 |
or ``\``, you can precede them with a backslash to remove their special |
| 108 |
meaning: ``\[`` or ``\\``. |
| 109 |
|
| 110 |
Some of the special sequences beginning with ``'\'`` represent |
| 111 |
predefined sets of characters that are often useful, such as the set |
| 112 |
of digits, the set of letters, or the set of anything that isn't |
| 113 |
whitespace. |
| 114 |
|
| 115 |
Let's take an example: ``\w`` matches any alphanumeric character. If |
| 116 |
the regex pattern is expressed in bytes, this is equivalent to the |
| 117 |
class ``[a-zA-Z0-9_]``. If the regex pattern is a string, ``\w`` will |
| 118 |
match all the characters marked as letters in the Unicode database |
| 119 |
provided by the :mod:`unicodedata` module. You can use the more |
| 120 |
restricted definition of ``\w`` in a string pattern by supplying the |
| 121 |
:const:`re.ASCII` flag when compiling the regular expression. |
| 122 |
|
| 123 |
The following list of special sequences isn't complete. For a complete |
| 124 |
list of sequences and expanded class definitions for Unicode string |
| 125 |
patterns, see the last part of :ref:`Regular Expression Syntax |
| 126 |
<re-syntax>` in the Standard Library reference. In general, the |
| 127 |
Unicode versions match any character that's in the appropriate |
| 128 |
category in the Unicode database. |
| 129 |
|
| 130 |
``\d`` |
| 131 |
Matches any decimal digit; this is equivalent to the class ``[0-9]``. |
| 132 |
|
| 133 |
``\D`` |
| 134 |
Matches any non-digit character; this is equivalent to the class ``[^0-9]``. |
| 135 |
|
| 136 |
``\s`` |
| 137 |
Matches any whitespace character; this is equivalent to the class ``[ |
| 138 |
\t\n\r\f\v]``. |
| 139 |
|
| 140 |
``\S`` |
| 141 |
Matches any non-whitespace character; this is equivalent to the class ``[^ |
| 142 |
\t\n\r\f\v]``. |
| 143 |
|
| 144 |
``\w`` |
| 145 |
Matches any alphanumeric character; this is equivalent to the class |
| 146 |
``[a-zA-Z0-9_]``. |
| 147 |
|
| 148 |
``\W`` |
| 149 |
Matches any non-alphanumeric character; this is equivalent to the class |
| 150 |
``[^a-zA-Z0-9_]``. |
| 151 |
|
| 152 |
These sequences can be included inside a character class. For example, |
| 153 |
``[\s,.]`` is a character class that will match any whitespace character, or |
| 154 |
``','`` or ``'.'``. |
| 155 |
|
| 156 |
The final metacharacter in this section is ``.``. It matches anything except a |
| 157 |
newline character, and there's an alternate mode (:const:`re.DOTALL`) where it will |
| 158 |
match even a newline. ``.`` is often used where you want to match "any |
| 159 |
character". |
| 160 |
|
| 161 |
|
| 162 |
Repeating things |
| 163 |
---------------- |
| 164 |
|
| 165 |
Being able to match varying sets of characters is the first thing regular |
| 166 |
expressions can do that isn't already possible with the methods available on |
| 167 |
strings. However, if that was the only additional capability of regexes, they |
| 168 |
wouldn't be much of an advance. Another capability is that you can specify that |
| 169 |
portions of the RE must be repeated a certain number of times. |
| 170 |
|
| 171 |
The first metacharacter for repeating things that we'll look at is ``*``. ``*`` |
| 172 |
doesn't match the literal character ``'*'``; instead, it specifies that the |
| 173 |
previous character can be matched zero or more times, instead of exactly once. |
| 174 |
|
| 175 |
For example, ``ca*t`` will match ``'ct'`` (0 ``'a'`` characters), ``'cat'`` (1 ``'a'``), |
| 176 |
``'caaat'`` (3 ``'a'`` characters), and so forth. |
| 177 |
|
| 178 |
Repetitions such as ``*`` are :dfn:`greedy`; when repeating a RE, the matching |
| 179 |
engine will try to repeat it as many times as possible. If later portions of the |
| 180 |
pattern don't match, the matching engine will then back up and try again with |
| 181 |
fewer repetitions. |
| 182 |
|
| 183 |
A step-by-step example will make this more obvious. Let's consider the |
| 184 |
expression ``a[bcd]*b``. This matches the letter ``'a'``, zero or more letters |
| 185 |
from the class ``[bcd]``, and finally ends with a ``'b'``. Now imagine matching |
| 186 |
this RE against the string ``'abcbd'``. |
| 187 |
|
| 188 |
+------+-----------+---------------------------------+ |
| 189 |
| Step | Matched | Explanation | |
| 190 |
+======+===========+=================================+ |
| 191 |
| 1 | ``a`` | The ``a`` in the RE matches. | |
| 192 |
+------+-----------+---------------------------------+ |
| 193 |
| 2 | ``abcbd`` | The engine matches ``[bcd]*``, | |
| 194 |
| | | going as far as it can, which | |
| 195 |
| | | is to the end of the string. | |
| 196 |
+------+-----------+---------------------------------+ |
| 197 |
| 3 | *Failure* | The engine tries to match | |
| 198 |
| | | ``b``, but the current position | |
| 199 |
| | | is at the end of the string, so | |
| 200 |
| | | it fails. | |
| 201 |
+------+-----------+---------------------------------+ |
| 202 |
| 4 | ``abcb`` | Back up, so that ``[bcd]*`` | |
| 203 |
| | | matches one less character. | |
| 204 |
+------+-----------+---------------------------------+ |
| 205 |
| 5 | *Failure* | Try ``b`` again, but the | |
| 206 |
| | | current position is at the last | |
| 207 |
| | | character, which is a ``'d'``. | |
| 208 |
+------+-----------+---------------------------------+ |
| 209 |
| 6 | ``abc`` | Back up again, so that | |
| 210 |
| | | ``[bcd]*`` is only matching | |
| 211 |
| | | ``bc``. | |
| 212 |
+------+-----------+---------------------------------+ |
| 213 |
| 7 | ``abcb`` | Try ``b`` again. This time | |
| 214 |
| | | the character at the | |
| 215 |
| | | current position is ``'b'``, so | |
| 216 |
| | | it succeeds. | |
| 217 |
+------+-----------+---------------------------------+ |
| 218 |
|
| 219 |
The end of the RE has now been reached, and it has matched ``'abcb'``. This |
| 220 |
demonstrates how the matching engine goes as far as it can at first, and if no |
| 221 |
match is found it will then progressively back up and retry the rest of the RE |
| 222 |
again and again. It will back up until it has tried zero matches for |
| 223 |
``[bcd]*``, and if that subsequently fails, the engine will conclude that the |
| 224 |
string doesn't match the RE at all. |
| 225 |
|
| 226 |
Another repeating metacharacter is ``+``, which matches one or more times. Pay |
| 227 |
careful attention to the difference between ``*`` and ``+``; ``*`` matches |
| 228 |
*zero* or more times, so whatever's being repeated may not be present at all, |
| 229 |
while ``+`` requires at least *one* occurrence. To use a similar example, |
| 230 |
``ca+t`` will match ``'cat'`` (1 ``'a'``), ``'caaat'`` (3 ``'a'``\ s), but won't |
| 231 |
match ``'ct'``. |
| 232 |
|
| 233 |
There are two more repeating operators or quantifiers. The question mark character, ``?``, |
| 234 |
matches either once or zero times; you can think of it as marking something as |
| 235 |
being optional. For example, ``home-?brew`` matches either ``'homebrew'`` or |
| 236 |
``'home-brew'``. |
| 237 |
|
| 238 |
The most complicated quantifier is ``{m,n}``, where *m* and *n* are |
| 239 |
decimal integers. This quantifier means there must be at least *m* repetitions, |
| 240 |
and at most *n*. For example, ``a/{1,3}b`` will match ``'a/b'``, ``'a//b'``, and |
| 241 |
``'a///b'``. It won't match ``'ab'``, which has no slashes, or ``'a////b'``, which |
| 242 |
has four. |
| 243 |
|
| 244 |
You can omit either *m* or *n*; in that case, a reasonable value is assumed for |
| 245 |
the missing value. Omitting *m* is interpreted as a lower limit of 0, while |
| 246 |
omitting *n* results in an upper bound of infinity. |
| 247 |
|
| 248 |
The simplest case ``{m}`` matches the preceding item exactly *m* times. |
| 249 |
For example, ``a/{2}b`` will only match ``'a//b'``. |
| 250 |
|
| 251 |
Readers of a reductionist bent may notice that the three other quantifiers can |
| 252 |
all be expressed using this notation. ``{0,}`` is the same as ``*``, ``{1,}`` |
| 253 |
is equivalent to ``+``, and ``{0,1}`` is the same as ``?``. It's better to use |
| 254 |
``*``, ``+``, or ``?`` when you can, simply because they're shorter and easier |
| 255 |
to read. |
| 256 |
|
| 257 |
|
| 258 |
Using regular expressions |
| 259 |
========================= |
| 260 |
|
| 261 |
Now that we've looked at some simple regular expressions, how do we actually use |
| 262 |
them in Python? The :mod:`re` module provides an interface to the regular |
| 263 |
expression engine, allowing you to compile REs into objects and then perform |
| 264 |
matches with them. |
| 265 |
|
| 266 |
|
| 267 |
Compiling regular expressions |
| 268 |
----------------------------- |
| 269 |
|
| 270 |
Regular expressions are compiled into pattern objects, which have |
| 271 |
methods for various operations such as searching for pattern matches or |
| 272 |
performing string substitutions. :: |
| 273 |
|
| 274 |
>>> import re |
| 275 |
>>> p = re.compile('ab*') |
| 276 |
>>> p |
| 277 |
re.compile('ab*') |
| 278 |
|
| 279 |
:func:`re.compile` also accepts an optional *flags* argument, used to enable |
| 280 |
various special features and syntax variations. We'll go over the available |
| 281 |
settings later, but for now a single example will do:: |
| 282 |
|
| 283 |
>>> p = re.compile('ab*', re.IGNORECASE) |
| 284 |
|
| 285 |
The RE is passed to :func:`re.compile` as a string. REs are handled as strings |
| 286 |
because regular expressions aren't part of the core Python language, and no |
| 287 |
special syntax was created for expressing them. (There are applications that |
| 288 |
don't need REs at all, so there's no need to bloat the language specification by |
| 289 |
including them.) Instead, the :mod:`re` module is simply a C extension module |
| 290 |
included with Python, just like the :mod:`socket` or :mod:`zlib` modules. |
| 291 |
|
| 292 |
Putting REs in strings keeps the Python language simpler, but has one |
| 293 |
disadvantage which is the topic of the next section. |
| 294 |
|
| 295 |
|
| 296 |
.. _the-backslash-plague: |
| 297 |
|
| 298 |
The backslash plague |
| 299 |
-------------------- |
| 300 |
|
| 301 |
As stated earlier, regular expressions use the backslash character (``'\'``) to |
| 302 |
indicate special forms or to allow special characters to be used without |
| 303 |
invoking their special meaning. This conflicts with Python's usage of the same |
| 304 |
character for the same purpose in string literals. |
| 305 |
|
| 306 |
Let's say you want to write a RE that matches the string ``\section``, which |
| 307 |
might be found in a LaTeX file. To figure out what to write in the program |
| 308 |
code, start with the desired string to be matched. Next, you must escape any |
| 309 |
backslashes and other metacharacters by preceding them with a backslash, |
| 310 |
resulting in the string ``\\section``. The resulting string that must be passed |
| 311 |
to :func:`re.compile` must be ``\\section``. However, to express this as a |
| 312 |
Python string literal, both backslashes must be escaped *again*. |
| 313 |
|
| 314 |
+-------------------+------------------------------------------+ |
| 315 |
| Characters | Stage | |
| 316 |
+===================+==========================================+ |
| 317 |
| ``\section`` | Text string to be matched | |
| 318 |
+-------------------+------------------------------------------+ |
| 319 |
| ``\\section`` | Escaped backslash for :func:`re.compile` | |
| 320 |
+-------------------+------------------------------------------+ |
| 321 |
| ``"\\\\section"`` | Escaped backslashes for a string literal | |
| 322 |
+-------------------+------------------------------------------+ |
| 323 |
|
| 324 |
In short, to match a literal backslash, one has to write ``'\\\\'`` as the RE |
| 325 |
string, because the regular expression must be ``\\``, and each backslash must |
| 326 |
be expressed as ``\\`` inside a regular Python string literal. In REs that |
| 327 |
feature backslashes repeatedly, this leads to lots of repeated backslashes and |
| 328 |
makes the resulting strings difficult to understand. |
| 329 |
|
| 330 |
The solution is to use Python's raw string notation for regular expressions; |
| 331 |
backslashes are not handled in any special way in a string literal prefixed with |
| 332 |
``'r'``, so ``r"\n"`` is a two-character string containing ``'\'`` and ``'n'``, |
| 333 |
while ``"\n"`` is a one-character string containing a newline. Regular |
| 334 |
expressions will often be written in Python code using this raw string notation. |
| 335 |
|
| 336 |
In addition, special escape sequences that are valid in regular expressions, |
| 337 |
but not valid as Python string literals, now result in a |
| 338 |
:exc:`SyntaxWarning` and will eventually become a :exc:`SyntaxError`, |
| 339 |
which means the sequences will be invalid if raw string notation or escaping |
| 340 |
the backslashes isn't used. |
| 341 |
|
| 342 |
|
| 343 |
+-------------------+------------------+ |
| 344 |
| Regular String | Raw string | |
| 345 |
+===================+==================+ |
| 346 |
| ``"ab*"`` | ``r"ab*"`` | |
| 347 |
+-------------------+------------------+ |
| 348 |
| ``"\\\\section"`` | ``r"\\section"`` | |
| 349 |
+-------------------+------------------+ |
| 350 |
| ``"\\w+\\s+\\1"`` | ``r"\w+\s+\1"`` | |
| 351 |
+-------------------+------------------+ |
| 352 |
|
| 353 |
|
| 354 |
Performing matches |
| 355 |
------------------ |
| 356 |
|
| 357 |
Once you have an object representing a compiled regular expression, what do you |
| 358 |
do with it? Pattern objects have several methods and attributes. |
| 359 |
Only the most significant ones will be covered here; consult the :mod:`re` docs |
| 360 |
for a complete listing. |
| 361 |
|
| 362 |
+------------------+-----------------------------------------------+ |
| 363 |
| Method/Attribute | Purpose | |
| 364 |
+==================+===============================================+ |
| 365 |
| ``match()`` | Determine if the RE matches at the beginning | |
| 366 |
| | of the string. | |
| 367 |
+------------------+-----------------------------------------------+ |
| 368 |
| ``search()`` | Scan through a string, looking for any | |
| 369 |
| | location where this RE matches. | |
| 370 |
+------------------+-----------------------------------------------+ |
| 371 |
| ``findall()`` | Find all substrings where the RE matches, and | |
| 372 |
| | return them as a list. | |
| 373 |
+------------------+-----------------------------------------------+ |
| 374 |
| ``finditer()`` | Find all substrings where the RE matches, and | |
| 375 |
| | return them as an :term:`iterator`. | |
| 376 |
+------------------+-----------------------------------------------+ |
| 377 |
|
| 378 |
:meth:`~re.Pattern.match` and :meth:`~re.Pattern.search` return ``None`` if no match can be found. If |
| 379 |
they're successful, a :ref:`match object <match-objects>` instance is returned, |
| 380 |
containing information about the match: where it starts and ends, the substring |
| 381 |
it matched, and more. |
| 382 |
|
| 383 |
You can learn about this by interactively experimenting with the :mod:`re` |
| 384 |
module. |
| 385 |
|
| 386 |
This HOWTO uses the standard Python interpreter for its examples. First, run the |
| 387 |
Python interpreter, import the :mod:`re` module, and compile a RE:: |
| 388 |
|
| 389 |
>>> import re |
| 390 |
>>> p = re.compile('[a-z]+') |
| 391 |
>>> p |
| 392 |
re.compile('[a-z]+') |
| 393 |
|
| 394 |
Now, you can try matching various strings against the RE ``[a-z]+``. An empty |
| 395 |
string shouldn't match at all, since ``+`` means 'one or more repetitions'. |
| 396 |
:meth:`~re.Pattern.match` should return ``None`` in this case, which will cause the |
| 397 |
interpreter to print no output. You can explicitly print the result of |
| 398 |
:meth:`!match` to make this clear. :: |
| 399 |
|
| 400 |
>>> p.match("") |
| 401 |
>>> print(p.match("")) |
| 402 |
None |
| 403 |
|
| 404 |
Now, let's try it on a string that it should match, such as ``tempo``. In this |
| 405 |
case, :meth:`~re.Pattern.match` will return a :ref:`match object <match-objects>`, so you |
| 406 |
should store the result in a variable for later use. :: |
| 407 |
|
| 408 |
>>> m = p.match('tempo') |
| 409 |
>>> m |
| 410 |
<re.Match object; span=(0, 5), match='tempo'> |
| 411 |
|
| 412 |
Now you can query the :ref:`match object <match-objects>` for information |
| 413 |
about the matching string. Match object instances |
| 414 |
also have several methods and attributes; the most important ones are: |
| 415 |
|
| 416 |
+------------------+--------------------------------------------+ |
| 417 |
| Method/Attribute | Purpose | |
| 418 |
+==================+============================================+ |
| 419 |
| ``group()`` | Return the string matched by the RE | |
| 420 |
+------------------+--------------------------------------------+ |
| 421 |
| ``start()`` | Return the starting position of the match | |
| 422 |
+------------------+--------------------------------------------+ |
| 423 |
| ``end()`` | Return the ending position of the match | |
| 424 |
+------------------+--------------------------------------------+ |
| 425 |
| ``span()`` | Return a tuple containing the (start, end) | |
| 426 |
| | positions of the match | |
| 427 |
+------------------+--------------------------------------------+ |
| 428 |
|
| 429 |
Trying these methods will soon clarify their meaning:: |
| 430 |
|
| 431 |
>>> m.group() |
| 432 |
'tempo' |
| 433 |
>>> m.start(), m.end() |
| 434 |
(0, 5) |
| 435 |
>>> m.span() |
| 436 |
(0, 5) |
| 437 |
|
| 438 |
:meth:`~re.Match.group` returns the substring that was matched by the RE. :meth:`~re.Match.start` |
| 439 |
and :meth:`~re.Match.end` return the starting and ending index of the match. :meth:`~re.Match.span` |
| 440 |
returns both start and end indexes in a single tuple. Since the :meth:`~re.Pattern.match` |
| 441 |
method only checks if the RE matches at the start of a string, :meth:`!start` |
| 442 |
will always be zero. However, the :meth:`~re.Pattern.search` method of patterns |
| 443 |
scans through the string, so the match may not start at zero in that |
| 444 |
case. :: |
| 445 |
|
| 446 |
>>> print(p.match('::: message')) |
| 447 |
None |
| 448 |
>>> m = p.search('::: message'); print(m) |
| 449 |
<re.Match object; span=(4, 11), match='message'> |
| 450 |
>>> m.group() |
| 451 |
'message' |
| 452 |
>>> m.span() |
| 453 |
(4, 11) |
| 454 |
|
| 455 |
In actual programs, the most common style is to store the |
| 456 |
:ref:`match object <match-objects>` in a variable, and then check if it was |
| 457 |
``None``. This usually looks like:: |
| 458 |
|
| 459 |
p = re.compile( ... ) |
| 460 |
m = p.match( 'string goes here' ) |
| 461 |
if m: |
| 462 |
print('Match found: ', m.group()) |
| 463 |
else: |
| 464 |
print('No match') |
| 465 |
|
| 466 |
Two pattern methods return all of the matches for a pattern. |
| 467 |
:meth:`~re.Pattern.findall` returns a list of matching strings:: |
| 468 |
|
| 469 |
>>> p = re.compile(r'\d+') |
| 470 |
>>> p.findall('12 drummers drumming, 11 pipers piping, 10 lords a-leaping') |
| 471 |
['12', '11', '10'] |
| 472 |
|
| 473 |
The ``r`` prefix, making the literal a raw string literal, is needed in this |
| 474 |
example because escape sequences in a normal "cooked" string literal that are |
| 475 |
not recognized by Python, as opposed to regular expressions, now result in a |
| 476 |
:exc:`SyntaxWarning` and will eventually become a :exc:`SyntaxError`. See |
| 477 |
:ref:`the-backslash-plague`. |
| 478 |
|
| 479 |
:meth:`~re.Pattern.findall` has to create the entire list before it can be returned as the |
| 480 |
result. The :meth:`~re.Pattern.finditer` method returns a sequence of |
| 481 |
:ref:`match object <match-objects>` instances as an :term:`iterator`:: |
| 482 |
|
| 483 |
>>> iterator = p.finditer('12 drummers drumming, 11 ... 10 ...') |
| 484 |
>>> iterator #doctest: +ELLIPSIS |
| 485 |
<callable_iterator object at 0x...> |
| 486 |
>>> for match in iterator: |
| 487 |
... print(match.span()) |
| 488 |
... |
| 489 |
(0, 2) |
| 490 |
(22, 24) |
| 491 |
(29, 31) |
| 492 |
|
| 493 |
|
| 494 |
Module-level functions |
| 495 |
---------------------- |
| 496 |
|
| 497 |
You don't have to create a pattern object and call its methods; the |
| 498 |
:mod:`re` module also provides top-level functions called :func:`~re.match`, |
| 499 |
:func:`~re.search`, :func:`~re.findall`, :func:`~re.sub`, and so forth. These functions |
| 500 |
take the same arguments as the corresponding pattern method with |
| 501 |
the RE string added as the first argument, and still return either ``None`` or a |
| 502 |
:ref:`match object <match-objects>` instance. :: |
| 503 |
|
| 504 |
>>> print(re.match(r'From\s+', 'Fromage amk')) |
| 505 |
None |
| 506 |
>>> re.match(r'From\s+', 'From amk Thu May 14 19:12:10 1998') #doctest: +ELLIPSIS |
| 507 |
<re.Match object; span=(0, 5), match='From '> |
| 508 |
|
| 509 |
Under the hood, these functions simply create a pattern object for you |
| 510 |
and call the appropriate method on it. They also store the compiled |
| 511 |
object in a cache, so future calls using the same RE won't need to |
| 512 |
parse the pattern again and again. |
| 513 |
|
| 514 |
Should you use these module-level functions, or should you get the |
| 515 |
pattern and call its methods yourself? If you're accessing a regex |
| 516 |
within a loop, pre-compiling it will save a few function calls. |
| 517 |
Outside of loops, there's not much difference thanks to the internal |
| 518 |
cache. |
| 519 |
|
| 520 |
|
| 521 |
Compilation flags |
| 522 |
----------------- |
| 523 |
|
| 524 |
.. currentmodule:: re |
| 525 |
|
| 526 |
Compilation flags let you modify some aspects of how regular expressions work. |
| 527 |
Flags are available in the :mod:`re` module under two names, a long name such as |
| 528 |
:const:`IGNORECASE` and a short, one-letter form such as :const:`I`. (If you're |
| 529 |
familiar with Perl's pattern modifiers, the one-letter forms use the same |
| 530 |
letters; the short form of :const:`re.VERBOSE` is :const:`re.X`, for example.) |
| 531 |
Multiple flags can be specified by bitwise OR-ing them; ``re.I | re.M`` sets |
| 532 |
both the :const:`I` and :const:`M` flags, for example. |
| 533 |
|
| 534 |
Here's a table of the available flags, followed by a more detailed explanation |
| 535 |
of each one. |
| 536 |
|
| 537 |
+---------------------------------+--------------------------------------------+ |
| 538 |
| Flag | Meaning | |
| 539 |
+=================================+============================================+ |
| 540 |
| :const:`ASCII`, :const:`A` | Makes several escapes like ``\w``, ``\b``, | |
| 541 |
| | ``\s`` and ``\d`` match only on ASCII | |
| 542 |
| | characters with the respective property. | |
| 543 |
+---------------------------------+--------------------------------------------+ |
| 544 |
| :const:`DOTALL`, :const:`S` | Make ``.`` match any character, including | |
| 545 |
| | newlines. | |
| 546 |
+---------------------------------+--------------------------------------------+ |
| 547 |
| :const:`IGNORECASE`, :const:`I` | Do case-insensitive matches. | |
| 548 |
+---------------------------------+--------------------------------------------+ |
| 549 |
| :const:`LOCALE`, :const:`L` | Do a locale-aware match. | |
| 550 |
+---------------------------------+--------------------------------------------+ |
| 551 |
| :const:`MULTILINE`, :const:`M` | Multi-line matching, affecting ``^`` and | |
| 552 |
| | ``$``. | |
| 553 |
+---------------------------------+--------------------------------------------+ |
| 554 |
| :const:`VERBOSE`, :const:`X` | Enable verbose REs, which can be organized | |
| 555 |
| (for 'extended') | more cleanly and understandably. | |
| 556 |
+---------------------------------+--------------------------------------------+ |
| 557 |
|
| 558 |
|
| 559 |
.. data:: I |
| 560 |
IGNORECASE |
| 561 |
:noindex: |
| 562 |
|
| 563 |
Perform case-insensitive matching; character class and literal strings will |
| 564 |
match letters by ignoring case. For example, ``[A-Z]`` will match lowercase |
| 565 |
letters, too. Full Unicode matching also works unless the :const:`ASCII` |
| 566 |
flag is used to disable non-ASCII matches. When the Unicode patterns |
| 567 |
``[a-z]`` or ``[A-Z]`` are used in combination with the :const:`IGNORECASE` |
| 568 |
flag, they will match the 52 ASCII letters and 4 additional non-ASCII |
| 569 |
letters: 'İ' (U+0130, Latin capital letter I with dot above), 'ı' (U+0131, |
| 570 |
Latin small letter dotless i), 'Å¿' (U+017F, Latin small letter long s) and |
| 571 |
'K' (U+212A, Kelvin sign). ``Spam`` will match ``'Spam'``, ``'spam'``, |
| 572 |
``'spAM'``, or ``'ſpam'`` (the latter is matched only in Unicode mode). |
| 573 |
This lowercasing doesn't take the current locale into account; |
| 574 |
it will if you also set the :const:`LOCALE` flag. |
| 575 |
|
| 576 |
|
| 577 |
.. data:: L |
| 578 |
LOCALE |
| 579 |
:noindex: |
| 580 |
|
| 581 |
Make ``\w``, ``\W``, ``\b``, ``\B`` and case-insensitive matching dependent |
| 582 |
on the current locale instead of the Unicode database. |
| 583 |
|
| 584 |
Locales are a feature of the C library intended to help in writing programs |
| 585 |
that take account of language differences. For example, if you're |
| 586 |
processing encoded French text, you'd want to be able to write ``\w+`` to |
| 587 |
match words, but ``\w`` only matches the character class ``[A-Za-z]`` in |
| 588 |
bytes patterns; it won't match bytes corresponding to ``é`` or ``ç``. |
| 589 |
If your system is configured properly and a French locale is selected, |
| 590 |
certain C functions will tell the program that the byte corresponding to |
| 591 |
``é`` should also be considered a letter. |
| 592 |
Setting the :const:`LOCALE` flag when compiling a regular expression will cause |
| 593 |
the resulting compiled object to use these C functions for ``\w``; this is |
| 594 |
slower, but also enables ``\w+`` to match French words as you'd expect. |
| 595 |
The use of this flag is discouraged in Python 3 as the locale mechanism |
| 596 |
is very unreliable, it only handles one "culture" at a time, and it only |
| 597 |
works with 8-bit locales. Unicode matching is already enabled by default |
| 598 |
in Python 3 for Unicode (str) patterns, and it is able to handle different |
| 599 |
locales/languages. |
| 600 |
|
| 601 |
|
| 602 |
.. data:: M |
| 603 |
MULTILINE |
| 604 |
:noindex: |
| 605 |
|
| 606 |
(``^`` and ``$`` haven't been explained yet; they'll be introduced in section |
| 607 |
:ref:`more-metacharacters`.) |
| 608 |
|
| 609 |
Usually ``^`` matches only at the beginning of the string, and ``$`` matches |
| 610 |
only at the end of the string and immediately before the newline (if any) at the |
| 611 |
end of the string. When this flag is specified, ``^`` matches at the beginning |
| 612 |
of the string and at the beginning of each line within the string, immediately |
| 613 |
following each newline. Similarly, the ``$`` metacharacter matches either at |
| 614 |
the end of the string and at the end of each line (immediately preceding each |
| 615 |
newline). |
| 616 |
|
| 617 |
|
| 618 |
.. data:: S |
| 619 |
DOTALL |
| 620 |
:noindex: |
| 621 |
|
| 622 |
Makes the ``'.'`` special character match any character at all, including a |
| 623 |
newline; without this flag, ``'.'`` will match anything *except* a newline. |
| 624 |
|
| 625 |
|
| 626 |
.. data:: A |
| 627 |
ASCII |
| 628 |
:noindex: |
| 629 |
|
| 630 |
Make ``\w``, ``\W``, ``\b``, ``\B``, ``\s`` and ``\S`` perform ASCII-only |
| 631 |
matching instead of full Unicode matching. This is only meaningful for |
| 632 |
Unicode patterns, and is ignored for byte patterns. |
| 633 |
|
| 634 |
|
| 635 |
.. data:: X |
| 636 |
VERBOSE |
| 637 |
:noindex: |
| 638 |
|
| 639 |
This flag allows you to write regular expressions that are more readable by |
| 640 |
granting you more flexibility in how you can format them. When this flag has |
| 641 |
been specified, whitespace within the RE string is ignored, except when the |
| 642 |
whitespace is in a character class or preceded by an unescaped backslash; this |
| 643 |
lets you organize and indent the RE more clearly. This flag also lets you put |
| 644 |
comments within a RE that will be ignored by the engine; comments are marked by |
| 645 |
a ``'#'`` that's neither in a character class nor preceded by an unescaped |
| 646 |
backslash. |
| 647 |
|
| 648 |
For example, here's a RE that uses :const:`re.VERBOSE`; see how much easier it |
| 649 |
is to read? :: |
| 650 |
|
| 651 |
charref = re.compile(r""" |
| 652 |
&[#] # Start of a numeric entity reference |
| 653 |
( |
| 654 |
0[0-7]+ # Octal form |
| 655 |
| [0-9]+ # Decimal form |
| 656 |
| x[0-9a-fA-F]+ # Hexadecimal form |
| 657 |
) |
| 658 |
; # Trailing semicolon |
| 659 |
""", re.VERBOSE) |
| 660 |
|
| 661 |
Without the verbose setting, the RE would look like this:: |
| 662 |
|
| 663 |
charref = re.compile("&#(0[0-7]+" |
| 664 |
"|[0-9]+" |
| 665 |
"|x[0-9a-fA-F]+);") |
| 666 |
|
| 667 |
In the above example, Python's automatic concatenation of string literals has |
| 668 |
been used to break up the RE into smaller pieces, but it's still more difficult |
| 669 |
to understand than the version using :const:`re.VERBOSE`. |
| 670 |
|
| 671 |
|
| 672 |
More pattern power |
| 673 |
================== |
| 674 |
|
| 675 |
So far we've only covered a part of the features of regular expressions. In |
| 676 |
this section, we'll cover some new metacharacters, and how to use groups to |
| 677 |
retrieve portions of the text that was matched. |
| 678 |
|
| 679 |
|
| 680 |
.. _more-metacharacters: |
| 681 |
|
| 682 |
More metacharacters |
| 683 |
------------------- |
| 684 |
|
| 685 |
There are some metacharacters that we haven't covered yet. Most of them will be |
| 686 |
covered in this section. |
| 687 |
|
| 688 |
Some of the remaining metacharacters to be discussed are :dfn:`zero-width |
| 689 |
assertions`. They don't cause the engine to advance through the string; |
| 690 |
instead, they consume no characters at all, and simply succeed or fail. For |
| 691 |
example, ``\b`` is an assertion that the current position is located at a word |
| 692 |
boundary; the position isn't changed by the ``\b`` at all. This means that |
| 693 |
zero-width assertions should never be repeated, because if they match once at a |
| 694 |
given location, they can obviously be matched an infinite number of times. |
| 695 |
|
| 696 |
``|`` |
| 697 |
Alternation, or the "or" operator. If *A* and *B* are regular expressions, |
| 698 |
``A|B`` will match any string that matches either *A* or *B*. ``|`` has very |
| 699 |
low precedence in order to make it work reasonably when you're alternating |
| 700 |
multi-character strings. ``Crow|Servo`` will match either ``'Crow'`` or ``'Servo'``, |
| 701 |
not ``'Cro'``, a ``'w'`` or an ``'S'``, and ``'ervo'``. |
| 702 |
|
| 703 |
To match a literal ``'|'``, use ``\|``, or enclose it inside a character class, |
| 704 |
as in ``[|]``. |
| 705 |
|
| 706 |
``^`` |
| 707 |
Matches at the beginning of lines. Unless the :const:`MULTILINE` flag has been |
| 708 |
set, this will only match at the beginning of the string. In :const:`MULTILINE` |
| 709 |
mode, this also matches immediately after each newline within the string. |
| 710 |
|
| 711 |
For example, if you wish to match the word ``From`` only at the beginning of a |
| 712 |
line, the RE to use is ``^From``. :: |
| 713 |
|
| 714 |
>>> print(re.search('^From', 'From Here to Eternity')) #doctest: +ELLIPSIS |
| 715 |
<re.Match object; span=(0, 4), match='From'> |
| 716 |
>>> print(re.search('^From', 'Reciting From Memory')) |
| 717 |
None |
| 718 |
|
| 719 |
To match a literal ``'^'``, use ``\^``. |
| 720 |
|
| 721 |
``$`` |
| 722 |
Matches at the end of a line, which is defined as either the end of the string, |
| 723 |
or any location followed by a newline character. :: |
| 724 |
|
| 725 |
>>> print(re.search('}$', '{block}')) #doctest: +ELLIPSIS |
| 726 |
<re.Match object; span=(6, 7), match='}'> |
| 727 |
>>> print(re.search('}$', '{block} ')) |
| 728 |
None |
| 729 |
>>> print(re.search('}$', '{block}\n')) #doctest: +ELLIPSIS |
| 730 |
<re.Match object; span=(6, 7), match='}'> |
| 731 |
|
| 732 |
To match a literal ``'$'``, use ``\$`` or enclose it inside a character class, |
| 733 |
as in ``[$]``. |
| 734 |
|
| 735 |
``\A`` |
| 736 |
Matches only at the start of the string. When not in :const:`MULTILINE` mode, |
| 737 |
``\A`` and ``^`` are effectively the same. In :const:`MULTILINE` mode, they're |
| 738 |
different: ``\A`` still matches only at the beginning of the string, but ``^`` |
| 739 |
may match at any location inside the string that follows a newline character. |
| 740 |
|
| 741 |
``\z`` |
| 742 |
Matches only at the end of the string. |
| 743 |
|
| 744 |
``\Z`` |
| 745 |
The same as ``\z``. For compatibility with old Python versions. |
| 746 |
|
| 747 |
``\b`` |
| 748 |
Word boundary. This is a zero-width assertion that matches only at the |
| 749 |
beginning or end of a word. A word is defined as a sequence of alphanumeric |
| 750 |
characters, so the end of a word is indicated by whitespace or a |
| 751 |
non-alphanumeric character. |
| 752 |
|
| 753 |
The following example matches ``class`` only when it's a complete word; it won't |
| 754 |
match when it's contained inside another word. :: |
| 755 |
|
| 756 |
>>> p = re.compile(r'\bclass\b') |
| 757 |
>>> print(p.search('no class at all')) |
| 758 |
<re.Match object; span=(3, 8), match='class'> |
| 759 |
>>> print(p.search('the declassified algorithm')) |
| 760 |
None |
| 761 |
>>> print(p.search('one subclass is')) |
| 762 |
None |
| 763 |
|
| 764 |
There are two subtleties you should remember when using this special sequence. |
| 765 |
First, this is the worst collision between Python's string literals and regular |
| 766 |
expression sequences. In Python's string literals, ``\b`` is the backspace |
| 767 |
character, ASCII value 8. If you're not using raw strings, then Python will |
| 768 |
convert the ``\b`` to a backspace, and your RE won't match as you expect it to. |
| 769 |
The following example looks the same as our previous RE, but omits the ``'r'`` |
| 770 |
in front of the RE string. :: |
| 771 |
|
| 772 |
>>> p = re.compile('\bclass\b') |
| 773 |
>>> print(p.search('no class at all')) |
| 774 |
None |
| 775 |
>>> print(p.search('\b' + 'class' + '\b')) |
| 776 |
<re.Match object; span=(0, 7), match='\x08class\x08'> |
| 777 |
|
| 778 |
Second, inside a character class, where there's no use for this assertion, |
| 779 |
``\b`` represents the backspace character, for compatibility with Python's |
| 780 |
string literals. |
| 781 |
|
| 782 |
``\B`` |
| 783 |
Another zero-width assertion, this is the opposite of ``\b``, only matching when |
| 784 |
the current position is not at a word boundary. |
| 785 |
|
| 786 |
|
| 787 |
Grouping |
| 788 |
-------- |
| 789 |
|
| 790 |
Frequently you need to obtain more information than just whether the RE matched |
| 791 |
or not. Regular expressions are often used to dissect strings by writing a RE |
| 792 |
divided into several subgroups which match different components of interest. |
| 793 |
For example, an RFC-822 header line is divided into a header name and a value, |
| 794 |
separated by a ``':'``, like this: |
| 795 |
|
| 796 |
.. code-block:: none |
| 797 |
|
| 798 |
From: [email protected] |
| 799 |
User-Agent: Thunderbird 1.5.0.9 (X11/20061227) |
| 800 |
MIME-Version: 1.0 |
| 801 |
To: [email protected] |
| 802 |
|
| 803 |
This can be handled by writing a regular expression which matches an entire |
| 804 |
header line, and has one group which matches the header name, and another group |
| 805 |
which matches the header's value. |
| 806 |
|
| 807 |
Groups are marked by the ``'('``, ``')'`` metacharacters. ``'('`` and ``')'`` |
| 808 |
have much the same meaning as they do in mathematical expressions; they group |
| 809 |
together the expressions contained inside them, and you can repeat the contents |
| 810 |
of a group with a quantifier, such as ``*``, ``+``, ``?``, or |
| 811 |
``{m,n}``. For example, ``(ab)*`` will match zero or more repetitions of |
| 812 |
``ab``. :: |
| 813 |
|
| 814 |
>>> p = re.compile('(ab)*') |
| 815 |
>>> print(p.match('ababababab').span()) |
| 816 |
(0, 10) |
| 817 |
|
| 818 |
Groups indicated with ``'('``, ``')'`` also capture the starting and ending |
| 819 |
index of the text that they match; this can be retrieved by passing an argument |
| 820 |
to :meth:`~re.Match.group`, :meth:`~re.Match.start`, :meth:`~re.Match.end`, and |
| 821 |
:meth:`~re.Match.span`. Groups are |
| 822 |
numbered starting with 0. Group 0 is always present; it's the whole RE, so |
| 823 |
:ref:`match object <match-objects>` methods all have group 0 as their default |
| 824 |
argument. Later we'll see how to express groups that don't capture the span |
| 825 |
of text that they match. :: |
| 826 |
|
| 827 |
>>> p = re.compile('(a)b') |
| 828 |
>>> m = p.match('ab') |
| 829 |
>>> m.group() |
| 830 |
'ab' |
| 831 |
>>> m.group(0) |
| 832 |
'ab' |
| 833 |
|
| 834 |
Subgroups are numbered from left to right, from 1 upward. Groups can be nested; |
| 835 |
to determine the number, just count the opening parenthesis characters, going |
| 836 |
from left to right. :: |
| 837 |
|
| 838 |
>>> p = re.compile('(a(b)c)d') |
| 839 |
>>> m = p.match('abcd') |
| 840 |
>>> m.group(0) |
| 841 |
'abcd' |
| 842 |
>>> m.group(1) |
| 843 |
'abc' |
| 844 |
>>> m.group(2) |
| 845 |
'b' |
| 846 |
|
| 847 |
:meth:`~re.Match.group` can be passed multiple group numbers at a time, in which case it |
| 848 |
will return a tuple containing the corresponding values for those groups. :: |
| 849 |
|
| 850 |
>>> m.group(2,1,2) |
| 851 |
('b', 'abc', 'b') |
| 852 |
|
| 853 |
The :meth:`~re.Match.groups` method returns a tuple containing the strings for all the |
| 854 |
subgroups, from 1 up to however many there are. :: |
| 855 |
|
| 856 |
>>> m.groups() |
| 857 |
('abc', 'b') |
| 858 |
|
| 859 |
Backreferences in a pattern allow you to specify that the contents of an earlier |
| 860 |
capturing group must also be found at the current location in the string. For |
| 861 |
example, ``\1`` will succeed if the exact contents of group 1 can be found at |
| 862 |
the current position, and fails otherwise. Remember that Python's string |
| 863 |
literals also use a backslash followed by numbers to allow including arbitrary |
| 864 |
characters in a string, so be sure to use a raw string when incorporating |
| 865 |
backreferences in a RE. |
| 866 |
|
| 867 |
For example, the following RE detects doubled words in a string. :: |
| 868 |
|
| 869 |
>>> p = re.compile(r'\b(\w+)\s+\1\b') |
| 870 |
>>> p.search('Paris in the the spring').group() |
| 871 |
'the the' |
| 872 |
|
| 873 |
Backreferences like this aren't often useful for just searching through a string |
| 874 |
--- there are few text formats which repeat data in this way --- but you'll soon |
| 875 |
find out that they're *very* useful when performing string substitutions. |
| 876 |
|
| 877 |
|
| 878 |
Non-capturing and named groups |
| 879 |
------------------------------ |
| 880 |
|
| 881 |
Elaborate REs may use many groups, both to capture substrings of interest, and |
| 882 |
to group and structure the RE itself. In complex REs, it becomes difficult to |
| 883 |
keep track of the group numbers. There are two features which help with this |
| 884 |
problem. Both of them use a common syntax for regular expression extensions, so |
| 885 |
we'll look at that first. |
| 886 |
|
| 887 |
Perl 5 is well known for its powerful additions to standard regular expressions. |
| 888 |
For these new features the Perl developers couldn't choose new single-keystroke metacharacters |
| 889 |
or new special sequences beginning with ``\`` without making Perl's regular |
| 890 |
expressions confusingly different from standard REs. If they chose ``&`` as a |
| 891 |
new metacharacter, for example, old expressions would be assuming that ``&`` was |
| 892 |
a regular character and wouldn't have escaped it by writing ``\&`` or ``[&]``. |
| 893 |
|
| 894 |
The solution chosen by the Perl developers was to use ``(?...)`` as the |
| 895 |
extension syntax. ``?`` immediately after a parenthesis was a syntax error |
| 896 |
because the ``?`` would have nothing to repeat, so this didn't introduce any |
| 897 |
compatibility problems. The characters immediately after the ``?`` indicate |
| 898 |
what extension is being used, so ``(?=foo)`` is one thing (a positive lookahead |
| 899 |
assertion) and ``(?:foo)`` is something else (a non-capturing group containing |
| 900 |
the subexpression ``foo``). |
| 901 |
|
| 902 |
Python supports several of Perl's extensions and adds an extension |
| 903 |
syntax to Perl's extension syntax. If the first character after the |
| 904 |
question mark is a ``P``, you know that it's an extension that's |
| 905 |
specific to Python. |
| 906 |
|
| 907 |
Now that we've looked at the general extension syntax, we can return |
| 908 |
to the features that simplify working with groups in complex REs. |
| 909 |
|
| 910 |
Sometimes you'll want to use a group to denote a part of a regular expression, |
| 911 |
but aren't interested in retrieving the group's contents. You can make this fact |
| 912 |
explicit by using a non-capturing group: ``(?:...)``, where you can replace the |
| 913 |
``...`` with any other regular expression. :: |
| 914 |
|
| 915 |
>>> m = re.match("([abc])+", "abc") |
| 916 |
>>> m.groups() |
| 917 |
('c',) |
| 918 |
>>> m = re.match("(?:[abc])+", "abc") |
| 919 |
>>> m.groups() |
| 920 |
() |
| 921 |
|
| 922 |
Except for the fact that you can't retrieve the contents of what the group |
| 923 |
matched, a non-capturing group behaves exactly the same as a capturing group; |
| 924 |
you can put anything inside it, repeat it with a repetition metacharacter such |
| 925 |
as ``*``, and nest it within other groups (capturing or non-capturing). |
| 926 |
``(?:...)`` is particularly useful when modifying an existing pattern, since you |
| 927 |
can add new groups without changing how all the other groups are numbered. It |
| 928 |
should be mentioned that there's no performance difference in searching between |
| 929 |
capturing and non-capturing groups; neither form is any faster than the other. |
| 930 |
|
| 931 |
A more significant feature is named groups: instead of referring to them by |
| 932 |
numbers, groups can be referenced by a name. |
| 933 |
|
| 934 |
The syntax for a named group is one of the Python-specific extensions: |
| 935 |
``(?P<name>...)``. *name* is, obviously, the name of the group. Named groups |
| 936 |
behave exactly like capturing groups, and additionally associate a name |
| 937 |
with a group. The :ref:`match object <match-objects>` methods that deal with |
| 938 |
capturing groups all accept either integers that refer to the group by number |
| 939 |
or strings that contain the desired group's name. Named groups are still |
| 940 |
given numbers, so you can retrieve information about a group in two ways:: |
| 941 |
|
| 942 |
>>> p = re.compile(r'(?P<word>\b\w+\b)') |
| 943 |
>>> m = p.search( '(((( Lots of punctuation )))' ) |
| 944 |
>>> m.group('word') |
| 945 |
'Lots' |
| 946 |
>>> m.group(1) |
| 947 |
'Lots' |
| 948 |
|
| 949 |
Additionally, you can retrieve named groups as a dictionary with |
| 950 |
:meth:`~re.Match.groupdict`:: |
| 951 |
|
| 952 |
>>> m = re.match(r'(?P<first>\w+) (?P<last>\w+)', 'Jane Doe') |
| 953 |
>>> m.groupdict() |
| 954 |
{'first': 'Jane', 'last': 'Doe'} |
| 955 |
|
| 956 |
Named groups are handy because they let you use easily remembered names, instead |
| 957 |
of having to remember numbers. Here's an example RE from the :mod:`imaplib` |
| 958 |
module:: |
| 959 |
|
| 960 |
InternalDate = re.compile(r'INTERNALDATE "' |
| 961 |
r'(?P<day>[ 123][0-9])-(?P<mon>[A-Z][a-z][a-z])-' |
| 962 |
r'(?P<year>[0-9][0-9][0-9][0-9])' |
| 963 |
r' (?P<hour>[0-9][0-9]):(?P<min>[0-9][0-9]):(?P<sec>[0-9][0-9])' |
| 964 |
r' (?P<zonen>[-+])(?P<zoneh>[0-9][0-9])(?P<zonem>[0-9][0-9])' |
| 965 |
r'"') |
| 966 |
|
| 967 |
It's obviously much easier to retrieve ``m.group('zonem')``, instead of having |
| 968 |
to remember to retrieve group 9. |
| 969 |
|
| 970 |
The syntax for backreferences in an expression such as ``(...)\1`` refers to the |
| 971 |
number of the group. There's naturally a variant that uses the group name |
| 972 |
instead of the number. This is another Python extension: ``(?P=name)`` indicates |
| 973 |
that the contents of the group called *name* should again be matched at the |
| 974 |
current point. The regular expression for finding doubled words, |
| 975 |
``\b(\w+)\s+\1\b`` can also be written as ``\b(?P<word>\w+)\s+(?P=word)\b``:: |
| 976 |
|
| 977 |
>>> p = re.compile(r'\b(?P<word>\w+)\s+(?P=word)\b') |
| 978 |
>>> p.search('Paris in the the spring').group() |
| 979 |
'the the' |
| 980 |
|
| 981 |
|
| 982 |
Lookahead assertions |
| 983 |
-------------------- |
| 984 |
|
| 985 |
Another zero-width assertion is the lookahead assertion. Lookahead assertions |
| 986 |
are available in both positive and negative form, and look like this: |
| 987 |
|
| 988 |
``(?=...)`` |
| 989 |
Positive lookahead assertion. This succeeds if the contained regular |
| 990 |
expression, represented here by ``...``, successfully matches at the current |
| 991 |
location, and fails otherwise. But, once the contained expression has been |
| 992 |
tried, the matching engine doesn't advance at all; the rest of the pattern is |
| 993 |
tried right where the assertion started. |
| 994 |
|
| 995 |
``(?!...)`` |
| 996 |
Negative lookahead assertion. This is the opposite of the positive assertion; |
| 997 |
it succeeds if the contained expression *doesn't* match at the current position |
| 998 |
in the string. |
| 999 |
|
| 1000 |
To make this concrete, let's look at a case where a lookahead is useful. |
| 1001 |
Consider a simple pattern to match a filename and split it apart into a base |
| 1002 |
name and an extension, separated by a ``.``. For example, in ``news.rc``, |
| 1003 |
``news`` is the base name, and ``rc`` is the filename's extension. |
| 1004 |
|
| 1005 |
The pattern to match this is quite simple: |
| 1006 |
|
| 1007 |
``.*[.].*$`` |
| 1008 |
|
| 1009 |
Notice that the ``.`` needs to be treated specially because it's a |
| 1010 |
metacharacter, so it's inside a character class to only match that |
| 1011 |
specific character. Also notice the trailing ``$``; this is added to |
| 1012 |
ensure that all the rest of the string must be included in the |
| 1013 |
extension. This regular expression matches ``foo.bar`` and |
| 1014 |
``autoexec.bat`` and ``sendmail.cf`` and ``printers.conf``. |
| 1015 |
|
| 1016 |
Now, consider complicating the problem a bit; what if you want to match |
| 1017 |
filenames where the extension is not ``bat``? Some incorrect attempts: |
| 1018 |
|
| 1019 |
``.*[.][^b].*$`` |
| 1020 |
|
| 1021 |
The first attempt above tries to exclude ``bat`` by requiring |
| 1022 |
that the first character of the extension is not a ``b``. This is wrong, |
| 1023 |
because the pattern also doesn't match ``foo.bar``. |
| 1024 |
|
| 1025 |
``.*[.]([^b]..|.[^a].|..[^t])$`` |
| 1026 |
|
| 1027 |
The expression gets messier when you try to patch up the first solution by |
| 1028 |
requiring one of the following cases to match: the first character of the |
| 1029 |
extension isn't ``b``; the second character isn't ``a``; or the third character |
| 1030 |
isn't ``t``. This accepts ``foo.bar`` and rejects ``autoexec.bat``, but it |
| 1031 |
requires a three-letter extension and won't accept a filename with a two-letter |
| 1032 |
extension such as ``sendmail.cf``. We'll complicate the pattern again in an |
| 1033 |
effort to fix it. |
| 1034 |
|
| 1035 |
``.*[.]([^b].?.?|.[^a]?.?|..?[^t]?)$`` |
| 1036 |
|
| 1037 |
In the third attempt, the second and third letters are all made optional in |
| 1038 |
order to allow matching extensions shorter than three characters, such as |
| 1039 |
``sendmail.cf``. |
| 1040 |
|
| 1041 |
The pattern's getting really complicated now, which makes it hard to read and |
| 1042 |
understand. Worse, if the problem changes and you want to exclude both ``bat`` |
| 1043 |
and ``exe`` as extensions, the pattern would get even more complicated and |
| 1044 |
confusing. |
| 1045 |
|
| 1046 |
A negative lookahead cuts through all this confusion: |
| 1047 |
|
| 1048 |
``.*[.](?!bat$)[^.]*$`` |
| 1049 |
|
| 1050 |
The negative lookahead means: if the expression ``bat`` |
| 1051 |
doesn't match at this point, try the rest of the pattern; if ``bat$`` does |
| 1052 |
match, the whole pattern will fail. The trailing ``$`` is required to ensure |
| 1053 |
that something like ``sample.batch``, where the extension only starts with |
| 1054 |
``bat``, will be allowed. The ``[^.]*`` makes sure that the pattern works |
| 1055 |
when there are multiple dots in the filename. |
| 1056 |
|
| 1057 |
Excluding another filename extension is now easy; simply add it as an |
| 1058 |
alternative inside the assertion. The following pattern excludes filenames that |
| 1059 |
end in either ``bat`` or ``exe``: |
| 1060 |
|
| 1061 |
``.*[.](?!bat$|exe$)[^.]*$`` |
| 1062 |
|
| 1063 |
|
| 1064 |
Modifying strings |
| 1065 |
================= |
| 1066 |
|
| 1067 |
Up to this point, we've simply performed searches against a static string. |
| 1068 |
Regular expressions are also commonly used to modify strings in various ways, |
| 1069 |
using the following pattern methods: |
| 1070 |
|
| 1071 |
+------------------+-----------------------------------------------+ |
| 1072 |
| Method/Attribute | Purpose | |
| 1073 |
+==================+===============================================+ |
| 1074 |
| ``split()`` | Split the string into a list, splitting it | |
| 1075 |
| | wherever the RE matches | |
| 1076 |
+------------------+-----------------------------------------------+ |
| 1077 |
| ``sub()`` | Find all substrings where the RE matches, and | |
| 1078 |
| | replace them with a different string | |
| 1079 |
+------------------+-----------------------------------------------+ |
| 1080 |
| ``subn()`` | Does the same thing as :meth:`!sub`, but | |
| 1081 |
| | returns the new string and the number of | |
| 1082 |
| | replacements | |
| 1083 |
+------------------+-----------------------------------------------+ |
| 1084 |
|
| 1085 |
|
| 1086 |
Splitting strings |
| 1087 |
----------------- |
| 1088 |
|
| 1089 |
The :meth:`~re.Pattern.split` method of a pattern splits a string apart |
| 1090 |
wherever the RE matches, returning a list of the pieces. It's similar to the |
| 1091 |
:meth:`~str.split` method of strings but provides much more generality in the |
| 1092 |
delimiters that you can split by; string :meth:`!split` only supports splitting by |
| 1093 |
whitespace or by a fixed string. As you'd expect, there's a module-level |
| 1094 |
:func:`re.split` function, too. |
| 1095 |
|
| 1096 |
|
| 1097 |
.. method:: .split(string [, maxsplit=0]) |
| 1098 |
:noindex: |
| 1099 |
|
| 1100 |
Split *string* by the matches of the regular expression. If capturing |
| 1101 |
parentheses are used in the RE, then their contents will also be returned as |
| 1102 |
part of the resulting list. If *maxsplit* is nonzero, at most *maxsplit* splits |
| 1103 |
are performed. |
| 1104 |
|
| 1105 |
You can limit the number of splits made, by passing a value for *maxsplit*. |
| 1106 |
When *maxsplit* is nonzero, at most *maxsplit* splits will be made, and the |
| 1107 |
remainder of the string is returned as the final element of the list. In the |
| 1108 |
following example, the delimiter is any sequence of non-alphanumeric characters. |
| 1109 |
:: |
| 1110 |
|
| 1111 |
>>> p = re.compile(r'\W+') |
| 1112 |
>>> p.split('This is a test, short and sweet, of split().') |
| 1113 |
['This', 'is', 'a', 'test', 'short', 'and', 'sweet', 'of', 'split', ''] |
| 1114 |
>>> p.split('This is a test, short and sweet, of split().', 3) |
| 1115 |
['This', 'is', 'a', 'test, short and sweet, of split().'] |
| 1116 |
|
| 1117 |
Sometimes you're not only interested in what the text between delimiters is, but |
| 1118 |
also need to know what the delimiter was. If capturing parentheses are used in |
| 1119 |
the RE, then their values are also returned as part of the list. Compare the |
| 1120 |
following calls:: |
| 1121 |
|
| 1122 |
>>> p = re.compile(r'\W+') |
| 1123 |
>>> p2 = re.compile(r'(\W+)') |
| 1124 |
>>> p.split('This... is a test.') |
| 1125 |
['This', 'is', 'a', 'test', ''] |
| 1126 |
>>> p2.split('This... is a test.') |
| 1127 |
['This', '... ', 'is', ' ', 'a', ' ', 'test', '.', ''] |
| 1128 |
|
| 1129 |
The module-level function :func:`re.split` adds the RE to be used as the first |
| 1130 |
argument, but is otherwise the same. :: |
| 1131 |
|
| 1132 |
>>> re.split(r'[\W]+', 'Words, words, words.') |
| 1133 |
['Words', 'words', 'words', ''] |
| 1134 |
>>> re.split(r'([\W]+)', 'Words, words, words.') |
| 1135 |
['Words', ', ', 'words', ', ', 'words', '.', ''] |
| 1136 |
>>> re.split(r'[\W]+', 'Words, words, words.', 1) |
| 1137 |
['Words', 'words, words.'] |
| 1138 |
|
| 1139 |
|
| 1140 |
Search and replace |
| 1141 |
------------------ |
| 1142 |
|
| 1143 |
Another common task is to find all the matches for a pattern, and replace them |
| 1144 |
with a different string. The :meth:`~re.Pattern.sub` method takes a replacement value, |
| 1145 |
which can be either a string or a function, and the string to be processed. |
| 1146 |
|
| 1147 |
.. method:: .sub(replacement, string[, count=0]) |
| 1148 |
:noindex: |
| 1149 |
|
| 1150 |
Returns the string obtained by replacing the leftmost non-overlapping |
| 1151 |
occurrences of the RE in *string* by the replacement *replacement*. If the |
| 1152 |
pattern isn't found, *string* is returned unchanged. |
| 1153 |
|
| 1154 |
The optional argument *count* is the maximum number of pattern occurrences to be |
| 1155 |
replaced; *count* must be a non-negative integer. The default value of 0 means |
| 1156 |
to replace all occurrences. |
| 1157 |
|
| 1158 |
Here's a simple example of using the :meth:`~re.Pattern.sub` method. It replaces colour |
| 1159 |
names with the word ``colour``:: |
| 1160 |
|
| 1161 |
>>> p = re.compile('(blue|white|red)') |
| 1162 |
>>> p.sub('colour', 'blue socks and red shoes') |
| 1163 |
'colour socks and colour shoes' |
| 1164 |
>>> p.sub('colour', 'blue socks and red shoes', count=1) |
| 1165 |
'colour socks and red shoes' |
| 1166 |
|
| 1167 |
The :meth:`~re.Pattern.subn` method does the same work, but returns a 2-tuple containing the |
| 1168 |
new string value and the number of replacements that were performed:: |
| 1169 |
|
| 1170 |
>>> p = re.compile('(blue|white|red)') |
| 1171 |
>>> p.subn('colour', 'blue socks and red shoes') |
| 1172 |
('colour socks and colour shoes', 2) |
| 1173 |
>>> p.subn('colour', 'no colours at all') |
| 1174 |
('no colours at all', 0) |
| 1175 |
|
| 1176 |
Empty matches are replaced only when they're not adjacent to a previous empty match. |
| 1177 |
:: |
| 1178 |
|
| 1179 |
>>> p = re.compile('x*') |
| 1180 |
>>> p.sub('-', 'abxd') |
| 1181 |
'-a-b--d-' |
| 1182 |
|
| 1183 |
If *replacement* is a string, any backslash escapes in it are processed. That |
| 1184 |
is, ``\n`` is converted to a single newline character, ``\r`` is converted to a |
| 1185 |
carriage return, and so forth. Unknown escapes such as ``\&`` are left alone. |
| 1186 |
Backreferences, such as ``\6``, are replaced with the substring matched by the |
| 1187 |
corresponding group in the RE. This lets you incorporate portions of the |
| 1188 |
original text in the resulting replacement string. |
| 1189 |
|
| 1190 |
This example matches the word ``section`` followed by a string enclosed in |
| 1191 |
``{``, ``}``, and changes ``section`` to ``subsection``:: |
| 1192 |
|
| 1193 |
>>> p = re.compile('section{ ( [^}]* ) }', re.VERBOSE) |
| 1194 |
>>> p.sub(r'subsection{\1}','section{First} section{second}') |
| 1195 |
'subsection{First} subsection{second}' |
| 1196 |
|
| 1197 |
There's also a syntax for referring to named groups as defined by the |
| 1198 |
``(?P<name>...)`` syntax. ``\g<name>`` will use the substring matched by the |
| 1199 |
group named ``name``, and ``\g<number>`` uses the corresponding group number. |
| 1200 |
``\g<2>`` is therefore equivalent to ``\2``, but isn't ambiguous in a |
| 1201 |
replacement string such as ``\g<2>0``. (``\20`` would be interpreted as a |
| 1202 |
reference to group 20, not a reference to group 2 followed by the literal |
| 1203 |
character ``'0'``.) The following substitutions are all equivalent, but use all |
| 1204 |
three variations of the replacement string. :: |
| 1205 |
|
| 1206 |
>>> p = re.compile('section{ (?P<name> [^}]* ) }', re.VERBOSE) |
| 1207 |
>>> p.sub(r'subsection{\1}','section{First}') |
| 1208 |
'subsection{First}' |
| 1209 |
>>> p.sub(r'subsection{\g<1>}','section{First}') |
| 1210 |
'subsection{First}' |
| 1211 |
>>> p.sub(r'subsection{\g<name>}','section{First}') |
| 1212 |
'subsection{First}' |
| 1213 |
|
| 1214 |
*replacement* can also be a function, which gives you even more control. If |
| 1215 |
*replacement* is a function, the function is called for every non-overlapping |
| 1216 |
occurrence of *pattern*. On each call, the function is passed a |
| 1217 |
:ref:`match object <match-objects>` argument for the match and can use this |
| 1218 |
information to compute the desired replacement string and return it. |
| 1219 |
|
| 1220 |
In the following example, the replacement function translates decimals into |
| 1221 |
hexadecimal:: |
| 1222 |
|
| 1223 |
>>> def hexrepl(match): |
| 1224 |
... "Return the hex string for a decimal number" |
| 1225 |
... value = int(match.group()) |
| 1226 |
... return hex(value) |
| 1227 |
... |
| 1228 |
>>> p = re.compile(r'\d+') |
| 1229 |
>>> p.sub(hexrepl, 'Call 65490 for printing, 49152 for user code.') |
| 1230 |
'Call 0xffd2 for printing, 0xc000 for user code.' |
| 1231 |
|
| 1232 |
When using the module-level :func:`re.sub` function, the pattern is passed as |
| 1233 |
the first argument. The pattern may be provided as an object or as a string; if |
| 1234 |
you need to specify regular expression flags, you must either use a |
| 1235 |
pattern object as the first parameter, or use embedded modifiers in the |
| 1236 |
pattern string, e.g. ``sub("(?i)b+", "x", "bbbb BBBB")`` returns ``'x x'``. |
| 1237 |
|
| 1238 |
|
| 1239 |
Common problems |
| 1240 |
=============== |
| 1241 |
|
| 1242 |
Regular expressions are a powerful tool for some applications, but in some ways |
| 1243 |
their behaviour isn't intuitive and at times they don't behave the way you may |
| 1244 |
expect them to. This section will point out some of the most common pitfalls. |
| 1245 |
|
| 1246 |
|
| 1247 |
Use string methods |
| 1248 |
------------------ |
| 1249 |
|
| 1250 |
Sometimes using the :mod:`re` module is a mistake. If you're matching a fixed |
| 1251 |
string, or a single character class, and you're not using any :mod:`re` features |
| 1252 |
such as the :const:`~re.IGNORECASE` flag, then the full power of regular expressions |
| 1253 |
may not be required. Strings have several methods for performing operations with |
| 1254 |
fixed strings and they're usually much faster, because the implementation is a |
| 1255 |
single small C loop that's been optimized for the purpose, instead of the large, |
| 1256 |
more generalized regular expression engine. |
| 1257 |
|
| 1258 |
One example might be replacing a single fixed string with another one; for |
| 1259 |
example, you might replace ``word`` with ``deed``. :func:`re.sub` seems like the |
| 1260 |
function to use for this, but consider the :meth:`~str.replace` method. Note that |
| 1261 |
:meth:`!replace` will also replace ``word`` inside words, turning ``swordfish`` |
| 1262 |
into ``sdeedfish``, but the naive RE ``word`` would have done that, too. (To |
| 1263 |
avoid performing the substitution on parts of words, the pattern would have to |
| 1264 |
be ``\bword\b``, in order to require that ``word`` have a word boundary on |
| 1265 |
either side. This takes the job beyond :meth:`!replace`'s abilities.) |
| 1266 |
|
| 1267 |
Another common task is deleting every occurrence of a single character from a |
| 1268 |
string or replacing it with another single character. You might do this with |
| 1269 |
something like ``re.sub('\n', ' ', S)``, but :meth:`~str.translate` is capable of |
| 1270 |
doing both tasks and will be faster than any regular expression operation can |
| 1271 |
be. |
| 1272 |
|
| 1273 |
In short, before turning to the :mod:`re` module, consider whether your problem |
| 1274 |
can be solved with a faster and simpler string method. |
| 1275 |
|
| 1276 |
|
| 1277 |
match() versus search() |
| 1278 |
----------------------- |
| 1279 |
|
| 1280 |
The :func:`~re.match` function only checks if the RE matches at the beginning of the |
| 1281 |
string while :func:`~re.search` will scan forward through the string for a match. |
| 1282 |
It's important to keep this distinction in mind. Remember, :func:`!match` will |
| 1283 |
only report a successful match which will start at 0; if the match wouldn't |
| 1284 |
start at zero, :func:`!match` will *not* report it. :: |
| 1285 |
|
| 1286 |
>>> print(re.match('super', 'superstition').span()) |
| 1287 |
(0, 5) |
| 1288 |
>>> print(re.match('super', 'insuperable')) |
| 1289 |
None |
| 1290 |
|
| 1291 |
On the other hand, :func:`~re.search` will scan forward through the string, |
| 1292 |
reporting the first match it finds. :: |
| 1293 |
|
| 1294 |
>>> print(re.search('super', 'superstition').span()) |
| 1295 |
(0, 5) |
| 1296 |
>>> print(re.search('super', 'insuperable').span()) |
| 1297 |
(2, 7) |
| 1298 |
|
| 1299 |
Sometimes you'll be tempted to keep using :func:`re.match`, and just add ``.*`` |
| 1300 |
to the front of your RE. Resist this temptation and use :func:`re.search` |
| 1301 |
instead. The regular expression compiler does some analysis of REs in order to |
| 1302 |
speed up the process of looking for a match. One such analysis figures out what |
| 1303 |
the first character of a match must be; for example, a pattern starting with |
| 1304 |
``Crow`` must match starting with a ``'C'``. The analysis lets the engine |
| 1305 |
quickly scan through the string looking for the starting character, only trying |
| 1306 |
the full match if a ``'C'`` is found. |
| 1307 |
|
| 1308 |
Adding ``.*`` defeats this optimization, requiring scanning to the end of the |
| 1309 |
string and then backtracking to find a match for the rest of the RE. Use |
| 1310 |
:func:`re.search` instead. |
| 1311 |
|
| 1312 |
|
| 1313 |
Greedy versus non-greedy |
| 1314 |
------------------------ |
| 1315 |
|
| 1316 |
When repeating a regular expression, as in ``a*``, the resulting action is to |
| 1317 |
consume as much of the pattern as possible. This fact often bites you when |
| 1318 |
you're trying to match a pair of balanced delimiters, such as the angle brackets |
| 1319 |
surrounding an HTML tag. The naive pattern for matching a single HTML tag |
| 1320 |
doesn't work because of the greedy nature of ``.*``. :: |
| 1321 |
|
| 1322 |
>>> s = '<html><head><title>Title</title>' |
| 1323 |
>>> len(s) |
| 1324 |
32 |
| 1325 |
>>> print(re.match('<.*>', s).span()) |
| 1326 |
(0, 32) |
| 1327 |
>>> print(re.match('<.*>', s).group()) |
| 1328 |
<html><head><title>Title</title> |
| 1329 |
|
| 1330 |
The RE matches the ``'<'`` in ``'<html>'``, and the ``.*`` consumes the rest of |
| 1331 |
the string. There's still more left in the RE, though, and the ``>`` can't |
| 1332 |
match at the end of the string, so the regular expression engine has to |
| 1333 |
backtrack character by character until it finds a match for the ``>``. The |
| 1334 |
final match extends from the ``'<'`` in ``'<html>'`` to the ``'>'`` in |
| 1335 |
``'</title>'``, which isn't what you want. |
| 1336 |
|
| 1337 |
In this case, the solution is to use the non-greedy quantifiers ``*?``, ``+?``, |
| 1338 |
``??``, or ``{m,n}?``, which match as *little* text as possible. In the above |
| 1339 |
example, the ``'>'`` is tried immediately after the first ``'<'`` matches, and |
| 1340 |
when it fails, the engine advances a character at a time, retrying the ``'>'`` |
| 1341 |
at every step. This produces just the right result:: |
| 1342 |
|
| 1343 |
>>> print(re.match('<.*?>', s).group()) |
| 1344 |
<html> |
| 1345 |
|
| 1346 |
(Note that parsing HTML or XML with regular expressions is painful. |
| 1347 |
Quick-and-dirty patterns will handle common cases, but HTML and XML have special |
| 1348 |
cases that will break the obvious regular expression; by the time you've written |
| 1349 |
a regular expression that handles all of the possible cases, the patterns will |
| 1350 |
be *very* complicated. Use an HTML or XML parser module for such tasks.) |
| 1351 |
|
| 1352 |
|
| 1353 |
Using re.VERBOSE |
| 1354 |
---------------- |
| 1355 |
|
| 1356 |
By now you've probably noticed that regular expressions are a very compact |
| 1357 |
notation, but they're not terribly readable. REs of moderate complexity can |
| 1358 |
become lengthy collections of backslashes, parentheses, and metacharacters, |
| 1359 |
making them difficult to read and understand. |
| 1360 |
|
| 1361 |
For such REs, specifying the :const:`re.VERBOSE` flag when compiling the regular |
| 1362 |
expression can be helpful, because it allows you to format the regular |
| 1363 |
expression more clearly. |
| 1364 |
|
| 1365 |
The ``re.VERBOSE`` flag has several effects. Whitespace in the regular |
| 1366 |
expression that *isn't* inside a character class is ignored. This means that an |
| 1367 |
expression such as ``dog | cat`` is equivalent to the less readable ``dog|cat``, |
| 1368 |
but ``[a b]`` will still match the characters ``'a'``, ``'b'``, or a space. In |
| 1369 |
addition, you can also put comments inside a RE; comments extend from a ``#`` |
| 1370 |
character to the next newline. When used with triple-quoted strings, this |
| 1371 |
enables REs to be formatted more neatly:: |
| 1372 |
|
| 1373 |
pat = re.compile(r""" |
| 1374 |
\s* # Skip leading whitespace |
| 1375 |
(?P<header>[^:]+) # Header name |
| 1376 |
\s* : # Whitespace, and a colon |
| 1377 |
(?P<value>.*?) # The header's value -- *? used to |
| 1378 |
# lose the following trailing whitespace |
| 1379 |
\s*$ # Trailing whitespace to end-of-line |
| 1380 |
""", re.VERBOSE) |
| 1381 |
|
| 1382 |
This is far more readable than:: |
| 1383 |
|
| 1384 |
pat = re.compile(r"\s*(?P<header>[^:]+)\s*:(?P<value>.*?)\s*$") |
| 1385 |
|
| 1386 |
|
| 1387 |
Feedback |
| 1388 |
======== |
| 1389 |
|
| 1390 |
Regular expressions are a complicated topic. Did this document help you |
| 1391 |
understand them? Were there parts that were unclear, or problems you |
| 1392 |
encountered that weren't covered here? If so, please send suggestions for |
| 1393 |
improvements to the :ref:`issue tracker <using-the-tracker>`. |
| 1394 |
|
| 1395 |
The most complete book on regular expressions is almost certainly Jeffrey |
| 1396 |
Friedl's Mastering Regular Expressions, published by O'Reilly. Unfortunately, |
| 1397 |
it exclusively concentrates on Perl and Java's flavours of regular expressions, |
| 1398 |
and doesn't contain any Python material at all, so it won't be useful as a |
| 1399 |
reference for programming in Python. (The first edition covered Python's |
| 1400 |
now-removed :mod:`!regex` module, which won't help you much.) Consider checking |
| 1401 |
it out from your library. |