Debian Codegraph @

manual/regexp_quick_reference.rst

raw
1 Quick reference for regexp syntax
2 =================================================
3
4 This checklist summarizes the most commonly used/hard to remember parts of the
5 regexp engine available in most parts of calibre.
6
7 .. contents:: Contents
8 :depth: 2
9 :local:
10
11
12 Character classes
13 ------------------
14
15 Character classes are useful to represent different groups of characters,
16 succinctly.
17
18 Examples:
19
20 +-----------------------+-----------------------------------------------------------------------------+
21 | **Representation** | **Class** |
22 | | |
23 +-----------------------+-----------------------------------------------------------------------------+
24 | ``[a-z]`` | Lowercase letters. Does not include characters with accent mark and |
25 | | ligatures |
26 +-----------------------+-----------------------------------------------------------------------------+
27 | ``[a-z0-9]`` | Lowercase letters from a to z or numbers from 0 to 9 |
28 +-----------------------+-----------------------------------------------------------------------------+
29 | ``[A-Za-z-]`` | Uppercase or lowercase letters, or a dash. To include the dash in a class, |
30 | | you must put it at the beginning or at the end so as not to confuse it |
31 | | with the hyphen that specifies a range of characters |
32 +-----------------------+-----------------------------------------------------------------------------+
33 | ``[^0-9]`` | Any character except a digit. The caret (^) placed at the beginning of the |
34 | | class excludes the characters of the class (complemented class) |
35 +-----------------------+-----------------------------------------------------------------------------+
36 | ``[[a-z]--[aeiouy]]`` | The lowercase consonants. A class can be included in a class. The |
37 | | characters ``--`` exclude what follows them |
38 +-----------------------+-----------------------------------------------------------------------------+
39 | ``[\w--[\d_]]`` | All letters (including foreign accented characters). Abbreviated classes |
40 | | can be used inside a class |
41 +-----------------------+-----------------------------------------------------------------------------+
42
43
44 Example::
45
46 <[^<>]+> to select an HTML tag
47
48 Shorthand character classes
49 ---------------------------
50
51 +---------------------+-------------------------------------------------------------------------------+
52 | **Representation** | **Class** |
53 | | |
54 +---------------------+-------------------------------------------------------------------------------+
55 | ``\d`` | A digit (same as ``[0-9]``) |
56 +---------------------+-------------------------------------------------------------------------------+
57 | ``\D`` | Any non-numeric character (same as ``[^0-9]``) |
58 +---------------------+-------------------------------------------------------------------------------+
59 | ``\w`` | An alphanumeric character (``[a-zA-Z0-9]``) including characters with accent |
60 | | mark and ligatures |
61 +---------------------+-------------------------------------------------------------------------------+
62 | ``\W`` | Any “non-word” character |
63 +---------------------+-------------------------------------------------------------------------------+
64 | ``\s`` | Space, non-breaking space, tab, carriage return |
65 +---------------------+-------------------------------------------------------------------------------+
66 | ``\S`` | Any “non-whitespace” character |
67 +---------------------+-------------------------------------------------------------------------------+
68 | ``.`` | Any character except newline. Use the “dot all” checkbox or the ``(?s)`` |
69 | | regexp modifier to include the newline character. |
70 +---------------------+-------------------------------------------------------------------------------+
71
72 The quantifiers
73 ---------------
74
75 +----------------+---------------------------------------------------------------------------+
76 | **Quantifier** | **Number of occurrences of the expression preceding the quantifier** |
77 | | |
78 +----------------+---------------------------------------------------------------------------+
79 | ``?`` | 0 or 1 occurrence of the expression. Same as ``{0,1}`` |
80 +----------------+---------------------------------------------------------------------------+
81 | ``+`` | 1 or more occurrences of the expression. Same as ``{1,}`` |
82 +----------------+---------------------------------------------------------------------------+
83 | ``*`` | 0, 1 or more occurrences of the expression. Same as ``{0,}`` |
84 +----------------+---------------------------------------------------------------------------+
85 | ``{n}`` | Exactly n occurrences of the expression |
86 +----------------+---------------------------------------------------------------------------+
87 | ``{min,max}`` | Number of occurrences between the minimum and maximum values included |
88 +----------------+---------------------------------------------------------------------------+
89 | ``{min,}`` | Number of occurrences between the minimum value included and infinity |
90 +----------------+---------------------------------------------------------------------------+
91 | ``{,max}`` | Number of occurrences between 0 and the maximum value included |
92 +----------------+---------------------------------------------------------------------------+
93
94
95
96 Greed
97 -----
98
99 By default, with quantifiers, the regular expression engine is greedy: it
100 extends the selection as much as possible. This often causes surprises, at
101 first. ``?`` follows a quantifier to make it lazy.
102 Avoid putting two in the same expression, the result can be unpredictable.
103
104 Beware of nesting quantifiers, for example, the pattern ``(a*)*``, as it
105 exponentially increases processing time.
106
107 Alternation
108 -----------
109
110 The ``|`` character in a regular expression is a logical ``OR``. It means
111 that either the preceding or the following expression can match.
112
113 Exclusion
114 ---------
115
116 Method 1
117
118 ``pattern_to_exclude(*SKIP)(*FAIL)|pattern_to_select``
119
120 Example:
121
122 ``"Blabla"(*SKIP)(*FAIL)|Blabla``
123
124 selects Blabla, in the strings Blabla or "Blabla or Blabla", but not in "Blabla".
125
126 Method 2
127
128 ``pattern_to_exclude\K|(pattern_to_select)``
129
130 ``"Blabla"\K|(Blabla)``
131
132 selects Blabla, in the strings Blabla or "Blabla or Blabla", but not in "Blabla".
133
134 Anchors
135 -------
136
137 An anchor is a way to match a logical location in a string, rather than a
138 character. The most useful anchors for text processing are:
139
140 ``\b``
141 Designates a word boundary, i.e. a transition from space to non-space
142 character. For example, you can use ``\bsurd`` to match ``the surd`` but
143 not ``absurd``.
144
145 ``^``
146 Matches the start of a line (in multi-line mode, which is the
147 default)
148
149 ``$``
150 Matches the end of a line (in multi-line mode, which is the default)
151
152 ``\K``
153 Resets the start position of the selection to its position in the pattern.
154 Some regexp engines (but not calibre) do not allow lookbehind of variable
155 length, especially with quantifiers. When you can use ``\K`` with these
156 engines, it also allows you to get rid of this limit by writing the
157 equivalent of a positive lookbehind of variable length.
158
159 Groups
160 ------
161
162 ``(expression)``
163 Capturing group, which stores the selection and can be recalled later
164 in the *search* or *replace* patterns with ``\n``, where ``n`` is the
165 sequence number of the capturing group (starting at 1 in reading order)
166
167 ``(?:expression)``
168 Group that does not capture the selection
169
170 ``(?>expression)``
171 Atomic Group: As soon as the expression is satisfied, the regexp engine
172 passes, and if the rest of the pattern fails, it will not backtrack to
173 try other combinations with the expression. Atomic groups do not
174 capture.
175
176 ``(?|expression)``
177 Branch reset group: the branches of the alternations included in the
178 expression share the same group numbers
179
180 ``(?<name>expression)``
181 Group named “name”. The selection can be recalled later in the *search*
182 pattern by ``(?P=name)`` and in the *replace* by ``\g<name>``. Two
183 different groups can use the same name.
184
185
186 Lookarounds
187 -----------
188
189 +----------------+---------------------------------------------------------+
190 | **Lookaround** | **Meaning** |
191 | | |
192 +----------------+---------------------------------------------------------+
193 | ``?=`` | Positive lookahead (to be placed after the selection) |
194 | | |
195 +----------------+---------------------------------------------------------+
196 | ``?!`` | Negative lookahead (to be placed after the selection) |
197 | | |
198 +----------------+---------------------------------------------------------+
199 | ``?<=`` | Positive lookbehind (to be placed before the selection) |
200 | | |
201 +----------------+---------------------------------------------------------+
202 | ``?<!`` | Negative lookbehind (to be placed before the selection) |
203 | | |
204 +----------------+---------------------------------------------------------+
205
206 Lookaheads and lookbehinds do not consume characters, they are zero length and
207 do not capture. They are atomic groups: as soon as the assertion is satisfied,
208 the regexp engine passes, and if the rest of the pattern fails, it will not
209 backtrack inside the lookaround to try other combinations.
210
211 When looking for multiple matches in a string, at the starting position of each
212 match attempt, a lookbehind can inspect the characters before the current
213 position. Therefore, on the string 123, the pattern ``(?<=\d)\d`` (a digit preceded
214 by a digit) should, in theory, select 2 and 3. On the other hand, ``\d\K\d`` can
215 only select 2, because the starting position after the first selection is
216 immediately before 3, and there are not enough digits for a second match.
217 Similarly, ``\d(\d)`` only captures 2. In calibre's regexp engine practice, the
218 positive lookbehind behaves in the same way, and selects only 2, contrary to
219 theory.
220
221 Groups can be placed inside lookarounds, but capture is rarely useful.
222 Nevertheless, if it is useful, it will be necessary to be very careful in the
223 use of a quantifier in a lookbehind: the greed associated with the absence of
224 backtracking can give a surprising capture. For this reason, use ``\K`` rather than
225 a positive lookbehind when you have a quantifier (or worse, several) in a
226 capturing group of the positive lookbehind.
227
228 Example of negative lookahead:
229
230 ``(?![^<>{}]*[>}])``
231
232 Placed at the end of the pattern prevents selecting within a tag or a style embedded in the file.
233
234 Whenever possible, it is always better to "anchor" the lookarounds, to reduce
235 the number of steps necessary to obtain the result.
236
237 Recursion
238 ---------
239
240 +--------------------+-----------------------------------------------------------------------------+
241 | **Representation** | **Meaning** |
242 | | |
243 +--------------------+-----------------------------------------------------------------------------+
244 | ``(?R)`` | Recursion of the entire pattern |
245 | | |
246 +--------------------+-----------------------------------------------------------------------------+
247 | ``(?1)`` | Recursion of the only pattern of the numbered capturing group, here group 1 |
248 | | |
249 +--------------------+-----------------------------------------------------------------------------+
250
251 Recursion is calling oneself. This is useful for balanced queries, such as
252 quoted strings, which can contain embedded quoted strings. Thus, if during the
253 processing of a string between double quotation marks, we encounter the
254 beginning of a new string between double quotation marks, well we know how to
255 do, and we call ourselves. Then we have a pattern like::
256
257 start-pattern(?>atomic sub-pattern|(?R))*end-pattern
258
259 To select a string between double quotation marks without stopping on an embedded string::
260
261 “((?>[^“”]+|(?R))*[^“”]+)”
262
263 This template can also be used to modify pairs of tags that can be
264 embedded, such as ``<div>`` tags.
265
266
267 Special characters
268 ------------------
269
270 +--------------------+-------------------+
271 | **Representation** | **Character** |
272 | | |
273 +--------------------+-------------------+
274 | ``\t`` | tabulation |
275 | | |
276 +--------------------+-------------------+
277 | ``\n`` | line break |
278 | | |
279 +--------------------+-------------------+
280 | ``\x20`` | (breakable) space |
281 | | |
282 +--------------------+-------------------+
283 | ``\xa0`` | no-break space |
284 | | |
285 +--------------------+-------------------+
286
287 Meta-characters
288 ---------------
289
290 Meta-characters are those that have a special meaning for the regexp engine. Of
291 these, twelve must be preceded by an escape character, the backslash (``\``), to
292 lose their special meaning and become a regular character again::
293
294 ^ . [ ] $ ( ) * + ? | \
295
296 Seven other meta-characters do not need to be preceded by a backslash (but can
297 be without any other consequence)::
298
299 { } ! < > = :
300
301
302 Special characters lose their status if they are used inside a class (between
303 brackets ``[]``). The closing bracket and the dash have a special status in a
304 class. Outside the class, the dash is a simple literal, the closing bracket
305 remains a meta-character.
306
307 The slash (/) and the number sign (or hash character) (#) are not
308 meta-characters, they don’t need to be escaped.
309
310 In some tools, like regex101.com with the Python engine, double quotes have the
311 special status of separator, and must be escaped, or the options changed. This
312 is not the case in the editor of calibre.
313
314 Modes
315 -----
316
317 ``(?s)``
318 Causes the dot (``.``) to match newline characters as well
319
320 ``(?m)``
321 Makes the ``^`` and ``$`` anchors match the start and end of lines
322 instead of the start and end of the entire string.