Debian Codegraph @

debian/rust-vendor/onig_sys/oniguruma/doc/RE

raw
1 Oniguruma Regular Expressions Version 6.9.9 2024/06/10
2
3 syntax: ONIG_SYNTAX_ONIGURUMA (default syntax)
4
5
6 1. Syntax elements
7
8 \ escape (enable or disable meta character)
9 | alternation
10 (...) group
11 [...] character class
12
13
14 2. Characters
15
16 \t horizontal tab (0x09)
17 \v vertical tab (0x0B)
18 \n newline (line feed) (0x0A)
19 \r carriage return (0x0D)
20 \b backspace (0x08)
21 \f form feed (0x0C)
22 \a bell (0x07)
23 \e escape (0x1B)
24 \nnn octal char (encoded byte value)
25 \xHH hexadecimal char (encoded byte value)
26 \x{7HHHHHHH} (1-8 digits) hexadecimal char (code point value)
27 \o{17777777777} (1-11 digits) octal char (code point value)
28 \uHHHH hexadecimal char (code point value)
29 \cx control char (code point value)
30 \C-x control char (code point value)
31 \M-x meta (x|0x80) (code point value)
32 \M-\C-x meta control char (code point value)
33
34 (* \b as backspace is effective in character class only)
35
36
37 2.1 Code point sequences
38
39 Hexadecimal code point (1-8 digits)
40 \x{7HHHHHHH 7HHHHHHH ... 7HHHHHHH}
41
42 Octal code point (1-11 digits)
43 \o{17777777777 17777777777 ... 17777777777}
44
45
46 3. Character types
47
48 . any character (except newline)
49
50 \w word character
51
52 Not Unicode:
53 alphanumeric, "_" and multibyte char.
54
55 Unicode:
56 General_Category -- (Letter|Mark|Number|Connector_Punctuation)
57
58 \W non-word char
59
60 \s whitespace char
61
62 Not Unicode:
63 \t, \n, \v, \f, \r, \x20
64
65 Unicode case:
66 U+0009, U+000A, U+000B, U+000C, U+000D, U+0085(NEL),
67 General_Category -- Line_Separator
68 -- Paragraph_Separator
69 -- Space_Separator
70
71 \S non-whitespace char
72
73 \d decimal digit char
74
75 Unicode: General_Category -- Decimal_Number
76
77 \D non-decimal-digit char
78
79 \h hexadecimal digit char [0-9a-fA-F]
80
81 \H non-hexdigit char
82
83 \R general newline (* can't be used in character-class)
84 "\r\n" or \n,\v,\f,\r (* but doesn't backtrack from \r\n to \r)
85
86 Unicode case:
87 "\r\n" or \n,\v,\f,\r or U+0085, U+2028, U+2029
88
89 \N negative newline (?-m:.)
90
91 \O true anychar (?m:.) (* original function)
92
93 \X Text Segment \X === (?>\O(?:\Y\O)*)
94
95 The meaning of this operator changes depending on the setting of
96 the option (?y{..}).
97
98 \X doesn't check whether matching start position is boundary or not.
99 Please write as \y\X if you want to ensure it.
100
101 [Extended Grapheme Cluster mode] (default)
102 Unicode case:
103 See [Unicode Standard Annex #29: http://unicode.org/reports/tr29/]
104
105 Not Unicode case: \X === (?>\r\n|\O)
106
107 [Word mode]
108 Currently, this mode is supported in Unicode only.
109 See [Unicode Standard Annex #29: http://unicode.org/reports/tr29/]
110
111
112 Character Property
113
114 * \p{property-name}
115 * \p{^property-name} (negative)
116 * \P{property-name} (negative)
117
118 property-name:
119
120 + works on all encodings
121 Alnum, Alpha, Blank, Cntrl, Digit, Graph, Lower,
122 Print, Punct, Space, Upper, XDigit, Word, ASCII
123
124 + works on EUC_JP, Shift_JIS
125 Hiragana, Katakana
126
127 + works on UTF8, UTF16, UTF32
128 See doc/UNICODE_PROPERTIES.
129
130
131
132 4. Quantifier
133
134 greedy
135
136 ? 1 or 0 times
137 * 0 or more times
138 + 1 or more times
139 {n,m} (n <= m) at least n but no more than m times
140 {n,} at least n times
141 {,n} at least 0 but no more than n times ({0,n})
142 {n} n times
143
144
145 reluctant
146
147 ?? 0 or 1 times
148 *? 0 or more times
149 +? 1 or more times
150 {n,m}? (n <= m) at least n but not more than m times
151 {n,}? at least n times
152 {,n}? at least 0 but not more than n times (== {0,n}?)
153
154 {n}? is reluctant operator in ONIG_SYNTAX_JAVA and ONIG_SYNTAX_PERL only.
155 (In that case, it doesn't make sense to write so.)
156 In default syntax, /a{n}?/ === /(?:a{n})?/
157
158
159 possessive (greedy and does not backtrack once match)
160
161 ?+ 1 or 0 times
162 *+ 0 or more times
163 ++ 1 or more times
164 {n,m} (n > m) at least m but not more than n times
165
166 {n,m}+, {n,}+, {n}+ are possessive operators in ONIG_SYNTAX_JAVA and
167 ONIG_SYNTAX_PERL only.
168
169 ex. /a*+/ === /(?>a*)/
170
171
172 5. Anchors
173
174 ^ beginning of the line
175 $ end of the line
176 \b word boundary
177 \B non-word boundary
178
179 \A beginning of string
180 \Z end of string, or before newline at the end
181 \z end of string
182 \G where the current search attempt begins
183 \K keep (keep start position of the result string)
184
185
186 \y Text Segment boundary
187 \Y Text Segment non-boundary
188
189 The meaning of these operators(\y, \Y) changes depending on the setting
190 of the option (?y{..}).
191
192 [Extended Grapheme Cluster mode] (default)
193 Unicode case:
194 See [Unicode Standard Annex #29: http://unicode.org/reports/tr29/]
195
196 Not Unicode:
197 All positions except between \r and \n.
198
199 [Word mode]
200 Currently, this mode is supported in Unicode only.
201 See [Unicode Standard Annex #29: http://unicode.org/reports/tr29/]
202
203
204
205 6. Character class
206
207 ^... negative class (lowest precedence)
208 x-y range from x to y
209 [...] set (character class in character class)
210 ..&&.. intersection (low precedence, only higher than ^)
211
212 ex. [a-w&&[^c-g]z] ==> ([a-w] AND ([^c-g] OR z)) ==> [abh-w]
213
214 * If you want to use '[', '-', or ']' as a normal character
215 in character class, you should escape them with '\'.
216
217
218 POSIX bracket ([:xxxxx:], negate [:^xxxxx:])
219
220 Not Unicode Case:
221
222 alnum alphabet or digit char
223 alpha alphabet
224 ascii code value: [0 - 127]
225 blank \t, \x20
226 cntrl
227 digit 0-9
228 graph include all of multibyte encoded characters
229 lower
230 print include all of multibyte encoded characters
231 punct
232 space \t, \n, \v, \f, \r, \x20
233 upper
234 xdigit 0-9, a-f, A-F
235 word alphanumeric, "_" and multibyte characters
236
237
238 Unicode Case:
239
240 alnum Alphabetic | Decimal_Number
241 alpha Alphabetic
242 ascii U+0000 - U+007F
243 blank Space_Separator | U+0009
244 cntrl U+0000 - U+001F, U+007F - U+009F
245 digit Decimal_Number
246 graph ^White_Space && ^[[:cntrl:]] && ^Unassigned && ^Surrogate
247 lower Lowercase
248 print [[:graph:]] | Space_Separator
249 punct Punctuation | Symbol
250 space White_Space
251 upper Uppercase
252 xdigit U+0030 - U+0039 | U+0041 - U+0046 | U+0061 - U+0066
253 (0-9, a-f, A-F)
254 word Alphabetic | Mark | Decimal_Number | Connector_Punctuation
255
256
257
258 7. Extended groups
259
260 (?#...) comment
261
262 (?imxWDSPy-imxWDSP:subexp) option on/off for subexp
263
264 i: ignore case
265 m: multi-line (dot (.) also matches newline)
266 x: extended form
267 W: ASCII only word (\w, \p{Word}, [[:word:]])
268 ASCII only word bound (\b)
269 D: ASCII only digit (\d, \p{Digit}, [[:digit:]])
270 S: ASCII only space (\s, \p{Space}, [[:space:]])
271 P: ASCII only POSIX properties (includes W,D,S)
272 (alnum, alpha, blank, cntrl, digit, graph,
273 lower, print, punct, space, upper, xdigit, word)
274
275 y{?}: Text Segment mode
276 This option changes the meaning of \X, \y, \Y.
277 Currently, this option is supported in Unicode only.
278
279 y{g}: Extended Grapheme Cluster mode (default)
280 y{w}: Word mode
281 See [Unicode Standard Annex #29]
282
283 (?imxWDSPy-imxWDSP) isolated option
284
285 * It makes a group to the next ')' or end of the pattern.
286 /ab(?i)c|def|gh/ == /ab(?i:c|def|gh)/
287
288 * (?i) option has no effect on word types (\w, \p{Word}). However, if the word types are used within a character class, it is valid. But, this would only be a concern when word types are used with the (?W) option.
289
290
291 /(?CIL).../, /(?CIL:...)/ whole option
292
293 This option must be placed in a position that
294 affects the entire regular expression.
295
296 C: ONIG_OPTION_DONT_CAPTURE_GROUP
297 I: ONIG_OPTION_IGNORECASE_IS_ASCII
298 L: ONIG_OPTION_FIND_LONGEST
299
300 (?:subexp) non-capturing group
301 (subexp) capturing group
302
303 (?=subexp) look-ahead
304 (?!subexp) negative look-ahead
305
306 (?<=subexp) look-behind
307 (?<!subexp) negative look-behind
308
309 * Cannot use Absent stopper (?~|expr) and Range clear
310 (?~|) operators in look-behind and negative look-behind.
311
312 * In look-behind and negative look-behind, support for
313 ignore-case option is limited. Only supports conversion
314 between single characters. (Does not support conversion
315 of multiple characters in Unicode)
316
317 (?>subexp) atomic group
318 no backtracks in subexp.
319
320 (?<name>subexp), (?'name'subexp)
321 define named group
322 (Each character of the name must be a word character.)
323
324 Not only a name but a number is assigned like a capturing
325 group.
326
327 Assigning the same name to two or more subexps is allowed.
328
329
330 <Callouts>
331
332 * Callouts of contents
333 (?{...contents...}) callout in progress
334 (?{...contents...}D) D is a direction flag char
335 D = 'X': in progress and retraction
336 '<': in retraction only
337 '>': in progress only
338 (?{...contents...}[tag]) tag assigned
339 (?{...contents...}[tag]D)
340
341 * Escape characters have no effects in contents.
342 * contents is not allowed to start with '{'.
343
344 (?{{{...contents...}}}) n times continuations '}' in contents is allowed in
345 (n+1) times continuations {{{...}}}.
346
347 Allowed tag string characters: _ A-Z a-z 0-9 (* first character: _ A-Z a-z)
348
349
350 * Callouts of name
351 (*name)
352 (*name{args...}) with args
353 (*name[tag]) tag assigned
354 (*name[tag]{args...})
355
356 Allowed name string characters: _ A-Z a-z 0-9 (* first character: _ A-Z a-z)
357 Allowed tag string characters: _ A-Z a-z 0-9 (* first character: _ A-Z a-z)
358
359
360 <Absent functions>
361
362 (?~absent) Absent repeater (* proposed by Tanaka Akira)
363 This works like .* (more precisely \O*), but it is
364 limited by the range that does not include the string
365 match with <absent>.
366 This is a written abbreviation of (?~|(?:absent)|\O*).
367 \O* is used as a repeater.
368
369 (?~|absent|exp) Absent expression (* original)
370 This works like "exp", but it is limited by the range
371 that does not include the string match with <absent>.
372
373 ex. (?~|345|\d*) "12345678" ==> "12", "1", ""
374
375 (?~|absent) Absent stopper (* original)
376 After passed this operator, string right range is limited
377 at the point that does not include the string match with
378 <absent>.
379
380 (?~|) Range clear
381 Clear the effects caused by Absent stoppers.
382
383 * Nested Absent functions are not supported and the behavior
384 is undefined.
385
386
387 <if-then-else>
388
389 (?(condition_exp)then_exp|else_exp) if-then-else
390 (?(condition_exp)then_exp) if-then
391
392 condition_exp can be a backreference number/name or a normal
393 regular expression.
394 When condition_exp is a backreference number/name, both then_exp and
395 else_exp can be omitted.
396 Then it works as a backreference validity checker.
397
398 [ Backreference validity checker ] (* original)
399
400 (?(n)), (?(-n)), (?(+n)), (?(n+level)) ...
401 (?(<n>)), (?('-n')), (?(<+n>)) ...
402 (?(<name>)), (?('name')), (?(<name+level>)) ...
403
404
405
406 8. Backreferences
407
408 When we say "backreference a group," it actually means, "re-match the same
409 text matched by the subexp in that group."
410
411 \n \k<n> \k'n' (n >= 1) backreference the nth group in the regexp
412 \k<-n> \k'-n' (n >= 1) backreference the nth group counting
413 backwards from the referring position
414 \k<+n> \k'+n' (n >= 1) backreference the nth group counting
415 forwards from the referring position
416 \k<name> \k'name' backreference a group with the specified name
417
418 When backreferencing with a name that is assigned to more than one groups,
419 the last group with the name is checked first, if not matched then the
420 previous one with the name, and so on, until there is a match.
421
422 * Backreference by number is forbidden if any named group is defined and
423 ONIG_OPTION_CAPTURE_GROUP is not set.
424
425
426 backreference with recursion level
427
428 (n >= 1, level >= 0)
429
430 \k<n+level> \k'n+level'
431 \k<n-level> \k'n-level'
432
433 \k<name+level> \k'name+level'
434 \k<name-level> \k'name-level'
435
436 Destine a group on the recursion level relative to the referring position.
437
438 ex 1.
439
440 /\A(?<a>|.|(?:(?<b>.)\g<a>\k<b>))\z/.match("reee")
441 /\A(?<a>|.|(?:(?<b>.)\g<a>\k<b+0>))\z/.match("reer")
442
443 \k<b+0> refers to the (?<b>.) on the same recursion level with it.
444
445 ex 2.
446
447 r = Regexp.compile(<<'__REGEXP__'.strip, Regexp::EXTENDED)
448 (?<element> \g<stag> \g<content>* \g<etag> ){0}
449 (?<stag> < \g<name> \s* > ){0}
450 (?<name> [a-zA-Z_:]+ ){0}
451 (?<content> [^<&]+ (\g<element> | [^<&]+)* ){0}
452 (?<etag> </ \k<name+1> >){0}
453 \g<element>
454 __REGEXP__
455
456 p r.match("<foo>f<bar>bbb</bar>f</foo>").captures
457
458
459 9. Subexp calls ("Tanaka Akira special") (* original function)
460
461 When we say "call a group," it actually means, "re-execute the subexp in
462 that group."
463
464 \g<n> \g'n' (n >= 1) call the nth group
465 \g<0> \g'0' call zero (call the total regexp)
466 \g<-n> \g'-n' (n >= 1) call the nth group counting backwards from
467 the calling position
468 \g<+n> \g'+n' (n >= 1) call the nth group counting forwards from
469 the calling position
470 \g<name> \g'name' call the group with the specified name
471
472 * Left-most recursive calls are not allowed.
473
474 ex. (?<name>a|\g<name>b) => error
475 (?<name>a|b\g<name>c) => OK
476
477 * Calls with a name that is assigned to more than one groups are not
478 allowed.
479
480 * Call by number is forbidden if any named group is defined and
481 ONIG_OPTION_CAPTURE_GROUP is not set.
482
483 * The option status of the called group is always effective.
484
485 ex. /(?-i:\g<name>)(?i:(?<name>a)){0}/.match("A")
486
487
488 10. Captured group
489
490 Behavior of an unnamed group (...) changes with the following conditions.
491 (But named group is not changed.)
492
493 case 1. /.../ (named group is not used, no option)
494
495 (...) is treated as a capturing group.
496
497 case 2. /.../g (named group is not used, 'g' option)
498
499 (...) is treated as a non-capturing group (?:...).
500
501 case 3. /..(?<name>..)../ (named group is used, no option)
502
503 (...) is treated as a non-capturing group.
504 numbered-backref/call is not allowed.
505
506 case 4. /..(?<name>..)../G (named group is used, 'G' option)
507
508 (...) is treated as a capturing group.
509 numbered-backref/call is allowed.
510
511 where
512 g: ONIG_OPTION_DONT_CAPTURE_GROUP
513 G: ONIG_OPTION_CAPTURE_GROUP
514
515 ('g' and 'G' options are argued in ruby-dev ML)
516
517
518
519 -----------------------------
520 A-1. Syntax-dependent options
521
522 + ONIG_SYNTAX_ONIGURUMA
523 (?m): dot (.) also matches newline
524
525 + ONIG_SYNTAX_PERL and ONIG_SYNTAX_JAVA
526 (?s): dot (.) also matches newline
527 (?m): ^ matches after newline, $ matches before newline
528
529
530 A-2. Original extensions
531
532 + hexadecimal digit char type \h, \H
533 + true anychar \O
534 + text segment boundary \y, \Y
535 + backreference validity checker (?(...))
536 + named group (?<name>...), (?'name'...)
537 + named backref \k<name>
538 + subexp call \g<name>, \g<group-num>
539 + absent expression (?~|...|...)
540 + absent stopper (?~|...)
541
542
543 A-3. Missing features compared with perl 5.8.0
544
545 + \N{name}
546 + \l,\u,\L,\U,\C
547 + (??{code})
548
549 * \Q...\E
550 This is effective on ONIG_SYNTAX_PERL and ONIG_SYNTAX_JAVA.
551
552
553 A-4. Differences with Japanized GNU regex(version 0.12) of Ruby 1.8
554
555 + add character property (\p{property}, \P{property})
556 + add hexadecimal digit char type (\h, \H)
557 + add look-behind
558 (?<=fixed-width-pattern), (?<!fixed-width-pattern)
559 + add possessive quantifier. ?+, *+, ++
560 + add operations in character class. [], &&
561 ('[' must be escaped as an usual char in character class.)
562 + add named group and subexp call.
563 + octal or hexadecimal number sequence can be treated as
564 a multibyte code char in character class if multibyte encoding
565 is specified.
566 (ex. [\xa1\xa2], [\xa1\xa7-\xa4\xa1])
567 + allow the range of single byte char and multibyte char in character
568 class.
569 ex. /[a-<<any EUC-JP character>>]/ in EUC-JP encoding.
570 + effect range of isolated option is to next ')'.
571 ex. (?:(?i)a|b) is interpreted as (?:(?i:a|b)), not (?:(?i:a)|b).
572 + isolated option is not transparent to previous pattern.
573 ex. a(?i)* is a syntax error pattern.
574 + allowed unpaired left brace as a normal character.
575 ex. /{/, /({)/, /a{2,3/ etc...
576 + negative POSIX bracket [:^xxxx:] is supported.
577 + POSIX bracket [:ascii:] is added.
578 + repeat of look-ahead is not allowed.
579 ex. /(?=a)*/, /(?!b){5}/
580 + Ignore case option is effective to escape sequence.
581 ex. /\x61/i =~ "A"
582 + In the range quantifier, the number of the minimum is optional.
583 /a{,n}/ == /a{0,n}/
584 The omission of both minimum and maximum values is not allowed.
585 /a{,}/
586 + /{n}?/ is not a reluctant quantifier.
587 /a{n}?/ == /(?:a{n})?/
588 + invalid back reference is checked and raises error.
589 /\1/, /(a)\2/
590 + Zero-width match in an infinite loop stops the repeat,
591 then changes of the capture group status are checked as stop condition.
592 /(?:()|())*\1\2/ =~ ""
593 /(?:\1a|())*/ =~ "a"
594
595 // END