Debian Codegraph @

upstream/mageia-cauldron/man1/perlfaq6.1

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man v6.0.2 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\"
56 .\" Required to disable full justification in groff 1.23.0.
57 .if n .ds AD l
58 .\" ========================================================================
59 .\"
60 .IX Title "PERLFAQ6 1"
61 .TH PERLFAQ6 1 2025-06-24 "perl v5.42.0" "Perl Programmers Reference Guide"
62 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
63 .\" way too many mistakes in technical documents.
64 .if n .ad l
65 .nh
66 .SH NAME
67 perlfaq6 \- Regular Expressions
68 .SH VERSION
69 .IX Header "VERSION"
70 version 5.20250619
71 .SH DESCRIPTION
72 .IX Header "DESCRIPTION"
73 This section is surprisingly small because the rest of the FAQ is
74 littered with answers involving regular expressions. For example,
75 decoding a URL and checking whether something is a number can be handled
76 with regular expressions, but those answers are found elsewhere in
77 this document (in perlfaq9: "How do I decode or create those %\-encodings
78 on the web" and perlfaq4: "How do I determine whether a scalar is
79 a number/whole/integer/float", to be precise).
80 .SS "How can I hope to use regular expressions without creating illegible and unmaintainable code?"
81 .IX Xref "regex, legibility regexp, legibility regular expression, legibility x"
82 .IX Subsection "How can I hope to use regular expressions without creating illegible and unmaintainable code?"
83 Three techniques can make regular expressions maintainable and
84 understandable.
85 .IP "Comments Outside the Regex" 4
86 .IX Item "Comments Outside the Regex"
87 Describe what you\*(Aqre doing and how you\*(Aqre doing it, using normal Perl
88 comments.
89 .Sp
90 .Vb 3
91 \& # turn the line into the first word, a colon, and the
92 \& # number of characters on the rest of the line
93 \& s/^(\ew+)(.*)/ lc($1) . ":" . length($2) /meg;
94 .Ve
95 .IP "Comments Inside the Regex" 4
96 .IX Item "Comments Inside the Regex"
97 The \f(CW\*(C`/x\*(C'\fR modifier causes whitespace to be ignored in a regex pattern
98 (except in a character class and a few other places), and also allows you to
99 use normal comments there, too. As you can imagine, whitespace and comments
100 help a lot.
101 .Sp
102 \&\f(CW\*(C`/x\*(C'\fR lets you turn this:
103 .Sp
104 .Vb 1
105 \& s{<(?:[^>\*(Aq"]*|".*?"|\*(Aq.*?\*(Aq)+>}{}gs;
106 .Ve
107 .Sp
108 into this:
109 .Sp
110 .Vb 10
111 \& s{ < # opening angle bracket
112 \& (?: # Non\-backreffing grouping paren
113 \& [^>\*(Aq"] * # 0 or more things that are neither > nor \*(Aq nor "
114 \& | # or else
115 \& ".*?" # a section between double quotes (stingy match)
116 \& | # or else
117 \& \*(Aq.*?\*(Aq # a section between single quotes (stingy match)
118 \& ) + # all occurring one or more times
119 \& > # closing angle bracket
120 \& }{}gsx; # replace with nothing, i.e. delete
121 .Ve
122 .Sp
123 It\*(Aqs still not quite so clear as prose, but it is very useful for
124 describing the meaning of each part of the pattern.
125 .IP "Different Delimiters" 4
126 .IX Item "Different Delimiters"
127 While we normally think of patterns as being delimited with \f(CW\*(C`/\*(C'\fR
128 characters, they can be delimited by almost any character. perlre
129 describes this. For example, the \f(CW\*(C`s///\*(C'\fR above uses braces as
130 delimiters. Selecting another delimiter can avoid quoting the
131 delimiter within the pattern:
132 .Sp
133 .Vb 2
134 \& s/\e/usr\e/local/\e/usr\e/share/g; # bad delimiter choice
135 \& s#/usr/local#/usr/share#g; # better
136 .Ve
137 .Sp
138 Using logically paired delimiters can be even more readable:
139 .Sp
140 .Vb 1
141 \& s{/usr/local/}{/usr/share}g; # better still
142 .Ve
143 .SS "I\*(Aqm having trouble matching over more than one line. What\*(Aqs wrong?"
144 .IX Xref "regex, multiline regexp, multiline regular expression, multiline"
145 .IX Subsection "I'm having trouble matching over more than one line. What's wrong?"
146 Either you don\*(Aqt have more than one line in the string you\*(Aqre looking
147 at (probably), or else you aren\*(Aqt using the correct modifier(s) on
148 your pattern (possibly).
149 .PP
150 There are many ways to get multiline data into a string. If you want
151 it to happen automatically while reading input, you\*(Aqll want to set $/
152 (probably to \*(Aq\*(Aq for paragraphs or \f(CW\*(C`undef\*(C'\fR for the whole file) to
153 allow you to read more than one line at a time.
154 .PP
155 Read perlre to help you decide which of \f(CW\*(C`/s\*(C'\fR and \f(CW\*(C`/m\*(C'\fR (or both)
156 you might want to use: \f(CW\*(C`/s\*(C'\fR allows dot to include newline, and \f(CW\*(C`/m\*(C'\fR
157 allows caret and dollar to match next to a newline, not just at the
158 end of the string. You do need to make sure that you\*(Aqve actually
159 got a multiline string in there.
160 .PP
161 For example, this program detects duplicate words, even when they span
162 line breaks (but not paragraph ones). For this example, we don\*(Aqt need
163 \&\f(CW\*(C`/s\*(C'\fR because we aren\*(Aqt using dot in a regular expression that we want
164 to cross line boundaries. Neither do we need \f(CW\*(C`/m\*(C'\fR because we don\*(Aqt
165 want caret or dollar to match at any point inside the record next
166 to newlines. But it\*(Aqs imperative that $/ be set to something other
167 than the default, or else we won\*(Aqt actually ever have a multiline
168 record read in.
169 .PP
170 .Vb 6
171 \& $/ = \*(Aq\*(Aq; # read in whole paragraph, not just one line
172 \& while ( <> ) {
173 \& while ( /\eb([\ew\*(Aq\-]+)(\es+\eg1)+\eb/gi ) { # word starts alpha
174 \& print "Duplicate $1 at paragraph $.\en";
175 \& }
176 \& }
177 .Ve
178 .PP
179 Here\*(Aqs some code that finds sentences that begin with "From " (which would
180 be mangled by many mailers):
181 .PP
182 .Vb 6
183 \& $/ = \*(Aq\*(Aq; # read in whole paragraph, not just one line
184 \& while ( <> ) {
185 \& while ( /^From /gm ) { # /m makes ^ match next to \en
186 \& print "leading From in paragraph $.\en";
187 \& }
188 \& }
189 .Ve
190 .PP
191 Here\*(Aqs code that finds everything between START and END in a paragraph:
192 .PP
193 .Vb 6
194 \& undef $/; # read in whole file, not just one line or paragraph
195 \& while ( <> ) {
196 \& while ( /START(.*?)END/sgm ) { # /s makes . cross line boundaries
197 \& print "$1\en";
198 \& }
199 \& }
200 .Ve
201 .SS "How can I pull out lines between two patterns that are themselves on different lines?"
202 .IX Xref ".."
203 .IX Subsection "How can I pull out lines between two patterns that are themselves on different lines?"
204 You can use Perl\*(Aqs somewhat exotic \f(CW\*(C`..\*(C'\fR operator (documented in
205 perlop):
206 .PP
207 .Vb 1
208 \& perl \-ne \*(Aqprint if /START/ .. /END/\*(Aq file1 file2 ...
209 .Ve
210 .PP
211 If you wanted text and not lines, you would use
212 .PP
213 .Vb 1
214 \& perl \-0777 \-ne \*(Aqprint "$1\en" while /START(.*?)END/gs\*(Aq file1 file2 ...
215 .Ve
216 .PP
217 But if you want nested occurrences of \f(CW\*(C`START\*(C'\fR through \f(CW\*(C`END\*(C'\fR, you\*(Aqll
218 run up against the problem described in the question in this section
219 on matching balanced text.
220 .PP
221 Here\*(Aqs another example of using \f(CW\*(C`..\*(C'\fR:
222 .PP
223 .Vb 7
224 \& while (<>) {
225 \& my $in_header = 1 .. /^$/;
226 \& my $in_body = /^$/ .. eof;
227 \& # now choose between them
228 \& } continue {
229 \& $. = 0 if eof; # fix $.
230 \& }
231 .Ve
232 .SS "How do I match XML, HTML, or other nasty, ugly things with a regex?"
233 .IX Xref "regex, XML regex, HTML XML HTML pain frustration sucking out, will to live"
234 .IX Subsection "How do I match XML, HTML, or other nasty, ugly things with a regex?"
235 Do not use regexes. Use a module and forget about the
236 regular expressions. The XML::LibXML, HTML::TokeParser and
237 HTML::TreeBuilder modules are good starts, although each namespace
238 has other parsing modules specialized for certain tasks and different
239 ways of doing it. Start at CPAN Search ( <http://metacpan.org/> )
240 and wonder at all the work people have done for you already! :)
241 .SS "I put a regular expression into $/ but it didn\*(Aqt work. What\*(Aqs wrong?"
242 .IX Xref "$ , regexes in $INPUT_RECORD_SEPARATOR, regexes in $RS, regexes in"
243 .IX Subsection "I put a regular expression into $/ but it didn't work. What's wrong?"
244 $/ has to be a string. You can use these examples if you really need to
245 do this.
246 .PP
247 If you have File::Stream, this is easy.
248 .PP
249 .Vb 1
250 \& use File::Stream;
251 \&
252 \& my $stream = File::Stream\->new(
253 \& $filehandle,
254 \& separator => qr/\es*,\es*/,
255 \& );
256 \&
257 \& print "$_\en" while <$stream>;
258 .Ve
259 .PP
260 If you don\*(Aqt have File::Stream, you have to do a little more work.
261 .PP
262 You can use the four\-argument form of sysread to continually add to
263 a buffer. After you add to the buffer, you check if you have a
264 complete line (using your regular expression).
265 .PP
266 .Vb 7
267 \& local $_ = "";
268 \& while( sysread FH, $_, 8192, length ) {
269 \& while( s/^((?s).*?)your_pattern// ) {
270 \& my $record = $1;
271 \& # do stuff here.
272 \& }
273 \& }
274 .Ve
275 .PP
276 You can do the same thing with foreach and a match using the
277 c flag and the \eG anchor, if you do not mind your entire file
278 being in memory at the end.
279 .PP
280 .Vb 7
281 \& local $_ = "";
282 \& while( sysread FH, $_, 8192, length ) {
283 \& foreach my $record ( m/\eG((?s).*?)your_pattern/gc ) {
284 \& # do stuff here.
285 \& }
286 \& substr( $_, 0, pos ) = "" if pos;
287 \& }
288 .Ve
289 .SS "How do I substitute case\-insensitively on the LHS while preserving case on the RHS?"
290 .IX Xref "replace, case preserving substitute, case preserving substitution, case preserving s, case preserving"
291 .IX Subsection "How do I substitute case-insensitively on the LHS while preserving case on the RHS?"
292 Here\*(Aqs a lovely Perlish solution by Larry Rosler. It exploits
293 properties of bitwise xor on ASCII strings.
294 .PP
295 .Vb 1
296 \& $_= "this is a TEsT case";
297 \&
298 \& $old = \*(Aqtest\*(Aq;
299 \& $new = \*(Aqsuccess\*(Aq;
300 \&
301 \& s{(\eQ$old\eE)}
302 \& { uc $new | (uc $1 ^ $1) .
303 \& (uc(substr $1, \-1) ^ substr $1, \-1) x
304 \& (length($new) \- length $1)
305 \& }egi;
306 \&
307 \& print;
308 .Ve
309 .PP
310 And here it is as a subroutine, modeled after the above:
311 .PP
312 .Vb 3
313 \& sub preserve_case {
314 \& my ($old, $new) = @_;
315 \& my $mask = uc $old ^ $old;
316 \&
317 \& uc $new | $mask .
318 \& substr($mask, \-1) x (length($new) \- length($old))
319 \& }
320 \&
321 \& $string = "this is a TEsT case";
322 \& $string =~ s/(test)/preserve_case($1, "success")/egi;
323 \& print "$string\en";
324 .Ve
325 .PP
326 This prints:
327 .PP
328 .Vb 1
329 \& this is a SUcCESS case
330 .Ve
331 .PP
332 As an alternative, to keep the case of the replacement word if it is
333 longer than the original, you can use this code, by Jeff Pinyan:
334 .PP
335 .Vb 3
336 \& sub preserve_case {
337 \& my ($from, $to) = @_;
338 \& my ($lf, $lt) = map length, @_;
339 \&
340 \& if ($lt < $lf) { $from = substr $from, 0, $lt }
341 \& else { $from .= substr $to, $lf }
342 \&
343 \& return uc $to | ($from ^ uc $from);
344 \& }
345 .Ve
346 .PP
347 This changes the sentence to "this is a SUcCess case."
348 .PP
349 Just to show that C programmers can write C in any programming language,
350 if you prefer a more C\-like solution, the following script makes the
351 substitution have the same case, letter by letter, as the original.
352 (It also happens to run about 240% slower than the Perlish solution runs.)
353 If the substitution has more characters than the string being substituted,
354 the case of the last character is used for the rest of the substitution.
355 .PP
356 .Vb 8
357 \& # Original by Nathan Torkington, massaged by Jeffrey Friedl
358 \& #
359 \& sub preserve_case
360 \& {
361 \& my ($old, $new) = @_;
362 \& my $state = 0; # 0 = no change; 1 = lc; 2 = uc
363 \& my ($i, $oldlen, $newlen, $c) = (0, length($old), length($new));
364 \& my $len = $oldlen < $newlen ? $oldlen : $newlen;
365 \&
366 \& for ($i = 0; $i < $len; $i++) {
367 \& if ($c = substr($old, $i, 1), $c =~ /[\eW\ed_]/) {
368 \& $state = 0;
369 \& } elsif (lc $c eq $c) {
370 \& substr($new, $i, 1) = lc(substr($new, $i, 1));
371 \& $state = 1;
372 \& } else {
373 \& substr($new, $i, 1) = uc(substr($new, $i, 1));
374 \& $state = 2;
375 \& }
376 \& }
377 \& # finish up with any remaining new (for when new is longer than old)
378 \& if ($newlen > $oldlen) {
379 \& if ($state == 1) {
380 \& substr($new, $oldlen) = lc(substr($new, $oldlen));
381 \& } elsif ($state == 2) {
382 \& substr($new, $oldlen) = uc(substr($new, $oldlen));
383 \& }
384 \& }
385 \& return $new;
386 \& }
387 .Ve
388 .ie n .SS "How can I make ""\ew"" match national character sets?"
389 .el .SS "How can I make \f(CW\ew\fP match national character sets?"
390 .IX Xref "\\w"
391 .IX Subsection "How can I make w match national character sets?"
392 Put \f(CW\*(C`use locale;\*(C'\fR in your script. The \ew character class is taken
393 from the current locale.
394 .PP
395 See perllocale for details.
396 .ie n .SS "How can I match a locale\-smart version of ""/[a\-zA\-Z]/""?"
397 .el .SS "How can I match a locale\-smart version of \f(CW/[a\-zA\-Z]/\fP?"
398 .IX Xref "alpha"
399 .IX Subsection "How can I match a locale-smart version of /[a-zA-Z]/?"
400 You can use the POSIX character class syntax \f(CW\*(C`/[[:alpha:]]/\*(C'\fR
401 documented in perlre.
402 .PP
403 No matter which locale you are in, the alphabetic characters are
404 the characters in \ew without the digits and the underscore.
405 As a regex, that looks like \f(CW\*(C`/[^\eW\ed_]/\*(C'\fR. Its complement,
406 the non\-alphabetics, is then everything in \eW along with
407 the digits and the underscore, or \f(CW\*(C`/[\eW\ed_]/\*(C'\fR.
408 .SS "How can I quote a variable to use in a regex?"
409 .IX Xref "regex, escaping regexp, escaping regular expression, escaping"
410 .IX Subsection "How can I quote a variable to use in a regex?"
411 The Perl parser will expand \f(CW$variable\fR and \f(CW@variable\fR references in
412 regular expressions unless the delimiter is a single quote. Remember,
413 too, that the right\-hand side of a \f(CW\*(C`s///\*(C'\fR substitution is considered
414 a double\-quoted string (see perlop for more details). Remember
415 also that any regex special characters will be acted on unless you
416 precede the substitution with \eQ. Here\*(Aqs an example:
417 .PP
418 .Vb 2
419 \& $string = "Placido P. Octopus";
420 \& $regex = "P.";
421 \&
422 \& $string =~ s/$regex/Polyp/;
423 \& # $string is now "Polypacido P. Octopus"
424 .Ve
425 .PP
426 Because \f(CW\*(C`.\*(C'\fR is special in regular expressions, and can match any
427 single character, the regex \f(CW\*(C`P.\*(C'\fR here has matched the <Pl> in the
428 original string.
429 .PP
430 To escape the special meaning of \f(CW\*(C`.\*(C'\fR, we use \f(CW\*(C`\eQ\*(C'\fR:
431 .PP
432 .Vb 2
433 \& $string = "Placido P. Octopus";
434 \& $regex = "P.";
435 \&
436 \& $string =~ s/\eQ$regex/Polyp/;
437 \& # $string is now "Placido Polyp Octopus"
438 .Ve
439 .PP
440 The use of \f(CW\*(C`\eQ\*(C'\fR causes the \f(CW\*(C`.\*(C'\fR in the regex to be treated as a
441 regular character, so that \f(CW\*(C`P.\*(C'\fR matches a \f(CW\*(C`P\*(C'\fR followed by a dot.
442 .ie n .SS "What is ""/o"" really for?"
443 .el .SS "What is \f(CW/o\fP really for?"
444 .IX Xref " o, regular expressions compile, regular expressions"
445 .IX Subsection "What is /o really for?"
446 (contributed by brian d foy)
447 .PP
448 The \f(CW\*(C`/o\*(C'\fR option for regular expressions (documented in perlop and
449 perlreref) tells Perl to compile the regular expression only once.
450 This is only useful when the pattern contains a variable. Perls 5.6
451 and later handle this automatically if the pattern does not change.
452 .PP
453 Since the match operator \f(CW\*(C`m//\*(C'\fR, the substitution operator \f(CW\*(C`s///\*(C'\fR,
454 and the regular expression quoting operator \f(CW\*(C`qr//\*(C'\fR are double\-quotish
455 constructs, you can interpolate variables into the pattern. See the
456 answer to "How can I quote a variable to use in a regex?" for more
457 details.
458 .PP
459 This example takes a regular expression from the argument list and
460 prints the lines of input that match it:
461 .PP
462 .Vb 1
463 \& my $pattern = shift @ARGV;
464 \&
465 \& while( <> ) {
466 \& print if m/$pattern/;
467 \& }
468 .Ve
469 .PP
470 Versions of Perl prior to 5.6 would recompile the regular expression
471 for each iteration, even if \f(CW$pattern\fR had not changed. The \f(CW\*(C`/o\*(C'\fR
472 would prevent this by telling Perl to compile the pattern the first
473 time, then reuse that for subsequent iterations:
474 .PP
475 .Vb 1
476 \& my $pattern = shift @ARGV;
477 \&
478 \& while( <> ) {
479 \& print if m/$pattern/o; # useful for Perl < 5.6
480 \& }
481 .Ve
482 .PP
483 In versions 5.6 and later, Perl won\*(Aqt recompile the regular expression
484 if the variable hasn\*(Aqt changed, so you probably don\*(Aqt need the \f(CW\*(C`/o\*(C'\fR
485 option. It doesn\*(Aqt hurt, but it doesn\*(Aqt help either. If you want any
486 version of Perl to compile the regular expression only once even if
487 the variable changes (thus, only using its initial value), you still
488 need the \f(CW\*(C`/o\*(C'\fR.
489 .PP
490 You can watch Perl\*(Aqs regular expression engine at work to verify for
491 yourself if Perl is recompiling a regular expression. The \f(CWuse re
492 \&\*(Aqdebug\*(Aq\fR pragma (comes with Perl 5.005 and later) shows the details.
493 With Perls before 5.6, you should see \f(CW\*(C`re\*(C'\fR reporting that it\*(Aqs
494 compiling the regular expression on each iteration. With Perl 5.6 or
495 later, you should only see \f(CW\*(C`re\*(C'\fR report that for the first iteration.
496 .PP
497 .Vb 1
498 \& use re \*(Aqdebug\*(Aq;
499 \&
500 \& my $regex = \*(AqPerl\*(Aq;
501 \& foreach ( qw(Perl Java Ruby Python) ) {
502 \& print STDERR "\-" x 73, "\en";
503 \& print STDERR "Trying $_...\en";
504 \& print STDERR "\et$_ is good!\en" if m/$regex/;
505 \& }
506 .Ve
507 .SS "How do I use a regular expression to strip C\-style comments from a file?"
508 .IX Subsection "How do I use a regular expression to strip C-style comments from a file?"
509 While this actually can be done, it\*(Aqs much harder than you\*(Aqd think.
510 For example, this one\-liner
511 .PP
512 .Vb 1
513 \& perl \-0777 \-pe \*(Aqs{/\e*.*?\e*/}{}gs\*(Aq foo.c
514 .Ve
515 .PP
516 will work in many but not all cases. You see, it\*(Aqs too simple\-minded for
517 certain kinds of C programs, in particular, those with what appear to be
518 comments in quoted strings. For that, you\*(Aqd need something like this,
519 created by Jeffrey Friedl and later modified by Fred Curtis.
520 .PP
521 .Vb 4
522 \& $/ = undef;
523 \& $_ = <>;
524 \& s#/\e*[^*]*\e*+([^/*][^*]*\e*+)*/|("(\e\e.|[^"\e\e])*"|\*(Aq(\e\e.|[^\*(Aq\e\e])*\*(Aq|.[^/"\*(Aq\e\e]*)#defined $2 ? $2 : ""#gse;
525 \& print;
526 .Ve
527 .PP
528 This could, of course, be more legibly written with the \f(CW\*(C`/x\*(C'\fR modifier, adding
529 whitespace and comments. Here it is expanded, courtesy of Fred Curtis.
530 .PP
531 .Vb 8
532 \& s{
533 \& /\e* ## Start of /* ... */ comment
534 \& [^*]*\e*+ ## Non\-* followed by 1\-or\-more *\*(Aqs
535 \& (
536 \& [^/*][^*]*\e*+
537 \& )* ## 0\-or\-more things which don\*(Aqt start with /
538 \& ## but do end with \*(Aq*\*(Aq
539 \& / ## End of /* ... */ comment
540 \&
541 \& | ## OR various things which aren\*(Aqt comments:
542 \&
543 \& (
544 \& " ## Start of " ... " string
545 \& (
546 \& \e\e. ## Escaped char
547 \& | ## OR
548 \& [^"\e\e] ## Non "\e
549 \& )*
550 \& " ## End of " ... " string
551 \&
552 \& | ## OR
553 \&
554 \& \*(Aq ## Start of \*(Aq ... \*(Aq string
555 \& (
556 \& \e\e. ## Escaped char
557 \& | ## OR
558 \& [^\*(Aq\e\e] ## Non \*(Aq\e
559 \& )*
560 \& \*(Aq ## End of \*(Aq ... \*(Aq string
561 \&
562 \& | ## OR
563 \&
564 \& . ## Anything other char
565 \& [^/"\*(Aq\e\e]* ## Chars which doesn\*(Aqt start a comment, string or escape
566 \& )
567 \& }{defined $2 ? $2 : ""}gxse;
568 .Ve
569 .PP
570 A slight modification also removes C++ comments, possibly spanning multiple lines
571 using a continuation character:
572 .PP
573 .Vb 1
574 \& s#/\e*[^*]*\e*+([^/*][^*]*\e*+)*/|//([^\e\e]|[^\en][\en]?)*?\en|("(\e\e.|[^"\e\e])*"|\*(Aq(\e\e.|[^\*(Aq\e\e])*\*(Aq|.[^/"\*(Aq\e\e]*)#defined $3 ? $3 : ""#gse;
575 .Ve
576 .SS "Can I use Perl regular expressions to match balanced text?"
577 .IX Xref "regex, matching balanced test regexp, matching balanced test regular expression, matching balanced test possessive PARNO Text::Balanced Regexp::Common backtracking recursion"
578 .IX Subsection "Can I use Perl regular expressions to match balanced text?"
579 (contributed by brian d foy)
580 .PP
581 Your first try should probably be the Text::Balanced module, which
582 is in the Perl standard library since Perl 5.8. It has a variety of
583 functions to deal with tricky text. The Regexp::Common module can
584 also help by providing canned patterns you can use.
585 .PP
586 As of Perl 5.10, you can match balanced text with regular expressions
587 using recursive patterns. Before Perl 5.10, you had to resort to
588 various tricks such as using Perl code in \f(CW\*(C`(??{})\*(C'\fR sequences.
589 .PP
590 Here\*(Aqs an example using a recursive regular expression. The goal is to
591 capture all of the text within angle brackets, including the text in
592 nested angle brackets. This sample text has two "major" groups: a
593 group with one level of nesting and a group with two levels of
594 nesting. There are five total groups in angle brackets:
595 .PP
596 .Vb 3
597 \& I have some <brackets in <nested brackets> > and
598 \& <another group <nested once <nested twice> > >
599 \& and that\*(Aqs it.
600 .Ve
601 .PP
602 The regular expression to match the balanced text uses two new (to
603 Perl 5.10) regular expression features. These are covered in perlre
604 and this example is a modified version of one in that documentation.
605 .PP
606 First, adding the new possessive \f(CW\*(C`+\*(C'\fR to any quantifier finds the
607 longest match and does not backtrack. That\*(Aqs important since you want
608 to handle any angle brackets through the recursion, not backtracking.
609 The group \f(CW\*(C`[^<>]++\*(C'\fR finds one or more non\-angle brackets without
610 backtracking.
611 .PP
612 Second, the new \f(CW\*(C`(?PARNO)\*(C'\fR refers to the sub\-pattern in the
613 particular capture group given by \f(CW\*(C`PARNO\*(C'\fR. In the following regex,
614 the first capture group finds (and remembers) the balanced text, and
615 you need that same pattern within the first buffer to get past the
616 nested text. That\*(Aqs the recursive part. The \f(CW\*(C`(?1)\*(C'\fR uses the pattern
617 in the outer capture group as an independent part of the regex.
618 .PP
619 Putting it all together, you have:
620 .PP
621 .Vb 1
622 \& #!/usr/local/bin/perl5.10.0
623 \&
624 \& my $string =<<"HERE";
625 \& I have some <brackets in <nested brackets> > and
626 \& <another group <nested once <nested twice> > >
627 \& and that\*(Aqs it.
628 \& HERE
629 \&
630 \& my @groups = $string =~ m/
631 \& ( # start of capture group 1
632 \& < # match an opening angle bracket
633 \& (?:
634 \& [^<>]++ # one or more non angle brackets, non backtracking
635 \& |
636 \& (?1) # found < or >, so recurse to capture group 1
637 \& )*
638 \& > # match a closing angle bracket
639 \& ) # end of capture group 1
640 \& /xg;
641 \&
642 \& $" = "\en\et";
643 \& print "Found:\en\et@groups\en";
644 .Ve
645 .PP
646 The output shows that Perl found the two major groups:
647 .PP
648 .Vb 3
649 \& Found:
650 \& <brackets in <nested brackets> >
651 \& <another group <nested once <nested twice> > >
652 .Ve
653 .PP
654 With a little extra work, you can get all of the groups in angle
655 brackets even if they are in other angle brackets too. Each time you
656 get a balanced match, remove its outer delimiter (that\*(Aqs the one you
657 just matched so don\*(Aqt match it again) and add it to a queue of strings
658 to process. Keep doing that until you get no matches:
659 .PP
660 .Vb 1
661 \& #!/usr/local/bin/perl5.10.0
662 \&
663 \& my @queue =<<"HERE";
664 \& I have some <brackets in <nested brackets> > and
665 \& <another group <nested once <nested twice> > >
666 \& and that\*(Aqs it.
667 \& HERE
668 \&
669 \& my $regex = qr/
670 \& ( # start of bracket 1
671 \& < # match an opening angle bracket
672 \& (?:
673 \& [^<>]++ # one or more non angle brackets, non backtracking
674 \& |
675 \& (?1) # recurse to bracket 1
676 \& )*
677 \& > # match a closing angle bracket
678 \& ) # end of bracket 1
679 \& /x;
680 \&
681 \& $" = "\en\et";
682 \&
683 \& while( @queue ) {
684 \& my $string = shift @queue;
685 \&
686 \& my @groups = $string =~ m/$regex/g;
687 \& print "Found:\en\et@groups\en\en" if @groups;
688 \&
689 \& unshift @queue, map { s/^<//; s/>$//; $_ } @groups;
690 \& }
691 .Ve
692 .PP
693 The output shows all of the groups. The outermost matches show up
694 first and the nested matches show up later:
695 .PP
696 .Vb 3
697 \& Found:
698 \& <brackets in <nested brackets> >
699 \& <another group <nested once <nested twice> > >
700 \&
701 \& Found:
702 \& <nested brackets>
703 \&
704 \& Found:
705 \& <nested once <nested twice> >
706 \&
707 \& Found:
708 \& <nested twice>
709 .Ve
710 .SS "What does it mean that regexes are greedy? How can I get around it?"
711 .IX Xref "greedy greediness"
712 .IX Subsection "What does it mean that regexes are greedy? How can I get around it?"
713 Most people mean that greedy regexes match as much as they can.
714 Technically speaking, it\*(Aqs actually the quantifiers (\f(CW\*(C`?\*(C'\fR, \f(CW\*(C`*\*(C'\fR, \f(CW\*(C`+\*(C'\fR,
715 \&\f(CW\*(C`{}\*(C'\fR) that are greedy rather than the whole pattern; Perl prefers local
716 greed and immediate gratification to overall greed. To get non\-greedy
717 versions of the same quantifiers, use (\f(CW\*(C`??\*(C'\fR, \f(CW\*(C`*?\*(C'\fR, \f(CW\*(C`+?\*(C'\fR, \f(CW\*(C`{}?\*(C'\fR).
718 .PP
719 An example:
720 .PP
721 .Vb 3
722 \& my $s1 = my $s2 = "I am very very cold";
723 \& $s1 =~ s/ve.*y //; # I am cold
724 \& $s2 =~ s/ve.*?y //; # I am very cold
725 .Ve
726 .PP
727 Notice how the second substitution stopped matching as soon as it
728 encountered "y ". The \f(CW\*(C`*?\*(C'\fR quantifier effectively tells the regular
729 expression engine to find a match as quickly as possible and pass
730 control on to whatever is next in line, as you would if you were
731 playing hot potato.
732 .SS "How do I process each word on each line?"
733 .IX Xref "word"
734 .IX Subsection "How do I process each word on each line?"
735 Use the split function:
736 .PP
737 .Vb 5
738 \& while (<>) {
739 \& foreach my $word ( split ) {
740 \& # do something with $word here
741 \& }
742 \& }
743 .Ve
744 .PP
745 Note that this isn\*(Aqt really a word in the English sense; it\*(Aqs just
746 chunks of consecutive non\-whitespace characters.
747 .PP
748 To work with only alphanumeric sequences (including underscores), you
749 might consider
750 .PP
751 .Vb 5
752 \& while (<>) {
753 \& foreach $word (m/(\ew+)/g) {
754 \& # do something with $word here
755 \& }
756 \& }
757 .Ve
758 .SS "How can I print out a word\-frequency or line\-frequency summary?"
759 .IX Subsection "How can I print out a word-frequency or line-frequency summary?"
760 To do this, you have to parse out each word in the input stream. We\*(Aqll
761 pretend that by word you mean chunk of alphabetics, hyphens, or
762 apostrophes, rather than the non\-whitespace chunk idea of a word given
763 in the previous question:
764 .PP
765 .Vb 6
766 \& my (%seen);
767 \& while (<>) {
768 \& while ( /(\eb[^\eW_\ed][\ew\*(Aq\-]+\eb)/g ) { # misses "\`sheep\*(Aq"
769 \& $seen{$1}++;
770 \& }
771 \& }
772 \&
773 \& while ( my ($word, $count) = each %seen ) {
774 \& print "$count $word\en";
775 \& }
776 .Ve
777 .PP
778 If you wanted to do the same thing for lines, you wouldn\*(Aqt need a
779 regular expression:
780 .PP
781 .Vb 1
782 \& my (%seen);
783 \&
784 \& while (<>) {
785 \& $seen{$_}++;
786 \& }
787 \&
788 \& while ( my ($line, $count) = each %seen ) {
789 \& print "$count $line";
790 \& }
791 .Ve
792 .PP
793 If you want these output in a sorted order, see perlfaq4: "How do I
794 sort a hash (optionally by value instead of key)?".
795 .SS "How can I do approximate matching?"
796 .IX Xref "match, approximate matching, approximate"
797 .IX Subsection "How can I do approximate matching?"
798 See the module String::Approx available from CPAN.
799 .SS "How do I efficiently match many regular expressions at once?"
800 .IX Xref "regex, efficiency regexp, efficiency regular expression, efficiency"
801 .IX Subsection "How do I efficiently match many regular expressions at once?"
802 (contributed by brian d foy)
803 .PP
804 You want to
805 avoid compiling a regular expression every time you want to match it.
806 In this example, perl must recompile the regular expression for every
807 iteration of the \f(CW\*(C`foreach\*(C'\fR loop since \f(CW$pattern\fR can change:
808 .PP
809 .Vb 1
810 \& my @patterns = qw( fo+ ba[rz] );
811 \&
812 \& LINE: while( my $line = <> ) {
813 \& foreach my $pattern ( @patterns ) {
814 \& if( $line =~ m/\eb$pattern\eb/i ) {
815 \& print $line;
816 \& next LINE;
817 \& }
818 \& }
819 \& }
820 .Ve
821 .PP
822 The \f(CW\*(C`qr//\*(C'\fR operator compiles a regular
823 expression, but doesn\*(Aqt apply it. When you use the pre\-compiled
824 version of the regex, perl does less work. In this example, I inserted
825 a \f(CW\*(C`map\*(C'\fR to turn each pattern into its pre\-compiled form. The rest of
826 the script is the same, but faster:
827 .PP
828 .Vb 1
829 \& my @patterns = map { qr/\eb$_\eb/i } qw( fo+ ba[rz] );
830 \&
831 \& LINE: while( my $line = <> ) {
832 \& foreach my $pattern ( @patterns ) {
833 \& if( $line =~ m/$pattern/ ) {
834 \& print $line;
835 \& next LINE;
836 \& }
837 \& }
838 \& }
839 .Ve
840 .PP
841 In some cases, you may be able to make several patterns into a single
842 regular expression. Beware of situations that require backtracking
843 though. In this example, the regex is only compiled once because
844 \&\f(CW$regex\fR doesn\*(Aqt change between iterations:
845 .PP
846 .Vb 1
847 \& my $regex = join \*(Aq|\*(Aq, qw( fo+ ba[rz] );
848 \&
849 \& while( my $line = <> ) {
850 \& print if $line =~ m/\eb(?:$regex)\eb/i;
851 \& }
852 .Ve
853 .PP
854 The function "list2re" in Data::Munge on CPAN can also be used to form
855 a single regex that matches a list of literal strings (not regexes).
856 .PP
857 For more details on regular expression efficiency, see \fIMastering
858 Regular Expressions\fR by Jeffrey Friedl. He explains how the regular
859 expressions engine works and why some patterns are surprisingly
860 inefficient. Once you understand how perl applies regular expressions,
861 you can tune them for individual situations.
862 .ie n .SS "Why don\*(Aqt word\-boundary searches with ""\eb"" work for me?"
863 .el .SS "Why don\*(Aqt word\-boundary searches with \f(CW\eb\fP work for me?"
864 .IX Xref "\\b"
865 .IX Subsection "Why don't word-boundary searches with b work for me?"
866 (contributed by brian d foy)
867 .PP
868 Ensure that you know what \eb really does: it\*(Aqs the boundary between a
869 word character, \ew, and something that isn\*(Aqt a word character. That
870 thing that isn\*(Aqt a word character might be \eW, but it can also be the
871 start or end of the string.
872 .PP
873 It\*(Aqs not (not!) the boundary between whitespace and non\-whitespace,
874 and it\*(Aqs not the stuff between words we use to create sentences.
875 .PP
876 In regex speak, a word boundary (\eb) is a "zero width assertion",
877 meaning that it doesn\*(Aqt represent a character in the string, but a
878 condition at a certain position.
879 .PP
880 For the regular expression, /\ebPerl\eb/, there has to be a word
881 boundary before the "P" and after the "l". As long as something other
882 than a word character precedes the "P" and succeeds the "l", the
883 pattern will match. These strings match /\ebPerl\eb/.
884 .PP
885 .Vb 4
886 \& "Perl" # no word char before "P" or after "l"
887 \& "Perl " # same as previous (space is not a word char)
888 \& "\*(AqPerl\*(Aq" # the "\*(Aq" char is not a word char
889 \& "Perl\*(Aqs" # no word char before "P", non\-word char after "l"
890 .Ve
891 .PP
892 These strings do not match /\ebPerl\eb/.
893 .PP
894 .Vb 2
895 \& "Perl_" # "_" is a word char!
896 \& "Perler" # no word char before "P", but one after "l"
897 .Ve
898 .PP
899 You don\*(Aqt have to use \eb to match words though. You can look for
900 non\-word characters surrounded by word characters. These strings
901 match the pattern /\eb\*(Aq\eb/.
902 .PP
903 .Vb 2
904 \& "don\*(Aqt" # the "\*(Aq" char is surrounded by "n" and "t"
905 \& "qep\*(Aqa\*(Aq" # the "\*(Aq" char is surrounded by "p" and "a"
906 .Ve
907 .PP
908 These strings do not match /\eb\*(Aq\eb/.
909 .PP
910 .Vb 1
911 \& "foo\*(Aq" # there is no word char after non\-word "\*(Aq"
912 .Ve
913 .PP
914 You can also use the complement of \eb, \eB, to specify that there
915 should not be a word boundary.
916 .PP
917 In the pattern /\eBam\eB/, there must be a word character before the "a"
918 and after the "m". These patterns match /\eBam\eB/:
919 .PP
920 .Vb 2
921 \& "llama" # "am" surrounded by word chars
922 \& "Samuel" # same
923 .Ve
924 .PP
925 These strings do not match /\eBam\eB/
926 .PP
927 .Vb 2
928 \& "Sam" # no word boundary before "a", but one after "m"
929 \& "I am Sam" # "am" surrounded by non\-word chars
930 .Ve
931 .SS "Why does using $&, $\`, or $\*(Aq slow my program down?"
932 .IX Xref "$MATCH $& $POSTMATCH $\\*(Aq $PREMATCH $\\`"
933 .IX Subsection "Why does using $&, $`, or $' slow my program down?"
934 (contributed by Anno Siegel)
935 .PP
936 Once Perl sees that you need one of these variables anywhere in the
937 program, it provides them on each and every pattern match. That means
938 that on every pattern match the entire string will be copied, part of it
939 to $\`, part to $&, and part to $\*(Aq. Thus the penalty is most severe with
940 long strings and patterns that match often. Avoid $&, $\*(Aq, and $\` if you
941 can, but if you can\*(Aqt, once you\*(Aqve used them at all, use them at will
942 because you\*(Aqve already paid the price. Remember that some algorithms
943 really appreciate them. As of the 5.005 release, the $& variable is no
944 longer "expensive" the way the other two are.
945 .PP
946 Since Perl 5.6.1 the special variables @\- and @+ can functionally replace
947 $\`, $& and $\*(Aq. These arrays contain pointers to the beginning and end
948 of each match (see perlvar for the full story), so they give you
949 essentially the same information, but without the risk of excessive
950 string copying.
951 .PP
952 Perl 5.10 added three specials, \f(CW\*(C`${^MATCH}\*(C'\fR, \f(CW\*(C`${^PREMATCH}\*(C'\fR, and
953 \&\f(CW\*(C`${^POSTMATCH}\*(C'\fR to do the same job but without the global performance
954 penalty. Perl 5.10 only sets these variables if you compile or execute the
955 regular expression with the \f(CW\*(C`/p\*(C'\fR modifier.
956 .ie n .SS "What good is ""\eG"" in a regular expression?"
957 .el .SS "What good is \f(CW\eG\fP in a regular expression?"
958 .IX Xref "\\G"
959 .IX Subsection "What good is G in a regular expression?"
960 You use the \f(CW\*(C`\eG\*(C'\fR anchor to start the next match on the same
961 string where the last match left off. The regular
962 expression engine cannot skip over any characters to find
963 the next match with this anchor, so \f(CW\*(C`\eG\*(C'\fR is similar to the
964 beginning of string anchor, \f(CW\*(C`^\*(C'\fR. The \f(CW\*(C`\eG\*(C'\fR anchor is typically
965 used with the \f(CW\*(C`g\*(C'\fR modifier. It uses the value of \f(CWpos()\fR
966 as the position to start the next match. As the match
967 operator makes successive matches, it updates \f(CWpos()\fR with the
968 position of the next character past the last match (or the
969 first character of the next match, depending on how you like
970 to look at it). Each string has its own \f(CWpos()\fR value.
971 .PP
972 Suppose you want to match all of consecutive pairs of digits
973 in a string like "1122a44" and stop matching when you
974 encounter non\-digits. You want to match \f(CW11\fR and \f(CW22\fR but
975 the letter \f(CW\*(C`a\*(C'\fR shows up between \f(CW22\fR and \f(CW44\fR and you want
976 to stop at \f(CW\*(C`a\*(C'\fR. Simply matching pairs of digits skips over
977 the \f(CW\*(C`a\*(C'\fR and still matches \f(CW44\fR.
978 .PP
979 .Vb 2
980 \& $_ = "1122a44";
981 \& my @pairs = m/(\ed\ed)/g; # qw( 11 22 44 )
982 .Ve
983 .PP
984 If you use the \f(CW\*(C`\eG\*(C'\fR anchor, you force the match after \f(CW22\fR to
985 start with the \f(CW\*(C`a\*(C'\fR. The regular expression cannot match
986 there since it does not find a digit, so the next match
987 fails and the match operator returns the pairs it already
988 found.
989 .PP
990 .Vb 2
991 \& $_ = "1122a44";
992 \& my @pairs = m/\eG(\ed\ed)/g; # qw( 11 22 )
993 .Ve
994 .PP
995 You can also use the \f(CW\*(C`\eG\*(C'\fR anchor in scalar context. You
996 still need the \f(CW\*(C`g\*(C'\fR modifier.
997 .PP
998 .Vb 4
999 \& $_ = "1122a44";
1000 \& while( m/\eG(\ed\ed)/g ) {
1001 \& print "Found $1\en";
1002 \& }
1003 .Ve
1004 .PP
1005 After the match fails at the letter \f(CW\*(C`a\*(C'\fR, perl resets \f(CWpos()\fR
1006 and the next match on the same string starts at the beginning.
1007 .PP
1008 .Vb 4
1009 \& $_ = "1122a44";
1010 \& while( m/\eG(\ed\ed)/g ) {
1011 \& print "Found $1\en";
1012 \& }
1013 \&
1014 \& print "Found $1 after while" if m/(\ed\ed)/g; # finds "11"
1015 .Ve
1016 .PP
1017 You can disable \f(CWpos()\fR resets on fail with the \f(CW\*(C`c\*(C'\fR modifier, documented
1018 in perlop and perlreref. Subsequent matches start where the last
1019 successful match ended (the value of \f(CWpos()\fR) even if a match on the
1020 same string has failed in the meantime. In this case, the match after
1021 the \f(CWwhile()\fR loop starts at the \f(CW\*(C`a\*(C'\fR (where the last match stopped),
1022 and since it does not use any anchor it can skip over the \f(CW\*(C`a\*(C'\fR to find
1023 \&\f(CW44\fR.
1024 .PP
1025 .Vb 4
1026 \& $_ = "1122a44";
1027 \& while( m/\eG(\ed\ed)/gc ) {
1028 \& print "Found $1\en";
1029 \& }
1030 \&
1031 \& print "Found $1 after while" if m/(\ed\ed)/g; # finds "44"
1032 .Ve
1033 .PP
1034 Typically you use the \f(CW\*(C`\eG\*(C'\fR anchor with the \f(CW\*(C`c\*(C'\fR modifier
1035 when you want to try a different match if one fails,
1036 such as in a tokenizer. Jeffrey Friedl offers this example
1037 which works in 5.004 or later.
1038 .PP
1039 .Vb 9
1040 \& while (<>) {
1041 \& chomp;
1042 \& PARSER: {
1043 \& m/ \eG( \ed+\eb )/gcx && do { print "number: $1\en"; redo; };
1044 \& m/ \eG( \ew+ )/gcx && do { print "word: $1\en"; redo; };
1045 \& m/ \eG( \es+ )/gcx && do { print "space: $1\en"; redo; };
1046 \& m/ \eG( [^\ew\ed]+ )/gcx && do { print "other: $1\en"; redo; };
1047 \& }
1048 \& }
1049 .Ve
1050 .PP
1051 For each line, the \f(CW\*(C`PARSER\*(C'\fR loop first tries to match a series
1052 of digits followed by a word boundary. This match has to
1053 start at the place the last match left off (or the beginning
1054 of the string on the first match). Since \f(CW\*(C`m/ \eG( \ed+\eb
1055 )/gcx\*(C'\fR uses the \f(CW\*(C`c\*(C'\fR modifier, if the string does not match that
1056 regular expression, perl does not reset \fBpos()\fR and the next
1057 match starts at the same position to try a different
1058 pattern.
1059 .SS "Are Perl regexes DFAs or NFAs? Are they POSIX compliant?"
1060 .IX Xref "DFA NFA POSIX"
1061 .IX Subsection "Are Perl regexes DFAs or NFAs? Are they POSIX compliant?"
1062 While it\*(Aqs true that Perl\*(Aqs regular expressions resemble the DFAs
1063 (deterministic finite automata) of the \fBegrep\fR\|(1) program, they are in
1064 fact implemented as NFAs (non\-deterministic finite automata) to allow
1065 backtracking and backreferencing. And they aren\*(Aqt POSIX\-style either,
1066 because those guarantee worst\-case behavior for all cases. (It seems
1067 that some people prefer guarantees of consistency, even when what\*(Aqs
1068 guaranteed is slowness.) See the book "Mastering Regular Expressions"
1069 (from O\*(AqReilly) by Jeffrey Friedl for all the details you could ever
1070 hope to know on these matters (a full citation appears in
1071 perlfaq2).
1072 .SS "What\*(Aqs wrong with using grep in a void context?"
1073 .IX Xref "grep"
1074 .IX Subsection "What's wrong with using grep in a void context?"
1075 The problem is that grep builds a return list, regardless of the context.
1076 This means you\*(Aqre making Perl go to the trouble of building a list that
1077 you then just throw away. If the list is large, you waste both time and space.
1078 If your intent is to iterate over the list, then use a for loop for this
1079 purpose.
1080 .PP
1081 In perls older than 5.8.1, map suffers from this problem as well.
1082 But since 5.8.1, this has been fixed, and map is context aware \- in void
1083 context, no lists are constructed.
1084 .SS "How can I match strings with multibyte characters?"
1085 .IX Xref "regex, and multibyte characters regexp, and multibyte characters regular expression, and multibyte characters martian encoding, Martian"
1086 .IX Subsection "How can I match strings with multibyte characters?"
1087 Starting from Perl 5.6 Perl has had some level of multibyte character
1088 support. Perl 5.8 or later is recommended. Supported multibyte
1089 character repertoires include Unicode, and legacy encodings
1090 through the Encode module. See perluniintro, perlunicode,
1091 and Encode.
1092 .PP
1093 If you are stuck with older Perls, you can do Unicode with the
1094 Unicode::String module, and character conversions using the
1095 Unicode::Map8 and Unicode::Map modules. If you are using
1096 Japanese encodings, you might try using the jperl 5.005_03.
1097 .PP
1098 Finally, the following set of approaches was offered by Jeffrey
1099 Friedl, whose article in issue #5 of The Perl Journal talks about
1100 this very matter.
1101 .PP
1102 Let\*(Aqs suppose you have some weird Martian encoding where pairs of
1103 ASCII uppercase letters encode single Martian letters (i.e. the two
1104 bytes "CV" make a single Martian letter, as do the two bytes "SG",
1105 "VS", "XX", etc.). Other bytes represent single characters, just like
1106 ASCII.
1107 .PP
1108 So, the string of Martian "I am CVSGXX!" uses 12 bytes to encode the
1109 nine characters \*(AqI\*(Aq, \*(Aq \*(Aq, \*(Aqa\*(Aq, \*(Aqm\*(Aq, \*(Aq \*(Aq, \*(AqCV\*(Aq, \*(AqSG\*(Aq, \*(AqXX\*(Aq, \*(Aq!\*(Aq.
1110 .PP
1111 Now, say you want to search for the single character \f(CW\*(C`/GX/\*(C'\fR. Perl
1112 doesn\*(Aqt know about Martian, so it\*(Aqll find the two bytes "GX" in the "I
1113 am CVSGXX!" string, even though that character isn\*(Aqt there: it just
1114 looks like it is because "SG" is next to "XX", but there\*(Aqs no real
1115 "GX". This is a big problem.
1116 .PP
1117 Here are a few ways, all painful, to deal with it:
1118 .PP
1119 .Vb 2
1120 \& # Make sure adjacent "martian" bytes are no longer adjacent.
1121 \& $martian =~ s/([A\-Z][A\-Z])/ $1 /g;
1122 \&
1123 \& print "found GX!\en" if $martian =~ /GX/;
1124 .Ve
1125 .PP
1126 Or like this:
1127 .PP
1128 .Vb 6
1129 \& my @chars = $martian =~ m/([A\-Z][A\-Z]|[^A\-Z])/g;
1130 \& # above is conceptually similar to: my @chars = $text =~ m/(.)/g;
1131 \& #
1132 \& foreach my $char (@chars) {
1133 \& print "found GX!\en", last if $char eq \*(AqGX\*(Aq;
1134 \& }
1135 .Ve
1136 .PP
1137 Or like this:
1138 .PP
1139 .Vb 6
1140 \& while ($martian =~ m/\eG([A\-Z][A\-Z]|.)/gs) { # \eG probably unneeded
1141 \& if ($1 eq \*(AqGX\*(Aq) {
1142 \& print "found GX!\en";
1143 \& last;
1144 \& }
1145 \& }
1146 .Ve
1147 .PP
1148 Here\*(Aqs another, slightly less painful, way to do it from Benjamin
1149 Goldberg, who uses a zero\-width negative look\-behind assertion.
1150 .PP
1151 .Vb 5
1152 \& print "found GX!\en" if $martian =~ m/
1153 \& (?<![A\-Z])
1154 \& (?:[A\-Z][A\-Z])*?
1155 \& GX
1156 \& /x;
1157 .Ve
1158 .PP
1159 This succeeds if the "martian" character GX is in the string, and fails
1160 otherwise. If you don\*(Aqt like using (?<!), a zero\-width negative
1161 look\-behind assertion, you can replace (?<![A\-Z]) with (?:^|[^A\-Z]).
1162 .PP
1163 It does have the drawback of putting the wrong thing in $\-[0] and $+[0],
1164 but this usually can be worked around.
1165 .SS "How do I match a regular expression that\*(Aqs in a variable?"
1166 .IX Xref "regex, in variable eval regex quotemeta \\Q, regex \\E, regex qr"
1167 .IX Subsection "How do I match a regular expression that's in a variable?"
1168 (contributed by brian d foy)
1169 .PP
1170 We don\*(Aqt have to hard\-code patterns into the match operator (or
1171 anything else that works with regular expressions). We can put the
1172 pattern in a variable for later use.
1173 .PP
1174 The match operator is a double quote context, so you can interpolate
1175 your variable just like a double quoted string. In this case, you
1176 read the regular expression as user input and store it in \f(CW$regex\fR.
1177 Once you have the pattern in \f(CW$regex\fR, you use that variable in the
1178 match operator.
1179 .PP
1180 .Vb 1
1181 \& chomp( my $regex = <STDIN> );
1182 \&
1183 \& if( $string =~ m/$regex/ ) { ... }
1184 .Ve
1185 .PP
1186 Any regular expression special characters in \f(CW$regex\fR are still
1187 special, and the pattern still has to be valid or Perl will complain.
1188 For instance, in this pattern there is an unpaired parenthesis.
1189 .PP
1190 .Vb 1
1191 \& my $regex = "Unmatched ( paren";
1192 \&
1193 \& "Two parens to bind them all" =~ m/$regex/;
1194 .Ve
1195 .PP
1196 When Perl compiles the regular expression, it treats the parenthesis
1197 as the start of a memory match. When it doesn\*(Aqt find the closing
1198 parenthesis, it complains:
1199 .PP
1200 .Vb 1
1201 \& Unmatched ( in regex; marked by <\-\- HERE in m/Unmatched ( <\-\- HERE paren/ at script line 3.
1202 .Ve
1203 .PP
1204 You can get around this in several ways depending on our situation.
1205 First, if you don\*(Aqt want any of the characters in the string to be
1206 special, you can escape them with \f(CW\*(C`quotemeta\*(C'\fR before you use the string.
1207 .PP
1208 .Vb 2
1209 \& chomp( my $regex = <STDIN> );
1210 \& $regex = quotemeta( $regex );
1211 \&
1212 \& if( $string =~ m/$regex/ ) { ... }
1213 .Ve
1214 .PP
1215 You can also do this directly in the match operator using the \f(CW\*(C`\eQ\*(C'\fR
1216 and \f(CW\*(C`\eE\*(C'\fR sequences. The \f(CW\*(C`\eQ\*(C'\fR tells Perl where to start escaping
1217 special characters, and the \f(CW\*(C`\eE\*(C'\fR tells it where to stop (see perlop
1218 for more details).
1219 .PP
1220 .Vb 1
1221 \& chomp( my $regex = <STDIN> );
1222 \&
1223 \& if( $string =~ m/\eQ$regex\eE/ ) { ... }
1224 .Ve
1225 .PP
1226 Alternately, you can use \f(CW\*(C`qr//\*(C'\fR, the regular expression quote operator (see
1227 perlop for more details). It quotes and perhaps compiles the pattern,
1228 and you can apply regular expression flags to the pattern.
1229 .PP
1230 .Vb 1
1231 \& chomp( my $input = <STDIN> );
1232 \&
1233 \& my $regex = qr/$input/is;
1234 \&
1235 \& $string =~ m/$regex/ # same as m/$input/is;
1236 .Ve
1237 .PP
1238 You might also want to trap any errors by wrapping an \f(CW\*(C`eval\*(C'\fR block
1239 around the whole thing.
1240 .PP
1241 .Vb 1
1242 \& chomp( my $input = <STDIN> );
1243 \&
1244 \& eval {
1245 \& if( $string =~ m/\eQ$input\eE/ ) { ... }
1246 \& };
1247 \& warn $@ if $@;
1248 .Ve
1249 .PP
1250 Or...
1251 .PP
1252 .Vb 7
1253 \& my $regex = eval { qr/$input/is };
1254 \& if( defined $regex ) {
1255 \& $string =~ m/$regex/;
1256 \& }
1257 \& else {
1258 \& warn $@;
1259 \& }
1260 .Ve
1261 .SH "AUTHOR AND COPYRIGHT"
1262 .IX Header "AUTHOR AND COPYRIGHT"
1263 Copyright (c) 1997\-2010 Tom Christiansen, Nathan Torkington, and
1264 other authors as noted. All rights reserved.
1265 .PP
1266 This documentation is free; you can redistribute it and/or modify it
1267 under the same terms as Perl itself.
1268 .PP
1269 Irrespective of its distribution, all code examples in this file
1270 are hereby placed into the public domain. You are permitted and
1271 encouraged to use this code in your own programs for fun
1272 or for profit as you see fit. A simple comment in the code giving
1273 credit would be courteous but is not required.