Debian Codegraph @

upstream/debian-unstable/man1/perlunicode.1

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man 5.0102 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\" ========================================================================
56 .\"
57 .IX Title "PERLUNICODE 1"
58 .TH PERLUNICODE 1 2025-11-16 "perl v5.40.1" "Perl Programmers Reference Guide"
59 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
60 .\" way too many mistakes in technical documents.
61 .if n .ad l
62 .nh
63 .SH NAME
64 perlunicode \- Unicode support in Perl
65 .SH DESCRIPTION
66 .IX Header "DESCRIPTION"
67 If you haven't already, before reading this document, you should become
68 familiar with both perlunitut and perluniintro.
69 .PP
70 Unicode aims to \fBUNI\fR\-fy the en\-\fBCODE\fR\-ings of all the world's
71 character sets into a single Standard. For quite a few of the various
72 coding standards that existed when Unicode was first created, converting
73 from each to Unicode essentially meant adding a constant to each code
74 point in the original standard, and converting back meant just
75 subtracting that same constant. For ASCII and ISO\-8859\-1, the constant
76 is 0. For ISO\-8859\-5, (Cyrillic) the constant is 864; for Hebrew
77 (ISO\-8859\-8), it's 1488; Thai (ISO\-8859\-11), 3424; and so forth. This
78 made it easy to do the conversions, and facilitated the adoption of
79 Unicode.
80 .PP
81 And it worked; nowadays, those legacy standards are rarely used. Most
82 everyone uses Unicode.
83 .PP
84 Unicode is a comprehensive standard. It specifies many things outside
85 the scope of Perl, such as how to display sequences of characters. For
86 a full discussion of all aspects of Unicode, see
87 <https://www.unicode.org>.
88 .SS "Important Caveats"
89 .IX Subsection "Important Caveats"
90 Even though some of this section may not be understandable to you on
91 first reading, we think it's important enough to highlight some of the
92 gotchas before delving further, so here goes:
93 .PP
94 Unicode support is an extensive requirement. While Perl does not
95 implement the Unicode standard or the accompanying technical reports
96 from cover to cover, Perl does support many Unicode features.
97 .PP
98 Also, the use of Unicode may present security issues that aren't
99 obvious, see "Security Implications of Unicode" below.
100 .ie n .IP "Safest if you ""use feature \*(Aqunicode_strings\*(Aq""" 4
101 .el .IP "Safest if you \f(CWuse feature \*(Aqunicode_strings\*(Aq\fR" 4
102 .IX Item "Safest if you use feature unicode_strings"
103 In order to preserve backward compatibility, Perl does not turn
104 on full internal Unicode support unless the pragma
105 \&\f(CW\*(C`use\ feature\ \*(Aqunicode_strings\*(Aq\*(C'\fR
106 is specified. (This is automatically
107 selected if you \f(CW\*(C`use\ v5.12\*(C'\fR or higher.) Failure to do this can
108 trigger unexpected surprises. See "The "Unicode Bug"" below.
109 .Sp
110 This pragma doesn't affect I/O. Nor does it change the internal
111 representation of strings, only their interpretation. There are still
112 several places where Unicode isn't fully supported, such as in
113 filenames.
114 .IP "Input and Output Layers" 4
115 .IX Item "Input and Output Layers"
116 Use the \f(CW:encoding(...)\fR layer to read from and write to
117 filehandles using the specified encoding. (See open.)
118 .IP "You must convert your non-ASCII, non\-UTF\-8 Perl scripts to be UTF\-8." 4
119 .IX Item "You must convert your non-ASCII, non-UTF-8 Perl scripts to be UTF-8."
120 The encoding module has been deprecated since perl 5.18 and the
121 perl internals it requires have been removed with perl 5.26.
122 .ie n .IP """use utf8"" still needed to enable UTF\-8 in scripts" 4
123 .el .IP "\f(CWuse utf8\fR still needed to enable UTF\-8 in scripts" 4
124 .IX Item "use utf8 still needed to enable UTF-8 in scripts"
125 If your Perl script is itself encoded in UTF\-8,
126 the \f(CW\*(C`use\ utf8\*(C'\fR pragma must be explicitly included to enable
127 recognition of that (in string or regular expression literals, or in
128 identifier names). \fBThis is the only time when an explicit \fR\f(CB\*(C`use\ utf8\*(C'\fR\fB is needed.\fR (See utf8).
129 .Sp
130 If a Perl script begins with the bytes that form the UTF\-8 encoding of
131 the Unicode BYTE ORDER MARK (\f(CW\*(C`BOM\*(C'\fR, see "Unicode Encodings"), those
132 bytes are completely ignored.
133 .IP "UTF\-16 scripts autodetected" 4
134 .IX Item "UTF-16 scripts autodetected"
135 If a Perl script begins with the Unicode \f(CW\*(C`BOM\*(C'\fR (UTF\-16LE,
136 UTF16\-BE), or if the script looks like non\-\f(CW\*(C`BOM\*(C'\fR\-marked
137 UTF\-16 of either endianness, Perl will correctly read in the script as
138 the appropriate Unicode encoding.
139 .SS "Byte and Character Semantics"
140 .IX Subsection "Byte and Character Semantics"
141 Before Unicode, most encodings used 8 bits (a single byte) to encode
142 each character. Thus a character was a byte, and a byte was a
143 character, and there could be only 256 or fewer possible characters.
144 "Byte Semantics" in the title of this section refers to
145 this behavior. There was no need to distinguish between "Byte" and
146 "Character".
147 .PP
148 Then along comes Unicode which has room for over a million characters
149 (and Perl allows for even more). This means that a character may
150 require more than a single byte to represent it, and so the two terms
151 are no longer equivalent. What matter are the characters as whole
152 entities, and not usually the bytes that comprise them. That's what the
153 term "Character Semantics" in the title of this section refers to.
154 .PP
155 Perl had to change internally to decouple "bytes" from "characters".
156 It is important that you too change your ideas, if you haven't already,
157 so that "byte" and "character" no longer mean the same thing in your
158 mind.
159 .PP
160 The basic building block of Perl strings has always been a "character".
161 The changes basically come down to that the implementation no longer
162 thinks that a character is always just a single byte.
163 .PP
164 There are various things to note:
165 .IP \(bu 4
166 String handling functions, for the most part, continue to operate in
167 terms of characters. \f(CWlength()\fR, for example, returns the number of
168 characters in a string, just as before. But that number no longer is
169 necessarily the same as the number of bytes in the string (there may be
170 more bytes than characters). The other such functions include
171 \&\f(CWchop()\fR, \f(CWchomp()\fR, \f(CWsubstr()\fR, \f(CWpos()\fR, \f(CWindex()\fR, \f(CWrindex()\fR,
172 \&\f(CWsort()\fR, \f(CWsprintf()\fR, and \f(CWwrite()\fR.
173 .Sp
174 The exceptions are:
175 .RS 4
176 .IP \(bu 4
177 the bit-oriented \f(CW\*(C`vec\*(C'\fR
178 .Sp
179 \
180 .IP \(bu 4
181 the byte-oriented \f(CW\*(C`pack\*(C'\fR/\f(CW\*(C`unpack\*(C'\fR \f(CW"C"\fR format
182 .Sp
183 However, the \f(CW\*(C`W\*(C'\fR specifier does operate on whole characters, as does the
184 \&\f(CW\*(C`U\*(C'\fR specifier.
185 .IP \(bu 4
186 some operators that interact with the platform's operating system
187 .Sp
188 Operators dealing with filenames are examples.
189 .IP \(bu 4
190 when the functions are called from within the scope of the
191 \&\f(CW\*(C`use\ bytes\*(C'\fR pragma
192 .Sp
193 Likely, you should use this only for debugging anyway.
194 .RE
195 .RS 4
196 .RE
197 .IP \(bu 4
198 Strings\-\-including hash keys\-\-and regular expression patterns may
199 contain characters that have ordinal values larger than 255.
200 .Sp
201 If you use a Unicode editor to edit your program, Unicode characters may
202 occur directly within the literal strings in UTF\-8 encoding, or UTF\-16.
203 (The former requires a \f(CW\*(C`use utf8\*(C'\fR, the latter may require a \f(CW\*(C`BOM\*(C'\fR.)
204 .Sp
205 "Creating Unicode" in perluniintro gives other ways to place non-ASCII
206 characters in your strings.
207 .IP \(bu 4
208 The \f(CWchr()\fR and \f(CWord()\fR functions work on whole characters.
209 .IP \(bu 4
210 Regular expressions match whole characters. For example, \f(CW"."\fR matches
211 a whole character instead of only a single byte.
212 .IP \(bu 4
213 The \f(CW\*(C`tr///\*(C'\fR operator translates whole characters. (Note that the
214 \&\f(CW\*(C`tr///CU\*(C'\fR functionality has been removed. For similar functionality to
215 that, see \f(CW\*(C`pack(\*(AqU0\*(Aq,\ ...)\*(C'\fR and \f(CW\*(C`pack(\*(AqC0\*(Aq,\ ...)\*(C'\fR).
216 .IP \(bu 4
217 \&\f(CW\*(C`scalar reverse()\*(C'\fR reverses by character rather than by byte.
218 .IP \(bu 4
219 The bit string operators, \f(CW\*(C`& | ^ ~\*(C'\fR and (starting in v5.22)
220 \&\f(CW\*(C`&. |. ^. ~.\*(C'\fR can operate on bit strings encoded in UTF\-8, but this
221 can give unexpected results if any of the strings contain code points
222 above 0xFF. Starting in v5.28, it is a fatal error to have such an
223 operand. Otherwise, the operation is performed on a non\-UTF\-8 copy of
224 the operand. If you're not sure about the encoding of a string,
225 downgrade it before using any of these operators; you can use
226 \&\f(CWutf8::utf8_downgrade()\fR.
227 .PP
228 The bottom line is that Perl has always practiced "Character Semantics",
229 but with the advent of Unicode, that is now different than "Byte
230 Semantics".
231 .SS "ASCII Rules versus Unicode Rules"
232 .IX Subsection "ASCII Rules versus Unicode Rules"
233 Before Unicode, when a character was a byte was a character,
234 Perl knew only about the 128 characters defined by ASCII, code points 0
235 through 127 (except for under \f(CW\*(C`use\ locale\*(C'\fR). That
236 left the code
237 points 128 to 255 as unassigned, and available for whatever use a
238 program might want. The only semantics they have is their ordinal
239 numbers, and that they are members of none of the non-negative character
240 classes. None are considered to match \f(CW\*(C`\ew\*(C'\fR for example, but all match
241 \&\f(CW\*(C`\eW\*(C'\fR.
242 .PP
243 Unicode, of course, assigns each of those code points a particular
244 meaning (along with ones above 255). To preserve backward
245 compatibility, Perl only uses the Unicode meanings when there is some
246 indication that Unicode is what is intended; otherwise the non-ASCII
247 code points remain treated as if they are unassigned.
248 .PP
249 Here are the ways that Perl knows that a string should be treated as
250 Unicode:
251 .IP \(bu 4
252 Within the scope of \f(CW\*(C`use\ utf8\*(C'\fR
253 .Sp
254 If the whole program is Unicode (signified by using 8\-bit \fBU\fRnicode
255 \&\fBT\fRransformation \fBF\fRormat), then all literal strings within it must be
256 Unicode.
257 .IP \(bu 4
258 Within the scope of
259 \&\f(CW\*(C`use\ feature\ \*(Aqunicode_strings\*(Aq\*(C'\fR
260 .Sp
261 This pragma was created so you can explicitly tell Perl that operations
262 executed within its scope are to use Unicode rules. More operations are
263 affected with newer perls. See "The "Unicode Bug"".
264 .IP \(bu 4
265 Within the scope of \f(CW\*(C`use\ v5.12\*(C'\fR or higher
266 .Sp
267 This implicitly turns on \f(CW\*(C`use\ feature\ \*(Aqunicode_strings\*(Aq\*(C'\fR.
268 .IP \(bu 4
269 Within the scope of
270 \&\f(CW\*(C`use\ locale\ \*(Aqnot_characters\*(Aq\*(C'\fR,
271 or \f(CW\*(C`use\ locale\*(C'\fR and the current
272 locale is a UTF\-8 locale.
273 .Sp
274 The former is defined to imply Unicode handling; and the latter
275 indicates a Unicode locale, hence a Unicode interpretation of all
276 strings within it.
277 .IP \(bu 4
278 When the string contains a Unicode-only code point
279 .Sp
280 Perl has never accepted code points above 255 without them being
281 Unicode, so their use implies Unicode for the whole string.
282 .IP \(bu 4
283 When the string contains a Unicode named code point \f(CW\*(C`\eN{...}\*(C'\fR
284 .Sp
285 The \f(CW\*(C`\eN{...}\*(C'\fR construct explicitly refers to a Unicode code point,
286 even if it is one that is also in ASCII. Therefore the string
287 containing it must be Unicode.
288 .IP \(bu 4
289 When the string has come from an external source marked as
290 Unicode
291 .Sp
292 The \f(CW\*(C`\-C\*(C'\fR command line option can
293 specify that certain inputs to the program are Unicode, and the values
294 of this can be read by your Perl code, see "${^UNICODE}" in perlvar.
295 .IP \(bu 4
296 When the string has been upgraded to UTF\-8
297 .Sp
298 The function \f(CWutf8::utf8_upgrade()\fR
299 can be explicitly used to permanently (unless a subsequent
300 \&\f(CWutf8::utf8_downgrade()\fR is called) cause a string to be treated as
301 Unicode.
302 .IP \(bu 4
303 There are additional methods for regular expression patterns
304 .Sp
305 A pattern that is compiled with the \f(CW\*(C`/u\*(C'\fR or \f(CW\*(C`/a\*(C'\fR modifiers is
306 treated as Unicode (though there are some restrictions with \f(CW\*(C`/a\*(C'\fR).
307 Under the \f(CW\*(C`/d\*(C'\fR and \f(CW\*(C`/l\*(C'\fR modifiers, there are several other
308 indications for Unicode; see "Character set modifiers" in perlre.
309 .PP
310 Note that all of the above are overridden within the scope of
311 \&\f(CW\*(C`use bytes\*(C'\fR; but you should be using this pragma only for
312 debugging.
313 .PP
314 Note also that some interactions with the platform's operating system
315 never use Unicode rules.
316 .PP
317 When Unicode rules are in effect:
318 .IP \(bu 4
319 Case translation operators use the Unicode case translation tables.
320 .Sp
321 Note that \f(CWuc()\fR, or \f(CW\*(C`\eU\*(C'\fR in interpolated strings, translates to
322 uppercase, while \f(CW\*(C`ucfirst\*(C'\fR, or \f(CW\*(C`\eu\*(C'\fR in interpolated strings,
323 translates to titlecase in languages that make the distinction (which is
324 equivalent to uppercase in languages without the distinction).
325 .Sp
326 There is a CPAN module, \f(CW\*(C`Unicode::Casing\*(C'\fR, which allows you to
327 define your own mappings to be used in \f(CWlc()\fR, \f(CWlcfirst()\fR, \f(CWuc()\fR,
328 \&\f(CWucfirst()\fR, and \f(CW\*(C`fc\*(C'\fR (or their double-quoted string inlined versions
329 such as \f(CW\*(C`\eU\*(C'\fR). (Prior to Perl 5.16, this functionality was partially
330 provided in the Perl core, but suffered from a number of insurmountable
331 drawbacks, so the CPAN module was written instead.)
332 .IP \(bu 4
333 Character classes in regular expressions match based on the character
334 properties specified in the Unicode properties database.
335 .Sp
336 \&\f(CW\*(C`\ew\*(C'\fR can be used to match a Japanese ideograph, for instance; and
337 \&\f(CW\*(C`[[:digit:]]\*(C'\fR a Bengali number.
338 .IP \(bu 4
339 Named Unicode properties, scripts, and block ranges may be used (like
340 bracketed character classes) by using the \f(CW\*(C`\ep{}\*(C'\fR "matches property"
341 construct and the \f(CW\*(C`\eP{}\*(C'\fR negation, "doesn't match property".
342 .Sp
343 See "Unicode Character Properties" for more details.
344 .Sp
345 You can define your own character properties and use them
346 in the regular expression with the \f(CW\*(C`\ep{}\*(C'\fR or \f(CW\*(C`\eP{}\*(C'\fR construct.
347 See "User-Defined Character Properties" for more details.
348 .SS "Extended Grapheme Clusters (Logical characters)"
349 .IX Subsection "Extended Grapheme Clusters (Logical characters)"
350 Consider a character, say \f(CW\*(C`H\*(C'\fR. It could appear with various marks around it,
351 such as an acute accent, or a circumflex, or various hooks, circles, arrows,
352 \&\fIetc.\fR, above, below, to one side or the other, \fIetc\fR. There are many
353 possibilities among the world's languages. The number of combinations is
354 astronomical, and if there were a character for each combination, it would
355 soon exhaust Unicode's more than a million possible characters. So Unicode
356 took a different approach: there is a character for the base \f(CW\*(C`H\*(C'\fR, and a
357 character for each of the possible marks, and these can be variously combined
358 to get a final logical character. So a logical character\-\-what appears to be a
359 single character\-\-can be a sequence of more than one individual characters.
360 The Unicode standard calls these "extended grapheme clusters" (which
361 is an improved version of the no-longer much used "grapheme cluster");
362 Perl furnishes the \f(CW\*(C`\eX\*(C'\fR regular expression construct to match such
363 sequences in their entirety.
364 .PP
365 But Unicode's intent is to unify the existing character set standards and
366 practices, and several pre-existing standards have single characters that
367 mean the same thing as some of these combinations, like ISO\-8859\-1,
368 which has quite a few of them. For example, \f(CW"LATIN CAPITAL LETTER E
369 WITH ACUTE"\fR was already in this standard when Unicode came along.
370 Unicode therefore added it to its repertoire as that single character.
371 But this character is considered by Unicode to be equivalent to the
372 sequence consisting of the character \f(CW"LATIN CAPITAL LETTER E"\fR
373 followed by the character \f(CW"COMBINING ACUTE ACCENT"\fR.
374 .PP
375 \&\f(CW"LATIN CAPITAL LETTER E WITH ACUTE"\fR is called a "pre-composed"
376 character, and its equivalence with the "E" and the "COMBINING ACCENT"
377 sequence is called canonical equivalence. All pre-composed characters
378 are said to have a decomposition (into the equivalent sequence), and the
379 decomposition type is also called canonical. A string may consist
380 as much as possible of precomposed characters, or it may consist of
381 entirely decomposed characters. Unicode calls these respectively,
382 "Normalization Form Composed" (NFC) and "Normalization Form Decomposed".
383 The \f(CW\*(C`Unicode::Normalize\*(C'\fR module contains functions that convert
384 between the two. A string may also have both composed characters and
385 decomposed characters; this module can be used to make it all one or the
386 other.
387 .PP
388 You may be presented with strings in any of these equivalent forms.
389 There is currently nothing in Perl 5 that ignores the differences. So
390 you'll have to specially handle it. The usual advice is to convert your
391 inputs to \f(CW\*(C`NFD\*(C'\fR before processing further.
392 .PP
393 For more detailed information, see <http://unicode.org/reports/tr15/>.
394 .SS "Unicode Character Properties"
395 .IX Subsection "Unicode Character Properties"
396 (The only time that Perl considers a sequence of individual code
397 points as a single logical character is in the \f(CW\*(C`\eX\*(C'\fR construct, already
398 mentioned above. Therefore "character" in this discussion means a single
399 Unicode code point.)
400 .PP
401 Very nearly all Unicode character properties are accessible through
402 regular expressions by using the \f(CW\*(C`\ep{}\*(C'\fR "matches property" construct
403 and the \f(CW\*(C`\eP{}\*(C'\fR "doesn't match property" for its negation.
404 .PP
405 For instance, \f(CW\*(C`\ep{Uppercase}\*(C'\fR matches any single character with the Unicode
406 \&\f(CW"Uppercase"\fR property, while \f(CW\*(C`\ep{L}\*(C'\fR matches any character with a
407 \&\f(CW\*(C`General_Category\*(C'\fR of \f(CW"L"\fR (letter) property (see
408 "General_Category" below). Brackets are not
409 required for single letter property names, so \f(CW\*(C`\ep{L}\*(C'\fR is equivalent to \f(CW\*(C`\epL\*(C'\fR.
410 .PP
411 More formally, \f(CW\*(C`\ep{Uppercase}\*(C'\fR matches any single character whose Unicode
412 \&\f(CW\*(C`Uppercase\*(C'\fR property value is \f(CW\*(C`True\*(C'\fR, and \f(CW\*(C`\eP{Uppercase}\*(C'\fR matches any character
413 whose \f(CW\*(C`Uppercase\*(C'\fR property value is \f(CW\*(C`False\*(C'\fR, and they could have been written as
414 \&\f(CW\*(C`\ep{Uppercase=True}\*(C'\fR and \f(CW\*(C`\ep{Uppercase=False}\*(C'\fR, respectively.
415 .PP
416 This formality is needed when properties are not binary; that is, if they can
417 take on more values than just \f(CW\*(C`True\*(C'\fR and \f(CW\*(C`False\*(C'\fR. For example, the
418 \&\f(CW\*(C`Bidi_Class\*(C'\fR property (see "Bidirectional Character Types" below),
419 can take on several different
420 values, such as \f(CW\*(C`Left\*(C'\fR, \f(CW\*(C`Right\*(C'\fR, \f(CW\*(C`Whitespace\*(C'\fR, and others. To match these, one needs
421 to specify both the property name (\f(CW\*(C`Bidi_Class\*(C'\fR), AND the value being
422 matched against
423 (\f(CW\*(C`Left\*(C'\fR, \f(CW\*(C`Right\*(C'\fR, \fIetc.\fR). This is done, as in the examples above, by having the
424 two components separated by an equal sign (or interchangeably, a colon), like
425 \&\f(CW\*(C`\ep{Bidi_Class: Left}\*(C'\fR.
426 .PP
427 All Unicode-defined character properties may be written in these compound forms
428 of \f(CW\*(C`\ep{\fR\f(CIproperty\fR\f(CW=\fR\f(CIvalue\fR\f(CW}\*(C'\fR or \f(CW\*(C`\ep{\fR\f(CIproperty\fR\f(CW:\fR\f(CIvalue\fR\f(CW}\*(C'\fR, but Perl provides some
429 additional properties that are written only in the single form, as well as
430 single-form short-cuts for all binary properties and certain others described
431 below, in which you may omit the property name and the equals or colon
432 separator.
433 .PP
434 Most Unicode character properties have at least two synonyms (or aliases if you
435 prefer): a short one that is easier to type and a longer one that is more
436 descriptive and hence easier to understand. Thus the \f(CW"L"\fR and
437 \&\f(CW"Letter"\fR properties above are equivalent and can be used
438 interchangeably. Likewise, \f(CW"Upper"\fR is a synonym for \f(CW"Uppercase"\fR,
439 and we could have written \f(CW\*(C`\ep{Uppercase}\*(C'\fR equivalently as \f(CW\*(C`\ep{Upper}\*(C'\fR.
440 Also, there are typically various synonyms for the values the property
441 can be. For binary properties, \f(CW"True"\fR has 3 synonyms: \f(CW"T"\fR,
442 \&\f(CW"Yes"\fR, and \f(CW"Y"\fR; and \f(CW"False"\fR has correspondingly \f(CW"F"\fR,
443 \&\f(CW"No"\fR, and \f(CW"N"\fR. But be careful. A short form of a value for one
444 property may not mean the same thing as the short form spelled the same
445 for another.
446 Thus, for the \f(CW"General_Category"\fR property, \f(CW"L"\fR means
447 \&\f(CW"Letter"\fR, but for the \f(CW\*(C`Bidi_Class\*(C'\fR
448 property, \f(CW"L"\fR means \f(CW"Left"\fR. A complete list of properties and
449 synonyms is in perluniprops.
450 .PP
451 Upper/lower case differences in property names and values are irrelevant;
452 thus \f(CW\*(C`\ep{Upper}\*(C'\fR means the same thing as \f(CW\*(C`\ep{upper}\*(C'\fR or even \f(CW\*(C`\ep{UpPeR}\*(C'\fR.
453 Similarly, you can add or subtract underscores anywhere in the middle of a
454 word, so that these are also equivalent to \f(CW\*(C`\ep{U_p_p_e_r}\*(C'\fR. And white space
455 is generally irrelevant adjacent to non-word characters, such as the
456 braces and the equals or colon separators, so \f(CW\*(C`\ep{ Upper }\*(C'\fR and
457 \&\f(CW\*(C`\ep{ Upper_case : Y }\*(C'\fR are equivalent to these as well. In fact, white
458 space and even hyphens can usually be added or deleted anywhere. So
459 even \f(CW\*(C`\ep{ Up\-per case = Yes}\*(C'\fR is equivalent. All this is called
460 "loose-matching" by Unicode. The "name" property has some restrictions
461 on this due to a few outlier names. Full details are given in
462 <https://www.unicode.org/reports/tr44/tr44\-24.html#UAX44\-LM2>.
463 .PP
464 The few places where stricter matching is
465 used is in the middle of numbers, the "name" property, and in the Perl
466 extension properties that begin or end with an underscore. Stricter
467 matching cares about white space (except adjacent to non-word
468 characters), hyphens, and non-interior underscores.
469 .PP
470 You can also use negation in both \f(CW\*(C`\ep{}\*(C'\fR and \f(CW\*(C`\eP{}\*(C'\fR by introducing a caret
471 (\f(CW\*(C`^\*(C'\fR) between the first brace and the property name: \f(CW\*(C`\ep{^Tamil}\*(C'\fR is
472 equal to \f(CW\*(C`\eP{Tamil}\*(C'\fR.
473 .PP
474 Almost all properties are immune to case-insensitive matching. That is,
475 adding a \f(CW\*(C`/i\*(C'\fR regular expression modifier does not change what they
476 match. There are two sets that are affected.
477 The first set is
478 \&\f(CW\*(C`Uppercase_Letter\*(C'\fR,
479 \&\f(CW\*(C`Lowercase_Letter\*(C'\fR,
480 and \f(CW\*(C`Titlecase_Letter\*(C'\fR,
481 all of which match \f(CW\*(C`Cased_Letter\*(C'\fR under \f(CW\*(C`/i\*(C'\fR matching.
482 And the second set is
483 \&\f(CW\*(C`Uppercase\*(C'\fR,
484 \&\f(CW\*(C`Lowercase\*(C'\fR,
485 and \f(CW\*(C`Titlecase\*(C'\fR,
486 all of which match \f(CW\*(C`Cased\*(C'\fR under \f(CW\*(C`/i\*(C'\fR matching.
487 This set also includes its subsets \f(CW\*(C`PosixUpper\*(C'\fR and \f(CW\*(C`PosixLower\*(C'\fR both
488 of which under \f(CW\*(C`/i\*(C'\fR match \f(CW\*(C`PosixAlpha\*(C'\fR.
489 (The difference between these sets is that some things, such as Roman
490 numerals, come in both upper and lower case so they are \f(CW\*(C`Cased\*(C'\fR, but
491 aren't considered letters, so they aren't \f(CW\*(C`Cased_Letter\*(C'\fR's.)
492 .PP
493 See "Beyond Unicode code points" for special considerations when
494 matching Unicode properties against non-Unicode code points.
495 .PP
496 \fR\f(BIGeneral_Category\fR\fI\fR
497 .IX Subsection "General_Category"
498 .PP
499 Every Unicode character is assigned a general category, which is the "most
500 usual categorization of a character" (from
501 <https://www.unicode.org/reports/tr44>).
502 .PP
503 The compound way of writing these is like \f(CW\*(C`\ep{General_Category=Number}\*(C'\fR
504 (short: \f(CW\*(C`\ep{gc:n}\*(C'\fR). But Perl furnishes shortcuts in which everything up
505 through the equal or colon separator is omitted. So you can instead just write
506 \&\f(CW\*(C`\epN\*(C'\fR.
507 .PP
508 Here are the short and long forms of the values the \f(CW\*(C`General Category\*(C'\fR property
509 can have:
510 .PP
511 .Vb 1
512 \& Short Long
513 \&
514 \& L Letter
515 \& LC, L& Cased_Letter (that is: [\ep{Ll}\ep{Lu}\ep{Lt}])
516 \& Lu Uppercase_Letter
517 \& Ll Lowercase_Letter
518 \& Lt Titlecase_Letter
519 \& Lm Modifier_Letter
520 \& Lo Other_Letter
521 \&
522 \& M Mark
523 \& Mn Nonspacing_Mark
524 \& Mc Spacing_Mark
525 \& Me Enclosing_Mark
526 \&
527 \& N Number
528 \& Nd Decimal_Number (also Digit)
529 \& Nl Letter_Number
530 \& No Other_Number
531 \&
532 \& P Punctuation (also Punct)
533 \& Pc Connector_Punctuation
534 \& Pd Dash_Punctuation
535 \& Ps Open_Punctuation
536 \& Pe Close_Punctuation
537 \& Pi Initial_Punctuation
538 \& (may behave like Ps or Pe depending on usage)
539 \& Pf Final_Punctuation
540 \& (may behave like Ps or Pe depending on usage)
541 \& Po Other_Punctuation
542 \&
543 \& S Symbol
544 \& Sm Math_Symbol
545 \& Sc Currency_Symbol
546 \& Sk Modifier_Symbol
547 \& So Other_Symbol
548 \&
549 \& Z Separator
550 \& Zs Space_Separator
551 \& Zl Line_Separator
552 \& Zp Paragraph_Separator
553 \&
554 \& C Other
555 \& Cc Control (also Cntrl)
556 \& Cf Format
557 \& Cs Surrogate
558 \& Co Private_Use
559 \& Cn Unassigned
560 .Ve
561 .PP
562 Single-letter properties match all characters in any of the
563 two-letter sub-properties starting with the same letter.
564 \&\f(CW\*(C`LC\*(C'\fR and \f(CW\*(C`L&\*(C'\fR are special: both are aliases for the set consisting of everything matched by \f(CW\*(C`Ll\*(C'\fR, \f(CW\*(C`Lu\*(C'\fR, and \f(CW\*(C`Lt\*(C'\fR.
565 .PP
566 \fR\f(BIBidirectional Character Types\fR\fI\fR
567 .IX Subsection "Bidirectional Character Types"
568 .PP
569 Because scripts differ in their directionality (Hebrew and Arabic are
570 written right to left, for example) Unicode supplies a \f(CW\*(C`Bidi_Class\*(C'\fR property.
571 Some of the values this property can have are:
572 .PP
573 .Vb 1
574 \& Value Meaning
575 \&
576 \& L Left\-to\-Right
577 \& LRE Left\-to\-Right Embedding
578 \& LRO Left\-to\-Right Override
579 \& R Right\-to\-Left
580 \& AL Arabic Letter
581 \& RLE Right\-to\-Left Embedding
582 \& RLO Right\-to\-Left Override
583 \& PDF Pop Directional Format
584 \& EN European Number
585 \& ES European Separator
586 \& ET European Terminator
587 \& AN Arabic Number
588 \& CS Common Separator
589 \& NSM Non\-Spacing Mark
590 \& BN Boundary Neutral
591 \& B Paragraph Separator
592 \& S Segment Separator
593 \& WS Whitespace
594 \& ON Other Neutrals
595 .Ve
596 .PP
597 This property is always written in the compound form.
598 For example, \f(CW\*(C`\ep{Bidi_Class:R}\*(C'\fR matches characters that are normally
599 written right to left. Unlike the
600 \&\f(CW"General_Category"\fR property, this
601 property can have more values added in a future Unicode release. Those
602 listed above comprised the complete set for many Unicode releases, but
603 others were added in Unicode 6.3; you can always find what the
604 current ones are in perluniprops. And
605 <https://www.unicode.org/reports/tr9/> describes how to use them.
606 .PP
607 \fR\f(BIScripts\fR\fI\fR
608 .IX Subsection "Scripts"
609 .PP
610 The world's languages are written in many different scripts. This sentence
611 (unless you're reading it in translation) is written in Latin, while Russian is
612 written in Cyrillic, and Greek is written in, well, Greek; Japanese mainly in
613 Hiragana or Katakana. There are many more.
614 .PP
615 The Unicode \f(CW\*(C`Script\*(C'\fR and \f(CW\*(C`Script_Extensions\*(C'\fR properties give what
616 script a given character is in. The \f(CW\*(C`Script_Extensions\*(C'\fR property is an
617 improved version of \f(CW\*(C`Script\*(C'\fR, as demonstrated below. Either property
618 can be specified with the compound form like
619 \&\f(CW\*(C`\ep{Script=Hebrew}\*(C'\fR (short: \f(CW\*(C`\ep{sc=hebr}\*(C'\fR), or
620 \&\f(CW\*(C`\ep{Script_Extensions=Javanese}\*(C'\fR (short: \f(CW\*(C`\ep{scx=java}\*(C'\fR).
621 In addition, Perl furnishes shortcuts for all
622 \&\f(CW\*(C`Script_Extensions\*(C'\fR property names. You can omit everything up through
623 the equals (or colon), and simply write \f(CW\*(C`\ep{Latin}\*(C'\fR or \f(CW\*(C`\eP{Cyrillic}\*(C'\fR.
624 (This is not true for \f(CW\*(C`Script\*(C'\fR, which is required to be
625 written in the compound form. Prior to Perl v5.26, the single form
626 returned the plain old \f(CW\*(C`Script\*(C'\fR version, but was changed because
627 \&\f(CW\*(C`Script_Extensions\*(C'\fR gives better results.)
628 .PP
629 The difference between these two properties involves characters that are
630 used in multiple scripts. For example the digits '0' through '9' are
631 used in many parts of the world. These are placed in a script named
632 \&\f(CW\*(C`Common\*(C'\fR. Other characters are used in just a few scripts. For
633 example, the \f(CW"KATAKANA\-HIRAGANA DOUBLE HYPHEN"\fR is used in both Japanese
634 scripts, Katakana and Hiragana, but nowhere else. The \f(CW\*(C`Script\*(C'\fR
635 property places all characters that are used in multiple scripts in the
636 \&\f(CW\*(C`Common\*(C'\fR script, while the \f(CW\*(C`Script_Extensions\*(C'\fR property places those
637 that are used in only a few scripts into each of those scripts; while
638 still using \f(CW\*(C`Common\*(C'\fR for those used in many scripts. Thus both these
639 match:
640 .PP
641 .Vb 2
642 \& "0" =~ /\ep{sc=Common}/ # Matches
643 \& "0" =~ /\ep{scx=Common}/ # Matches
644 .Ve
645 .PP
646 and only the first of these match:
647 .PP
648 .Vb 2
649 \& "\eN{KATAKANA\-HIRAGANA DOUBLE HYPHEN}" =~ /\ep{sc=Common} # Matches
650 \& "\eN{KATAKANA\-HIRAGANA DOUBLE HYPHEN}" =~ /\ep{scx=Common} # No match
651 .Ve
652 .PP
653 And only the last two of these match:
654 .PP
655 .Vb 4
656 \& "\eN{KATAKANA\-HIRAGANA DOUBLE HYPHEN}" =~ /\ep{sc=Hiragana} # No match
657 \& "\eN{KATAKANA\-HIRAGANA DOUBLE HYPHEN}" =~ /\ep{sc=Katakana} # No match
658 \& "\eN{KATAKANA\-HIRAGANA DOUBLE HYPHEN}" =~ /\ep{scx=Hiragana} # Matches
659 \& "\eN{KATAKANA\-HIRAGANA DOUBLE HYPHEN}" =~ /\ep{scx=Katakana} # Matches
660 .Ve
661 .PP
662 \&\f(CW\*(C`Script_Extensions\*(C'\fR is thus an improved \f(CW\*(C`Script\*(C'\fR, in which there are
663 fewer characters in the \f(CW\*(C`Common\*(C'\fR script, and correspondingly more in
664 other scripts. It is new in Unicode version 6.0, and its data are likely
665 to change significantly in later releases, as things get sorted out.
666 New code should probably be using \f(CW\*(C`Script_Extensions\*(C'\fR and not plain
667 \&\f(CW\*(C`Script\*(C'\fR. If you compile perl with a Unicode release that doesn't have
668 \&\f(CW\*(C`Script_Extensions\*(C'\fR, the single form Perl extensions will instead refer
669 to the plain \f(CW\*(C`Script\*(C'\fR property. If you compile with a version of
670 Unicode that doesn't have the \f(CW\*(C`Script\*(C'\fR property, these extensions will
671 not be defined at all.
672 .PP
673 (Actually, besides \f(CW\*(C`Common\*(C'\fR, the \f(CW\*(C`Inherited\*(C'\fR script, contains
674 characters that are used in multiple scripts. These are modifier
675 characters which inherit the script value
676 of the controlling character. Some of these are used in many scripts,
677 and so go into \f(CW\*(C`Inherited\*(C'\fR in both \f(CW\*(C`Script\*(C'\fR and \f(CW\*(C`Script_Extensions\*(C'\fR.
678 Others are used in just a few scripts, so are in \f(CW\*(C`Inherited\*(C'\fR in
679 \&\f(CW\*(C`Script\*(C'\fR, but not in \f(CW\*(C`Script_Extensions\*(C'\fR.)
680 .PP
681 It is worth stressing that there are several different sets of digits in
682 Unicode that are equivalent to 0\-9 and are matchable by \f(CW\*(C`\ed\*(C'\fR in a
683 regular expression. If they are used in a single language only, they
684 are in that language's \f(CW\*(C`Script\*(C'\fR and \f(CW\*(C`Script_Extensions\*(C'\fR. If they are
685 used in more than one script, they will be in \f(CW\*(C`sc=Common\*(C'\fR, but only
686 if they are used in many scripts should they be in \f(CW\*(C`scx=Common\*(C'\fR.
687 .PP
688 The explanation above has omitted some detail; refer to UAX#24 "Unicode
689 Script Property": <https://www.unicode.org/reports/tr24>.
690 .PP
691 A complete list of scripts and their shortcuts is in perluniprops.
692 .PP
693 \fR\f(BIUse of the \fR\f(CB"Is"\fR\f(BI Prefix\fR\fI\fR
694 .IX Subsection "Use of the ""Is"" Prefix"
695 .PP
696 For backward compatibility (with ancient Perl 5.6), all properties writable
697 without using the compound form mentioned
698 so far may have \f(CW\*(C`Is\*(C'\fR or \f(CW\*(C`Is_\*(C'\fR prepended to their name, so \f(CW\*(C`\eP{Is_Lu}\*(C'\fR, for
699 example, is equal to \f(CW\*(C`\eP{Lu}\*(C'\fR, and \f(CW\*(C`\ep{IsScript:Arabic}\*(C'\fR is equal to
700 \&\f(CW\*(C`\ep{Arabic}\*(C'\fR.
701 .PP
702 \fR\f(BIBlocks\fR\fI\fR
703 .IX Subsection "Blocks"
704 .PP
705 In addition to \fBscripts\fR, Unicode also defines \fBblocks\fR of
706 characters. The difference between scripts and blocks is that the
707 concept of scripts is closer to natural languages, while the concept
708 of blocks is more of an artificial grouping based on groups of Unicode
709 characters with consecutive ordinal values. For example, the \f(CW"Basic Latin"\fR
710 block is all the characters whose ordinals are between 0 and 127, inclusive; in
711 other words, the ASCII characters. The \f(CW"Latin"\fR script contains some letters
712 from this as well as several other blocks, like \f(CW"Latin\-1 Supplement"\fR,
713 \&\f(CW"Latin Extended\-A"\fR, \fIetc.\fR, but it does not contain all the characters from
714 those blocks. It does not, for example, contain the digits 0\-9, because
715 those digits are shared across many scripts, and hence are in the
716 \&\f(CW\*(C`Common\*(C'\fR script.
717 .PP
718 For more about scripts versus blocks, see UAX#24 "Unicode Script Property":
719 <https://www.unicode.org/reports/tr24>
720 .PP
721 The \f(CW\*(C`Script_Extensions\*(C'\fR or \f(CW\*(C`Script\*(C'\fR properties are likely to be the
722 ones you want to use when processing
723 natural language; the \f(CW\*(C`Block\*(C'\fR property may occasionally be useful in working
724 with the nuts and bolts of Unicode.
725 .PP
726 Block names are matched in the compound form, like \f(CW\*(C`\ep{Block: Arrows}\*(C'\fR or
727 \&\f(CW\*(C`\ep{Blk=Hebrew}\*(C'\fR. Unlike most other properties, only a few block names have a
728 Unicode-defined short name.
729 .PP
730 Perl also defines single form synonyms for the block property in cases
731 where these do not conflict with something else. But don't use any of
732 these, because they are unstable. Since these are Perl extensions, they
733 are subordinate to official Unicode property names; Unicode doesn't know
734 nor care about Perl's extensions. It may happen that a name that
735 currently means the Perl extension will later be changed without warning
736 to mean a different Unicode property in a future version of the perl
737 interpreter that uses a later Unicode release, and your code would no
738 longer work. The extensions are mentioned here for completeness: Take
739 the block name and prefix it with one of: \f(CW\*(C`In\*(C'\fR (for example
740 \&\f(CW\*(C`\ep{Blk=Arrows}\*(C'\fR can currently be written as \f(CW\*(C`\ep{In_Arrows}\*(C'\fR); or
741 sometimes \f(CW\*(C`Is\*(C'\fR (like \f(CW\*(C`\ep{Is_Arrows}\*(C'\fR); or sometimes no prefix at all
742 (\f(CW\*(C`\ep{Arrows}\*(C'\fR). As of this writing (Unicode 9.0) there are no
743 conflicts with using the \f(CW\*(C`In_\*(C'\fR prefix, but there are plenty with the
744 other two forms. For example, \f(CW\*(C`\ep{Is_Hebrew}\*(C'\fR and \f(CW\*(C`\ep{Hebrew}\*(C'\fR mean
745 \&\f(CW\*(C`\ep{Script_Extensions=Hebrew}\*(C'\fR which is NOT the same thing as
746 \&\f(CW\*(C`\ep{Blk=Hebrew}\*(C'\fR. Our
747 advice used to be to use the \f(CW\*(C`In_\*(C'\fR prefix as a single form way of
748 specifying a block. But Unicode 8.0 added properties whose names begin
749 with \f(CW\*(C`In\*(C'\fR, and it's now clear that it's only luck that's so far
750 prevented a conflict. Using \f(CW\*(C`In\*(C'\fR is only marginally less typing than
751 \&\f(CW\*(C`Blk:\*(C'\fR, and the latter's meaning is clearer anyway, and guaranteed to
752 never conflict. So don't take chances. Use \f(CW\*(C`\ep{Blk=foo}\*(C'\fR for new
753 code. And be sure that block is what you really really want to do. In
754 most cases scripts are what you want instead.
755 .PP
756 A complete list of blocks is in perluniprops.
757 .PP
758 \fR\f(BIOther Properties\fR\fI\fR
759 .IX Subsection "Other Properties"
760 .PP
761 There are many more properties than the very basic ones described here.
762 A complete list is in perluniprops.
763 .PP
764 Unicode defines all its properties in the compound form, so all single-form
765 properties are Perl extensions. Most of these are just synonyms for the
766 Unicode ones, but some are genuine extensions, including several that are in
767 the compound form. And quite a few of these are actually recommended by Unicode
768 (in <https://www.unicode.org/reports/tr18>).
769 .PP
770 This section gives some details on all extensions that aren't just
771 synonyms for compound-form Unicode properties
772 (for those properties, you'll have to refer to the
773 Unicode Standard <https://www.unicode.org/reports/tr44>.
774 .ie n .IP "\fR\fB""\ep{All}""\fR\fB\fR" 4
775 .el .IP \fR\f(CB\ep{All}\fR\fB\fR 4
776 .IX Item "p{All}"
777 This matches every possible code point. It is equivalent to \f(CW\*(C`qr/./s\*(C'\fR.
778 Unlike all the other non-user-defined \f(CW\*(C`\ep{}\*(C'\fR property matches, no
779 warning is ever generated if this is property is matched against a
780 non-Unicode code point (see "Beyond Unicode code points" below).
781 .ie n .IP "\fR\fB""\ep{Alnum}""\fR\fB\fR" 4
782 .el .IP \fR\f(CB\ep{Alnum}\fR\fB\fR 4
783 .IX Item "p{Alnum}"
784 This matches any \f(CW\*(C`\ep{Alphabetic}\*(C'\fR or \f(CW\*(C`\ep{Decimal_Number}\*(C'\fR character.
785 .ie n .IP "\fR\fB""\ep{Any}""\fR\fB\fR" 4
786 .el .IP \fR\f(CB\ep{Any}\fR\fB\fR 4
787 .IX Item "p{Any}"
788 This matches any of the 1_114_112 Unicode code points. It is a synonym
789 for \f(CW\*(C`\ep{Unicode}\*(C'\fR.
790 .ie n .IP "\fR\fB""\ep{ASCII}""\fR\fB\fR" 4
791 .el .IP \fR\f(CB\ep{ASCII}\fR\fB\fR 4
792 .IX Item "p{ASCII}"
793 This matches any of the 128 characters in the US-ASCII character set,
794 which is a subset of Unicode.
795 .ie n .IP "\fR\fB""\ep{Assigned}""\fR\fB\fR" 4
796 .el .IP \fR\f(CB\ep{Assigned}\fR\fB\fR 4
797 .IX Item "p{Assigned}"
798 This matches any assigned code point; that is, any code point whose general
799 category is not \f(CW\*(C`Unassigned\*(C'\fR (or equivalently, not \f(CW\*(C`Cn\*(C'\fR).
800 .ie n .IP "\fR\fB""\ep{Blank}""\fR\fB\fR" 4
801 .el .IP \fR\f(CB\ep{Blank}\fR\fB\fR 4
802 .IX Item "p{Blank}"
803 This is the same as \f(CW\*(C`\eh\*(C'\fR and \f(CW\*(C`\ep{HorizSpace}\*(C'\fR: A character that changes the
804 spacing horizontally.
805 .ie n .IP "\fR\fB""\ep{Decomposition_Type: Non_Canonical}""\fR\fB\fR (Short: ""\ep{Dt=NonCanon}"")" 4
806 .el .IP "\fR\f(CB\ep{Decomposition_Type: Non_Canonical}\fR\fB\fR (Short: \f(CW\ep{Dt=NonCanon}\fR)" 4
807 .IX Item "p{Decomposition_Type: Non_Canonical} (Short: p{Dt=NonCanon})"
808 Matches a character that has any of the non-canonical decomposition
809 types. Canonical decompositions are introduced in the
810 "Extended Grapheme Clusters (Logical characters)" section above.
811 However, many more characters have a different type of decomposition,
812 generically called "compatible" decompositions, or "non-canonical". The
813 sequences that form these decompositions are not considered canonically
814 equivalent to the pre-composed character. An example is the
815 \&\f(CW"SUPERSCRIPT ONE"\fR. It is somewhat like a regular digit 1, but not
816 exactly; its decomposition into the digit 1 is called a "compatible"
817 decomposition, specifically a "super" (for "superscript") decomposition.
818 There are several such compatibility decompositions (see
819 <https://www.unicode.org/reports/tr44>). \f(CW\*(C`\ep{Dt:\ Non_Canon}\*(C'\fR is a
820 Perl extension that uses just one name to refer to the union of all of
821 them.
822 .Sp
823 Most Unicode characters don't have a decomposition, so their
824 decomposition type is \f(CW"None"\fR. Hence, \f(CW\*(C`Non_Canonical\*(C'\fR is equivalent
825 to
826 .Sp
827 .Vb 1
828 \& qr/(?[ \eP{DT=Canonical} \- \ep{DT=None} ])/
829 .Ve
830 .Sp
831 (Note that one of the non-canonical decompositions is named "compat",
832 which could perhaps have been better named "miscellaneous". It includes
833 just the things that Unicode couldn't figure out a better generic name
834 for.)
835 .ie n .IP "\fR\fB""\ep{Graph}""\fR\fB\fR" 4
836 .el .IP \fR\f(CB\ep{Graph}\fR\fB\fR 4
837 .IX Item "p{Graph}"
838 Matches any character that is graphic. Theoretically, this means a character
839 that on a printer would cause ink to be used.
840 .ie n .IP "\fR\fB""\ep{HorizSpace}""\fR\fB\fR" 4
841 .el .IP \fR\f(CB\ep{HorizSpace}\fR\fB\fR 4
842 .IX Item "p{HorizSpace}"
843 This is the same as \f(CW\*(C`\eh\*(C'\fR and \f(CW\*(C`\ep{Blank}\*(C'\fR: a character that changes the
844 spacing horizontally.
845 .ie n .IP "\fR\fB""\ep{In=*}""\fR\fB\fR" 4
846 .el .IP \fR\f(CB\ep{In=*}\fR\fB\fR 4
847 .IX Item "p{In=*}"
848 This is a synonym for \f(CW\*(C`\ep{Present_In=*}\*(C'\fR
849 .ie n .IP "\fR\fB""\ep{PerlSpace}""\fR\fB\fR" 4
850 .el .IP \fR\f(CB\ep{PerlSpace}\fR\fB\fR 4
851 .IX Item "p{PerlSpace}"
852 This is the same as \f(CW\*(C`\es\*(C'\fR, restricted to ASCII, namely \f(CW\*(C`[\ \ef\en\er\et]\*(C'\fR
853 and starting in Perl v5.18, a vertical tab.
854 .Sp
855 Mnemonic: Perl's (original) space
856 .ie n .IP "\fR\fB""\ep{PerlWord}""\fR\fB\fR" 4
857 .el .IP \fR\f(CB\ep{PerlWord}\fR\fB\fR 4
858 .IX Item "p{PerlWord}"
859 This is the same as \f(CW\*(C`\ew\*(C'\fR, restricted to ASCII, namely \f(CW\*(C`[A\-Za\-z0\-9_]\*(C'\fR
860 .Sp
861 Mnemonic: Perl's (original) word.
862 .ie n .IP "\fR\fB""\ep{Posix...}""\fR\fB\fR" 4
863 .el .IP \fR\f(CB\ep{Posix...}\fR\fB\fR 4
864 .IX Item "p{Posix...}"
865 There are several of these, which are equivalents, using the \f(CW\*(C`\ep{}\*(C'\fR
866 notation, for Posix classes and are described in
867 "POSIX Character Classes" in perlrecharclass.
868 .ie n .IP "\fR\fB""\ep{Present_In: *}""\fR\fB\fR (Short: ""\ep{In=*}"")" 4
869 .el .IP "\fR\f(CB\ep{Present_In: *}\fR\fB\fR (Short: \f(CW\ep{In=*}\fR)" 4
870 .IX Item "p{Present_In: *} (Short: p{In=*})"
871 This property is used when you need to know in what Unicode version(s) a
872 character is.
873 .Sp
874 The "*" above stands for some Unicode version number, such as
875 \&\f(CW1.1\fR or \f(CW12.0\fR; or the "*" can also be \f(CW\*(C`Unassigned\*(C'\fR. This property will
876 match the code points whose final disposition has been settled as of the
877 Unicode release given by the version number; \f(CW\*(C`\ep{Present_In: Unassigned}\*(C'\fR
878 will match those code points whose meaning has yet to be assigned.
879 .Sp
880 For example, \f(CW\*(C`U+0041\*(C'\fR \f(CW"LATIN CAPITAL LETTER A"\fR was present in the very first
881 Unicode release available, which is \f(CW1.1\fR, so this property is true for all
882 valid "*" versions. On the other hand, \f(CW\*(C`U+1EFF\*(C'\fR was not assigned until version
883 5.1 when it became \f(CW"LATIN SMALL LETTER Y WITH LOOP"\fR, so the only "*" that
884 would match it are 5.1, 5.2, and later.
885 .Sp
886 Unicode furnishes the \f(CW\*(C`Age\*(C'\fR property from which this is derived. The problem
887 with Age is that a strict interpretation of it (which Perl takes) has it
888 matching the precise release a code point's meaning is introduced in. Thus
889 \&\f(CW\*(C`U+0041\*(C'\fR would match only 1.1; and \f(CW\*(C`U+1EFF\*(C'\fR only 5.1. This is not usually what
890 you want.
891 .Sp
892 Some non-Perl implementations of the Age property may change its meaning to be
893 the same as the Perl \f(CW\*(C`Present_In\*(C'\fR property; just be aware of that.
894 .Sp
895 Another confusion with both these properties is that the definition is not
896 that the code point has been \fIassigned\fR, but that the meaning of the code point
897 has been \fIdetermined\fR. This is because 66 code points will always be
898 unassigned, and so the \f(CW\*(C`Age\*(C'\fR for them is the Unicode version in which the decision
899 to make them so was made. For example, \f(CW\*(C`U+FDD0\*(C'\fR is to be permanently
900 unassigned to a character, and the decision to do that was made in version 3.1,
901 so \f(CW\*(C`\ep{Age=3.1}\*(C'\fR matches this character, as also does \f(CW\*(C`\ep{Present_In: 3.1}\*(C'\fR and up.
902 .ie n .IP "\fR\fB""\ep{Print}""\fR\fB\fR" 4
903 .el .IP \fR\f(CB\ep{Print}\fR\fB\fR 4
904 .IX Item "p{Print}"
905 This matches any character that is graphical or blank, except controls.
906 .ie n .IP "\fR\fB""\ep{SpacePerl}""\fR\fB\fR" 4
907 .el .IP \fR\f(CB\ep{SpacePerl}\fR\fB\fR 4
908 .IX Item "p{SpacePerl}"
909 This is the same as \f(CW\*(C`\es\*(C'\fR, including beyond ASCII.
910 .Sp
911 Mnemonic: Space, as modified by Perl. (It doesn't include the vertical tab
912 until v5.18, which both the Posix standard and Unicode consider white space.)
913 .ie n .IP "\fR\fB""\ep{Title}""\fR\fB\fR and \fB\fR\fB""\ep{Titlecase}""\fR\fB\fR" 4
914 .el .IP "\fR\f(CB\ep{Title}\fR\fB\fR and \fB\fR\f(CB\ep{Titlecase}\fR\fB\fR" 4
915 .IX Item "p{Title} and p{Titlecase}"
916 Under case-sensitive matching, these both match the same code points as
917 \&\f(CW\*(C`\ep{General Category=Titlecase_Letter}\*(C'\fR (\f(CW\*(C`\ep{gc=lt}\*(C'\fR). The difference
918 is that under \f(CW\*(C`/i\*(C'\fR caseless matching, these match the same as
919 \&\f(CW\*(C`\ep{Cased}\*(C'\fR, whereas \f(CW\*(C`\ep{gc=lt}\*(C'\fR matches \f(CW\*(C`\ep{Cased_Letter\*(C'\fR).
920 .ie n .IP "\fR\fB""\ep{Unicode}""\fR\fB\fR" 4
921 .el .IP \fR\f(CB\ep{Unicode}\fR\fB\fR 4
922 .IX Item "p{Unicode}"
923 This matches any of the 1_114_112 Unicode code points.
924 \&\f(CW\*(C`\ep{Any}\*(C'\fR.
925 .ie n .IP "\fR\fB""\ep{VertSpace}""\fR\fB\fR" 4
926 .el .IP \fR\f(CB\ep{VertSpace}\fR\fB\fR 4
927 .IX Item "p{VertSpace}"
928 This is the same as \f(CW\*(C`\ev\*(C'\fR: A character that changes the spacing vertically.
929 .ie n .IP "\fR\fB""\ep{Word}""\fR\fB\fR" 4
930 .el .IP \fR\f(CB\ep{Word}\fR\fB\fR 4
931 .IX Item "p{Word}"
932 This is the same as \f(CW\*(C`\ew\*(C'\fR, including over 100_000 characters beyond ASCII.
933 .ie n .IP "\fR\fB""\ep{XPosix...}""\fR\fB\fR" 4
934 .el .IP \fR\f(CB\ep{XPosix...}\fR\fB\fR 4
935 .IX Item "p{XPosix...}"
936 There are several of these, which are the standard Posix classes
937 extended to the full Unicode range. They are described in
938 "POSIX Character Classes" in perlrecharclass.
939 .ie n .SS "Comparison of ""\eN{...}"" and ""\ep{name=...}"""
940 .el .SS "Comparison of \f(CW\eN{...}\fP and \f(CW\ep{name=...}\fP"
941 .IX Subsection "Comparison of N{...} and p{name=...}"
942 Starting in Perl 5.32, you can specify a character by its name in
943 regular expression patterns using \f(CW\*(C`\ep{name=...}\*(C'\fR. This is in addition
944 to the longstanding method of using \f(CW\*(C`\eN{...}\*(C'\fR. The following
945 summarizes the differences between these two:
946 .PP
947 .Vb 6
948 \& \eN{...} \ep{Name=...}
949 \& can interpolate only with eval yes [1]
950 \& custom names yes no [2]
951 \& name aliases yes yes [3]
952 \& named sequences yes yes [4]
953 \& name value parsing exact Unicode loose [5]
954 .Ve
955 .IP [1] 4
956 .IX Item "[1]"
957 The ability to interpolate means you can do something like
958 .Sp
959 .Vb 1
960 \& qr/\ep{na=latin capital letter $which}/
961 .Ve
962 .Sp
963 and specify \f(CW$which\fR elsewhere.
964 .IP [2] 4
965 .IX Item "[2]"
966 You can create your own names for characters, and override official
967 ones when using \f(CW\*(C`\eN{...}\*(C'\fR. See "CUSTOM ALIASES" in charnames.
968 .IP [3] 4
969 .IX Item "[3]"
970 Some characters have multiple names (synonyms).
971 .IP [4] 4
972 .IX Item "[4]"
973 Some particular sequences of characters are given a single name, in
974 addition to their individual ones.
975 .IP [5] 4
976 .IX Item "[5]"
977 Exact name value matching means you have to specify case, hyphens,
978 underscores, and spaces precisely in the name you want. Loose matching
979 follows the Unicode rules
980 <https://www.unicode.org/reports/tr44/tr44\-24.html#UAX44\-LM2>,
981 where these are mostly irrelevant. Except for a few outlier character
982 names, these are the same rules as are already used for any other
983 \&\f(CW\*(C`\ep{...}\*(C'\fR property.
984 .SS "Wildcards in Property Values"
985 .IX Subsection "Wildcards in Property Values"
986 Starting in Perl 5.30, it is possible to do something like this:
987 .PP
988 .Vb 1
989 \& qr!\ep{numeric_value=/\eA[0\-5]\ez/}!
990 .Ve
991 .PP
992 or, by abbreviating and adding \f(CW\*(C`/x\*(C'\fR,
993 .PP
994 .Vb 1
995 \& qr! \ep{nv= /(?x) \eA [0\-5] \ez / }!
996 .Ve
997 .PP
998 This matches all code points whose numeric value is one of 0, 1, 2, 3,
999 4, or 5. This particular example could instead have been written as
1000 .PP
1001 .Vb 1
1002 \& qr! \eA [ \ep{nv=0}\ep{nv=1}\ep{nv=2}\ep{nv=3}\ep{nv=4}\ep{nv=5} ] \ez !xx
1003 .Ve
1004 .PP
1005 in earlier perls, so in this case this feature just makes things easier
1006 and shorter to write. If we hadn't included the \f(CW\*(C`\eA\*(C'\fR and \f(CW\*(C`\ez\*(C'\fR, these
1007 would have matched things like \f(CW\*(C`1/2\*(C'\fR because that contains a 1 (as
1008 well as a 2). As written, it matches things like subscripts that have
1009 these numeric values. If we only wanted the decimal digits with those
1010 numeric values, we could say,
1011 .PP
1012 .Vb 1
1013 \& qr! (?[ \ed & \ep{nv=/[0\-5]/ ]) }!x
1014 .Ve
1015 .PP
1016 The \f(CW\*(C`\ed\*(C'\fR gets rid of needing to anchor the pattern, since it forces the
1017 result to only match \f(CW\*(C`[0\-9]\*(C'\fR, and the \f(CW\*(C`[0\-5]\*(C'\fR further restricts it.
1018 .PP
1019 The text in the above examples enclosed between the \f(CW"/"\fR
1020 characters can be just about any regular expression. It is independent
1021 of the main pattern, so doesn't share any capturing groups, \fIetc\fR. The
1022 delimiters for it must be ASCII punctuation, but it may NOT be
1023 delimited by \f(CW"{"\fR, nor \f(CW"}"\fR nor contain a literal \f(CW"}"\fR, as that
1024 delimits the end of the enclosing \f(CW\*(C`\ep{}\*(C'\fR. Like any pattern, certain
1025 other delimiters are terminated by their mirror images. These are
1026 \&\f(CW"("\fR, \f(CW\*(C`"[\*(C'\fR", and \f(CW"<"\fR. If the delimiter is any of \f(CW"\-"\fR,
1027 \&\f(CW"_"\fR, \f(CW"+"\fR, or \f(CW"\e"\fR, or is the same delimiter as is used for the
1028 enclosing pattern, it must be preceded by a backslash escape, both
1029 fore and aft.
1030 .PP
1031 Beware of using \f(CW"$"\fR to indicate to match the end of the string. It
1032 can too easily be interpreted as being a punctuation variable, like
1033 \&\f(CW$/\fR.
1034 .PP
1035 No modifiers may follow the final delimiter. Instead, use
1036 "(?adlupimnsx\-imnsx)" in perlre and/or
1037 "(?adluimnsx\-imnsx:pattern)" in perlre to specify modifiers.
1038 However, certain modifiers are illegal in your wildcard subpattern.
1039 The only character set modifier specifiable is \f(CW\*(C`/aa\*(C'\fR;
1040 any other character set, and \f(CW\*(C`\-m\*(C'\fR, and \f(CW\*(C`p\*(C'\fR, and \f(CW\*(C`s\*(C'\fR are all illegal.
1041 Specifying modifiers like \f(CW\*(C`qr/.../gc\*(C'\fR that aren't legal in the
1042 \&\f(CW\*(C`(?...)\*(C'\fR notation normally raise a warning, but with wildcard
1043 subpatterns, their use is an error. The \f(CW\*(C`m\*(C'\fR modifier is ineffective;
1044 everything that matches will be a single line.
1045 .PP
1046 By default, your pattern is matched case-insensitively, as if \f(CW\*(C`/i\*(C'\fR had
1047 been specified. You can change this by saying \f(CW\*(C`(?\-i)\*(C'\fR in your pattern.
1048 .PP
1049 There are also certain operations that are illegal. You can't nest
1050 \&\f(CW\*(C`\ep{...}\*(C'\fR and \f(CW\*(C`\eP{...}\*(C'\fR calls within a wildcard subpattern, and \f(CW\*(C`\eG\*(C'\fR
1051 doesn't make sense, so is also prohibited.
1052 .PP
1053 And the \f(CW\*(C`*\*(C'\fR quantifier (or its equivalent \f(CW\*(C`(0,}\*(C'\fR) is illegal.
1054 .PP
1055 This feature is not available when the left-hand side is prefixed by
1056 \&\f(CW\*(C`Is_\*(C'\fR, nor for any form that is marked as "Discouraged" in
1057 "Discouraged" in perluniprops.
1058 .PP
1059 This experimental feature has been added to begin to implement
1060 <https://www.unicode.org/reports/tr18/#Wildcard_Properties>. Using it
1061 will raise a (default-on) warning in the
1062 \&\f(CW\*(C`experimental::uniprop_wildcards\*(C'\fR category. We reserve the right to
1063 change its operation as we gain experience.
1064 .PP
1065 Your subpattern can be just about anything, but for it to have some
1066 utility, it should match when called with either or both of
1067 a) the full name of the property value with underscores (and/or spaces
1068 in the Block property) and some things uppercase; or b) the property
1069 value in all lowercase with spaces and underscores squeezed out. For
1070 example,
1071 .PP
1072 .Vb 2
1073 \& qr!\ep{Blk=/Old I.*/}!
1074 \& qr!\ep{Blk=/oldi.*/}!
1075 .Ve
1076 .PP
1077 would match the same things.
1078 .PP
1079 Another example that shows that within \f(CW\*(C`\ep{...}\*(C'\fR, \f(CW\*(C`/x\*(C'\fR isn't needed to
1080 have spaces:
1081 .PP
1082 .Vb 1
1083 \& qr!\ep{scx= /Hebrew|Greek/ }!
1084 .Ve
1085 .PP
1086 To be safe, we should have anchored the above example, to prevent
1087 matches for something like \f(CW\*(C`Hebrew_Braille\*(C'\fR, but there aren't
1088 any script names like that, so far.
1089 A warning is issued if none of the legal values for a property are
1090 matched by your pattern. It's likely that a future release will raise a
1091 warning if your pattern ends up causing every possible code point to
1092 match.
1093 .PP
1094 Starting in 5.32, the Name, Name Aliases, and Named Sequences properties
1095 are allowed to be matched. They are considered to be a single
1096 combination property, just as has long been the case for \f(CW\*(C`\eN{}\*(C'\fR. Loose
1097 matching doesn't work in exactly the same way for these as it does for
1098 the values of other properties. The rules are given in
1099 <https://www.unicode.org/reports/tr44/tr44\-24.html#UAX44\-LM2>. As a
1100 result, Perl doesn't try loose matching for you, like it does in other
1101 properties. All letters in names are uppercase, but you can add \f(CW\*(C`(?i)\*(C'\fR
1102 to your subpattern to ignore case. If you're uncertain where a blank
1103 is, you can use \f(CW\*(C` ?\*(C'\fR in your subpattern. No character name contains an
1104 underscore, so don't bother trying to match one. The use of hyphens is
1105 particularly problematic; refer to the above link. But note that, as of
1106 Unicode 13.0, the only script in modern usage which has weirdnesses with
1107 these is Tibetan; also the two Korean characters U+116C HANGUL JUNGSEONG
1108 OE and U+1180 HANGUL JUNGSEONG O\-E. Unicode makes no promises to not
1109 add hyphen-problematic names in the future.
1110 .PP
1111 Using wildcards on these is resource intensive, given the hundreds of
1112 thousands of legal names that must be checked against.
1113 .PP
1114 An example of using Name property wildcards is
1115 .PP
1116 .Vb 1
1117 \& qr!\ep{name=/(SMILING|GRINNING) FACE/}!
1118 .Ve
1119 .PP
1120 Another is
1121 .PP
1122 .Vb 1
1123 \& qr/(?[ \ep{name=\e/CJK\e/} \- \ep{ideographic} ])/
1124 .Ve
1125 .PP
1126 which is the 200\-ish (as of Unicode 13.0) CJK characters that aren't
1127 ideographs.
1128 .PP
1129 There are certain properties that wildcard subpatterns don't currently
1130 work with. These are:
1131 .PP
1132 .Vb 9
1133 \& Bidi Mirroring Glyph
1134 \& Bidi Paired Bracket
1135 \& Case Folding
1136 \& Decomposition Mapping
1137 \& Equivalent Unified Ideograph
1138 \& Lowercase Mapping
1139 \& NFKC Case Fold
1140 \& Titlecase Mapping
1141 \& Uppercase Mapping
1142 .Ve
1143 .PP
1144 Nor is the \f(CW\*(C`@\fR\f(CIunicode_property\fR\f(CW@\*(C'\fR form implemented.
1145 .PP
1146 Here's a complete example of matching IPV4 internet protocol addresses
1147 in any (single) script
1148 .PP
1149 .Vb 1
1150 \& no warnings \*(Aqexperimental::uniprop_wildcards\*(Aq;
1151 \&
1152 \& # Can match a substring, so this intermediate regex needs to have
1153 \& # context or anchoring in its final use. Using nt=de yields decimal
1154 \& # digits. When specifying a subset of these, we must include \ed to
1155 \& # prevent things like U+00B2 SUPERSCRIPT TWO from matching
1156 \& my $zero_through_255 =
1157 \& qr/ \eb (*sr: # All from same sript
1158 \& (?[ \ep{nv=0} & \ed ])* # Optional leading zeros
1159 \& ( # Then one of:
1160 \& \ed{1,2} # 0 \- 99
1161 \& | (?[ \ep{nv=1} & \ed ]) \ed{2} # 100 \- 199
1162 \& | (?[ \ep{nv=2} & \ed ])
1163 \& ( (?[ \ep{nv=:[0\-4]:} & \ed ]) \ed # 200 \- 249
1164 \& | (?[ \ep{nv=5} & \ed ])
1165 \& (?[ \ep{nv=:[0\-5]:} & \ed ]) # 250 \- 255
1166 \& )
1167 \& )
1168 \& )
1169 \& \eb
1170 \& /x;
1171 \&
1172 \& my $ipv4 = qr/ \eA (*sr: $zero_through_255
1173 \& (?: [.] $zero_through_255 ) {3}
1174 \& )
1175 \& \ez
1176 \& /x;
1177 .Ve
1178 .SS "User-Defined Character Properties"
1179 .IX Subsection "User-Defined Character Properties"
1180 You can define your own binary character properties by defining subroutines
1181 whose names begin with \f(CW"In"\fR or \f(CW"Is"\fR. (The regex sets feature
1182 "(?[ ])" in perlre provides an alternative which allows more complex
1183 definitions.) The subroutines can be defined in any
1184 package. They override any Unicode properties expressed as the same
1185 names. The user-defined properties can be used in the regular
1186 expression
1187 \&\f(CW\*(C`\ep{}\*(C'\fR and \f(CW\*(C`\eP{}\*(C'\fR constructs; if you are using a user-defined property from a
1188 package other than the one you are in, you must specify its package in the
1189 \&\f(CW\*(C`\ep{}\*(C'\fR or \f(CW\*(C`\eP{}\*(C'\fR construct.
1190 .PP
1191 .Vb 3
1192 \& # assuming property IsForeign defined in Lang::
1193 \& package main; # property package name required
1194 \& if ($txt =~ /\ep{Lang::IsForeign}+/) { ... }
1195 \&
1196 \& package Lang; # property package name not required
1197 \& if ($txt =~ /\ep{IsForeign}+/) { ... }
1198 .Ve
1199 .PP
1200 The subroutines are passed a single parameter, which is 0 if
1201 case-sensitive matching is in effect and non-zero if caseless matching
1202 is in effect. The subroutine may return different values depending on
1203 the value of the flag. But the subroutine is never called more than
1204 once for each flag value (zero vs non-zero). The return value is saved
1205 and used instead of calling the sub ever again. If the sub is defined
1206 at the time the pattern is compiled, it will be called then; if not, it
1207 will be called the first time its value (for that flag) is needed during
1208 execution.
1209 .PP
1210 Note that if the regular expression is tainted, then Perl will die rather
1211 than calling the subroutine when the name of the subroutine is
1212 determined by the tainted data.
1213 .PP
1214 The subroutines must return a specially-formatted string, with one
1215 or more newline-separated lines. Each line must be one of the following:
1216 .IP \(bu 4
1217 A single hexadecimal number denoting a code point to include.
1218 .IP \(bu 4
1219 Two hexadecimal numbers separated by horizontal whitespace (space or
1220 tabular characters) denoting a range of code points to include. The
1221 second number must not be smaller than the first.
1222 .IP \(bu 4
1223 Something to include, prefixed by \f(CW"+"\fR: a built-in character
1224 property (prefixed by \f(CW"utf8::"\fR) or a fully qualified (including package
1225 name) user-defined character property,
1226 to represent all the characters in that property; two hexadecimal code
1227 points for a range; or a single hexadecimal code point.
1228 .IP \(bu 4
1229 Something to exclude, prefixed by \f(CW"\-"\fR: an existing character
1230 property (prefixed by \f(CW"utf8::"\fR) or a fully qualified (including package
1231 name) user-defined character property,
1232 to represent all the characters in that property; two hexadecimal code
1233 points for a range; or a single hexadecimal code point.
1234 .IP \(bu 4
1235 Something to negate, prefixed \f(CW"!"\fR: an existing character
1236 property (prefixed by \f(CW"utf8::"\fR) or a fully qualified (including package
1237 name) user-defined character property,
1238 to represent all the characters in that property; two hexadecimal code
1239 points for a range; or a single hexadecimal code point.
1240 .IP \(bu 4
1241 Something to intersect with, prefixed by \f(CW"&"\fR: an existing character
1242 property (prefixed by \f(CW"utf8::"\fR) or a fully qualified (including package
1243 name) user-defined character property,
1244 for all the characters except the characters in the property; two
1245 hexadecimal code points for a range; or a single hexadecimal code point.
1246 .PP
1247 For example, to define a property that covers both the Japanese
1248 syllabaries (hiragana and katakana), you can define
1249 .PP
1250 .Vb 6
1251 \& sub InKana {
1252 \& return <<END;
1253 \& 3040\et309F
1254 \& 30A0\et30FF
1255 \& END
1256 \& }
1257 .Ve
1258 .PP
1259 Imagine that the here-doc end marker is at the beginning of the line.
1260 Now you can use \f(CW\*(C`\ep{InKana}\*(C'\fR and \f(CW\*(C`\eP{InKana}\*(C'\fR.
1261 .PP
1262 You could also have used the existing block property names:
1263 .PP
1264 .Vb 6
1265 \& sub InKana {
1266 \& return <<\*(AqEND\*(Aq;
1267 \& +utf8::InHiragana
1268 \& +utf8::InKatakana
1269 \& END
1270 \& }
1271 .Ve
1272 .PP
1273 Suppose you wanted to match only the allocated characters,
1274 not the raw block ranges: in other words, you want to remove
1275 the unassigned characters:
1276 .PP
1277 .Vb 7
1278 \& sub InKana {
1279 \& return <<\*(AqEND\*(Aq;
1280 \& +utf8::InHiragana
1281 \& +utf8::InKatakana
1282 \& \-utf8::IsCn
1283 \& END
1284 \& }
1285 .Ve
1286 .PP
1287 The negation is useful for defining (surprise!) negated classes.
1288 .PP
1289 .Vb 7
1290 \& sub InNotKana {
1291 \& return <<\*(AqEND\*(Aq;
1292 \& !utf8::InHiragana
1293 \& \-utf8::InKatakana
1294 \& +utf8::IsCn
1295 \& END
1296 \& }
1297 .Ve
1298 .PP
1299 This will match all non-Unicode code points, since every one of them is
1300 not in Kana. You can use intersection to exclude these, if desired, as
1301 this modified example shows:
1302 .PP
1303 .Vb 8
1304 \& sub InNotKana {
1305 \& return <<\*(AqEND\*(Aq;
1306 \& !utf8::InHiragana
1307 \& \-utf8::InKatakana
1308 \& +utf8::IsCn
1309 \& &utf8::Any
1310 \& END
1311 \& }
1312 .Ve
1313 .PP
1314 \&\f(CW&utf8::Any\fR must be the last line in the definition.
1315 .PP
1316 Intersection is used generally for getting the common characters matched
1317 by two (or more) classes. It's important to remember not to use \f(CW"&"\fR for
1318 the first set; that would be intersecting with nothing, resulting in an
1319 empty set. (Similarly using \f(CW"\-"\fR for the first set does nothing).
1320 .PP
1321 Unlike non-user-defined \f(CW\*(C`\ep{}\*(C'\fR property matches, no warning is ever
1322 generated if these properties are matched against a non-Unicode code
1323 point (see "Beyond Unicode code points" below).
1324 .SS "User-Defined Case Mappings (for serious hackers only)"
1325 .IX Subsection "User-Defined Case Mappings (for serious hackers only)"
1326 \&\fBThis feature has been removed as of Perl 5.16.\fR
1327 The CPAN module \f(CW\*(C`Unicode::Casing\*(C'\fR provides better functionality without
1328 the drawbacks that this feature had. If you are using a Perl earlier
1329 than 5.16, this feature was most fully documented in the 5.14 version of
1330 this pod:
1331 <http://perldoc.perl.org/5.14.0/perlunicode.html#User\-Defined\-Case\-Mappings\-%28for\-serious\-hackers\-only%29>
1332 .SS "Character Encodings for Input and Output"
1333 .IX Subsection "Character Encodings for Input and Output"
1334 See Encode.
1335 .SS "Unicode Regular Expression Support Level"
1336 .IX Subsection "Unicode Regular Expression Support Level"
1337 The following list of Unicode supported features for regular expressions describes
1338 all features currently directly supported by core Perl. The references
1339 to "Level \fIN\fR" and the section numbers refer to
1340 UTS#18 "Unicode Regular Expressions" <https://www.unicode.org/reports/tr18>,
1341 version 18, October 2016.
1342 .PP
1343 \fILevel 1 \- Basic Unicode Support\fR
1344 .IX Subsection "Level 1 - Basic Unicode Support"
1345 .PP
1346 .Vb 8
1347 \& RL1.1 Hex Notation \- Done [1]
1348 \& RL1.2 Properties \- Done [2]
1349 \& RL1.2a Compatibility Properties \- Done [3]
1350 \& RL1.3 Subtraction and Intersection \- Done [4]
1351 \& RL1.4 Simple Word Boundaries \- Done [5]
1352 \& RL1.5 Simple Loose Matches \- Done [6]
1353 \& RL1.6 Line Boundaries \- Partial [7]
1354 \& RL1.7 Supplementary Code Points \- Done [8]
1355 .Ve
1356 .ie n .IP "[1] ""\eN{U+...}"" and ""\ex{...}""" 4
1357 .el .IP "[1] \f(CW\eN{U+...}\fR and \f(CW\ex{...}\fR" 4
1358 .IX Item "[1] N{U+...} and x{...}"
1359 .PD 0
1360 .ie n .IP "[2] ""\ep{...}"" ""\eP{...}"". This requirement is for a minimal list of properties. Perl supports these. See R2.7 for other properties." 4
1361 .el .IP "[2] \f(CW\ep{...}\fR \f(CW\eP{...}\fR. This requirement is for a minimal list of properties. Perl supports these. See R2.7 for other properties." 4
1362 .IX Item "[2] p{...} P{...}. This requirement is for a minimal list of properties. Perl supports these. See R2.7 for other properties."
1363 .IP [3] 4
1364 .IX Item "[3]"
1365 .PD
1366 Perl has \f(CW\*(C`\ed\*(C'\fR \f(CW\*(C`\eD\*(C'\fR \f(CW\*(C`\es\*(C'\fR \f(CW\*(C`\eS\*(C'\fR \f(CW\*(C`\ew\*(C'\fR \f(CW\*(C`\eW\*(C'\fR \f(CW\*(C`\eX\*(C'\fR \f(CW\*(C`[:\fR\f(CIprop\fR\f(CW:]\*(C'\fR
1367 \&\f(CW\*(C`[:^\fR\f(CIprop\fR\f(CW:]\*(C'\fR, plus all the properties specified by
1368 <https://www.unicode.org/reports/tr18/#Compatibility_Properties>. These
1369 are described above in "Other Properties"
1370 .IP [4] 4
1371 .IX Item "[4]"
1372 The regex sets feature \f(CW"(?[...])"\fR starting in v5.18 accomplishes
1373 this. See "(?[ ])" in perlre.
1374 .ie n .IP "[5] ""\eb"" ""\eB"" meet most, but not all, the details of this requirement, but ""\eb{wb}"" and ""\eB{wb}"" do, as well as the stricter R2.3." 4
1375 .el .IP "[5] \f(CW\eb\fR \f(CW\eB\fR meet most, but not all, the details of this requirement, but \f(CW\eb{wb}\fR and \f(CW\eB{wb}\fR do, as well as the stricter R2.3." 4
1376 .IX Item "[5] b B meet most, but not all, the details of this requirement, but b{wb} and B{wb} do, as well as the stricter R2.3."
1377 .PD 0
1378 .IP [6] 4
1379 .IX Item "[6]"
1380 .PD
1381 Note that Perl does Full case-folding in matching, not Simple:
1382 .Sp
1383 For example \f(CW\*(C`U+1F88\*(C'\fR is equivalent to \f(CW\*(C`U+1F00 U+03B9\*(C'\fR, instead of just
1384 \&\f(CW\*(C`U+1F80\*(C'\fR. This difference matters mainly for certain Greek capital
1385 letters with certain modifiers: the Full case-folding decomposes the
1386 letter, while the Simple case-folding would map it to a single
1387 character.
1388 .IP [7] 4
1389 .IX Item "[7]"
1390 The reason this is considered to be only partially implemented is that
1391 Perl has \f(CW\*(C`qr/\eb{lb}/\*(C'\fR and
1392 \&\f(CW\*(C`Unicode::LineBreak\*(C'\fR that are conformant with
1393 UAX#14 "Unicode Line Breaking Algorithm" <https://www.unicode.org/reports/tr14>.
1394 The regular expression construct provides default behavior, while the
1395 heavier-weight module provides customizable line breaking.
1396 .Sp
1397 But Perl treats \f(CW\*(C`\en\*(C'\fR as the start\- and end-line
1398 delimiter, whereas Unicode specifies more characters that should be
1399 so-interpreted.
1400 .Sp
1401 These are:
1402 .Sp
1403 .Vb 6
1404 \& VT U+000B (\ev in C)
1405 \& FF U+000C (\ef)
1406 \& CR U+000D (\er)
1407 \& NEL U+0085
1408 \& LS U+2028
1409 \& PS U+2029
1410 .Ve
1411 .Sp
1412 \&\f(CW\*(C`^\*(C'\fR and \f(CW\*(C`$\*(C'\fR in regular expression patterns are supposed to match all
1413 these, but don't.
1414 These characters also don't, but should, affect \f(CW\*(C`<>\*(C'\fR \f(CW$.\fR, and
1415 script line numbers.
1416 .Sp
1417 Also, lines should not be split within \f(CW\*(C`CRLF\*(C'\fR (i.e. there is no
1418 empty line between \f(CW\*(C`\er\*(C'\fR and \f(CW\*(C`\en\*(C'\fR). For \f(CW\*(C`CRLF\*(C'\fR, try the \f(CW\*(C`:crlf\*(C'\fR
1419 layer (see PerlIO).
1420 .ie n .IP "[8] UTF\-8/UTF\-EBDDIC used in Perl allows not only ""U+10000"" to ""U+10FFFF"" but also beyond ""U+10FFFF""" 4
1421 .el .IP "[8] UTF\-8/UTF\-EBDDIC used in Perl allows not only \f(CWU+10000\fR to \f(CWU+10FFFF\fR but also beyond \f(CWU+10FFFF\fR" 4
1422 .IX Item "[8] UTF-8/UTF-EBDDIC used in Perl allows not only U+10000 to U+10FFFF but also beyond U+10FFFF"
1423 .PP
1424 \fILevel 2 \- Extended Unicode Support\fR
1425 .IX Subsection "Level 2 - Extended Unicode Support"
1426 .PP
1427 .Vb 10
1428 \& RL2.1 Canonical Equivalents \- Retracted [9]
1429 \& by Unicode
1430 \& RL2.2 Extended Grapheme Clusters and \- Partial [10]
1431 \& Character Classes with Strings
1432 \& RL2.3 Default Word Boundaries \- Done [11]
1433 \& RL2.4 Default Case Conversion \- Done
1434 \& RL2.5 Name Properties \- Done
1435 \& RL2.6 Wildcards in Property Values \- Partial [12]
1436 \& RL2.7 Full Properties \- Partial [13]
1437 \& RL2.8 Optional Properties \- Partial [14]
1438 .Ve
1439 .IP "[9] Unicode has rewritten this portion of UTS#18 to say that getting canonical equivalence (see UAX#15 ""Unicode Normalization Forms"" <https://www.unicode.org/reports/tr15>) is basically to be done at the programmer level. Use NFD to write both your regular expressions and text to match them against (you can use Unicode::Normalize)." 4
1440 .IX Item "[9] Unicode has rewritten this portion of UTS#18 to say that getting canonical equivalence (see UAX#15 ""Unicode Normalization Forms"" <https://www.unicode.org/reports/tr15>) is basically to be done at the programmer level. Use NFD to write both your regular expressions and text to match them against (you can use Unicode::Normalize)."
1441 .PD 0
1442 .ie n .IP "[10] Perl has ""\eX"" and ""\eb{gcb}"". Unicode has retracted their ""Grapheme Cluster Mode"", and recently added string properties, which Perl does not yet support." 4
1443 .el .IP "[10] Perl has \f(CW\eX\fR and \f(CW\eb{gcb}\fR. Unicode has retracted their ""Grapheme Cluster Mode"", and recently added string properties, which Perl does not yet support." 4
1444 .IX Item "[10] Perl has X and b{gcb}. Unicode has retracted their ""Grapheme Cluster Mode"", and recently added string properties, which Perl does not yet support."
1445 .IP "[11] see UAX#29 ""Unicode Text Segmentation"" <https://www.unicode.org/reports/tr29>," 4
1446 .IX Item "[11] see UAX#29 ""Unicode Text Segmentation"" <https://www.unicode.org/reports/tr29>,"
1447 .IP "[12] see ""Wildcards in Property Values"" above." 4
1448 .IX Item "[12] see ""Wildcards in Property Values"" above."
1449 .IP "[13] Perl supports all the properties in the Unicode Character Database (UCD). It does not yet support the listed properties that come from other Unicode sources." 4
1450 .IX Item "[13] Perl supports all the properties in the Unicode Character Database (UCD). It does not yet support the listed properties that come from other Unicode sources."
1451 .IP "[14] The only optional property that Perl supports is Named Sequence. None of these properties are in the UCD." 4
1452 .IX Item "[14] The only optional property that Perl supports is Named Sequence. None of these properties are in the UCD."
1453 .PD
1454 .PP
1455 \fILevel 3 \- Tailored Support\fR
1456 .IX Subsection "Level 3 - Tailored Support"
1457 .PP
1458 This has been retracted by Unicode.
1459 .SS "Unicode Encodings"
1460 .IX Subsection "Unicode Encodings"
1461 Unicode characters are assigned to \fIcode points\fR, which are abstract
1462 numbers. To use these numbers, various encodings are needed.
1463 .IP \(bu 4
1464 UTF\-8
1465 .Sp
1466 UTF\-8 is a variable-length (1 to 4 bytes), byte-order independent
1467 encoding. In most of Perl's documentation, including elsewhere in this
1468 document, the term "UTF\-8" means also "UTF-EBCDIC". But in this section,
1469 "UTF\-8" refers only to the encoding used on ASCII platforms. It is a
1470 superset of 7\-bit US-ASCII, so anything encoded in ASCII has the
1471 identical representation when encoded in UTF\-8.
1472 .Sp
1473 The following table is from Unicode 3.2.
1474 .Sp
1475 .Vb 1
1476 \& Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
1477 \&
1478 \& U+0000..U+007F 00..7F
1479 \& U+0080..U+07FF * C2..DF 80..BF
1480 \& U+0800..U+0FFF E0 * A0..BF 80..BF
1481 \& U+1000..U+CFFF E1..EC 80..BF 80..BF
1482 \& U+D000..U+D7FF ED 80..9F 80..BF
1483 \& U+D800..U+DFFF +++++ utf16 surrogates, not legal utf8 +++++
1484 \& U+E000..U+FFFF EE..EF 80..BF 80..BF
1485 \& U+10000..U+3FFFF F0 * 90..BF 80..BF 80..BF
1486 \& U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF
1487 \& U+100000..U+10FFFF F4 80..8F 80..BF 80..BF
1488 .Ve
1489 .Sp
1490 Note the gaps marked by "*" before several of the byte entries above. These are
1491 caused by legal UTF\-8 avoiding non-shortest encodings: it is technically
1492 possible to UTF\-8\-encode a single code point in different ways, but that is
1493 explicitly forbidden, and the shortest possible encoding should always be used
1494 (and that is what Perl does).
1495 .Sp
1496 Another way to look at it is via bits:
1497 .Sp
1498 .Vb 1
1499 \& Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte
1500 \&
1501 \& 0aaaaaaa 0aaaaaaa
1502 \& 00000bbbbbaaaaaa 110bbbbb 10aaaaaa
1503 \& ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa
1504 \& 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa
1505 .Ve
1506 .Sp
1507 As you can see, the continuation bytes all begin with \f(CW"10"\fR, and the
1508 leading bits of the start byte tell how many bytes there are in the
1509 encoded character.
1510 .Sp
1511 The original UTF\-8 specification allowed up to 6 bytes, to allow
1512 encoding of numbers up to \f(CW\*(C`0x7FFF_FFFF\*(C'\fR. Perl continues to allow those,
1513 and has extended that up to 13 bytes to encode code points up to what
1514 can fit in a 64\-bit word. However, Perl will warn if you output any of
1515 these as being non-portable; and under strict UTF\-8 input protocols,
1516 they are forbidden. In addition, it is now illegal to use a code point
1517 larger than what a signed integer variable on your system can hold. On
1518 32\-bit ASCII systems, this means \f(CW\*(C`0x7FFF_FFFF\*(C'\fR is the legal maximum
1519 (much higher on 64\-bit systems).
1520 .IP \(bu 4
1521 UTF-EBCDIC
1522 .Sp
1523 Like UTF\-8, but EBCDIC-safe, in the way that UTF\-8 is ASCII-safe.
1524 This means that all the basic characters (which includes all
1525 those that have ASCII equivalents (like \f(CW"A"\fR, \f(CW"0"\fR, \f(CW"%"\fR, \fIetc.\fR)
1526 are the same in both EBCDIC and UTF-EBCDIC.)
1527 .Sp
1528 UTF-EBCDIC is used on EBCDIC platforms. It generally requires more
1529 bytes to represent a given code point than UTF\-8 does; the largest
1530 Unicode code points take 5 bytes to represent (instead of 4 in UTF\-8),
1531 and, extended for 64\-bit words, it uses 14 bytes instead of 13 bytes in
1532 UTF\-8.
1533 .IP \(bu 4
1534 UTF\-16, UTF\-16BE, UTF\-16LE, Surrogates, and \f(CW\*(C`BOM\*(C'\fR's (Byte Order Marks)
1535 .Sp
1536 The followings items are mostly for reference and general Unicode
1537 knowledge, Perl doesn't use these constructs internally.
1538 .Sp
1539 Like UTF\-8, UTF\-16 is a variable-width encoding, but where
1540 UTF\-8 uses 8\-bit code units, UTF\-16 uses 16\-bit code units.
1541 All code points occupy either 2 or 4 bytes in UTF\-16: code points
1542 \&\f(CW\*(C`U+0000..U+FFFF\*(C'\fR are stored in a single 16\-bit unit, and code
1543 points \f(CW\*(C`U+10000..U+10FFFF\*(C'\fR in two 16\-bit units. The latter case is
1544 using \fIsurrogates\fR, the first 16\-bit unit being the \fIhigh
1545 surrogate\fR, and the second being the \fIlow surrogate\fR.
1546 .Sp
1547 Surrogates are code points set aside to encode the \f(CW\*(C`U+10000..U+10FFFF\*(C'\fR
1548 range of Unicode code points in pairs of 16\-bit units. The \fIhigh
1549 surrogates\fR are the range \f(CW\*(C`U+D800..U+DBFF\*(C'\fR and the \fIlow surrogates\fR
1550 are the range \f(CW\*(C`U+DC00..U+DFFF\*(C'\fR. The surrogate encoding is
1551 .Sp
1552 .Vb 2
1553 \& $hi = ($uni \- 0x10000) / 0x400 + 0xD800;
1554 \& $lo = ($uni \- 0x10000) % 0x400 + 0xDC00;
1555 .Ve
1556 .Sp
1557 and the decoding is
1558 .Sp
1559 .Vb 1
1560 \& $uni = 0x10000 + ($hi \- 0xD800) * 0x400 + ($lo \- 0xDC00);
1561 .Ve
1562 .Sp
1563 Because of the 16\-bitness, UTF\-16 is byte-order dependent. UTF\-16
1564 itself can be used for in-memory computations, but if storage or
1565 transfer is required either UTF\-16BE (big-endian) or UTF\-16LE
1566 (little-endian) encodings must be chosen.
1567 .Sp
1568 This introduces another problem: what if you just know that your data
1569 is UTF\-16, but you don't know which endianness? Byte Order Marks, or
1570 \&\f(CW\*(C`BOM\*(C'\fR's, are a solution to this. A special character has been reserved
1571 in Unicode to function as a byte order marker: the character with the
1572 code point \f(CW\*(C`U+FEFF\*(C'\fR is the \f(CW\*(C`BOM\*(C'\fR.
1573 .Sp
1574 The trick is that if you read a \f(CW\*(C`BOM\*(C'\fR, you will know the byte order,
1575 since if it was written on a big-endian platform, you will read the
1576 bytes \f(CW\*(C`0xFE 0xFF\*(C'\fR, but if it was written on a little-endian platform,
1577 you will read the bytes \f(CW\*(C`0xFF 0xFE\*(C'\fR. (And if the originating platform
1578 was writing in ASCII platform UTF\-8, you will read the bytes
1579 \&\f(CW\*(C`0xEF 0xBB 0xBF\*(C'\fR.)
1580 .Sp
1581 The way this trick works is that the character with the code point
1582 \&\f(CW\*(C`U+FFFE\*(C'\fR is not supposed to be in input streams, so the
1583 sequence of bytes \f(CW\*(C`0xFF 0xFE\*(C'\fR is unambiguously "\f(CW\*(C`BOM\*(C'\fR, represented in
1584 little-endian format" and cannot be \f(CW\*(C`U+FFFE\*(C'\fR, represented in big-endian
1585 format".
1586 .Sp
1587 Surrogates have no meaning in Unicode outside their use in pairs to
1588 represent other code points. However, Perl allows them to be
1589 represented individually internally, for example by saying
1590 \&\f(CWchr(0xD801)\fR, so that all code points, not just those valid for open
1591 interchange, are
1592 representable. Unicode does define semantics for them, such as their
1593 \&\f(CW"General_Category"\fR is \f(CW"Cs"\fR. But because their use is somewhat dangerous,
1594 Perl will warn (using the warning category \f(CW"surrogate"\fR, which is a
1595 sub-category of \f(CW"utf8"\fR) if an attempt is made
1596 to do things like take the lower case of one, or match
1597 case-insensitively, or to output them. (But don't try this on Perls
1598 before 5.14.)
1599 .IP \(bu 4
1600 UTF\-32, UTF\-32BE, UTF\-32LE
1601 .Sp
1602 The UTF\-32 family is pretty much like the UTF\-16 family, except that
1603 the units are 32\-bit, and therefore the surrogate scheme is not
1604 needed. UTF\-32 is a fixed-width encoding. The \f(CW\*(C`BOM\*(C'\fR signatures are
1605 \&\f(CW\*(C`0x00 0x00 0xFE 0xFF\*(C'\fR for BE and \f(CW\*(C`0xFF 0xFE 0x00 0x00\*(C'\fR for LE.
1606 .IP \(bu 4
1607 UCS\-2, UCS\-4
1608 .Sp
1609 Legacy, fixed-width encodings defined by the ISO 10646 standard. UCS\-2 is a 16\-bit
1610 encoding. Unlike UTF\-16, UCS\-2 is not extensible beyond \f(CW\*(C`U+FFFF\*(C'\fR,
1611 because it does not use surrogates. UCS\-4 is a 32\-bit encoding,
1612 functionally identical to UTF\-32 (the difference being that
1613 UCS\-4 forbids neither surrogates nor code points larger than \f(CW\*(C`0x10_FFFF\*(C'\fR).
1614 .IP \(bu 4
1615 UTF\-7
1616 .Sp
1617 A seven-bit safe (non-eight-bit) encoding, which is useful if the
1618 transport or storage is not eight-bit safe. Defined by RFC 2152.
1619 .SS "Noncharacter code points"
1620 .IX Subsection "Noncharacter code points"
1621 66 code points are set aside in Unicode as "noncharacter code points".
1622 These all have the \f(CW\*(C`Unassigned\*(C'\fR (\f(CW\*(C`Cn\*(C'\fR) \f(CW"General_Category"\fR, and
1623 no character will ever be assigned to any of them. They are the 32 code
1624 points between \f(CW\*(C`U+FDD0\*(C'\fR and \f(CW\*(C`U+FDEF\*(C'\fR inclusive, and the 34 code
1625 points:
1626 .PP
1627 .Vb 7
1628 \& U+FFFE U+FFFF
1629 \& U+1FFFE U+1FFFF
1630 \& U+2FFFE U+2FFFF
1631 \& ...
1632 \& U+EFFFE U+EFFFF
1633 \& U+FFFFE U+FFFFF
1634 \& U+10FFFE U+10FFFF
1635 .Ve
1636 .PP
1637 Until Unicode 7.0, the noncharacters were "\fBforbidden\fR for use in open
1638 interchange of Unicode text data", so that code that processed those
1639 streams could use these code points as sentinels that could be mixed in
1640 with character data, and would always be distinguishable from that data.
1641 (Emphasis above and in the next paragraph are added in this document.)
1642 .PP
1643 Unicode 7.0 changed the wording so that they are "\fBnot recommended\fR for
1644 use in open interchange of Unicode text data". The 7.0 Standard goes on
1645 to say:
1646 .Sp
1647 .RS 4
1648 "If a noncharacter is received in open interchange, an application is
1649 not required to interpret it in any way. It is good practice, however,
1650 to recognize it as a noncharacter and to take appropriate action, such
1651 as replacing it with \f(CW\*(C`U+FFFD\*(C'\fR replacement character, to indicate the
1652 problem in the text. It is not recommended to simply delete
1653 noncharacter code points from such text, because of the potential
1654 security issues caused by deleting uninterpreted characters. (See
1655 conformance clause C7 in Section 3.2, Conformance Requirements, and
1656 Unicode Technical Report #36, "Unicode Security
1657 Considerations" <https://www.unicode.org/reports/tr36/#Substituting_for_Ill_Formed_Subsequences>)."
1658 .RE
1659 .PP
1660 This change was made because it was found that various commercial tools
1661 like editors, or for things like source code control, had been written
1662 so that they would not handle program files that used these code points,
1663 effectively precluding their use almost entirely! And that was never
1664 the intent. They've always been meant to be usable within an
1665 application, or cooperating set of applications, at will.
1666 .PP
1667 If you're writing code, such as an editor, that is supposed to be able
1668 to handle any Unicode text data, then you shouldn't be using these code
1669 points yourself, and instead allow them in the input. If you need
1670 sentinels, they should instead be something that isn't legal Unicode.
1671 For UTF\-8 data, you can use the bytes 0xC0 and 0xC1 as sentinels, as
1672 they never appear in well-formed UTF\-8. (There are equivalents for
1673 UTF-EBCDIC). You can also store your Unicode code points in integer
1674 variables and use negative values as sentinels.
1675 .PP
1676 If you're not writing such a tool, then whether you accept noncharacters
1677 as input is up to you (though the Standard recommends that you not). If
1678 you do strict input stream checking with Perl, these code points
1679 continue to be forbidden. This is to maintain backward compatibility
1680 (otherwise potential security holes could open up, as an unsuspecting
1681 application that was written assuming the noncharacters would be
1682 filtered out before getting to it, could now, without warning, start
1683 getting them). To do strict checking, you can use the layer
1684 \&\f(CW:encoding(\*(AqUTF\-8\*(Aq)\fR.
1685 .PP
1686 Perl continues to warn (using the warning category \f(CW"nonchar"\fR, which
1687 is a sub-category of \f(CW"utf8"\fR) if an attempt is made to output
1688 noncharacters.
1689 .SS "Beyond Unicode code points"
1690 .IX Subsection "Beyond Unicode code points"
1691 The maximum Unicode code point is \f(CW\*(C`U+10FFFF\*(C'\fR, and Unicode only defines
1692 operations on code points up through that. But Perl works on code
1693 points up to the maximum permissible signed number available on the
1694 platform. However, Perl will not accept these from input streams unless
1695 lax rules are being used, and will warn (using the warning category
1696 \&\f(CW"non_unicode"\fR, which is a sub-category of \f(CW"utf8"\fR) if any are output.
1697 .PP
1698 Since Unicode rules are not defined on these code points, if a
1699 Unicode-defined operation is done on them, Perl uses what we believe are
1700 sensible rules, while generally warning, using the \f(CW"non_unicode"\fR
1701 category. For example, \f(CWuc("\ex{11_0000}")\fR will generate such a
1702 warning, returning the input parameter as its result, since Perl defines
1703 the uppercase of every non-Unicode code point to be the code point
1704 itself. (All the case changing operations, not just uppercasing, work
1705 this way.)
1706 .PP
1707 The situation with matching Unicode properties in regular expressions,
1708 the \f(CW\*(C`\ep{}\*(C'\fR and \f(CW\*(C`\eP{}\*(C'\fR constructs, against these code points is not as
1709 clear cut, and how these are handled has changed as we've gained
1710 experience.
1711 .PP
1712 One possibility is to treat any match against these code points as
1713 undefined. But since Perl doesn't have the concept of a match being
1714 undefined, it converts this to failing or \f(CW\*(C`FALSE\*(C'\fR. This is almost, but
1715 not quite, what Perl did from v5.14 (when use of these code points
1716 became generally reliable) through v5.18. The difference is that Perl
1717 treated all \f(CW\*(C`\ep{}\*(C'\fR matches as failing, but all \f(CW\*(C`\eP{}\*(C'\fR matches as
1718 succeeding.
1719 .PP
1720 One problem with this is that it leads to unexpected, and confusing
1721 results in some cases:
1722 .PP
1723 .Vb 2
1724 \& chr(0x110000) =~ \ep{ASCII_Hex_Digit=True} # Failed on <= v5.18
1725 \& chr(0x110000) =~ \ep{ASCII_Hex_Digit=False} # Failed! on <= v5.18
1726 .Ve
1727 .PP
1728 That is, it treated both matches as undefined, and converted that to
1729 false (raising a warning on each). The first case is the expected
1730 result, but the second is likely counterintuitive: "How could both be
1731 false when they are complements?" Another problem was that the
1732 implementation optimized many Unicode property matches down to already
1733 existing simpler, faster operations, which don't raise the warning. We
1734 chose to not forgo those optimizations, which help the vast majority of
1735 matches, just to generate a warning for the unlikely event that an
1736 above-Unicode code point is being matched against.
1737 .PP
1738 As a result of these problems, starting in v5.20, what Perl does is
1739 to treat non-Unicode code points as just typical unassigned Unicode
1740 characters, and matches accordingly. (Note: Unicode has atypical
1741 unassigned code points. For example, it has noncharacter code points,
1742 and ones that, when they do get assigned, are destined to be written
1743 Right-to-left, as Arabic and Hebrew are. Perl assumes that no
1744 non-Unicode code point has any atypical properties.)
1745 .PP
1746 Perl, in most cases, will raise a warning when matching an above-Unicode
1747 code point against a Unicode property when the result is \f(CW\*(C`TRUE\*(C'\fR for
1748 \&\f(CW\*(C`\ep{}\*(C'\fR, and \f(CW\*(C`FALSE\*(C'\fR for \f(CW\*(C`\eP{}\*(C'\fR. For example:
1749 .PP
1750 .Vb 2
1751 \& chr(0x110000) =~ \ep{ASCII_Hex_Digit=True} # Fails, no warning
1752 \& chr(0x110000) =~ \ep{ASCII_Hex_Digit=False} # Succeeds, with warning
1753 .Ve
1754 .PP
1755 In both these examples, the character being matched is non-Unicode, so
1756 Unicode doesn't define how it should match. It clearly isn't an ASCII
1757 hex digit, so the first example clearly should fail, and so it does,
1758 with no warning. But it is arguable that the second example should have
1759 an undefined, hence \f(CW\*(C`FALSE\*(C'\fR, result. So a warning is raised for it.
1760 .PP
1761 Thus the warning is raised for many fewer cases than in earlier Perls,
1762 and only when what the result is could be arguable. It turns out that
1763 none of the optimizations made by Perl (or are ever likely to be made)
1764 cause the warning to be skipped, so it solves both problems of Perl's
1765 earlier approach. The most commonly used property that is affected by
1766 this change is \f(CW\*(C`\ep{Unassigned}\*(C'\fR which is a short form for
1767 \&\f(CW\*(C`\ep{General_Category=Unassigned}\*(C'\fR. Starting in v5.20, all non-Unicode
1768 code points are considered \f(CW\*(C`Unassigned\*(C'\fR. In earlier releases the
1769 matches failed because the result was considered undefined.
1770 .PP
1771 The only place where the warning is not raised when it might ought to
1772 have been is if optimizations cause the whole pattern match to not even
1773 be attempted. For example, Perl may figure out that for a string to
1774 match a certain regular expression pattern, the string has to contain
1775 the substring \f(CW"foobar"\fR. Before attempting the match, Perl may look
1776 for that substring, and if not found, immediately fail the match without
1777 actually trying it; so no warning gets generated even if the string
1778 contains an above-Unicode code point.
1779 .PP
1780 This behavior is more "Do what I mean" than in earlier Perls for most
1781 applications. But it catches fewer issues for code that needs to be
1782 strictly Unicode compliant. Therefore there is an additional mode of
1783 operation available to accommodate such code. This mode is enabled if a
1784 regular expression pattern is compiled within the lexical scope where
1785 the \f(CW"non_unicode"\fR warning class has been made fatal, say by:
1786 .PP
1787 .Vb 1
1788 \& use warnings FATAL => "non_unicode"
1789 .Ve
1790 .PP
1791 (see warnings). In this mode of operation, Perl will raise the
1792 warning for all matches against a non-Unicode code point (not just the
1793 arguable ones), and it skips the optimizations that might cause the
1794 warning to not be output. (It currently still won't warn if the match
1795 isn't even attempted, like in the \f(CW"foobar"\fR example above.)
1796 .PP
1797 In summary, Perl now normally treats non-Unicode code points as typical
1798 Unicode unassigned code points for regular expression matches, raising a
1799 warning only when it is arguable what the result should be. However, if
1800 this warning has been made fatal, it isn't skipped.
1801 .PP
1802 There is one exception to all this. \f(CW\*(C`\ep{All}\*(C'\fR looks like a Unicode
1803 property, but it is a Perl extension that is defined to be true for all
1804 possible code points, Unicode or not, so no warning is ever generated
1805 when matching this against a non-Unicode code point. (Prior to v5.20,
1806 it was an exact synonym for \f(CW\*(C`\ep{Any}\*(C'\fR, matching code points \f(CW0\fR
1807 through \f(CW0x10FFFF\fR.)
1808 .SS "Security Implications of Unicode"
1809 .IX Subsection "Security Implications of Unicode"
1810 First, read
1811 Unicode Security Considerations <https://www.unicode.org/reports/tr36>.
1812 .PP
1813 Also, note the following:
1814 .IP \(bu 4
1815 Malformed UTF\-8
1816 .Sp
1817 UTF\-8 is very structured, so many combinations of bytes are invalid. In
1818 the past, Perl tried to soldier on and make some sense of invalid
1819 combinations, but this can lead to security holes, so now, if the Perl
1820 core needs to process an invalid combination, it will either raise a
1821 fatal error, or will replace those bytes by the sequence that forms the
1822 Unicode REPLACEMENT CHARACTER, for which purpose Unicode created it.
1823 .Sp
1824 Every code point can be represented by more than one possible
1825 syntactically valid UTF\-8 sequence. Early on, both Unicode and Perl
1826 considered any of these to be valid, but now, all sequences longer
1827 than the shortest possible one are considered to be malformed.
1828 .Sp
1829 Unicode considers many code points to be illegal, or to be avoided.
1830 Perl generally accepts them, once they have passed through any input
1831 filters that may try to exclude them. These have been discussed above
1832 (see "Surrogates" under UTF\-16 in "Unicode Encodings",
1833 "Noncharacter code points", and "Beyond Unicode code points").
1834 .IP \(bu 4
1835 Regular expression pattern matching may surprise you if you're not
1836 accustomed to Unicode. Starting in Perl 5.14, several pattern
1837 modifiers are available to control this, called the character set
1838 modifiers. Details are given in "Character set modifiers" in perlre.
1839 .PP
1840 As discussed elsewhere, Perl has one foot (two hooves?) planted in
1841 each of two worlds: the old world of ASCII and single-byte locales, and
1842 the new world of Unicode, upgrading when necessary.
1843 If your legacy code does not explicitly use Unicode, no automatic
1844 switch-over to Unicode should happen.
1845 .SS "Unicode in Perl on EBCDIC"
1846 .IX Subsection "Unicode in Perl on EBCDIC"
1847 Unicode is supported on EBCDIC platforms. See perlebcdic.
1848 .PP
1849 Unless ASCII vs. EBCDIC issues are specifically being discussed,
1850 references to UTF\-8 encoding in this document and elsewhere should be
1851 read as meaning UTF-EBCDIC on EBCDIC platforms.
1852 See "Unicode and UTF" in perlebcdic.
1853 .PP
1854 Because UTF-EBCDIC is so similar to UTF\-8, the differences are mostly
1855 hidden from you; \f(CW\*(C`use\ utf8\*(C'\fR (and NOT something like
1856 \&\f(CW\*(C`use\ utfebcdic\*(C'\fR) declares the script is in the platform's
1857 "native" 8\-bit encoding of Unicode. (Similarly for the \f(CW":utf8"\fR
1858 layer.)
1859 .SS Locales
1860 .IX Subsection "Locales"
1861 See "Unicode and UTF\-8" in perllocale
1862 .SS "When Unicode Does Not Happen"
1863 .IX Subsection "When Unicode Does Not Happen"
1864 There are still many places where Unicode (in some encoding or
1865 another) could be given as arguments or received as results, or both in
1866 Perl, but it is not, in spite of Perl having extensive ways to input and
1867 output in Unicode, and a few other "entry points" like the \f(CW@ARGV\fR
1868 array (which can sometimes be interpreted as UTF\-8).
1869 .PP
1870 The following are such interfaces. Also, see "The "Unicode Bug"".
1871 For all of these interfaces Perl
1872 currently (as of v5.16.0) simply assumes byte strings both as arguments
1873 and results, or UTF\-8 strings if the (deprecated) \f(CW\*(C`encoding\*(C'\fR pragma has been used.
1874 .PP
1875 One reason that Perl does not attempt to resolve the role of Unicode in
1876 these situations is that the answers are highly dependent on the operating
1877 system and the file system(s). For example, whether filenames can be
1878 in Unicode and in exactly what kind of encoding, is not exactly a
1879 portable concept. Similarly for \f(CW\*(C`qx\*(C'\fR and \f(CW\*(C`system\*(C'\fR: how well will the
1880 "command-line interface" (and which of them?) handle Unicode?
1881 .IP \(bu 4
1882 \&\f(CW\*(C`chdir\*(C'\fR, \f(CW\*(C`chmod\*(C'\fR, \f(CW\*(C`chown\*(C'\fR, \f(CW\*(C`chroot\*(C'\fR, \f(CW\*(C`exec\*(C'\fR, \f(CW\*(C`link\*(C'\fR, \f(CW\*(C`lstat\*(C'\fR, \f(CW\*(C`mkdir\*(C'\fR,
1883 \&\f(CW\*(C`rename\*(C'\fR, \f(CW\*(C`rmdir\*(C'\fR, \f(CW\*(C`stat\*(C'\fR, \f(CW\*(C`symlink\*(C'\fR, \f(CW\*(C`truncate\*(C'\fR, \f(CW\*(C`unlink\*(C'\fR, \f(CW\*(C`utime\*(C'\fR, \f(CW\*(C`\-X\*(C'\fR
1884 .IP \(bu 4
1885 \&\f(CW%ENV\fR
1886 .IP \(bu 4
1887 \&\f(CW\*(C`glob\*(C'\fR (aka the \f(CW\*(C`<*>\*(C'\fR)
1888 .IP \(bu 4
1889 \&\f(CW\*(C`open\*(C'\fR, \f(CW\*(C`opendir\*(C'\fR, \f(CW\*(C`sysopen\*(C'\fR
1890 .IP \(bu 4
1891 \&\f(CW\*(C`qx\*(C'\fR (aka the backtick operator), \f(CW\*(C`system\*(C'\fR
1892 .IP \(bu 4
1893 \&\f(CW\*(C`readdir\*(C'\fR, \f(CW\*(C`readlink\*(C'\fR
1894 .SS "The ""Unicode Bug"""
1895 .IX Subsection "The ""Unicode Bug"""
1896 The term, "Unicode bug" has been applied to an inconsistency with the
1897 code points in the \f(CW\*(C`Latin\-1 Supplement\*(C'\fR block, that is, between
1898 128 and 255. Without a locale specified, unlike all other characters or
1899 code points, these characters can have very different semantics
1900 depending on the rules in effect. (Characters whose code points are
1901 above 255 force Unicode rules; whereas the rules for ASCII characters
1902 are the same under both ASCII and Unicode rules.)
1903 .PP
1904 Under Unicode rules, these upper\-Latin1 characters are interpreted as
1905 Unicode code points, which means they have the same semantics as Latin\-1
1906 (ISO\-8859\-1) and C1 controls.
1907 .PP
1908 As explained in "ASCII Rules versus Unicode Rules", under ASCII rules,
1909 they are considered to be unassigned characters.
1910 .PP
1911 This can lead to unexpected results. For example, a string's
1912 semantics can suddenly change if a code point above 255 is appended to
1913 it, which changes the rules from ASCII to Unicode. As an
1914 example, consider the following program and its output:
1915 .PP
1916 .Vb 11
1917 \& $ perl \-le\*(Aq
1918 \& no feature "unicode_strings";
1919 \& $s1 = "\exC2";
1920 \& $s2 = "\ex{2660}";
1921 \& for ($s1, $s2, $s1.$s2) {
1922 \& print /\ew/ || 0;
1923 \& }
1924 \& \*(Aq
1925 \& 0
1926 \& 0
1927 \& 1
1928 .Ve
1929 .PP
1930 If there's no \f(CW\*(C`\ew\*(C'\fR in \f(CW\*(C`s1\*(C'\fR nor in \f(CW\*(C`s2\*(C'\fR, why does their concatenation
1931 have one?
1932 .PP
1933 This anomaly stems from Perl's attempt to not disturb older programs that
1934 didn't use Unicode, along with Perl's desire to add Unicode support
1935 seamlessly. But the result turned out to not be seamless. (By the way,
1936 you can choose to be warned when things like this happen. See
1937 \&\f(CW\*(C`encoding::warnings\*(C'\fR.)
1938 .PP
1939 \&\f(CW\*(C`use\ feature\ \*(Aqunicode_strings\*(Aq\*(C'\fR
1940 was added, starting in Perl v5.12, to address this problem. It affects
1941 these things:
1942 .IP \(bu 4
1943 Changing the case of a scalar, that is, using \f(CWuc()\fR, \f(CWucfirst()\fR, \f(CWlc()\fR,
1944 and \f(CWlcfirst()\fR, or \f(CW\*(C`\eL\*(C'\fR, \f(CW\*(C`\eU\*(C'\fR, \f(CW\*(C`\eu\*(C'\fR and \f(CW\*(C`\el\*(C'\fR in double-quotish
1945 contexts, such as regular expression substitutions.
1946 .Sp
1947 Under \f(CW\*(C`unicode_strings\*(C'\fR starting in Perl 5.12.0, Unicode rules are
1948 generally used. See "lc" in perlfunc for details on how this works
1949 in combination with various other pragmas.
1950 .IP \(bu 4
1951 Using caseless (\f(CW\*(C`/i\*(C'\fR) regular expression matching.
1952 .Sp
1953 Starting in Perl 5.14.0, regular expressions compiled within
1954 the scope of \f(CW\*(C`unicode_strings\*(C'\fR use Unicode rules
1955 even when executed or compiled into larger
1956 regular expressions outside the scope.
1957 .IP \(bu 4
1958 Matching any of several properties in regular expressions.
1959 .Sp
1960 These properties are \f(CW\*(C`\eb\*(C'\fR (without braces), \f(CW\*(C`\eB\*(C'\fR (without braces),
1961 \&\f(CW\*(C`\es\*(C'\fR, \f(CW\*(C`\eS\*(C'\fR, \f(CW\*(C`\ew\*(C'\fR, \f(CW\*(C`\eW\*(C'\fR, and all the Posix character classes
1962 \&\fIexcept\fR \f(CW\*(C`[[:ascii:]]\*(C'\fR.
1963 .Sp
1964 Starting in Perl 5.14.0, regular expressions compiled within
1965 the scope of \f(CW\*(C`unicode_strings\*(C'\fR use Unicode rules
1966 even when executed or compiled into larger
1967 regular expressions outside the scope.
1968 .IP \(bu 4
1969 In \f(CW\*(C`quotemeta\*(C'\fR or its inline equivalent \f(CW\*(C`\eQ\*(C'\fR.
1970 .Sp
1971 Starting in Perl 5.16.0, consistent quoting rules are used within the
1972 scope of \f(CW\*(C`unicode_strings\*(C'\fR, as described in "quotemeta" in perlfunc.
1973 Prior to that, or outside its scope, no code points above 127 are quoted
1974 in UTF\-8 encoded strings, but in byte encoded strings, code points
1975 between 128\-255 are always quoted.
1976 .IP \(bu 4
1977 In the \f(CW\*(C`..\*(C'\fR or range operator.
1978 .Sp
1979 Starting in Perl 5.26.0, the range operator on strings treats their lengths
1980 consistently within the scope of \f(CW\*(C`unicode_strings\*(C'\fR. Prior to that, or
1981 outside its scope, it could produce strings whose length in characters
1982 exceeded that of the right-hand side, where the right-hand side took up more
1983 bytes than the correct range endpoint.
1984 .IP \(bu 4
1985 In \f(CW\*(C`split\*(C'\fR's special-case whitespace splitting.
1986 .Sp
1987 Starting in Perl 5.28.0, the \f(CW\*(C`split\*(C'\fR function with a pattern specified as
1988 a string containing a single space handles whitespace characters consistently
1989 within the scope of \f(CW\*(C`unicode_strings\*(C'\fR. Prior to that, or outside its scope,
1990 characters that are whitespace according to Unicode rules but not according to
1991 ASCII rules were treated as field contents rather than field separators when
1992 they appear in byte-encoded strings.
1993 .PP
1994 You can see from the above that the effect of \f(CW\*(C`unicode_strings\*(C'\fR
1995 increased over several Perl releases. (And Perl's support for Unicode
1996 continues to improve; it's best to use the latest available release in
1997 order to get the most complete and accurate results possible.) Note that
1998 \&\f(CW\*(C`unicode_strings\*(C'\fR is automatically chosen if you \f(CW\*(C`use\ v5.12\*(C'\fR or
1999 higher.
2000 .PP
2001 For Perls earlier than those described above, or when a string is passed
2002 to a function outside the scope of \f(CW\*(C`unicode_strings\*(C'\fR, see the next section.
2003 .SS "Forcing Unicode in Perl (Or Unforcing Unicode in Perl)"
2004 .IX Subsection "Forcing Unicode in Perl (Or Unforcing Unicode in Perl)"
2005 Sometimes (see "When Unicode Does Not Happen" or "The "Unicode Bug"")
2006 there are situations where you simply need to force a byte
2007 string into UTF\-8, or vice versa. The standard module Encode can be
2008 used for this, or the low-level calls
2009 \&\f(CWutf8::upgrade($bytestring)\fR and
2010 \&\f(CW\*(C`utf8::downgrade($utf8string[, FAIL_OK])\*(C'\fR.
2011 .PP
2012 Note that \f(CWutf8::downgrade()\fR can fail if the string contains characters
2013 that don't fit into a byte.
2014 .PP
2015 Calling either function on a string that already is in the desired state is a
2016 no-op.
2017 .PP
2018 "ASCII Rules versus Unicode Rules" gives all the ways that a string is
2019 made to use Unicode rules.
2020 .SS "Using Unicode in XS"
2021 .IX Subsection "Using Unicode in XS"
2022 See "Unicode Support" in perlguts for an introduction to Unicode at
2023 the XS level, and "Unicode Support" in perlapi for the API details.
2024 .SS "Hacking Perl to work on earlier Unicode versions (for very serious hackers only)"
2025 .IX Subsection "Hacking Perl to work on earlier Unicode versions (for very serious hackers only)"
2026 Perl by default comes with the latest supported Unicode version built-in, but
2027 the goal is to allow you to change to use any earlier one. In Perls
2028 v5.20 and v5.22, however, the earliest usable version is Unicode 5.1.
2029 Perl v5.18 and v5.24 are able to handle all earlier versions.
2030 .PP
2031 Download the files in the desired version of Unicode from the Unicode web
2032 site <https://www.unicode.org>). These should replace the existing files in
2033 \&\fIlib/unicore\fR in the Perl source tree. Follow the instructions in
2034 \&\fIREADME.perl\fR in that directory to change some of their names, and then build
2035 perl (see INSTALL).
2036 .SS "Porting code from perl\-5.6.X"
2037 .IX Subsection "Porting code from perl-5.6.X"
2038 Perls starting in 5.8 have a different Unicode model from 5.6. In 5.6 the
2039 programmer was required to use the \f(CW\*(C`utf8\*(C'\fR pragma to declare that a
2040 given scope expected to deal with Unicode data and had to make sure that
2041 only Unicode data were reaching that scope. If you have code that is
2042 working with 5.6, you will need some of the following adjustments to
2043 your code. The examples are written such that the code will continue to
2044 work under 5.6, so you should be safe to try them out.
2045 .IP \(bu 3
2046 A filehandle that should read or write UTF\-8
2047 .Sp
2048 .Vb 3
2049 \& if ($] > 5.008) {
2050 \& binmode $fh, ":encoding(UTF\-8)";
2051 \& }
2052 .Ve
2053 .IP \(bu 3
2054 A scalar that is going to be passed to some extension
2055 .Sp
2056 Be it \f(CW\*(C`Compress::Zlib\*(C'\fR, \f(CW\*(C`Apache::Request\*(C'\fR or any extension that has no
2057 mention of Unicode in the manpage, you need to make sure that the
2058 UTF8 flag is stripped off. Note that at the time of this writing
2059 (January 2012) the mentioned modules are not UTF\-8\-aware. Please
2060 check the documentation to verify if this is still true.
2061 .Sp
2062 .Vb 4
2063 \& if ($] > 5.008) {
2064 \& require Encode;
2065 \& $val = Encode::encode("UTF\-8", $val); # make octets
2066 \& }
2067 .Ve
2068 .IP \(bu 3
2069 A scalar we got back from an extension
2070 .Sp
2071 If you believe the scalar comes back as UTF\-8, you will most likely
2072 want the UTF8 flag restored:
2073 .Sp
2074 .Vb 4
2075 \& if ($] > 5.008) {
2076 \& require Encode;
2077 \& $val = Encode::decode("UTF\-8", $val);
2078 \& }
2079 .Ve
2080 .IP \(bu 3
2081 Same thing, if you are really sure it is UTF\-8
2082 .Sp
2083 .Vb 4
2084 \& if ($] > 5.008) {
2085 \& require Encode;
2086 \& Encode::_utf8_on($val);
2087 \& }
2088 .Ve
2089 .IP \(bu 3
2090 A wrapper for DBI \f(CW\*(C`fetchrow_array\*(C'\fR and \f(CW\*(C`fetchrow_hashref\*(C'\fR
2091 .Sp
2092 When the database contains only UTF\-8, a wrapper function or method is
2093 a convenient way to replace all your \f(CW\*(C`fetchrow_array\*(C'\fR and
2094 \&\f(CW\*(C`fetchrow_hashref\*(C'\fR calls. A wrapper function will also make it easier to
2095 adapt to future enhancements in your database driver. Note that at the
2096 time of this writing (January 2012), the DBI has no standardized way
2097 to deal with UTF\-8 data. Please check the DBI documentation to verify if
2098 that is still true.
2099 .Sp
2100 .Vb 10
2101 \& sub fetchrow {
2102 \& # $what is one of fetchrow_{array,hashref}
2103 \& my($self, $sth, $what) = @_;
2104 \& if ($] < 5.008) {
2105 \& return $sth\->$what;
2106 \& } else {
2107 \& require Encode;
2108 \& if (wantarray) {
2109 \& my @arr = $sth\->$what;
2110 \& for (@arr) {
2111 \& defined && /[^\e000\-\e177]/ && Encode::_utf8_on($_);
2112 \& }
2113 \& return @arr;
2114 \& } else {
2115 \& my $ret = $sth\->$what;
2116 \& if (ref $ret) {
2117 \& for my $k (keys %$ret) {
2118 \& defined
2119 \& && /[^\e000\-\e177]/
2120 \& && Encode::_utf8_on($_) for $ret\->{$k};
2121 \& }
2122 \& return $ret;
2123 \& } else {
2124 \& defined && /[^\e000\-\e177]/ && Encode::_utf8_on($_) for $ret;
2125 \& return $ret;
2126 \& }
2127 \& }
2128 \& }
2129 \& }
2130 .Ve
2131 .IP \(bu 3
2132 A large scalar that you know can only contain ASCII
2133 .Sp
2134 Scalars that contain only ASCII and are marked as UTF\-8 are sometimes
2135 a drag to your program. If you recognize such a situation, just remove
2136 the UTF8 flag:
2137 .Sp
2138 .Vb 1
2139 \& utf8::downgrade($val) if $] > 5.008;
2140 .Ve
2141 .SH BUGS
2142 .IX Header "BUGS"
2143 See also "The "Unicode Bug"" above.
2144 .SS "Interaction with Extensions"
2145 .IX Subsection "Interaction with Extensions"
2146 When Perl exchanges data with an extension, the extension should be
2147 able to understand the UTF8 flag and act accordingly. If the
2148 extension doesn't recognize that flag, it's likely that the extension
2149 will return incorrectly-flagged data.
2150 .PP
2151 So if you're working with Unicode data, consult the documentation of
2152 every module you're using if there are any issues with Unicode data
2153 exchange. If the documentation does not talk about Unicode at all,
2154 suspect the worst and probably look at the source to learn how the
2155 module is implemented. Modules written completely in Perl shouldn't
2156 cause problems. Modules that directly or indirectly access code written
2157 in other programming languages are at risk.
2158 .PP
2159 For affected functions, the simple strategy to avoid data corruption is
2160 to always make the encoding of the exchanged data explicit. Choose an
2161 encoding that you know the extension can handle. Convert arguments passed
2162 to the extensions to that encoding and convert results back from that
2163 encoding. Write wrapper functions that do the conversions for you, so
2164 you can later change the functions when the extension catches up.
2165 .PP
2166 To provide an example, let's say the popular \f(CW\*(C`Foo::Bar::escape_html\*(C'\fR
2167 function doesn't deal with Unicode data yet. The wrapper function
2168 would convert the argument to raw UTF\-8 and convert the result back to
2169 Perl's internal representation like so:
2170 .PP
2171 .Vb 6
2172 \& sub my_escape_html ($) {
2173 \& my($what) = shift;
2174 \& return unless defined $what;
2175 \& Encode::decode("UTF\-8", Foo::Bar::escape_html(
2176 \& Encode::encode("UTF\-8", $what)));
2177 \& }
2178 .Ve
2179 .PP
2180 Sometimes, when the extension does not convert data but just stores
2181 and retrieves it, you will be able to use the otherwise
2182 dangerous \f(CWEncode::_utf8_on()\fR function. Let's say
2183 the popular \f(CW\*(C`Foo::Bar\*(C'\fR extension, written in C, provides a \f(CW\*(C`param\*(C'\fR
2184 method that lets you store and retrieve data according to these prototypes:
2185 .PP
2186 .Vb 2
2187 \& $self\->param($name, $value); # set a scalar
2188 \& $value = $self\->param($name); # retrieve a scalar
2189 .Ve
2190 .PP
2191 If it does not yet provide support for any encoding, one could write a
2192 derived class with such a \f(CW\*(C`param\*(C'\fR method:
2193 .PP
2194 .Vb 12
2195 \& sub param {
2196 \& my($self,$name,$value) = @_;
2197 \& utf8::upgrade($name); # make sure it is UTF\-8 encoded
2198 \& if (defined $value) {
2199 \& utf8::upgrade($value); # make sure it is UTF\-8 encoded
2200 \& return $self\->SUPER::param($name,$value);
2201 \& } else {
2202 \& my $ret = $self\->SUPER::param($name);
2203 \& Encode::_utf8_on($ret); # we know, it is UTF\-8 encoded
2204 \& return $ret;
2205 \& }
2206 \& }
2207 .Ve
2208 .PP
2209 Some extensions provide filters on data entry/exit points, such as
2210 \&\f(CW\*(C`DB_File::filter_store_key\*(C'\fR and family. Look out for such filters in
2211 the documentation of your extensions; they can make the transition to
2212 Unicode data much easier.
2213 .SS Speed
2214 .IX Subsection "Speed"
2215 Some functions are slower when working on UTF\-8 encoded strings than
2216 on byte encoded strings. All functions that need to hop over
2217 characters such as \f(CWlength()\fR, \f(CWsubstr()\fR or \f(CWindex()\fR, or matching
2218 regular expressions can work \fBmuch\fR faster when the underlying data are
2219 byte-encoded.
2220 .PP
2221 In Perl 5.8.0 the slowness was often quite spectacular; in Perl 5.8.1
2222 a caching scheme was introduced which improved the situation. In general,
2223 operations with UTF\-8 encoded strings are still slower. As an example,
2224 the Unicode properties (character classes) like \f(CW\*(C`\ep{Nd}\*(C'\fR are known to
2225 be quite a bit slower (5\-20 times) than their simpler counterparts
2226 like \f(CW\*(C`[0\-9]\*(C'\fR (then again, there are hundreds of Unicode characters matching
2227 \&\f(CW\*(C`Nd\*(C'\fR compared with the 10 ASCII characters matching \f(CW\*(C`[0\-9]\*(C'\fR).
2228 .SH "SEE ALSO"
2229 .IX Header "SEE ALSO"
2230 perlunitut, perluniintro, perluniprops, Encode, open, utf8, bytes,
2231 perlretut, "${^UNICODE}" in perlvar,
2232 <https://www.unicode.org/reports/tr44>).