| 1 |
.\" -*- mode: troff; coding: utf-8 -*- |
| 2 |
.\" Automatically generated by Pod::Man 5.0102 (Pod::Simple 3.45) |
| 3 |
.\" |
| 4 |
.\" Standard preamble: |
| 5 |
.\" ======================================================================== |
| 6 |
.de Sp \" Vertical space (when we can't use .PP) |
| 7 |
.if t .sp .5v |
| 8 |
.if n .sp |
| 9 |
.. |
| 10 |
.de Vb \" Begin verbatim text |
| 11 |
.ft CW |
| 12 |
.nf |
| 13 |
.ne \\$1 |
| 14 |
.. |
| 15 |
.de Ve \" End verbatim text |
| 16 |
.ft R |
| 17 |
.fi |
| 18 |
.. |
| 19 |
.\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>. |
| 20 |
.ie n \{\ |
| 21 |
. ds C` "" |
| 22 |
. ds C' "" |
| 23 |
'br\} |
| 24 |
.el\{\ |
| 25 |
. ds C` |
| 26 |
. ds C' |
| 27 |
'br\} |
| 28 |
.\" |
| 29 |
.\" Escape single quotes in literal strings from groff's Unicode transform. |
| 30 |
.ie \n(.g .ds Aq \(aq |
| 31 |
.el .ds Aq ' |
| 32 |
.\" |
| 33 |
.\" If the F register is >0, we'll generate index entries on stderr for |
| 34 |
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index |
| 35 |
.\" entries marked with X<> in POD. Of course, you'll have to process the |
| 36 |
.\" output yourself in some meaningful fashion. |
| 37 |
.\" |
| 38 |
.\" Avoid warning from groff about undefined register 'F'. |
| 39 |
.de IX |
| 40 |
.. |
| 41 |
.nr rF 0 |
| 42 |
.if \n(.g .if rF .nr rF 1 |
| 43 |
.if (\n(rF:(\n(.g==0)) \{\ |
| 44 |
. if \nF \{\ |
| 45 |
. de IX |
| 46 |
. tm Index:\\$1\t\\n%\t"\\$2" |
| 47 |
.. |
| 48 |
. if !\nF==2 \{\ |
| 49 |
. nr % 0 |
| 50 |
. nr F 2 |
| 51 |
. \} |
| 52 |
. \} |
| 53 |
.\} |
| 54 |
.rr rF |
| 55 |
.\" ======================================================================== |
| 56 |
.\" |
| 57 |
.IX Title "PERLUNICODE 1" |
| 58 |
.TH PERLUNICODE 1 2025-11-16 "perl v5.40.1" "Perl Programmers Reference Guide" |
| 59 |
.\" For nroff, turn off justification. Always turn off hyphenation; it makes |
| 60 |
.\" way too many mistakes in technical documents. |
| 61 |
.if n .ad l |
| 62 |
.nh |
| 63 |
.SH NAME |
| 64 |
perlunicode \- Unicode support in Perl |
| 65 |
.SH DESCRIPTION |
| 66 |
.IX Header "DESCRIPTION" |
| 67 |
If you haven't already, before reading this document, you should become |
| 68 |
familiar with both perlunitut and perluniintro. |
| 69 |
.PP |
| 70 |
Unicode aims to \fBUNI\fR\-fy the en\-\fBCODE\fR\-ings of all the world's |
| 71 |
character sets into a single Standard. For quite a few of the various |
| 72 |
coding standards that existed when Unicode was first created, converting |
| 73 |
from each to Unicode essentially meant adding a constant to each code |
| 74 |
point in the original standard, and converting back meant just |
| 75 |
subtracting that same constant. For ASCII and ISO\-8859\-1, the constant |
| 76 |
is 0. For ISO\-8859\-5, (Cyrillic) the constant is 864; for Hebrew |
| 77 |
(ISO\-8859\-8), it's 1488; Thai (ISO\-8859\-11), 3424; and so forth. This |
| 78 |
made it easy to do the conversions, and facilitated the adoption of |
| 79 |
Unicode. |
| 80 |
.PP |
| 81 |
And it worked; nowadays, those legacy standards are rarely used. Most |
| 82 |
everyone uses Unicode. |
| 83 |
.PP |
| 84 |
Unicode is a comprehensive standard. It specifies many things outside |
| 85 |
the scope of Perl, such as how to display sequences of characters. For |
| 86 |
a full discussion of all aspects of Unicode, see |
| 87 |
<https://www.unicode.org>. |
| 88 |
.SS "Important Caveats" |
| 89 |
.IX Subsection "Important Caveats" |
| 90 |
Even though some of this section may not be understandable to you on |
| 91 |
first reading, we think it's important enough to highlight some of the |
| 92 |
gotchas before delving further, so here goes: |
| 93 |
.PP |
| 94 |
Unicode support is an extensive requirement. While Perl does not |
| 95 |
implement the Unicode standard or the accompanying technical reports |
| 96 |
from cover to cover, Perl does support many Unicode features. |
| 97 |
.PP |
| 98 |
Also, the use of Unicode may present security issues that aren't |
| 99 |
obvious, see "Security Implications of Unicode" below. |
| 100 |
.ie n .IP "Safest if you ""use feature \*(Aqunicode_strings\*(Aq""" 4 |
| 101 |
.el .IP "Safest if you \f(CWuse feature \*(Aqunicode_strings\*(Aq\fR" 4 |
| 102 |
.IX Item "Safest if you use feature unicode_strings" |
| 103 |
In order to preserve backward compatibility, Perl does not turn |
| 104 |
on full internal Unicode support unless the pragma |
| 105 |
\&\f(CW\*(C`use\ feature\ \*(Aqunicode_strings\*(Aq\*(C'\fR |
| 106 |
is specified. (This is automatically |
| 107 |
selected if you \f(CW\*(C`use\ v5.12\*(C'\fR or higher.) Failure to do this can |
| 108 |
trigger unexpected surprises. See "The "Unicode Bug"" below. |
| 109 |
.Sp |
| 110 |
This pragma doesn't affect I/O. Nor does it change the internal |
| 111 |
representation of strings, only their interpretation. There are still |
| 112 |
several places where Unicode isn't fully supported, such as in |
| 113 |
filenames. |
| 114 |
.IP "Input and Output Layers" 4 |
| 115 |
.IX Item "Input and Output Layers" |
| 116 |
Use the \f(CW:encoding(...)\fR layer to read from and write to |
| 117 |
filehandles using the specified encoding. (See open.) |
| 118 |
.IP "You must convert your non-ASCII, non\-UTF\-8 Perl scripts to be UTF\-8." 4 |
| 119 |
.IX Item "You must convert your non-ASCII, non-UTF-8 Perl scripts to be UTF-8." |
| 120 |
The encoding module has been deprecated since perl 5.18 and the |
| 121 |
perl internals it requires have been removed with perl 5.26. |
| 122 |
.ie n .IP """use utf8"" still needed to enable UTF\-8 in scripts" 4 |
| 123 |
.el .IP "\f(CWuse utf8\fR still needed to enable UTF\-8 in scripts" 4 |
| 124 |
.IX Item "use utf8 still needed to enable UTF-8 in scripts" |
| 125 |
If your Perl script is itself encoded in UTF\-8, |
| 126 |
the \f(CW\*(C`use\ utf8\*(C'\fR pragma must be explicitly included to enable |
| 127 |
recognition of that (in string or regular expression literals, or in |
| 128 |
identifier names). \fBThis is the only time when an explicit \fR\f(CB\*(C`use\ utf8\*(C'\fR\fB is needed.\fR (See utf8). |
| 129 |
.Sp |
| 130 |
If a Perl script begins with the bytes that form the UTF\-8 encoding of |
| 131 |
the Unicode BYTE ORDER MARK (\f(CW\*(C`BOM\*(C'\fR, see "Unicode Encodings"), those |
| 132 |
bytes are completely ignored. |
| 133 |
.IP "UTF\-16 scripts autodetected" 4 |
| 134 |
.IX Item "UTF-16 scripts autodetected" |
| 135 |
If a Perl script begins with the Unicode \f(CW\*(C`BOM\*(C'\fR (UTF\-16LE, |
| 136 |
UTF16\-BE), or if the script looks like non\-\f(CW\*(C`BOM\*(C'\fR\-marked |
| 137 |
UTF\-16 of either endianness, Perl will correctly read in the script as |
| 138 |
the appropriate Unicode encoding. |
| 139 |
.SS "Byte and Character Semantics" |
| 140 |
.IX Subsection "Byte and Character Semantics" |
| 141 |
Before Unicode, most encodings used 8 bits (a single byte) to encode |
| 142 |
each character. Thus a character was a byte, and a byte was a |
| 143 |
character, and there could be only 256 or fewer possible characters. |
| 144 |
"Byte Semantics" in the title of this section refers to |
| 145 |
this behavior. There was no need to distinguish between "Byte" and |
| 146 |
"Character". |
| 147 |
.PP |
| 148 |
Then along comes Unicode which has room for over a million characters |
| 149 |
(and Perl allows for even more). This means that a character may |
| 150 |
require more than a single byte to represent it, and so the two terms |
| 151 |
are no longer equivalent. What matter are the characters as whole |
| 152 |
entities, and not usually the bytes that comprise them. That's what the |
| 153 |
term "Character Semantics" in the title of this section refers to. |
| 154 |
.PP |
| 155 |
Perl had to change internally to decouple "bytes" from "characters". |
| 156 |
It is important that you too change your ideas, if you haven't already, |
| 157 |
so that "byte" and "character" no longer mean the same thing in your |
| 158 |
mind. |
| 159 |
.PP |
| 160 |
The basic building block of Perl strings has always been a "character". |
| 161 |
The changes basically come down to that the implementation no longer |
| 162 |
thinks that a character is always just a single byte. |
| 163 |
.PP |
| 164 |
There are various things to note: |
| 165 |
.IP \(bu 4 |
| 166 |
String handling functions, for the most part, continue to operate in |
| 167 |
terms of characters. \f(CWlength()\fR, for example, returns the number of |
| 168 |
characters in a string, just as before. But that number no longer is |
| 169 |
necessarily the same as the number of bytes in the string (there may be |
| 170 |
more bytes than characters). The other such functions include |
| 171 |
\&\f(CWchop()\fR, \f(CWchomp()\fR, \f(CWsubstr()\fR, \f(CWpos()\fR, \f(CWindex()\fR, \f(CWrindex()\fR, |
| 172 |
\&\f(CWsort()\fR, \f(CWsprintf()\fR, and \f(CWwrite()\fR. |
| 173 |
.Sp |
| 174 |
The exceptions are: |
| 175 |
.RS 4 |
| 176 |
.IP \(bu 4 |
| 177 |
the bit-oriented \f(CW\*(C`vec\*(C'\fR |
| 178 |
.Sp |
| 179 |
\ |
| 180 |
.IP \(bu 4 |
| 181 |
the byte-oriented \f(CW\*(C`pack\*(C'\fR/\f(CW\*(C`unpack\*(C'\fR \f(CW"C"\fR format |
| 182 |
.Sp |
| 183 |
However, the \f(CW\*(C`W\*(C'\fR specifier does operate on whole characters, as does the |
| 184 |
\&\f(CW\*(C`U\*(C'\fR specifier. |
| 185 |
.IP \(bu 4 |
| 186 |
some operators that interact with the platform's operating system |
| 187 |
.Sp |
| 188 |
Operators dealing with filenames are examples. |
| 189 |
.IP \(bu 4 |
| 190 |
when the functions are called from within the scope of the |
| 191 |
\&\f(CW\*(C`use\ bytes\*(C'\fR pragma |
| 192 |
.Sp |
| 193 |
Likely, you should use this only for debugging anyway. |
| 194 |
.RE |
| 195 |
.RS 4 |
| 196 |
.RE |
| 197 |
.IP \(bu 4 |
| 198 |
Strings\-\-including hash keys\-\-and regular expression patterns may |
| 199 |
contain characters that have ordinal values larger than 255. |
| 200 |
.Sp |
| 201 |
If you use a Unicode editor to edit your program, Unicode characters may |
| 202 |
occur directly within the literal strings in UTF\-8 encoding, or UTF\-16. |
| 203 |
(The former requires a \f(CW\*(C`use utf8\*(C'\fR, the latter may require a \f(CW\*(C`BOM\*(C'\fR.) |
| 204 |
.Sp |
| 205 |
"Creating Unicode" in perluniintro gives other ways to place non-ASCII |
| 206 |
characters in your strings. |
| 207 |
.IP \(bu 4 |
| 208 |
The \f(CWchr()\fR and \f(CWord()\fR functions work on whole characters. |
| 209 |
.IP \(bu 4 |
| 210 |
Regular expressions match whole characters. For example, \f(CW"."\fR matches |
| 211 |
a whole character instead of only a single byte. |
| 212 |
.IP \(bu 4 |
| 213 |
The \f(CW\*(C`tr///\*(C'\fR operator translates whole characters. (Note that the |
| 214 |
\&\f(CW\*(C`tr///CU\*(C'\fR functionality has been removed. For similar functionality to |
| 215 |
that, see \f(CW\*(C`pack(\*(AqU0\*(Aq,\ ...)\*(C'\fR and \f(CW\*(C`pack(\*(AqC0\*(Aq,\ ...)\*(C'\fR). |
| 216 |
.IP \(bu 4 |
| 217 |
\&\f(CW\*(C`scalar reverse()\*(C'\fR reverses by character rather than by byte. |
| 218 |
.IP \(bu 4 |
| 219 |
The bit string operators, \f(CW\*(C`& | ^ ~\*(C'\fR and (starting in v5.22) |
| 220 |
\&\f(CW\*(C`&. |. ^. ~.\*(C'\fR can operate on bit strings encoded in UTF\-8, but this |
| 221 |
can give unexpected results if any of the strings contain code points |
| 222 |
above 0xFF. Starting in v5.28, it is a fatal error to have such an |
| 223 |
operand. Otherwise, the operation is performed on a non\-UTF\-8 copy of |
| 224 |
the operand. If you're not sure about the encoding of a string, |
| 225 |
downgrade it before using any of these operators; you can use |
| 226 |
\&\f(CWutf8::utf8_downgrade()\fR. |
| 227 |
.PP |
| 228 |
The bottom line is that Perl has always practiced "Character Semantics", |
| 229 |
but with the advent of Unicode, that is now different than "Byte |
| 230 |
Semantics". |
| 231 |
.SS "ASCII Rules versus Unicode Rules" |
| 232 |
.IX Subsection "ASCII Rules versus Unicode Rules" |
| 233 |
Before Unicode, when a character was a byte was a character, |
| 234 |
Perl knew only about the 128 characters defined by ASCII, code points 0 |
| 235 |
through 127 (except for under \f(CW\*(C`use\ locale\*(C'\fR). That |
| 236 |
left the code |
| 237 |
points 128 to 255 as unassigned, and available for whatever use a |
| 238 |
program might want. The only semantics they have is their ordinal |
| 239 |
numbers, and that they are members of none of the non-negative character |
| 240 |
classes. None are considered to match \f(CW\*(C`\ew\*(C'\fR for example, but all match |
| 241 |
\&\f(CW\*(C`\eW\*(C'\fR. |
| 242 |
.PP |
| 243 |
Unicode, of course, assigns each of those code points a particular |
| 244 |
meaning (along with ones above 255). To preserve backward |
| 245 |
compatibility, Perl only uses the Unicode meanings when there is some |
| 246 |
indication that Unicode is what is intended; otherwise the non-ASCII |
| 247 |
code points remain treated as if they are unassigned. |
| 248 |
.PP |
| 249 |
Here are the ways that Perl knows that a string should be treated as |
| 250 |
Unicode: |
| 251 |
.IP \(bu 4 |
| 252 |
Within the scope of \f(CW\*(C`use\ utf8\*(C'\fR |
| 253 |
.Sp |
| 254 |
If the whole program is Unicode (signified by using 8\-bit \fBU\fRnicode |
| 255 |
\&\fBT\fRransformation \fBF\fRormat), then all literal strings within it must be |
| 256 |
Unicode. |
| 257 |
.IP \(bu 4 |
| 258 |
Within the scope of |
| 259 |
\&\f(CW\*(C`use\ feature\ \*(Aqunicode_strings\*(Aq\*(C'\fR |
| 260 |
.Sp |
| 261 |
This pragma was created so you can explicitly tell Perl that operations |
| 262 |
executed within its scope are to use Unicode rules. More operations are |
| 263 |
affected with newer perls. See "The "Unicode Bug"". |
| 264 |
.IP \(bu 4 |
| 265 |
Within the scope of \f(CW\*(C`use\ v5.12\*(C'\fR or higher |
| 266 |
.Sp |
| 267 |
This implicitly turns on \f(CW\*(C`use\ feature\ \*(Aqunicode_strings\*(Aq\*(C'\fR. |
| 268 |
.IP \(bu 4 |
| 269 |
Within the scope of |
| 270 |
\&\f(CW\*(C`use\ locale\ \*(Aqnot_characters\*(Aq\*(C'\fR, |
| 271 |
or \f(CW\*(C`use\ locale\*(C'\fR and the current |
| 272 |
locale is a UTF\-8 locale. |
| 273 |
.Sp |
| 274 |
The former is defined to imply Unicode handling; and the latter |
| 275 |
indicates a Unicode locale, hence a Unicode interpretation of all |
| 276 |
strings within it. |
| 277 |
.IP \(bu 4 |
| 278 |
When the string contains a Unicode-only code point |
| 279 |
.Sp |
| 280 |
Perl has never accepted code points above 255 without them being |
| 281 |
Unicode, so their use implies Unicode for the whole string. |
| 282 |
.IP \(bu 4 |
| 283 |
When the string contains a Unicode named code point \f(CW\*(C`\eN{...}\*(C'\fR |
| 284 |
.Sp |
| 285 |
The \f(CW\*(C`\eN{...}\*(C'\fR construct explicitly refers to a Unicode code point, |
| 286 |
even if it is one that is also in ASCII. Therefore the string |
| 287 |
containing it must be Unicode. |
| 288 |
.IP \(bu 4 |
| 289 |
When the string has come from an external source marked as |
| 290 |
Unicode |
| 291 |
.Sp |
| 292 |
The \f(CW\*(C`\-C\*(C'\fR command line option can |
| 293 |
specify that certain inputs to the program are Unicode, and the values |
| 294 |
of this can be read by your Perl code, see "${^UNICODE}" in perlvar. |
| 295 |
.IP \(bu 4 |
| 296 |
When the string has been upgraded to UTF\-8 |
| 297 |
.Sp |
| 298 |
The function \f(CWutf8::utf8_upgrade()\fR |
| 299 |
can be explicitly used to permanently (unless a subsequent |
| 300 |
\&\f(CWutf8::utf8_downgrade()\fR is called) cause a string to be treated as |
| 301 |
Unicode. |
| 302 |
.IP \(bu 4 |
| 303 |
There are additional methods for regular expression patterns |
| 304 |
.Sp |
| 305 |
A pattern that is compiled with the \f(CW\*(C`/u\*(C'\fR or \f(CW\*(C`/a\*(C'\fR modifiers is |
| 306 |
treated as Unicode (though there are some restrictions with \f(CW\*(C`/a\*(C'\fR). |
| 307 |
Under the \f(CW\*(C`/d\*(C'\fR and \f(CW\*(C`/l\*(C'\fR modifiers, there are several other |
| 308 |
indications for Unicode; see "Character set modifiers" in perlre. |
| 309 |
.PP |
| 310 |
Note that all of the above are overridden within the scope of |
| 311 |
\&\f(CW\*(C`use bytes\*(C'\fR; but you should be using this pragma only for |
| 312 |
debugging. |
| 313 |
.PP |
| 314 |
Note also that some interactions with the platform's operating system |
| 315 |
never use Unicode rules. |
| 316 |
.PP |
| 317 |
When Unicode rules are in effect: |
| 318 |
.IP \(bu 4 |
| 319 |
Case translation operators use the Unicode case translation tables. |
| 320 |
.Sp |
| 321 |
Note that \f(CWuc()\fR, or \f(CW\*(C`\eU\*(C'\fR in interpolated strings, translates to |
| 322 |
uppercase, while \f(CW\*(C`ucfirst\*(C'\fR, or \f(CW\*(C`\eu\*(C'\fR in interpolated strings, |
| 323 |
translates to titlecase in languages that make the distinction (which is |
| 324 |
equivalent to uppercase in languages without the distinction). |
| 325 |
.Sp |
| 326 |
There is a CPAN module, \f(CW\*(C`Unicode::Casing\*(C'\fR, which allows you to |
| 327 |
define your own mappings to be used in \f(CWlc()\fR, \f(CWlcfirst()\fR, \f(CWuc()\fR, |
| 328 |
\&\f(CWucfirst()\fR, and \f(CW\*(C`fc\*(C'\fR (or their double-quoted string inlined versions |
| 329 |
such as \f(CW\*(C`\eU\*(C'\fR). (Prior to Perl 5.16, this functionality was partially |
| 330 |
provided in the Perl core, but suffered from a number of insurmountable |
| 331 |
drawbacks, so the CPAN module was written instead.) |
| 332 |
.IP \(bu 4 |
| 333 |
Character classes in regular expressions match based on the character |
| 334 |
properties specified in the Unicode properties database. |
| 335 |
.Sp |
| 336 |
\&\f(CW\*(C`\ew\*(C'\fR can be used to match a Japanese ideograph, for instance; and |
| 337 |
\&\f(CW\*(C`[[:digit:]]\*(C'\fR a Bengali number. |
| 338 |
.IP \(bu 4 |
| 339 |
Named Unicode properties, scripts, and block ranges may be used (like |
| 340 |
bracketed character classes) by using the \f(CW\*(C`\ep{}\*(C'\fR "matches property" |
| 341 |
construct and the \f(CW\*(C`\eP{}\*(C'\fR negation, "doesn't match property". |
| 342 |
.Sp |
| 343 |
See "Unicode Character Properties" for more details. |
| 344 |
.Sp |
| 345 |
You can define your own character properties and use them |
| 346 |
in the regular expression with the \f(CW\*(C`\ep{}\*(C'\fR or \f(CW\*(C`\eP{}\*(C'\fR construct. |
| 347 |
See "User-Defined Character Properties" for more details. |
| 348 |
.SS "Extended Grapheme Clusters (Logical characters)" |
| 349 |
.IX Subsection "Extended Grapheme Clusters (Logical characters)" |
| 350 |
Consider a character, say \f(CW\*(C`H\*(C'\fR. It could appear with various marks around it, |
| 351 |
such as an acute accent, or a circumflex, or various hooks, circles, arrows, |
| 352 |
\&\fIetc.\fR, above, below, to one side or the other, \fIetc\fR. There are many |
| 353 |
possibilities among the world's languages. The number of combinations is |
| 354 |
astronomical, and if there were a character for each combination, it would |
| 355 |
soon exhaust Unicode's more than a million possible characters. So Unicode |
| 356 |
took a different approach: there is a character for the base \f(CW\*(C`H\*(C'\fR, and a |
| 357 |
character for each of the possible marks, and these can be variously combined |
| 358 |
to get a final logical character. So a logical character\-\-what appears to be a |
| 359 |
single character\-\-can be a sequence of more than one individual characters. |
| 360 |
The Unicode standard calls these "extended grapheme clusters" (which |
| 361 |
is an improved version of the no-longer much used "grapheme cluster"); |
| 362 |
Perl furnishes the \f(CW\*(C`\eX\*(C'\fR regular expression construct to match such |
| 363 |
sequences in their entirety. |
| 364 |
.PP |
| 365 |
But Unicode's intent is to unify the existing character set standards and |
| 366 |
practices, and several pre-existing standards have single characters that |
| 367 |
mean the same thing as some of these combinations, like ISO\-8859\-1, |
| 368 |
which has quite a few of them. For example, \f(CW"LATIN CAPITAL LETTER E |
| 369 |
WITH ACUTE"\fR was already in this standard when Unicode came along. |
| 370 |
Unicode therefore added it to its repertoire as that single character. |
| 371 |
But this character is considered by Unicode to be equivalent to the |
| 372 |
sequence consisting of the character \f(CW"LATIN CAPITAL LETTER E"\fR |
| 373 |
followed by the character \f(CW"COMBINING ACUTE ACCENT"\fR. |
| 374 |
.PP |
| 375 |
\&\f(CW"LATIN CAPITAL LETTER E WITH ACUTE"\fR is called a "pre-composed" |
| 376 |
character, and its equivalence with the "E" and the "COMBINING ACCENT" |
| 377 |
sequence is called canonical equivalence. All pre-composed characters |
| 378 |
are said to have a decomposition (into the equivalent sequence), and the |
| 379 |
decomposition type is also called canonical. A string may consist |
| 380 |
as much as possible of precomposed characters, or it may consist of |
| 381 |
entirely decomposed characters. Unicode calls these respectively, |
| 382 |
"Normalization Form Composed" (NFC) and "Normalization Form Decomposed". |
| 383 |
The \f(CW\*(C`Unicode::Normalize\*(C'\fR module contains functions that convert |
| 384 |
between the two. A string may also have both composed characters and |
| 385 |
decomposed characters; this module can be used to make it all one or the |
| 386 |
other. |
| 387 |
.PP |
| 388 |
You may be presented with strings in any of these equivalent forms. |
| 389 |
There is currently nothing in Perl 5 that ignores the differences. So |
| 390 |
you'll have to specially handle it. The usual advice is to convert your |
| 391 |
inputs to \f(CW\*(C`NFD\*(C'\fR before processing further. |
| 392 |
.PP |
| 393 |
For more detailed information, see <http://unicode.org/reports/tr15/>. |
| 394 |
.SS "Unicode Character Properties" |
| 395 |
.IX Subsection "Unicode Character Properties" |
| 396 |
(The only time that Perl considers a sequence of individual code |
| 397 |
points as a single logical character is in the \f(CW\*(C`\eX\*(C'\fR construct, already |
| 398 |
mentioned above. Therefore "character" in this discussion means a single |
| 399 |
Unicode code point.) |
| 400 |
.PP |
| 401 |
Very nearly all Unicode character properties are accessible through |
| 402 |
regular expressions by using the \f(CW\*(C`\ep{}\*(C'\fR "matches property" construct |
| 403 |
and the \f(CW\*(C`\eP{}\*(C'\fR "doesn't match property" for its negation. |
| 404 |
.PP |
| 405 |
For instance, \f(CW\*(C`\ep{Uppercase}\*(C'\fR matches any single character with the Unicode |
| 406 |
\&\f(CW"Uppercase"\fR property, while \f(CW\*(C`\ep{L}\*(C'\fR matches any character with a |
| 407 |
\&\f(CW\*(C`General_Category\*(C'\fR of \f(CW"L"\fR (letter) property (see |
| 408 |
"General_Category" below). Brackets are not |
| 409 |
required for single letter property names, so \f(CW\*(C`\ep{L}\*(C'\fR is equivalent to \f(CW\*(C`\epL\*(C'\fR. |
| 410 |
.PP |
| 411 |
More formally, \f(CW\*(C`\ep{Uppercase}\*(C'\fR matches any single character whose Unicode |
| 412 |
\&\f(CW\*(C`Uppercase\*(C'\fR property value is \f(CW\*(C`True\*(C'\fR, and \f(CW\*(C`\eP{Uppercase}\*(C'\fR matches any character |
| 413 |
whose \f(CW\*(C`Uppercase\*(C'\fR property value is \f(CW\*(C`False\*(C'\fR, and they could have been written as |
| 414 |
\&\f(CW\*(C`\ep{Uppercase=True}\*(C'\fR and \f(CW\*(C`\ep{Uppercase=False}\*(C'\fR, respectively. |
| 415 |
.PP |
| 416 |
This formality is needed when properties are not binary; that is, if they can |
| 417 |
take on more values than just \f(CW\*(C`True\*(C'\fR and \f(CW\*(C`False\*(C'\fR. For example, the |
| 418 |
\&\f(CW\*(C`Bidi_Class\*(C'\fR property (see "Bidirectional Character Types" below), |
| 419 |
can take on several different |
| 420 |
values, such as \f(CW\*(C`Left\*(C'\fR, \f(CW\*(C`Right\*(C'\fR, \f(CW\*(C`Whitespace\*(C'\fR, and others. To match these, one needs |
| 421 |
to specify both the property name (\f(CW\*(C`Bidi_Class\*(C'\fR), AND the value being |
| 422 |
matched against |
| 423 |
(\f(CW\*(C`Left\*(C'\fR, \f(CW\*(C`Right\*(C'\fR, \fIetc.\fR). This is done, as in the examples above, by having the |
| 424 |
two components separated by an equal sign (or interchangeably, a colon), like |
| 425 |
\&\f(CW\*(C`\ep{Bidi_Class: Left}\*(C'\fR. |
| 426 |
.PP |
| 427 |
All Unicode-defined character properties may be written in these compound forms |
| 428 |
of \f(CW\*(C`\ep{\fR\f(CIproperty\fR\f(CW=\fR\f(CIvalue\fR\f(CW}\*(C'\fR or \f(CW\*(C`\ep{\fR\f(CIproperty\fR\f(CW:\fR\f(CIvalue\fR\f(CW}\*(C'\fR, but Perl provides some |
| 429 |
additional properties that are written only in the single form, as well as |
| 430 |
single-form short-cuts for all binary properties and certain others described |
| 431 |
below, in which you may omit the property name and the equals or colon |
| 432 |
separator. |
| 433 |
.PP |
| 434 |
Most Unicode character properties have at least two synonyms (or aliases if you |
| 435 |
prefer): a short one that is easier to type and a longer one that is more |
| 436 |
descriptive and hence easier to understand. Thus the \f(CW"L"\fR and |
| 437 |
\&\f(CW"Letter"\fR properties above are equivalent and can be used |
| 438 |
interchangeably. Likewise, \f(CW"Upper"\fR is a synonym for \f(CW"Uppercase"\fR, |
| 439 |
and we could have written \f(CW\*(C`\ep{Uppercase}\*(C'\fR equivalently as \f(CW\*(C`\ep{Upper}\*(C'\fR. |
| 440 |
Also, there are typically various synonyms for the values the property |
| 441 |
can be. For binary properties, \f(CW"True"\fR has 3 synonyms: \f(CW"T"\fR, |
| 442 |
\&\f(CW"Yes"\fR, and \f(CW"Y"\fR; and \f(CW"False"\fR has correspondingly \f(CW"F"\fR, |
| 443 |
\&\f(CW"No"\fR, and \f(CW"N"\fR. But be careful. A short form of a value for one |
| 444 |
property may not mean the same thing as the short form spelled the same |
| 445 |
for another. |
| 446 |
Thus, for the \f(CW"General_Category"\fR property, \f(CW"L"\fR means |
| 447 |
\&\f(CW"Letter"\fR, but for the \f(CW\*(C`Bidi_Class\*(C'\fR |
| 448 |
property, \f(CW"L"\fR means \f(CW"Left"\fR. A complete list of properties and |
| 449 |
synonyms is in perluniprops. |
| 450 |
.PP |
| 451 |
Upper/lower case differences in property names and values are irrelevant; |
| 452 |
thus \f(CW\*(C`\ep{Upper}\*(C'\fR means the same thing as \f(CW\*(C`\ep{upper}\*(C'\fR or even \f(CW\*(C`\ep{UpPeR}\*(C'\fR. |
| 453 |
Similarly, you can add or subtract underscores anywhere in the middle of a |
| 454 |
word, so that these are also equivalent to \f(CW\*(C`\ep{U_p_p_e_r}\*(C'\fR. And white space |
| 455 |
is generally irrelevant adjacent to non-word characters, such as the |
| 456 |
braces and the equals or colon separators, so \f(CW\*(C`\ep{ Upper }\*(C'\fR and |
| 457 |
\&\f(CW\*(C`\ep{ Upper_case : Y }\*(C'\fR are equivalent to these as well. In fact, white |
| 458 |
space and even hyphens can usually be added or deleted anywhere. So |
| 459 |
even \f(CW\*(C`\ep{ Up\-per case = Yes}\*(C'\fR is equivalent. All this is called |
| 460 |
"loose-matching" by Unicode. The "name" property has some restrictions |
| 461 |
on this due to a few outlier names. Full details are given in |
| 462 |
<https://www.unicode.org/reports/tr44/tr44\-24.html#UAX44\-LM2>. |
| 463 |
.PP |
| 464 |
The few places where stricter matching is |
| 465 |
used is in the middle of numbers, the "name" property, and in the Perl |
| 466 |
extension properties that begin or end with an underscore. Stricter |
| 467 |
matching cares about white space (except adjacent to non-word |
| 468 |
characters), hyphens, and non-interior underscores. |
| 469 |
.PP |
| 470 |
You can also use negation in both \f(CW\*(C`\ep{}\*(C'\fR and \f(CW\*(C`\eP{}\*(C'\fR by introducing a caret |
| 471 |
(\f(CW\*(C`^\*(C'\fR) between the first brace and the property name: \f(CW\*(C`\ep{^Tamil}\*(C'\fR is |
| 472 |
equal to \f(CW\*(C`\eP{Tamil}\*(C'\fR. |
| 473 |
.PP |
| 474 |
Almost all properties are immune to case-insensitive matching. That is, |
| 475 |
adding a \f(CW\*(C`/i\*(C'\fR regular expression modifier does not change what they |
| 476 |
match. There are two sets that are affected. |
| 477 |
The first set is |
| 478 |
\&\f(CW\*(C`Uppercase_Letter\*(C'\fR, |
| 479 |
\&\f(CW\*(C`Lowercase_Letter\*(C'\fR, |
| 480 |
and \f(CW\*(C`Titlecase_Letter\*(C'\fR, |
| 481 |
all of which match \f(CW\*(C`Cased_Letter\*(C'\fR under \f(CW\*(C`/i\*(C'\fR matching. |
| 482 |
And the second set is |
| 483 |
\&\f(CW\*(C`Uppercase\*(C'\fR, |
| 484 |
\&\f(CW\*(C`Lowercase\*(C'\fR, |
| 485 |
and \f(CW\*(C`Titlecase\*(C'\fR, |
| 486 |
all of which match \f(CW\*(C`Cased\*(C'\fR under \f(CW\*(C`/i\*(C'\fR matching. |
| 487 |
This set also includes its subsets \f(CW\*(C`PosixUpper\*(C'\fR and \f(CW\*(C`PosixLower\*(C'\fR both |
| 488 |
of which under \f(CW\*(C`/i\*(C'\fR match \f(CW\*(C`PosixAlpha\*(C'\fR. |
| 489 |
(The difference between these sets is that some things, such as Roman |
| 490 |
numerals, come in both upper and lower case so they are \f(CW\*(C`Cased\*(C'\fR, but |
| 491 |
aren't considered letters, so they aren't \f(CW\*(C`Cased_Letter\*(C'\fR's.) |
| 492 |
.PP |
| 493 |
See "Beyond Unicode code points" for special considerations when |
| 494 |
matching Unicode properties against non-Unicode code points. |
| 495 |
.PP |
| 496 |
\fR\f(BIGeneral_Category\fR\fI\fR |
| 497 |
.IX Subsection "General_Category" |
| 498 |
.PP |
| 499 |
Every Unicode character is assigned a general category, which is the "most |
| 500 |
usual categorization of a character" (from |
| 501 |
<https://www.unicode.org/reports/tr44>). |
| 502 |
.PP |
| 503 |
The compound way of writing these is like \f(CW\*(C`\ep{General_Category=Number}\*(C'\fR |
| 504 |
(short: \f(CW\*(C`\ep{gc:n}\*(C'\fR). But Perl furnishes shortcuts in which everything up |
| 505 |
through the equal or colon separator is omitted. So you can instead just write |
| 506 |
\&\f(CW\*(C`\epN\*(C'\fR. |
| 507 |
.PP |
| 508 |
Here are the short and long forms of the values the \f(CW\*(C`General Category\*(C'\fR property |
| 509 |
can have: |
| 510 |
.PP |
| 511 |
.Vb 1 |
| 512 |
\& Short Long |
| 513 |
\& |
| 514 |
\& L Letter |
| 515 |
\& LC, L& Cased_Letter (that is: [\ep{Ll}\ep{Lu}\ep{Lt}]) |
| 516 |
\& Lu Uppercase_Letter |
| 517 |
\& Ll Lowercase_Letter |
| 518 |
\& Lt Titlecase_Letter |
| 519 |
\& Lm Modifier_Letter |
| 520 |
\& Lo Other_Letter |
| 521 |
\& |
| 522 |
\& M Mark |
| 523 |
\& Mn Nonspacing_Mark |
| 524 |
\& Mc Spacing_Mark |
| 525 |
\& Me Enclosing_Mark |
| 526 |
\& |
| 527 |
\& N Number |
| 528 |
\& Nd Decimal_Number (also Digit) |
| 529 |
\& Nl Letter_Number |
| 530 |
\& No Other_Number |
| 531 |
\& |
| 532 |
\& P Punctuation (also Punct) |
| 533 |
\& Pc Connector_Punctuation |
| 534 |
\& Pd Dash_Punctuation |
| 535 |
\& Ps Open_Punctuation |
| 536 |
\& Pe Close_Punctuation |
| 537 |
\& Pi Initial_Punctuation |
| 538 |
\& (may behave like Ps or Pe depending on usage) |
| 539 |
\& Pf Final_Punctuation |
| 540 |
\& (may behave like Ps or Pe depending on usage) |
| 541 |
\& Po Other_Punctuation |
| 542 |
\& |
| 543 |
\& S Symbol |
| 544 |
\& Sm Math_Symbol |
| 545 |
\& Sc Currency_Symbol |
| 546 |
\& Sk Modifier_Symbol |
| 547 |
\& So Other_Symbol |
| 548 |
\& |
| 549 |
\& Z Separator |
| 550 |
\& Zs Space_Separator |
| 551 |
\& Zl Line_Separator |
| 552 |
\& Zp Paragraph_Separator |
| 553 |
\& |
| 554 |
\& C Other |
| 555 |
\& Cc Control (also Cntrl) |
| 556 |
\& Cf Format |
| 557 |
\& Cs Surrogate |
| 558 |
\& Co Private_Use |
| 559 |
\& Cn Unassigned |
| 560 |
.Ve |
| 561 |
.PP |
| 562 |
Single-letter properties match all characters in any of the |
| 563 |
two-letter sub-properties starting with the same letter. |
| 564 |
\&\f(CW\*(C`LC\*(C'\fR and \f(CW\*(C`L&\*(C'\fR are special: both are aliases for the set consisting of everything matched by \f(CW\*(C`Ll\*(C'\fR, \f(CW\*(C`Lu\*(C'\fR, and \f(CW\*(C`Lt\*(C'\fR. |
| 565 |
.PP |
| 566 |
\fR\f(BIBidirectional Character Types\fR\fI\fR |
| 567 |
.IX Subsection "Bidirectional Character Types" |
| 568 |
.PP |
| 569 |
Because scripts differ in their directionality (Hebrew and Arabic are |
| 570 |
written right to left, for example) Unicode supplies a \f(CW\*(C`Bidi_Class\*(C'\fR property. |
| 571 |
Some of the values this property can have are: |
| 572 |
.PP |
| 573 |
.Vb 1 |
| 574 |
\& Value Meaning |
| 575 |
\& |
| 576 |
\& L Left\-to\-Right |
| 577 |
\& LRE Left\-to\-Right Embedding |
| 578 |
\& LRO Left\-to\-Right Override |
| 579 |
\& R Right\-to\-Left |
| 580 |
\& AL Arabic Letter |
| 581 |
\& RLE Right\-to\-Left Embedding |
| 582 |
\& RLO Right\-to\-Left Override |
| 583 |
\& PDF Pop Directional Format |
| 584 |
\& EN European Number |
| 585 |
\& ES European Separator |
| 586 |
\& ET European Terminator |
| 587 |
\& AN Arabic Number |
| 588 |
\& CS Common Separator |
| 589 |
\& NSM Non\-Spacing Mark |
| 590 |
\& BN Boundary Neutral |
| 591 |
\& B Paragraph Separator |
| 592 |
\& S Segment Separator |
| 593 |
\& WS Whitespace |
| 594 |
\& ON Other Neutrals |
| 595 |
.Ve |
| 596 |
.PP |
| 597 |
This property is always written in the compound form. |
| 598 |
For example, \f(CW\*(C`\ep{Bidi_Class:R}\*(C'\fR matches characters that are normally |
| 599 |
written right to left. Unlike the |
| 600 |
\&\f(CW"General_Category"\fR property, this |
| 601 |
property can have more values added in a future Unicode release. Those |
| 602 |
listed above comprised the complete set for many Unicode releases, but |
| 603 |
others were added in Unicode 6.3; you can always find what the |
| 604 |
current ones are in perluniprops. And |
| 605 |
<https://www.unicode.org/reports/tr9/> describes how to use them. |
| 606 |
.PP |
| 607 |
\fR\f(BIScripts\fR\fI\fR |
| 608 |
.IX Subsection "Scripts" |
| 609 |
.PP |
| 610 |
The world's languages are written in many different scripts. This sentence |
| 611 |
(unless you're reading it in translation) is written in Latin, while Russian is |
| 612 |
written in Cyrillic, and Greek is written in, well, Greek; Japanese mainly in |
| 613 |
Hiragana or Katakana. There are many more. |
| 614 |
.PP |
| 615 |
The Unicode \f(CW\*(C`Script\*(C'\fR and \f(CW\*(C`Script_Extensions\*(C'\fR properties give what |
| 616 |
script a given character is in. The \f(CW\*(C`Script_Extensions\*(C'\fR property is an |
| 617 |
improved version of \f(CW\*(C`Script\*(C'\fR, as demonstrated below. Either property |
| 618 |
can be specified with the compound form like |
| 619 |
\&\f(CW\*(C`\ep{Script=Hebrew}\*(C'\fR (short: \f(CW\*(C`\ep{sc=hebr}\*(C'\fR), or |
| 620 |
\&\f(CW\*(C`\ep{Script_Extensions=Javanese}\*(C'\fR (short: \f(CW\*(C`\ep{scx=java}\*(C'\fR). |
| 621 |
In addition, Perl furnishes shortcuts for all |
| 622 |
\&\f(CW\*(C`Script_Extensions\*(C'\fR property names. You can omit everything up through |
| 623 |
the equals (or colon), and simply write \f(CW\*(C`\ep{Latin}\*(C'\fR or \f(CW\*(C`\eP{Cyrillic}\*(C'\fR. |
| 624 |
(This is not true for \f(CW\*(C`Script\*(C'\fR, which is required to be |
| 625 |
written in the compound form. Prior to Perl v5.26, the single form |
| 626 |
returned the plain old \f(CW\*(C`Script\*(C'\fR version, but was changed because |
| 627 |
\&\f(CW\*(C`Script_Extensions\*(C'\fR gives better results.) |
| 628 |
.PP |
| 629 |
The difference between these two properties involves characters that are |
| 630 |
used in multiple scripts. For example the digits '0' through '9' are |
| 631 |
used in many parts of the world. These are placed in a script named |
| 632 |
\&\f(CW\*(C`Common\*(C'\fR. Other characters are used in just a few scripts. For |
| 633 |
example, the \f(CW"KATAKANA\-HIRAGANA DOUBLE HYPHEN"\fR is used in both Japanese |
| 634 |
scripts, Katakana and Hiragana, but nowhere else. The \f(CW\*(C`Script\*(C'\fR |
| 635 |
property places all characters that are used in multiple scripts in the |
| 636 |
\&\f(CW\*(C`Common\*(C'\fR script, while the \f(CW\*(C`Script_Extensions\*(C'\fR property places those |
| 637 |
that are used in only a few scripts into each of those scripts; while |
| 638 |
still using \f(CW\*(C`Common\*(C'\fR for those used in many scripts. Thus both these |
| 639 |
match: |
| 640 |
.PP |
| 641 |
.Vb 2 |
| 642 |
\& "0" =~ /\ep{sc=Common}/ # Matches |
| 643 |
\& "0" =~ /\ep{scx=Common}/ # Matches |
| 644 |
.Ve |
| 645 |
.PP |
| 646 |
and only the first of these match: |
| 647 |
.PP |
| 648 |
.Vb 2 |
| 649 |
\& "\eN{KATAKANA\-HIRAGANA DOUBLE HYPHEN}" =~ /\ep{sc=Common} # Matches |
| 650 |
\& "\eN{KATAKANA\-HIRAGANA DOUBLE HYPHEN}" =~ /\ep{scx=Common} # No match |
| 651 |
.Ve |
| 652 |
.PP |
| 653 |
And only the last two of these match: |
| 654 |
.PP |
| 655 |
.Vb 4 |
| 656 |
\& "\eN{KATAKANA\-HIRAGANA DOUBLE HYPHEN}" =~ /\ep{sc=Hiragana} # No match |
| 657 |
\& "\eN{KATAKANA\-HIRAGANA DOUBLE HYPHEN}" =~ /\ep{sc=Katakana} # No match |
| 658 |
\& "\eN{KATAKANA\-HIRAGANA DOUBLE HYPHEN}" =~ /\ep{scx=Hiragana} # Matches |
| 659 |
\& "\eN{KATAKANA\-HIRAGANA DOUBLE HYPHEN}" =~ /\ep{scx=Katakana} # Matches |
| 660 |
.Ve |
| 661 |
.PP |
| 662 |
\&\f(CW\*(C`Script_Extensions\*(C'\fR is thus an improved \f(CW\*(C`Script\*(C'\fR, in which there are |
| 663 |
fewer characters in the \f(CW\*(C`Common\*(C'\fR script, and correspondingly more in |
| 664 |
other scripts. It is new in Unicode version 6.0, and its data are likely |
| 665 |
to change significantly in later releases, as things get sorted out. |
| 666 |
New code should probably be using \f(CW\*(C`Script_Extensions\*(C'\fR and not plain |
| 667 |
\&\f(CW\*(C`Script\*(C'\fR. If you compile perl with a Unicode release that doesn't have |
| 668 |
\&\f(CW\*(C`Script_Extensions\*(C'\fR, the single form Perl extensions will instead refer |
| 669 |
to the plain \f(CW\*(C`Script\*(C'\fR property. If you compile with a version of |
| 670 |
Unicode that doesn't have the \f(CW\*(C`Script\*(C'\fR property, these extensions will |
| 671 |
not be defined at all. |
| 672 |
.PP |
| 673 |
(Actually, besides \f(CW\*(C`Common\*(C'\fR, the \f(CW\*(C`Inherited\*(C'\fR script, contains |
| 674 |
characters that are used in multiple scripts. These are modifier |
| 675 |
characters which inherit the script value |
| 676 |
of the controlling character. Some of these are used in many scripts, |
| 677 |
and so go into \f(CW\*(C`Inherited\*(C'\fR in both \f(CW\*(C`Script\*(C'\fR and \f(CW\*(C`Script_Extensions\*(C'\fR. |
| 678 |
Others are used in just a few scripts, so are in \f(CW\*(C`Inherited\*(C'\fR in |
| 679 |
\&\f(CW\*(C`Script\*(C'\fR, but not in \f(CW\*(C`Script_Extensions\*(C'\fR.) |
| 680 |
.PP |
| 681 |
It is worth stressing that there are several different sets of digits in |
| 682 |
Unicode that are equivalent to 0\-9 and are matchable by \f(CW\*(C`\ed\*(C'\fR in a |
| 683 |
regular expression. If they are used in a single language only, they |
| 684 |
are in that language's \f(CW\*(C`Script\*(C'\fR and \f(CW\*(C`Script_Extensions\*(C'\fR. If they are |
| 685 |
used in more than one script, they will be in \f(CW\*(C`sc=Common\*(C'\fR, but only |
| 686 |
if they are used in many scripts should they be in \f(CW\*(C`scx=Common\*(C'\fR. |
| 687 |
.PP |
| 688 |
The explanation above has omitted some detail; refer to UAX#24 "Unicode |
| 689 |
Script Property": <https://www.unicode.org/reports/tr24>. |
| 690 |
.PP |
| 691 |
A complete list of scripts and their shortcuts is in perluniprops. |
| 692 |
.PP |
| 693 |
\fR\f(BIUse of the \fR\f(CB"Is"\fR\f(BI Prefix\fR\fI\fR |
| 694 |
.IX Subsection "Use of the ""Is"" Prefix" |
| 695 |
.PP |
| 696 |
For backward compatibility (with ancient Perl 5.6), all properties writable |
| 697 |
without using the compound form mentioned |
| 698 |
so far may have \f(CW\*(C`Is\*(C'\fR or \f(CW\*(C`Is_\*(C'\fR prepended to their name, so \f(CW\*(C`\eP{Is_Lu}\*(C'\fR, for |
| 699 |
example, is equal to \f(CW\*(C`\eP{Lu}\*(C'\fR, and \f(CW\*(C`\ep{IsScript:Arabic}\*(C'\fR is equal to |
| 700 |
\&\f(CW\*(C`\ep{Arabic}\*(C'\fR. |
| 701 |
.PP |
| 702 |
\fR\f(BIBlocks\fR\fI\fR |
| 703 |
.IX Subsection "Blocks" |
| 704 |
.PP |
| 705 |
In addition to \fBscripts\fR, Unicode also defines \fBblocks\fR of |
| 706 |
characters. The difference between scripts and blocks is that the |
| 707 |
concept of scripts is closer to natural languages, while the concept |
| 708 |
of blocks is more of an artificial grouping based on groups of Unicode |
| 709 |
characters with consecutive ordinal values. For example, the \f(CW"Basic Latin"\fR |
| 710 |
block is all the characters whose ordinals are between 0 and 127, inclusive; in |
| 711 |
other words, the ASCII characters. The \f(CW"Latin"\fR script contains some letters |
| 712 |
from this as well as several other blocks, like \f(CW"Latin\-1 Supplement"\fR, |
| 713 |
\&\f(CW"Latin Extended\-A"\fR, \fIetc.\fR, but it does not contain all the characters from |
| 714 |
those blocks. It does not, for example, contain the digits 0\-9, because |
| 715 |
those digits are shared across many scripts, and hence are in the |
| 716 |
\&\f(CW\*(C`Common\*(C'\fR script. |
| 717 |
.PP |
| 718 |
For more about scripts versus blocks, see UAX#24 "Unicode Script Property": |
| 719 |
<https://www.unicode.org/reports/tr24> |
| 720 |
.PP |
| 721 |
The \f(CW\*(C`Script_Extensions\*(C'\fR or \f(CW\*(C`Script\*(C'\fR properties are likely to be the |
| 722 |
ones you want to use when processing |
| 723 |
natural language; the \f(CW\*(C`Block\*(C'\fR property may occasionally be useful in working |
| 724 |
with the nuts and bolts of Unicode. |
| 725 |
.PP |
| 726 |
Block names are matched in the compound form, like \f(CW\*(C`\ep{Block: Arrows}\*(C'\fR or |
| 727 |
\&\f(CW\*(C`\ep{Blk=Hebrew}\*(C'\fR. Unlike most other properties, only a few block names have a |
| 728 |
Unicode-defined short name. |
| 729 |
.PP |
| 730 |
Perl also defines single form synonyms for the block property in cases |
| 731 |
where these do not conflict with something else. But don't use any of |
| 732 |
these, because they are unstable. Since these are Perl extensions, they |
| 733 |
are subordinate to official Unicode property names; Unicode doesn't know |
| 734 |
nor care about Perl's extensions. It may happen that a name that |
| 735 |
currently means the Perl extension will later be changed without warning |
| 736 |
to mean a different Unicode property in a future version of the perl |
| 737 |
interpreter that uses a later Unicode release, and your code would no |
| 738 |
longer work. The extensions are mentioned here for completeness: Take |
| 739 |
the block name and prefix it with one of: \f(CW\*(C`In\*(C'\fR (for example |
| 740 |
\&\f(CW\*(C`\ep{Blk=Arrows}\*(C'\fR can currently be written as \f(CW\*(C`\ep{In_Arrows}\*(C'\fR); or |
| 741 |
sometimes \f(CW\*(C`Is\*(C'\fR (like \f(CW\*(C`\ep{Is_Arrows}\*(C'\fR); or sometimes no prefix at all |
| 742 |
(\f(CW\*(C`\ep{Arrows}\*(C'\fR). As of this writing (Unicode 9.0) there are no |
| 743 |
conflicts with using the \f(CW\*(C`In_\*(C'\fR prefix, but there are plenty with the |
| 744 |
other two forms. For example, \f(CW\*(C`\ep{Is_Hebrew}\*(C'\fR and \f(CW\*(C`\ep{Hebrew}\*(C'\fR mean |
| 745 |
\&\f(CW\*(C`\ep{Script_Extensions=Hebrew}\*(C'\fR which is NOT the same thing as |
| 746 |
\&\f(CW\*(C`\ep{Blk=Hebrew}\*(C'\fR. Our |
| 747 |
advice used to be to use the \f(CW\*(C`In_\*(C'\fR prefix as a single form way of |
| 748 |
specifying a block. But Unicode 8.0 added properties whose names begin |
| 749 |
with \f(CW\*(C`In\*(C'\fR, and it's now clear that it's only luck that's so far |
| 750 |
prevented a conflict. Using \f(CW\*(C`In\*(C'\fR is only marginally less typing than |
| 751 |
\&\f(CW\*(C`Blk:\*(C'\fR, and the latter's meaning is clearer anyway, and guaranteed to |
| 752 |
never conflict. So don't take chances. Use \f(CW\*(C`\ep{Blk=foo}\*(C'\fR for new |
| 753 |
code. And be sure that block is what you really really want to do. In |
| 754 |
most cases scripts are what you want instead. |
| 755 |
.PP |
| 756 |
A complete list of blocks is in perluniprops. |
| 757 |
.PP |
| 758 |
\fR\f(BIOther Properties\fR\fI\fR |
| 759 |
.IX Subsection "Other Properties" |
| 760 |
.PP |
| 761 |
There are many more properties than the very basic ones described here. |
| 762 |
A complete list is in perluniprops. |
| 763 |
.PP |
| 764 |
Unicode defines all its properties in the compound form, so all single-form |
| 765 |
properties are Perl extensions. Most of these are just synonyms for the |
| 766 |
Unicode ones, but some are genuine extensions, including several that are in |
| 767 |
the compound form. And quite a few of these are actually recommended by Unicode |
| 768 |
(in <https://www.unicode.org/reports/tr18>). |
| 769 |
.PP |
| 770 |
This section gives some details on all extensions that aren't just |
| 771 |
synonyms for compound-form Unicode properties |
| 772 |
(for those properties, you'll have to refer to the |
| 773 |
Unicode Standard <https://www.unicode.org/reports/tr44>. |
| 774 |
.ie n .IP "\fR\fB""\ep{All}""\fR\fB\fR" 4 |
| 775 |
.el .IP \fR\f(CB\ep{All}\fR\fB\fR 4 |
| 776 |
.IX Item "p{All}" |
| 777 |
This matches every possible code point. It is equivalent to \f(CW\*(C`qr/./s\*(C'\fR. |
| 778 |
Unlike all the other non-user-defined \f(CW\*(C`\ep{}\*(C'\fR property matches, no |
| 779 |
warning is ever generated if this is property is matched against a |
| 780 |
non-Unicode code point (see "Beyond Unicode code points" below). |
| 781 |
.ie n .IP "\fR\fB""\ep{Alnum}""\fR\fB\fR" 4 |
| 782 |
.el .IP \fR\f(CB\ep{Alnum}\fR\fB\fR 4 |
| 783 |
.IX Item "p{Alnum}" |
| 784 |
This matches any \f(CW\*(C`\ep{Alphabetic}\*(C'\fR or \f(CW\*(C`\ep{Decimal_Number}\*(C'\fR character. |
| 785 |
.ie n .IP "\fR\fB""\ep{Any}""\fR\fB\fR" 4 |
| 786 |
.el .IP \fR\f(CB\ep{Any}\fR\fB\fR 4 |
| 787 |
.IX Item "p{Any}" |
| 788 |
This matches any of the 1_114_112 Unicode code points. It is a synonym |
| 789 |
for \f(CW\*(C`\ep{Unicode}\*(C'\fR. |
| 790 |
.ie n .IP "\fR\fB""\ep{ASCII}""\fR\fB\fR" 4 |
| 791 |
.el .IP \fR\f(CB\ep{ASCII}\fR\fB\fR 4 |
| 792 |
.IX Item "p{ASCII}" |
| 793 |
This matches any of the 128 characters in the US-ASCII character set, |
| 794 |
which is a subset of Unicode. |
| 795 |
.ie n .IP "\fR\fB""\ep{Assigned}""\fR\fB\fR" 4 |
| 796 |
.el .IP \fR\f(CB\ep{Assigned}\fR\fB\fR 4 |
| 797 |
.IX Item "p{Assigned}" |
| 798 |
This matches any assigned code point; that is, any code point whose general |
| 799 |
category is not \f(CW\*(C`Unassigned\*(C'\fR (or equivalently, not \f(CW\*(C`Cn\*(C'\fR). |
| 800 |
.ie n .IP "\fR\fB""\ep{Blank}""\fR\fB\fR" 4 |
| 801 |
.el .IP \fR\f(CB\ep{Blank}\fR\fB\fR 4 |
| 802 |
.IX Item "p{Blank}" |
| 803 |
This is the same as \f(CW\*(C`\eh\*(C'\fR and \f(CW\*(C`\ep{HorizSpace}\*(C'\fR: A character that changes the |
| 804 |
spacing horizontally. |
| 805 |
.ie n .IP "\fR\fB""\ep{Decomposition_Type: Non_Canonical}""\fR\fB\fR (Short: ""\ep{Dt=NonCanon}"")" 4 |
| 806 |
.el .IP "\fR\f(CB\ep{Decomposition_Type: Non_Canonical}\fR\fB\fR (Short: \f(CW\ep{Dt=NonCanon}\fR)" 4 |
| 807 |
.IX Item "p{Decomposition_Type: Non_Canonical} (Short: p{Dt=NonCanon})" |
| 808 |
Matches a character that has any of the non-canonical decomposition |
| 809 |
types. Canonical decompositions are introduced in the |
| 810 |
"Extended Grapheme Clusters (Logical characters)" section above. |
| 811 |
However, many more characters have a different type of decomposition, |
| 812 |
generically called "compatible" decompositions, or "non-canonical". The |
| 813 |
sequences that form these decompositions are not considered canonically |
| 814 |
equivalent to the pre-composed character. An example is the |
| 815 |
\&\f(CW"SUPERSCRIPT ONE"\fR. It is somewhat like a regular digit 1, but not |
| 816 |
exactly; its decomposition into the digit 1 is called a "compatible" |
| 817 |
decomposition, specifically a "super" (for "superscript") decomposition. |
| 818 |
There are several such compatibility decompositions (see |
| 819 |
<https://www.unicode.org/reports/tr44>). \f(CW\*(C`\ep{Dt:\ Non_Canon}\*(C'\fR is a |
| 820 |
Perl extension that uses just one name to refer to the union of all of |
| 821 |
them. |
| 822 |
.Sp |
| 823 |
Most Unicode characters don't have a decomposition, so their |
| 824 |
decomposition type is \f(CW"None"\fR. Hence, \f(CW\*(C`Non_Canonical\*(C'\fR is equivalent |
| 825 |
to |
| 826 |
.Sp |
| 827 |
.Vb 1 |
| 828 |
\& qr/(?[ \eP{DT=Canonical} \- \ep{DT=None} ])/ |
| 829 |
.Ve |
| 830 |
.Sp |
| 831 |
(Note that one of the non-canonical decompositions is named "compat", |
| 832 |
which could perhaps have been better named "miscellaneous". It includes |
| 833 |
just the things that Unicode couldn't figure out a better generic name |
| 834 |
for.) |
| 835 |
.ie n .IP "\fR\fB""\ep{Graph}""\fR\fB\fR" 4 |
| 836 |
.el .IP \fR\f(CB\ep{Graph}\fR\fB\fR 4 |
| 837 |
.IX Item "p{Graph}" |
| 838 |
Matches any character that is graphic. Theoretically, this means a character |
| 839 |
that on a printer would cause ink to be used. |
| 840 |
.ie n .IP "\fR\fB""\ep{HorizSpace}""\fR\fB\fR" 4 |
| 841 |
.el .IP \fR\f(CB\ep{HorizSpace}\fR\fB\fR 4 |
| 842 |
.IX Item "p{HorizSpace}" |
| 843 |
This is the same as \f(CW\*(C`\eh\*(C'\fR and \f(CW\*(C`\ep{Blank}\*(C'\fR: a character that changes the |
| 844 |
spacing horizontally. |
| 845 |
.ie n .IP "\fR\fB""\ep{In=*}""\fR\fB\fR" 4 |
| 846 |
.el .IP \fR\f(CB\ep{In=*}\fR\fB\fR 4 |
| 847 |
.IX Item "p{In=*}" |
| 848 |
This is a synonym for \f(CW\*(C`\ep{Present_In=*}\*(C'\fR |
| 849 |
.ie n .IP "\fR\fB""\ep{PerlSpace}""\fR\fB\fR" 4 |
| 850 |
.el .IP \fR\f(CB\ep{PerlSpace}\fR\fB\fR 4 |
| 851 |
.IX Item "p{PerlSpace}" |
| 852 |
This is the same as \f(CW\*(C`\es\*(C'\fR, restricted to ASCII, namely \f(CW\*(C`[\ \ef\en\er\et]\*(C'\fR |
| 853 |
and starting in Perl v5.18, a vertical tab. |
| 854 |
.Sp |
| 855 |
Mnemonic: Perl's (original) space |
| 856 |
.ie n .IP "\fR\fB""\ep{PerlWord}""\fR\fB\fR" 4 |
| 857 |
.el .IP \fR\f(CB\ep{PerlWord}\fR\fB\fR 4 |
| 858 |
.IX Item "p{PerlWord}" |
| 859 |
This is the same as \f(CW\*(C`\ew\*(C'\fR, restricted to ASCII, namely \f(CW\*(C`[A\-Za\-z0\-9_]\*(C'\fR |
| 860 |
.Sp |
| 861 |
Mnemonic: Perl's (original) word. |
| 862 |
.ie n .IP "\fR\fB""\ep{Posix...}""\fR\fB\fR" 4 |
| 863 |
.el .IP \fR\f(CB\ep{Posix...}\fR\fB\fR 4 |
| 864 |
.IX Item "p{Posix...}" |
| 865 |
There are several of these, which are equivalents, using the \f(CW\*(C`\ep{}\*(C'\fR |
| 866 |
notation, for Posix classes and are described in |
| 867 |
"POSIX Character Classes" in perlrecharclass. |
| 868 |
.ie n .IP "\fR\fB""\ep{Present_In: *}""\fR\fB\fR (Short: ""\ep{In=*}"")" 4 |
| 869 |
.el .IP "\fR\f(CB\ep{Present_In: *}\fR\fB\fR (Short: \f(CW\ep{In=*}\fR)" 4 |
| 870 |
.IX Item "p{Present_In: *} (Short: p{In=*})" |
| 871 |
This property is used when you need to know in what Unicode version(s) a |
| 872 |
character is. |
| 873 |
.Sp |
| 874 |
The "*" above stands for some Unicode version number, such as |
| 875 |
\&\f(CW1.1\fR or \f(CW12.0\fR; or the "*" can also be \f(CW\*(C`Unassigned\*(C'\fR. This property will |
| 876 |
match the code points whose final disposition has been settled as of the |
| 877 |
Unicode release given by the version number; \f(CW\*(C`\ep{Present_In: Unassigned}\*(C'\fR |
| 878 |
will match those code points whose meaning has yet to be assigned. |
| 879 |
.Sp |
| 880 |
For example, \f(CW\*(C`U+0041\*(C'\fR \f(CW"LATIN CAPITAL LETTER A"\fR was present in the very first |
| 881 |
Unicode release available, which is \f(CW1.1\fR, so this property is true for all |
| 882 |
valid "*" versions. On the other hand, \f(CW\*(C`U+1EFF\*(C'\fR was not assigned until version |
| 883 |
5.1 when it became \f(CW"LATIN SMALL LETTER Y WITH LOOP"\fR, so the only "*" that |
| 884 |
would match it are 5.1, 5.2, and later. |
| 885 |
.Sp |
| 886 |
Unicode furnishes the \f(CW\*(C`Age\*(C'\fR property from which this is derived. The problem |
| 887 |
with Age is that a strict interpretation of it (which Perl takes) has it |
| 888 |
matching the precise release a code point's meaning is introduced in. Thus |
| 889 |
\&\f(CW\*(C`U+0041\*(C'\fR would match only 1.1; and \f(CW\*(C`U+1EFF\*(C'\fR only 5.1. This is not usually what |
| 890 |
you want. |
| 891 |
.Sp |
| 892 |
Some non-Perl implementations of the Age property may change its meaning to be |
| 893 |
the same as the Perl \f(CW\*(C`Present_In\*(C'\fR property; just be aware of that. |
| 894 |
.Sp |
| 895 |
Another confusion with both these properties is that the definition is not |
| 896 |
that the code point has been \fIassigned\fR, but that the meaning of the code point |
| 897 |
has been \fIdetermined\fR. This is because 66 code points will always be |
| 898 |
unassigned, and so the \f(CW\*(C`Age\*(C'\fR for them is the Unicode version in which the decision |
| 899 |
to make them so was made. For example, \f(CW\*(C`U+FDD0\*(C'\fR is to be permanently |
| 900 |
unassigned to a character, and the decision to do that was made in version 3.1, |
| 901 |
so \f(CW\*(C`\ep{Age=3.1}\*(C'\fR matches this character, as also does \f(CW\*(C`\ep{Present_In: 3.1}\*(C'\fR and up. |
| 902 |
.ie n .IP "\fR\fB""\ep{Print}""\fR\fB\fR" 4 |
| 903 |
.el .IP \fR\f(CB\ep{Print}\fR\fB\fR 4 |
| 904 |
.IX Item "p{Print}" |
| 905 |
This matches any character that is graphical or blank, except controls. |
| 906 |
.ie n .IP "\fR\fB""\ep{SpacePerl}""\fR\fB\fR" 4 |
| 907 |
.el .IP \fR\f(CB\ep{SpacePerl}\fR\fB\fR 4 |
| 908 |
.IX Item "p{SpacePerl}" |
| 909 |
This is the same as \f(CW\*(C`\es\*(C'\fR, including beyond ASCII. |
| 910 |
.Sp |
| 911 |
Mnemonic: Space, as modified by Perl. (It doesn't include the vertical tab |
| 912 |
until v5.18, which both the Posix standard and Unicode consider white space.) |
| 913 |
.ie n .IP "\fR\fB""\ep{Title}""\fR\fB\fR and \fB\fR\fB""\ep{Titlecase}""\fR\fB\fR" 4 |
| 914 |
.el .IP "\fR\f(CB\ep{Title}\fR\fB\fR and \fB\fR\f(CB\ep{Titlecase}\fR\fB\fR" 4 |
| 915 |
.IX Item "p{Title} and p{Titlecase}" |
| 916 |
Under case-sensitive matching, these both match the same code points as |
| 917 |
\&\f(CW\*(C`\ep{General Category=Titlecase_Letter}\*(C'\fR (\f(CW\*(C`\ep{gc=lt}\*(C'\fR). The difference |
| 918 |
is that under \f(CW\*(C`/i\*(C'\fR caseless matching, these match the same as |
| 919 |
\&\f(CW\*(C`\ep{Cased}\*(C'\fR, whereas \f(CW\*(C`\ep{gc=lt}\*(C'\fR matches \f(CW\*(C`\ep{Cased_Letter\*(C'\fR). |
| 920 |
.ie n .IP "\fR\fB""\ep{Unicode}""\fR\fB\fR" 4 |
| 921 |
.el .IP \fR\f(CB\ep{Unicode}\fR\fB\fR 4 |
| 922 |
.IX Item "p{Unicode}" |
| 923 |
This matches any of the 1_114_112 Unicode code points. |
| 924 |
\&\f(CW\*(C`\ep{Any}\*(C'\fR. |
| 925 |
.ie n .IP "\fR\fB""\ep{VertSpace}""\fR\fB\fR" 4 |
| 926 |
.el .IP \fR\f(CB\ep{VertSpace}\fR\fB\fR 4 |
| 927 |
.IX Item "p{VertSpace}" |
| 928 |
This is the same as \f(CW\*(C`\ev\*(C'\fR: A character that changes the spacing vertically. |
| 929 |
.ie n .IP "\fR\fB""\ep{Word}""\fR\fB\fR" 4 |
| 930 |
.el .IP \fR\f(CB\ep{Word}\fR\fB\fR 4 |
| 931 |
.IX Item "p{Word}" |
| 932 |
This is the same as \f(CW\*(C`\ew\*(C'\fR, including over 100_000 characters beyond ASCII. |
| 933 |
.ie n .IP "\fR\fB""\ep{XPosix...}""\fR\fB\fR" 4 |
| 934 |
.el .IP \fR\f(CB\ep{XPosix...}\fR\fB\fR 4 |
| 935 |
.IX Item "p{XPosix...}" |
| 936 |
There are several of these, which are the standard Posix classes |
| 937 |
extended to the full Unicode range. They are described in |
| 938 |
"POSIX Character Classes" in perlrecharclass. |
| 939 |
.ie n .SS "Comparison of ""\eN{...}"" and ""\ep{name=...}""" |
| 940 |
.el .SS "Comparison of \f(CW\eN{...}\fP and \f(CW\ep{name=...}\fP" |
| 941 |
.IX Subsection "Comparison of N{...} and p{name=...}" |
| 942 |
Starting in Perl 5.32, you can specify a character by its name in |
| 943 |
regular expression patterns using \f(CW\*(C`\ep{name=...}\*(C'\fR. This is in addition |
| 944 |
to the longstanding method of using \f(CW\*(C`\eN{...}\*(C'\fR. The following |
| 945 |
summarizes the differences between these two: |
| 946 |
.PP |
| 947 |
.Vb 6 |
| 948 |
\& \eN{...} \ep{Name=...} |
| 949 |
\& can interpolate only with eval yes [1] |
| 950 |
\& custom names yes no [2] |
| 951 |
\& name aliases yes yes [3] |
| 952 |
\& named sequences yes yes [4] |
| 953 |
\& name value parsing exact Unicode loose [5] |
| 954 |
.Ve |
| 955 |
.IP [1] 4 |
| 956 |
.IX Item "[1]" |
| 957 |
The ability to interpolate means you can do something like |
| 958 |
.Sp |
| 959 |
.Vb 1 |
| 960 |
\& qr/\ep{na=latin capital letter $which}/ |
| 961 |
.Ve |
| 962 |
.Sp |
| 963 |
and specify \f(CW$which\fR elsewhere. |
| 964 |
.IP [2] 4 |
| 965 |
.IX Item "[2]" |
| 966 |
You can create your own names for characters, and override official |
| 967 |
ones when using \f(CW\*(C`\eN{...}\*(C'\fR. See "CUSTOM ALIASES" in charnames. |
| 968 |
.IP [3] 4 |
| 969 |
.IX Item "[3]" |
| 970 |
Some characters have multiple names (synonyms). |
| 971 |
.IP [4] 4 |
| 972 |
.IX Item "[4]" |
| 973 |
Some particular sequences of characters are given a single name, in |
| 974 |
addition to their individual ones. |
| 975 |
.IP [5] 4 |
| 976 |
.IX Item "[5]" |
| 977 |
Exact name value matching means you have to specify case, hyphens, |
| 978 |
underscores, and spaces precisely in the name you want. Loose matching |
| 979 |
follows the Unicode rules |
| 980 |
<https://www.unicode.org/reports/tr44/tr44\-24.html#UAX44\-LM2>, |
| 981 |
where these are mostly irrelevant. Except for a few outlier character |
| 982 |
names, these are the same rules as are already used for any other |
| 983 |
\&\f(CW\*(C`\ep{...}\*(C'\fR property. |
| 984 |
.SS "Wildcards in Property Values" |
| 985 |
.IX Subsection "Wildcards in Property Values" |
| 986 |
Starting in Perl 5.30, it is possible to do something like this: |
| 987 |
.PP |
| 988 |
.Vb 1 |
| 989 |
\& qr!\ep{numeric_value=/\eA[0\-5]\ez/}! |
| 990 |
.Ve |
| 991 |
.PP |
| 992 |
or, by abbreviating and adding \f(CW\*(C`/x\*(C'\fR, |
| 993 |
.PP |
| 994 |
.Vb 1 |
| 995 |
\& qr! \ep{nv= /(?x) \eA [0\-5] \ez / }! |
| 996 |
.Ve |
| 997 |
.PP |
| 998 |
This matches all code points whose numeric value is one of 0, 1, 2, 3, |
| 999 |
4, or 5. This particular example could instead have been written as |
| 1000 |
.PP |
| 1001 |
.Vb 1 |
| 1002 |
\& qr! \eA [ \ep{nv=0}\ep{nv=1}\ep{nv=2}\ep{nv=3}\ep{nv=4}\ep{nv=5} ] \ez !xx |
| 1003 |
.Ve |
| 1004 |
.PP |
| 1005 |
in earlier perls, so in this case this feature just makes things easier |
| 1006 |
and shorter to write. If we hadn't included the \f(CW\*(C`\eA\*(C'\fR and \f(CW\*(C`\ez\*(C'\fR, these |
| 1007 |
would have matched things like \f(CW\*(C`1/2\*(C'\fR because that contains a 1 (as |
| 1008 |
well as a 2). As written, it matches things like subscripts that have |
| 1009 |
these numeric values. If we only wanted the decimal digits with those |
| 1010 |
numeric values, we could say, |
| 1011 |
.PP |
| 1012 |
.Vb 1 |
| 1013 |
\& qr! (?[ \ed & \ep{nv=/[0\-5]/ ]) }!x |
| 1014 |
.Ve |
| 1015 |
.PP |
| 1016 |
The \f(CW\*(C`\ed\*(C'\fR gets rid of needing to anchor the pattern, since it forces the |
| 1017 |
result to only match \f(CW\*(C`[0\-9]\*(C'\fR, and the \f(CW\*(C`[0\-5]\*(C'\fR further restricts it. |
| 1018 |
.PP |
| 1019 |
The text in the above examples enclosed between the \f(CW"/"\fR |
| 1020 |
characters can be just about any regular expression. It is independent |
| 1021 |
of the main pattern, so doesn't share any capturing groups, \fIetc\fR. The |
| 1022 |
delimiters for it must be ASCII punctuation, but it may NOT be |
| 1023 |
delimited by \f(CW"{"\fR, nor \f(CW"}"\fR nor contain a literal \f(CW"}"\fR, as that |
| 1024 |
delimits the end of the enclosing \f(CW\*(C`\ep{}\*(C'\fR. Like any pattern, certain |
| 1025 |
other delimiters are terminated by their mirror images. These are |
| 1026 |
\&\f(CW"("\fR, \f(CW\*(C`"[\*(C'\fR", and \f(CW"<"\fR. If the delimiter is any of \f(CW"\-"\fR, |
| 1027 |
\&\f(CW"_"\fR, \f(CW"+"\fR, or \f(CW"\e"\fR, or is the same delimiter as is used for the |
| 1028 |
enclosing pattern, it must be preceded by a backslash escape, both |
| 1029 |
fore and aft. |
| 1030 |
.PP |
| 1031 |
Beware of using \f(CW"$"\fR to indicate to match the end of the string. It |
| 1032 |
can too easily be interpreted as being a punctuation variable, like |
| 1033 |
\&\f(CW$/\fR. |
| 1034 |
.PP |
| 1035 |
No modifiers may follow the final delimiter. Instead, use |
| 1036 |
"(?adlupimnsx\-imnsx)" in perlre and/or |
| 1037 |
"(?adluimnsx\-imnsx:pattern)" in perlre to specify modifiers. |
| 1038 |
However, certain modifiers are illegal in your wildcard subpattern. |
| 1039 |
The only character set modifier specifiable is \f(CW\*(C`/aa\*(C'\fR; |
| 1040 |
any other character set, and \f(CW\*(C`\-m\*(C'\fR, and \f(CW\*(C`p\*(C'\fR, and \f(CW\*(C`s\*(C'\fR are all illegal. |
| 1041 |
Specifying modifiers like \f(CW\*(C`qr/.../gc\*(C'\fR that aren't legal in the |
| 1042 |
\&\f(CW\*(C`(?...)\*(C'\fR notation normally raise a warning, but with wildcard |
| 1043 |
subpatterns, their use is an error. The \f(CW\*(C`m\*(C'\fR modifier is ineffective; |
| 1044 |
everything that matches will be a single line. |
| 1045 |
.PP |
| 1046 |
By default, your pattern is matched case-insensitively, as if \f(CW\*(C`/i\*(C'\fR had |
| 1047 |
been specified. You can change this by saying \f(CW\*(C`(?\-i)\*(C'\fR in your pattern. |
| 1048 |
.PP |
| 1049 |
There are also certain operations that are illegal. You can't nest |
| 1050 |
\&\f(CW\*(C`\ep{...}\*(C'\fR and \f(CW\*(C`\eP{...}\*(C'\fR calls within a wildcard subpattern, and \f(CW\*(C`\eG\*(C'\fR |
| 1051 |
doesn't make sense, so is also prohibited. |
| 1052 |
.PP |
| 1053 |
And the \f(CW\*(C`*\*(C'\fR quantifier (or its equivalent \f(CW\*(C`(0,}\*(C'\fR) is illegal. |
| 1054 |
.PP |
| 1055 |
This feature is not available when the left-hand side is prefixed by |
| 1056 |
\&\f(CW\*(C`Is_\*(C'\fR, nor for any form that is marked as "Discouraged" in |
| 1057 |
"Discouraged" in perluniprops. |
| 1058 |
.PP |
| 1059 |
This experimental feature has been added to begin to implement |
| 1060 |
<https://www.unicode.org/reports/tr18/#Wildcard_Properties>. Using it |
| 1061 |
will raise a (default-on) warning in the |
| 1062 |
\&\f(CW\*(C`experimental::uniprop_wildcards\*(C'\fR category. We reserve the right to |
| 1063 |
change its operation as we gain experience. |
| 1064 |
.PP |
| 1065 |
Your subpattern can be just about anything, but for it to have some |
| 1066 |
utility, it should match when called with either or both of |
| 1067 |
a) the full name of the property value with underscores (and/or spaces |
| 1068 |
in the Block property) and some things uppercase; or b) the property |
| 1069 |
value in all lowercase with spaces and underscores squeezed out. For |
| 1070 |
example, |
| 1071 |
.PP |
| 1072 |
.Vb 2 |
| 1073 |
\& qr!\ep{Blk=/Old I.*/}! |
| 1074 |
\& qr!\ep{Blk=/oldi.*/}! |
| 1075 |
.Ve |
| 1076 |
.PP |
| 1077 |
would match the same things. |
| 1078 |
.PP |
| 1079 |
Another example that shows that within \f(CW\*(C`\ep{...}\*(C'\fR, \f(CW\*(C`/x\*(C'\fR isn't needed to |
| 1080 |
have spaces: |
| 1081 |
.PP |
| 1082 |
.Vb 1 |
| 1083 |
\& qr!\ep{scx= /Hebrew|Greek/ }! |
| 1084 |
.Ve |
| 1085 |
.PP |
| 1086 |
To be safe, we should have anchored the above example, to prevent |
| 1087 |
matches for something like \f(CW\*(C`Hebrew_Braille\*(C'\fR, but there aren't |
| 1088 |
any script names like that, so far. |
| 1089 |
A warning is issued if none of the legal values for a property are |
| 1090 |
matched by your pattern. It's likely that a future release will raise a |
| 1091 |
warning if your pattern ends up causing every possible code point to |
| 1092 |
match. |
| 1093 |
.PP |
| 1094 |
Starting in 5.32, the Name, Name Aliases, and Named Sequences properties |
| 1095 |
are allowed to be matched. They are considered to be a single |
| 1096 |
combination property, just as has long been the case for \f(CW\*(C`\eN{}\*(C'\fR. Loose |
| 1097 |
matching doesn't work in exactly the same way for these as it does for |
| 1098 |
the values of other properties. The rules are given in |
| 1099 |
<https://www.unicode.org/reports/tr44/tr44\-24.html#UAX44\-LM2>. As a |
| 1100 |
result, Perl doesn't try loose matching for you, like it does in other |
| 1101 |
properties. All letters in names are uppercase, but you can add \f(CW\*(C`(?i)\*(C'\fR |
| 1102 |
to your subpattern to ignore case. If you're uncertain where a blank |
| 1103 |
is, you can use \f(CW\*(C` ?\*(C'\fR in your subpattern. No character name contains an |
| 1104 |
underscore, so don't bother trying to match one. The use of hyphens is |
| 1105 |
particularly problematic; refer to the above link. But note that, as of |
| 1106 |
Unicode 13.0, the only script in modern usage which has weirdnesses with |
| 1107 |
these is Tibetan; also the two Korean characters U+116C HANGUL JUNGSEONG |
| 1108 |
OE and U+1180 HANGUL JUNGSEONG O\-E. Unicode makes no promises to not |
| 1109 |
add hyphen-problematic names in the future. |
| 1110 |
.PP |
| 1111 |
Using wildcards on these is resource intensive, given the hundreds of |
| 1112 |
thousands of legal names that must be checked against. |
| 1113 |
.PP |
| 1114 |
An example of using Name property wildcards is |
| 1115 |
.PP |
| 1116 |
.Vb 1 |
| 1117 |
\& qr!\ep{name=/(SMILING|GRINNING) FACE/}! |
| 1118 |
.Ve |
| 1119 |
.PP |
| 1120 |
Another is |
| 1121 |
.PP |
| 1122 |
.Vb 1 |
| 1123 |
\& qr/(?[ \ep{name=\e/CJK\e/} \- \ep{ideographic} ])/ |
| 1124 |
.Ve |
| 1125 |
.PP |
| 1126 |
which is the 200\-ish (as of Unicode 13.0) CJK characters that aren't |
| 1127 |
ideographs. |
| 1128 |
.PP |
| 1129 |
There are certain properties that wildcard subpatterns don't currently |
| 1130 |
work with. These are: |
| 1131 |
.PP |
| 1132 |
.Vb 9 |
| 1133 |
\& Bidi Mirroring Glyph |
| 1134 |
\& Bidi Paired Bracket |
| 1135 |
\& Case Folding |
| 1136 |
\& Decomposition Mapping |
| 1137 |
\& Equivalent Unified Ideograph |
| 1138 |
\& Lowercase Mapping |
| 1139 |
\& NFKC Case Fold |
| 1140 |
\& Titlecase Mapping |
| 1141 |
\& Uppercase Mapping |
| 1142 |
.Ve |
| 1143 |
.PP |
| 1144 |
Nor is the \f(CW\*(C`@\fR\f(CIunicode_property\fR\f(CW@\*(C'\fR form implemented. |
| 1145 |
.PP |
| 1146 |
Here's a complete example of matching IPV4 internet protocol addresses |
| 1147 |
in any (single) script |
| 1148 |
.PP |
| 1149 |
.Vb 1 |
| 1150 |
\& no warnings \*(Aqexperimental::uniprop_wildcards\*(Aq; |
| 1151 |
\& |
| 1152 |
\& # Can match a substring, so this intermediate regex needs to have |
| 1153 |
\& # context or anchoring in its final use. Using nt=de yields decimal |
| 1154 |
\& # digits. When specifying a subset of these, we must include \ed to |
| 1155 |
\& # prevent things like U+00B2 SUPERSCRIPT TWO from matching |
| 1156 |
\& my $zero_through_255 = |
| 1157 |
\& qr/ \eb (*sr: # All from same sript |
| 1158 |
\& (?[ \ep{nv=0} & \ed ])* # Optional leading zeros |
| 1159 |
\& ( # Then one of: |
| 1160 |
\& \ed{1,2} # 0 \- 99 |
| 1161 |
\& | (?[ \ep{nv=1} & \ed ]) \ed{2} # 100 \- 199 |
| 1162 |
\& | (?[ \ep{nv=2} & \ed ]) |
| 1163 |
\& ( (?[ \ep{nv=:[0\-4]:} & \ed ]) \ed # 200 \- 249 |
| 1164 |
\& | (?[ \ep{nv=5} & \ed ]) |
| 1165 |
\& (?[ \ep{nv=:[0\-5]:} & \ed ]) # 250 \- 255 |
| 1166 |
\& ) |
| 1167 |
\& ) |
| 1168 |
\& ) |
| 1169 |
\& \eb |
| 1170 |
\& /x; |
| 1171 |
\& |
| 1172 |
\& my $ipv4 = qr/ \eA (*sr: $zero_through_255 |
| 1173 |
\& (?: [.] $zero_through_255 ) {3} |
| 1174 |
\& ) |
| 1175 |
\& \ez |
| 1176 |
\& /x; |
| 1177 |
.Ve |
| 1178 |
.SS "User-Defined Character Properties" |
| 1179 |
.IX Subsection "User-Defined Character Properties" |
| 1180 |
You can define your own binary character properties by defining subroutines |
| 1181 |
whose names begin with \f(CW"In"\fR or \f(CW"Is"\fR. (The regex sets feature |
| 1182 |
"(?[ ])" in perlre provides an alternative which allows more complex |
| 1183 |
definitions.) The subroutines can be defined in any |
| 1184 |
package. They override any Unicode properties expressed as the same |
| 1185 |
names. The user-defined properties can be used in the regular |
| 1186 |
expression |
| 1187 |
\&\f(CW\*(C`\ep{}\*(C'\fR and \f(CW\*(C`\eP{}\*(C'\fR constructs; if you are using a user-defined property from a |
| 1188 |
package other than the one you are in, you must specify its package in the |
| 1189 |
\&\f(CW\*(C`\ep{}\*(C'\fR or \f(CW\*(C`\eP{}\*(C'\fR construct. |
| 1190 |
.PP |
| 1191 |
.Vb 3 |
| 1192 |
\& # assuming property IsForeign defined in Lang:: |
| 1193 |
\& package main; # property package name required |
| 1194 |
\& if ($txt =~ /\ep{Lang::IsForeign}+/) { ... } |
| 1195 |
\& |
| 1196 |
\& package Lang; # property package name not required |
| 1197 |
\& if ($txt =~ /\ep{IsForeign}+/) { ... } |
| 1198 |
.Ve |
| 1199 |
.PP |
| 1200 |
The subroutines are passed a single parameter, which is 0 if |
| 1201 |
case-sensitive matching is in effect and non-zero if caseless matching |
| 1202 |
is in effect. The subroutine may return different values depending on |
| 1203 |
the value of the flag. But the subroutine is never called more than |
| 1204 |
once for each flag value (zero vs non-zero). The return value is saved |
| 1205 |
and used instead of calling the sub ever again. If the sub is defined |
| 1206 |
at the time the pattern is compiled, it will be called then; if not, it |
| 1207 |
will be called the first time its value (for that flag) is needed during |
| 1208 |
execution. |
| 1209 |
.PP |
| 1210 |
Note that if the regular expression is tainted, then Perl will die rather |
| 1211 |
than calling the subroutine when the name of the subroutine is |
| 1212 |
determined by the tainted data. |
| 1213 |
.PP |
| 1214 |
The subroutines must return a specially-formatted string, with one |
| 1215 |
or more newline-separated lines. Each line must be one of the following: |
| 1216 |
.IP \(bu 4 |
| 1217 |
A single hexadecimal number denoting a code point to include. |
| 1218 |
.IP \(bu 4 |
| 1219 |
Two hexadecimal numbers separated by horizontal whitespace (space or |
| 1220 |
tabular characters) denoting a range of code points to include. The |
| 1221 |
second number must not be smaller than the first. |
| 1222 |
.IP \(bu 4 |
| 1223 |
Something to include, prefixed by \f(CW"+"\fR: a built-in character |
| 1224 |
property (prefixed by \f(CW"utf8::"\fR) or a fully qualified (including package |
| 1225 |
name) user-defined character property, |
| 1226 |
to represent all the characters in that property; two hexadecimal code |
| 1227 |
points for a range; or a single hexadecimal code point. |
| 1228 |
.IP \(bu 4 |
| 1229 |
Something to exclude, prefixed by \f(CW"\-"\fR: an existing character |
| 1230 |
property (prefixed by \f(CW"utf8::"\fR) or a fully qualified (including package |
| 1231 |
name) user-defined character property, |
| 1232 |
to represent all the characters in that property; two hexadecimal code |
| 1233 |
points for a range; or a single hexadecimal code point. |
| 1234 |
.IP \(bu 4 |
| 1235 |
Something to negate, prefixed \f(CW"!"\fR: an existing character |
| 1236 |
property (prefixed by \f(CW"utf8::"\fR) or a fully qualified (including package |
| 1237 |
name) user-defined character property, |
| 1238 |
to represent all the characters in that property; two hexadecimal code |
| 1239 |
points for a range; or a single hexadecimal code point. |
| 1240 |
.IP \(bu 4 |
| 1241 |
Something to intersect with, prefixed by \f(CW"&"\fR: an existing character |
| 1242 |
property (prefixed by \f(CW"utf8::"\fR) or a fully qualified (including package |
| 1243 |
name) user-defined character property, |
| 1244 |
for all the characters except the characters in the property; two |
| 1245 |
hexadecimal code points for a range; or a single hexadecimal code point. |
| 1246 |
.PP |
| 1247 |
For example, to define a property that covers both the Japanese |
| 1248 |
syllabaries (hiragana and katakana), you can define |
| 1249 |
.PP |
| 1250 |
.Vb 6 |
| 1251 |
\& sub InKana { |
| 1252 |
\& return <<END; |
| 1253 |
\& 3040\et309F |
| 1254 |
\& 30A0\et30FF |
| 1255 |
\& END |
| 1256 |
\& } |
| 1257 |
.Ve |
| 1258 |
.PP |
| 1259 |
Imagine that the here-doc end marker is at the beginning of the line. |
| 1260 |
Now you can use \f(CW\*(C`\ep{InKana}\*(C'\fR and \f(CW\*(C`\eP{InKana}\*(C'\fR. |
| 1261 |
.PP |
| 1262 |
You could also have used the existing block property names: |
| 1263 |
.PP |
| 1264 |
.Vb 6 |
| 1265 |
\& sub InKana { |
| 1266 |
\& return <<\*(AqEND\*(Aq; |
| 1267 |
\& +utf8::InHiragana |
| 1268 |
\& +utf8::InKatakana |
| 1269 |
\& END |
| 1270 |
\& } |
| 1271 |
.Ve |
| 1272 |
.PP |
| 1273 |
Suppose you wanted to match only the allocated characters, |
| 1274 |
not the raw block ranges: in other words, you want to remove |
| 1275 |
the unassigned characters: |
| 1276 |
.PP |
| 1277 |
.Vb 7 |
| 1278 |
\& sub InKana { |
| 1279 |
\& return <<\*(AqEND\*(Aq; |
| 1280 |
\& +utf8::InHiragana |
| 1281 |
\& +utf8::InKatakana |
| 1282 |
\& \-utf8::IsCn |
| 1283 |
\& END |
| 1284 |
\& } |
| 1285 |
.Ve |
| 1286 |
.PP |
| 1287 |
The negation is useful for defining (surprise!) negated classes. |
| 1288 |
.PP |
| 1289 |
.Vb 7 |
| 1290 |
\& sub InNotKana { |
| 1291 |
\& return <<\*(AqEND\*(Aq; |
| 1292 |
\& !utf8::InHiragana |
| 1293 |
\& \-utf8::InKatakana |
| 1294 |
\& +utf8::IsCn |
| 1295 |
\& END |
| 1296 |
\& } |
| 1297 |
.Ve |
| 1298 |
.PP |
| 1299 |
This will match all non-Unicode code points, since every one of them is |
| 1300 |
not in Kana. You can use intersection to exclude these, if desired, as |
| 1301 |
this modified example shows: |
| 1302 |
.PP |
| 1303 |
.Vb 8 |
| 1304 |
\& sub InNotKana { |
| 1305 |
\& return <<\*(AqEND\*(Aq; |
| 1306 |
\& !utf8::InHiragana |
| 1307 |
\& \-utf8::InKatakana |
| 1308 |
\& +utf8::IsCn |
| 1309 |
\& &utf8::Any |
| 1310 |
\& END |
| 1311 |
\& } |
| 1312 |
.Ve |
| 1313 |
.PP |
| 1314 |
\&\f(CW&utf8::Any\fR must be the last line in the definition. |
| 1315 |
.PP |
| 1316 |
Intersection is used generally for getting the common characters matched |
| 1317 |
by two (or more) classes. It's important to remember not to use \f(CW"&"\fR for |
| 1318 |
the first set; that would be intersecting with nothing, resulting in an |
| 1319 |
empty set. (Similarly using \f(CW"\-"\fR for the first set does nothing). |
| 1320 |
.PP |
| 1321 |
Unlike non-user-defined \f(CW\*(C`\ep{}\*(C'\fR property matches, no warning is ever |
| 1322 |
generated if these properties are matched against a non-Unicode code |
| 1323 |
point (see "Beyond Unicode code points" below). |
| 1324 |
.SS "User-Defined Case Mappings (for serious hackers only)" |
| 1325 |
.IX Subsection "User-Defined Case Mappings (for serious hackers only)" |
| 1326 |
\&\fBThis feature has been removed as of Perl 5.16.\fR |
| 1327 |
The CPAN module \f(CW\*(C`Unicode::Casing\*(C'\fR provides better functionality without |
| 1328 |
the drawbacks that this feature had. If you are using a Perl earlier |
| 1329 |
than 5.16, this feature was most fully documented in the 5.14 version of |
| 1330 |
this pod: |
| 1331 |
<http://perldoc.perl.org/5.14.0/perlunicode.html#User\-Defined\-Case\-Mappings\-%28for\-serious\-hackers\-only%29> |
| 1332 |
.SS "Character Encodings for Input and Output" |
| 1333 |
.IX Subsection "Character Encodings for Input and Output" |
| 1334 |
See Encode. |
| 1335 |
.SS "Unicode Regular Expression Support Level" |
| 1336 |
.IX Subsection "Unicode Regular Expression Support Level" |
| 1337 |
The following list of Unicode supported features for regular expressions describes |
| 1338 |
all features currently directly supported by core Perl. The references |
| 1339 |
to "Level \fIN\fR" and the section numbers refer to |
| 1340 |
UTS#18 "Unicode Regular Expressions" <https://www.unicode.org/reports/tr18>, |
| 1341 |
version 18, October 2016. |
| 1342 |
.PP |
| 1343 |
\fILevel 1 \- Basic Unicode Support\fR |
| 1344 |
.IX Subsection "Level 1 - Basic Unicode Support" |
| 1345 |
.PP |
| 1346 |
.Vb 8 |
| 1347 |
\& RL1.1 Hex Notation \- Done [1] |
| 1348 |
\& RL1.2 Properties \- Done [2] |
| 1349 |
\& RL1.2a Compatibility Properties \- Done [3] |
| 1350 |
\& RL1.3 Subtraction and Intersection \- Done [4] |
| 1351 |
\& RL1.4 Simple Word Boundaries \- Done [5] |
| 1352 |
\& RL1.5 Simple Loose Matches \- Done [6] |
| 1353 |
\& RL1.6 Line Boundaries \- Partial [7] |
| 1354 |
\& RL1.7 Supplementary Code Points \- Done [8] |
| 1355 |
.Ve |
| 1356 |
.ie n .IP "[1] ""\eN{U+...}"" and ""\ex{...}""" 4 |
| 1357 |
.el .IP "[1] \f(CW\eN{U+...}\fR and \f(CW\ex{...}\fR" 4 |
| 1358 |
.IX Item "[1] N{U+...} and x{...}" |
| 1359 |
.PD 0 |
| 1360 |
.ie n .IP "[2] ""\ep{...}"" ""\eP{...}"". This requirement is for a minimal list of properties. Perl supports these. See R2.7 for other properties." 4 |
| 1361 |
.el .IP "[2] \f(CW\ep{...}\fR \f(CW\eP{...}\fR. This requirement is for a minimal list of properties. Perl supports these. See R2.7 for other properties." 4 |
| 1362 |
.IX Item "[2] p{...} P{...}. This requirement is for a minimal list of properties. Perl supports these. See R2.7 for other properties." |
| 1363 |
.IP [3] 4 |
| 1364 |
.IX Item "[3]" |
| 1365 |
.PD |
| 1366 |
Perl has \f(CW\*(C`\ed\*(C'\fR \f(CW\*(C`\eD\*(C'\fR \f(CW\*(C`\es\*(C'\fR \f(CW\*(C`\eS\*(C'\fR \f(CW\*(C`\ew\*(C'\fR \f(CW\*(C`\eW\*(C'\fR \f(CW\*(C`\eX\*(C'\fR \f(CW\*(C`[:\fR\f(CIprop\fR\f(CW:]\*(C'\fR |
| 1367 |
\&\f(CW\*(C`[:^\fR\f(CIprop\fR\f(CW:]\*(C'\fR, plus all the properties specified by |
| 1368 |
<https://www.unicode.org/reports/tr18/#Compatibility_Properties>. These |
| 1369 |
are described above in "Other Properties" |
| 1370 |
.IP [4] 4 |
| 1371 |
.IX Item "[4]" |
| 1372 |
The regex sets feature \f(CW"(?[...])"\fR starting in v5.18 accomplishes |
| 1373 |
this. See "(?[ ])" in perlre. |
| 1374 |
.ie n .IP "[5] ""\eb"" ""\eB"" meet most, but not all, the details of this requirement, but ""\eb{wb}"" and ""\eB{wb}"" do, as well as the stricter R2.3." 4 |
| 1375 |
.el .IP "[5] \f(CW\eb\fR \f(CW\eB\fR meet most, but not all, the details of this requirement, but \f(CW\eb{wb}\fR and \f(CW\eB{wb}\fR do, as well as the stricter R2.3." 4 |
| 1376 |
.IX Item "[5] b B meet most, but not all, the details of this requirement, but b{wb} and B{wb} do, as well as the stricter R2.3." |
| 1377 |
.PD 0 |
| 1378 |
.IP [6] 4 |
| 1379 |
.IX Item "[6]" |
| 1380 |
.PD |
| 1381 |
Note that Perl does Full case-folding in matching, not Simple: |
| 1382 |
.Sp |
| 1383 |
For example \f(CW\*(C`U+1F88\*(C'\fR is equivalent to \f(CW\*(C`U+1F00 U+03B9\*(C'\fR, instead of just |
| 1384 |
\&\f(CW\*(C`U+1F80\*(C'\fR. This difference matters mainly for certain Greek capital |
| 1385 |
letters with certain modifiers: the Full case-folding decomposes the |
| 1386 |
letter, while the Simple case-folding would map it to a single |
| 1387 |
character. |
| 1388 |
.IP [7] 4 |
| 1389 |
.IX Item "[7]" |
| 1390 |
The reason this is considered to be only partially implemented is that |
| 1391 |
Perl has \f(CW\*(C`qr/\eb{lb}/\*(C'\fR and |
| 1392 |
\&\f(CW\*(C`Unicode::LineBreak\*(C'\fR that are conformant with |
| 1393 |
UAX#14 "Unicode Line Breaking Algorithm" <https://www.unicode.org/reports/tr14>. |
| 1394 |
The regular expression construct provides default behavior, while the |
| 1395 |
heavier-weight module provides customizable line breaking. |
| 1396 |
.Sp |
| 1397 |
But Perl treats \f(CW\*(C`\en\*(C'\fR as the start\- and end-line |
| 1398 |
delimiter, whereas Unicode specifies more characters that should be |
| 1399 |
so-interpreted. |
| 1400 |
.Sp |
| 1401 |
These are: |
| 1402 |
.Sp |
| 1403 |
.Vb 6 |
| 1404 |
\& VT U+000B (\ev in C) |
| 1405 |
\& FF U+000C (\ef) |
| 1406 |
\& CR U+000D (\er) |
| 1407 |
\& NEL U+0085 |
| 1408 |
\& LS U+2028 |
| 1409 |
\& PS U+2029 |
| 1410 |
.Ve |
| 1411 |
.Sp |
| 1412 |
\&\f(CW\*(C`^\*(C'\fR and \f(CW\*(C`$\*(C'\fR in regular expression patterns are supposed to match all |
| 1413 |
these, but don't. |
| 1414 |
These characters also don't, but should, affect \f(CW\*(C`<>\*(C'\fR \f(CW$.\fR, and |
| 1415 |
script line numbers. |
| 1416 |
.Sp |
| 1417 |
Also, lines should not be split within \f(CW\*(C`CRLF\*(C'\fR (i.e. there is no |
| 1418 |
empty line between \f(CW\*(C`\er\*(C'\fR and \f(CW\*(C`\en\*(C'\fR). For \f(CW\*(C`CRLF\*(C'\fR, try the \f(CW\*(C`:crlf\*(C'\fR |
| 1419 |
layer (see PerlIO). |
| 1420 |
.ie n .IP "[8] UTF\-8/UTF\-EBDDIC used in Perl allows not only ""U+10000"" to ""U+10FFFF"" but also beyond ""U+10FFFF""" 4 |
| 1421 |
.el .IP "[8] UTF\-8/UTF\-EBDDIC used in Perl allows not only \f(CWU+10000\fR to \f(CWU+10FFFF\fR but also beyond \f(CWU+10FFFF\fR" 4 |
| 1422 |
.IX Item "[8] UTF-8/UTF-EBDDIC used in Perl allows not only U+10000 to U+10FFFF but also beyond U+10FFFF" |
| 1423 |
.PP |
| 1424 |
\fILevel 2 \- Extended Unicode Support\fR |
| 1425 |
.IX Subsection "Level 2 - Extended Unicode Support" |
| 1426 |
.PP |
| 1427 |
.Vb 10 |
| 1428 |
\& RL2.1 Canonical Equivalents \- Retracted [9] |
| 1429 |
\& by Unicode |
| 1430 |
\& RL2.2 Extended Grapheme Clusters and \- Partial [10] |
| 1431 |
\& Character Classes with Strings |
| 1432 |
\& RL2.3 Default Word Boundaries \- Done [11] |
| 1433 |
\& RL2.4 Default Case Conversion \- Done |
| 1434 |
\& RL2.5 Name Properties \- Done |
| 1435 |
\& RL2.6 Wildcards in Property Values \- Partial [12] |
| 1436 |
\& RL2.7 Full Properties \- Partial [13] |
| 1437 |
\& RL2.8 Optional Properties \- Partial [14] |
| 1438 |
.Ve |
| 1439 |
.IP "[9] Unicode has rewritten this portion of UTS#18 to say that getting canonical equivalence (see UAX#15 ""Unicode Normalization Forms"" <https://www.unicode.org/reports/tr15>) is basically to be done at the programmer level. Use NFD to write both your regular expressions and text to match them against (you can use Unicode::Normalize)." 4 |
| 1440 |
.IX Item "[9] Unicode has rewritten this portion of UTS#18 to say that getting canonical equivalence (see UAX#15 ""Unicode Normalization Forms"" <https://www.unicode.org/reports/tr15>) is basically to be done at the programmer level. Use NFD to write both your regular expressions and text to match them against (you can use Unicode::Normalize)." |
| 1441 |
.PD 0 |
| 1442 |
.ie n .IP "[10] Perl has ""\eX"" and ""\eb{gcb}"". Unicode has retracted their ""Grapheme Cluster Mode"", and recently added string properties, which Perl does not yet support." 4 |
| 1443 |
.el .IP "[10] Perl has \f(CW\eX\fR and \f(CW\eb{gcb}\fR. Unicode has retracted their ""Grapheme Cluster Mode"", and recently added string properties, which Perl does not yet support." 4 |
| 1444 |
.IX Item "[10] Perl has X and b{gcb}. Unicode has retracted their ""Grapheme Cluster Mode"", and recently added string properties, which Perl does not yet support." |
| 1445 |
.IP "[11] see UAX#29 ""Unicode Text Segmentation"" <https://www.unicode.org/reports/tr29>," 4 |
| 1446 |
.IX Item "[11] see UAX#29 ""Unicode Text Segmentation"" <https://www.unicode.org/reports/tr29>," |
| 1447 |
.IP "[12] see ""Wildcards in Property Values"" above." 4 |
| 1448 |
.IX Item "[12] see ""Wildcards in Property Values"" above." |
| 1449 |
.IP "[13] Perl supports all the properties in the Unicode Character Database (UCD). It does not yet support the listed properties that come from other Unicode sources." 4 |
| 1450 |
.IX Item "[13] Perl supports all the properties in the Unicode Character Database (UCD). It does not yet support the listed properties that come from other Unicode sources." |
| 1451 |
.IP "[14] The only optional property that Perl supports is Named Sequence. None of these properties are in the UCD." 4 |
| 1452 |
.IX Item "[14] The only optional property that Perl supports is Named Sequence. None of these properties are in the UCD." |
| 1453 |
.PD |
| 1454 |
.PP |
| 1455 |
\fILevel 3 \- Tailored Support\fR |
| 1456 |
.IX Subsection "Level 3 - Tailored Support" |
| 1457 |
.PP |
| 1458 |
This has been retracted by Unicode. |
| 1459 |
.SS "Unicode Encodings" |
| 1460 |
.IX Subsection "Unicode Encodings" |
| 1461 |
Unicode characters are assigned to \fIcode points\fR, which are abstract |
| 1462 |
numbers. To use these numbers, various encodings are needed. |
| 1463 |
.IP \(bu 4 |
| 1464 |
UTF\-8 |
| 1465 |
.Sp |
| 1466 |
UTF\-8 is a variable-length (1 to 4 bytes), byte-order independent |
| 1467 |
encoding. In most of Perl's documentation, including elsewhere in this |
| 1468 |
document, the term "UTF\-8" means also "UTF-EBCDIC". But in this section, |
| 1469 |
"UTF\-8" refers only to the encoding used on ASCII platforms. It is a |
| 1470 |
superset of 7\-bit US-ASCII, so anything encoded in ASCII has the |
| 1471 |
identical representation when encoded in UTF\-8. |
| 1472 |
.Sp |
| 1473 |
The following table is from Unicode 3.2. |
| 1474 |
.Sp |
| 1475 |
.Vb 1 |
| 1476 |
\& Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte |
| 1477 |
\& |
| 1478 |
\& U+0000..U+007F 00..7F |
| 1479 |
\& U+0080..U+07FF * C2..DF 80..BF |
| 1480 |
\& U+0800..U+0FFF E0 * A0..BF 80..BF |
| 1481 |
\& U+1000..U+CFFF E1..EC 80..BF 80..BF |
| 1482 |
\& U+D000..U+D7FF ED 80..9F 80..BF |
| 1483 |
\& U+D800..U+DFFF +++++ utf16 surrogates, not legal utf8 +++++ |
| 1484 |
\& U+E000..U+FFFF EE..EF 80..BF 80..BF |
| 1485 |
\& U+10000..U+3FFFF F0 * 90..BF 80..BF 80..BF |
| 1486 |
\& U+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF |
| 1487 |
\& U+100000..U+10FFFF F4 80..8F 80..BF 80..BF |
| 1488 |
.Ve |
| 1489 |
.Sp |
| 1490 |
Note the gaps marked by "*" before several of the byte entries above. These are |
| 1491 |
caused by legal UTF\-8 avoiding non-shortest encodings: it is technically |
| 1492 |
possible to UTF\-8\-encode a single code point in different ways, but that is |
| 1493 |
explicitly forbidden, and the shortest possible encoding should always be used |
| 1494 |
(and that is what Perl does). |
| 1495 |
.Sp |
| 1496 |
Another way to look at it is via bits: |
| 1497 |
.Sp |
| 1498 |
.Vb 1 |
| 1499 |
\& Code Points 1st Byte 2nd Byte 3rd Byte 4th Byte |
| 1500 |
\& |
| 1501 |
\& 0aaaaaaa 0aaaaaaa |
| 1502 |
\& 00000bbbbbaaaaaa 110bbbbb 10aaaaaa |
| 1503 |
\& ccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa |
| 1504 |
\& 00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa |
| 1505 |
.Ve |
| 1506 |
.Sp |
| 1507 |
As you can see, the continuation bytes all begin with \f(CW"10"\fR, and the |
| 1508 |
leading bits of the start byte tell how many bytes there are in the |
| 1509 |
encoded character. |
| 1510 |
.Sp |
| 1511 |
The original UTF\-8 specification allowed up to 6 bytes, to allow |
| 1512 |
encoding of numbers up to \f(CW\*(C`0x7FFF_FFFF\*(C'\fR. Perl continues to allow those, |
| 1513 |
and has extended that up to 13 bytes to encode code points up to what |
| 1514 |
can fit in a 64\-bit word. However, Perl will warn if you output any of |
| 1515 |
these as being non-portable; and under strict UTF\-8 input protocols, |
| 1516 |
they are forbidden. In addition, it is now illegal to use a code point |
| 1517 |
larger than what a signed integer variable on your system can hold. On |
| 1518 |
32\-bit ASCII systems, this means \f(CW\*(C`0x7FFF_FFFF\*(C'\fR is the legal maximum |
| 1519 |
(much higher on 64\-bit systems). |
| 1520 |
.IP \(bu 4 |
| 1521 |
UTF-EBCDIC |
| 1522 |
.Sp |
| 1523 |
Like UTF\-8, but EBCDIC-safe, in the way that UTF\-8 is ASCII-safe. |
| 1524 |
This means that all the basic characters (which includes all |
| 1525 |
those that have ASCII equivalents (like \f(CW"A"\fR, \f(CW"0"\fR, \f(CW"%"\fR, \fIetc.\fR) |
| 1526 |
are the same in both EBCDIC and UTF-EBCDIC.) |
| 1527 |
.Sp |
| 1528 |
UTF-EBCDIC is used on EBCDIC platforms. It generally requires more |
| 1529 |
bytes to represent a given code point than UTF\-8 does; the largest |
| 1530 |
Unicode code points take 5 bytes to represent (instead of 4 in UTF\-8), |
| 1531 |
and, extended for 64\-bit words, it uses 14 bytes instead of 13 bytes in |
| 1532 |
UTF\-8. |
| 1533 |
.IP \(bu 4 |
| 1534 |
UTF\-16, UTF\-16BE, UTF\-16LE, Surrogates, and \f(CW\*(C`BOM\*(C'\fR's (Byte Order Marks) |
| 1535 |
.Sp |
| 1536 |
The followings items are mostly for reference and general Unicode |
| 1537 |
knowledge, Perl doesn't use these constructs internally. |
| 1538 |
.Sp |
| 1539 |
Like UTF\-8, UTF\-16 is a variable-width encoding, but where |
| 1540 |
UTF\-8 uses 8\-bit code units, UTF\-16 uses 16\-bit code units. |
| 1541 |
All code points occupy either 2 or 4 bytes in UTF\-16: code points |
| 1542 |
\&\f(CW\*(C`U+0000..U+FFFF\*(C'\fR are stored in a single 16\-bit unit, and code |
| 1543 |
points \f(CW\*(C`U+10000..U+10FFFF\*(C'\fR in two 16\-bit units. The latter case is |
| 1544 |
using \fIsurrogates\fR, the first 16\-bit unit being the \fIhigh |
| 1545 |
surrogate\fR, and the second being the \fIlow surrogate\fR. |
| 1546 |
.Sp |
| 1547 |
Surrogates are code points set aside to encode the \f(CW\*(C`U+10000..U+10FFFF\*(C'\fR |
| 1548 |
range of Unicode code points in pairs of 16\-bit units. The \fIhigh |
| 1549 |
surrogates\fR are the range \f(CW\*(C`U+D800..U+DBFF\*(C'\fR and the \fIlow surrogates\fR |
| 1550 |
are the range \f(CW\*(C`U+DC00..U+DFFF\*(C'\fR. The surrogate encoding is |
| 1551 |
.Sp |
| 1552 |
.Vb 2 |
| 1553 |
\& $hi = ($uni \- 0x10000) / 0x400 + 0xD800; |
| 1554 |
\& $lo = ($uni \- 0x10000) % 0x400 + 0xDC00; |
| 1555 |
.Ve |
| 1556 |
.Sp |
| 1557 |
and the decoding is |
| 1558 |
.Sp |
| 1559 |
.Vb 1 |
| 1560 |
\& $uni = 0x10000 + ($hi \- 0xD800) * 0x400 + ($lo \- 0xDC00); |
| 1561 |
.Ve |
| 1562 |
.Sp |
| 1563 |
Because of the 16\-bitness, UTF\-16 is byte-order dependent. UTF\-16 |
| 1564 |
itself can be used for in-memory computations, but if storage or |
| 1565 |
transfer is required either UTF\-16BE (big-endian) or UTF\-16LE |
| 1566 |
(little-endian) encodings must be chosen. |
| 1567 |
.Sp |
| 1568 |
This introduces another problem: what if you just know that your data |
| 1569 |
is UTF\-16, but you don't know which endianness? Byte Order Marks, or |
| 1570 |
\&\f(CW\*(C`BOM\*(C'\fR's, are a solution to this. A special character has been reserved |
| 1571 |
in Unicode to function as a byte order marker: the character with the |
| 1572 |
code point \f(CW\*(C`U+FEFF\*(C'\fR is the \f(CW\*(C`BOM\*(C'\fR. |
| 1573 |
.Sp |
| 1574 |
The trick is that if you read a \f(CW\*(C`BOM\*(C'\fR, you will know the byte order, |
| 1575 |
since if it was written on a big-endian platform, you will read the |
| 1576 |
bytes \f(CW\*(C`0xFE 0xFF\*(C'\fR, but if it was written on a little-endian platform, |
| 1577 |
you will read the bytes \f(CW\*(C`0xFF 0xFE\*(C'\fR. (And if the originating platform |
| 1578 |
was writing in ASCII platform UTF\-8, you will read the bytes |
| 1579 |
\&\f(CW\*(C`0xEF 0xBB 0xBF\*(C'\fR.) |
| 1580 |
.Sp |
| 1581 |
The way this trick works is that the character with the code point |
| 1582 |
\&\f(CW\*(C`U+FFFE\*(C'\fR is not supposed to be in input streams, so the |
| 1583 |
sequence of bytes \f(CW\*(C`0xFF 0xFE\*(C'\fR is unambiguously "\f(CW\*(C`BOM\*(C'\fR, represented in |
| 1584 |
little-endian format" and cannot be \f(CW\*(C`U+FFFE\*(C'\fR, represented in big-endian |
| 1585 |
format". |
| 1586 |
.Sp |
| 1587 |
Surrogates have no meaning in Unicode outside their use in pairs to |
| 1588 |
represent other code points. However, Perl allows them to be |
| 1589 |
represented individually internally, for example by saying |
| 1590 |
\&\f(CWchr(0xD801)\fR, so that all code points, not just those valid for open |
| 1591 |
interchange, are |
| 1592 |
representable. Unicode does define semantics for them, such as their |
| 1593 |
\&\f(CW"General_Category"\fR is \f(CW"Cs"\fR. But because their use is somewhat dangerous, |
| 1594 |
Perl will warn (using the warning category \f(CW"surrogate"\fR, which is a |
| 1595 |
sub-category of \f(CW"utf8"\fR) if an attempt is made |
| 1596 |
to do things like take the lower case of one, or match |
| 1597 |
case-insensitively, or to output them. (But don't try this on Perls |
| 1598 |
before 5.14.) |
| 1599 |
.IP \(bu 4 |
| 1600 |
UTF\-32, UTF\-32BE, UTF\-32LE |
| 1601 |
.Sp |
| 1602 |
The UTF\-32 family is pretty much like the UTF\-16 family, except that |
| 1603 |
the units are 32\-bit, and therefore the surrogate scheme is not |
| 1604 |
needed. UTF\-32 is a fixed-width encoding. The \f(CW\*(C`BOM\*(C'\fR signatures are |
| 1605 |
\&\f(CW\*(C`0x00 0x00 0xFE 0xFF\*(C'\fR for BE and \f(CW\*(C`0xFF 0xFE 0x00 0x00\*(C'\fR for LE. |
| 1606 |
.IP \(bu 4 |
| 1607 |
UCS\-2, UCS\-4 |
| 1608 |
.Sp |
| 1609 |
Legacy, fixed-width encodings defined by the ISO 10646 standard. UCS\-2 is a 16\-bit |
| 1610 |
encoding. Unlike UTF\-16, UCS\-2 is not extensible beyond \f(CW\*(C`U+FFFF\*(C'\fR, |
| 1611 |
because it does not use surrogates. UCS\-4 is a 32\-bit encoding, |
| 1612 |
functionally identical to UTF\-32 (the difference being that |
| 1613 |
UCS\-4 forbids neither surrogates nor code points larger than \f(CW\*(C`0x10_FFFF\*(C'\fR). |
| 1614 |
.IP \(bu 4 |
| 1615 |
UTF\-7 |
| 1616 |
.Sp |
| 1617 |
A seven-bit safe (non-eight-bit) encoding, which is useful if the |
| 1618 |
transport or storage is not eight-bit safe. Defined by RFC 2152. |
| 1619 |
.SS "Noncharacter code points" |
| 1620 |
.IX Subsection "Noncharacter code points" |
| 1621 |
66 code points are set aside in Unicode as "noncharacter code points". |
| 1622 |
These all have the \f(CW\*(C`Unassigned\*(C'\fR (\f(CW\*(C`Cn\*(C'\fR) \f(CW"General_Category"\fR, and |
| 1623 |
no character will ever be assigned to any of them. They are the 32 code |
| 1624 |
points between \f(CW\*(C`U+FDD0\*(C'\fR and \f(CW\*(C`U+FDEF\*(C'\fR inclusive, and the 34 code |
| 1625 |
points: |
| 1626 |
.PP |
| 1627 |
.Vb 7 |
| 1628 |
\& U+FFFE U+FFFF |
| 1629 |
\& U+1FFFE U+1FFFF |
| 1630 |
\& U+2FFFE U+2FFFF |
| 1631 |
\& ... |
| 1632 |
\& U+EFFFE U+EFFFF |
| 1633 |
\& U+FFFFE U+FFFFF |
| 1634 |
\& U+10FFFE U+10FFFF |
| 1635 |
.Ve |
| 1636 |
.PP |
| 1637 |
Until Unicode 7.0, the noncharacters were "\fBforbidden\fR for use in open |
| 1638 |
interchange of Unicode text data", so that code that processed those |
| 1639 |
streams could use these code points as sentinels that could be mixed in |
| 1640 |
with character data, and would always be distinguishable from that data. |
| 1641 |
(Emphasis above and in the next paragraph are added in this document.) |
| 1642 |
.PP |
| 1643 |
Unicode 7.0 changed the wording so that they are "\fBnot recommended\fR for |
| 1644 |
use in open interchange of Unicode text data". The 7.0 Standard goes on |
| 1645 |
to say: |
| 1646 |
.Sp |
| 1647 |
.RS 4 |
| 1648 |
"If a noncharacter is received in open interchange, an application is |
| 1649 |
not required to interpret it in any way. It is good practice, however, |
| 1650 |
to recognize it as a noncharacter and to take appropriate action, such |
| 1651 |
as replacing it with \f(CW\*(C`U+FFFD\*(C'\fR replacement character, to indicate the |
| 1652 |
problem in the text. It is not recommended to simply delete |
| 1653 |
noncharacter code points from such text, because of the potential |
| 1654 |
security issues caused by deleting uninterpreted characters. (See |
| 1655 |
conformance clause C7 in Section 3.2, Conformance Requirements, and |
| 1656 |
Unicode Technical Report #36, "Unicode Security |
| 1657 |
Considerations" <https://www.unicode.org/reports/tr36/#Substituting_for_Ill_Formed_Subsequences>)." |
| 1658 |
.RE |
| 1659 |
.PP |
| 1660 |
This change was made because it was found that various commercial tools |
| 1661 |
like editors, or for things like source code control, had been written |
| 1662 |
so that they would not handle program files that used these code points, |
| 1663 |
effectively precluding their use almost entirely! And that was never |
| 1664 |
the intent. They've always been meant to be usable within an |
| 1665 |
application, or cooperating set of applications, at will. |
| 1666 |
.PP |
| 1667 |
If you're writing code, such as an editor, that is supposed to be able |
| 1668 |
to handle any Unicode text data, then you shouldn't be using these code |
| 1669 |
points yourself, and instead allow them in the input. If you need |
| 1670 |
sentinels, they should instead be something that isn't legal Unicode. |
| 1671 |
For UTF\-8 data, you can use the bytes 0xC0 and 0xC1 as sentinels, as |
| 1672 |
they never appear in well-formed UTF\-8. (There are equivalents for |
| 1673 |
UTF-EBCDIC). You can also store your Unicode code points in integer |
| 1674 |
variables and use negative values as sentinels. |
| 1675 |
.PP |
| 1676 |
If you're not writing such a tool, then whether you accept noncharacters |
| 1677 |
as input is up to you (though the Standard recommends that you not). If |
| 1678 |
you do strict input stream checking with Perl, these code points |
| 1679 |
continue to be forbidden. This is to maintain backward compatibility |
| 1680 |
(otherwise potential security holes could open up, as an unsuspecting |
| 1681 |
application that was written assuming the noncharacters would be |
| 1682 |
filtered out before getting to it, could now, without warning, start |
| 1683 |
getting them). To do strict checking, you can use the layer |
| 1684 |
\&\f(CW:encoding(\*(AqUTF\-8\*(Aq)\fR. |
| 1685 |
.PP |
| 1686 |
Perl continues to warn (using the warning category \f(CW"nonchar"\fR, which |
| 1687 |
is a sub-category of \f(CW"utf8"\fR) if an attempt is made to output |
| 1688 |
noncharacters. |
| 1689 |
.SS "Beyond Unicode code points" |
| 1690 |
.IX Subsection "Beyond Unicode code points" |
| 1691 |
The maximum Unicode code point is \f(CW\*(C`U+10FFFF\*(C'\fR, and Unicode only defines |
| 1692 |
operations on code points up through that. But Perl works on code |
| 1693 |
points up to the maximum permissible signed number available on the |
| 1694 |
platform. However, Perl will not accept these from input streams unless |
| 1695 |
lax rules are being used, and will warn (using the warning category |
| 1696 |
\&\f(CW"non_unicode"\fR, which is a sub-category of \f(CW"utf8"\fR) if any are output. |
| 1697 |
.PP |
| 1698 |
Since Unicode rules are not defined on these code points, if a |
| 1699 |
Unicode-defined operation is done on them, Perl uses what we believe are |
| 1700 |
sensible rules, while generally warning, using the \f(CW"non_unicode"\fR |
| 1701 |
category. For example, \f(CWuc("\ex{11_0000}")\fR will generate such a |
| 1702 |
warning, returning the input parameter as its result, since Perl defines |
| 1703 |
the uppercase of every non-Unicode code point to be the code point |
| 1704 |
itself. (All the case changing operations, not just uppercasing, work |
| 1705 |
this way.) |
| 1706 |
.PP |
| 1707 |
The situation with matching Unicode properties in regular expressions, |
| 1708 |
the \f(CW\*(C`\ep{}\*(C'\fR and \f(CW\*(C`\eP{}\*(C'\fR constructs, against these code points is not as |
| 1709 |
clear cut, and how these are handled has changed as we've gained |
| 1710 |
experience. |
| 1711 |
.PP |
| 1712 |
One possibility is to treat any match against these code points as |
| 1713 |
undefined. But since Perl doesn't have the concept of a match being |
| 1714 |
undefined, it converts this to failing or \f(CW\*(C`FALSE\*(C'\fR. This is almost, but |
| 1715 |
not quite, what Perl did from v5.14 (when use of these code points |
| 1716 |
became generally reliable) through v5.18. The difference is that Perl |
| 1717 |
treated all \f(CW\*(C`\ep{}\*(C'\fR matches as failing, but all \f(CW\*(C`\eP{}\*(C'\fR matches as |
| 1718 |
succeeding. |
| 1719 |
.PP |
| 1720 |
One problem with this is that it leads to unexpected, and confusing |
| 1721 |
results in some cases: |
| 1722 |
.PP |
| 1723 |
.Vb 2 |
| 1724 |
\& chr(0x110000) =~ \ep{ASCII_Hex_Digit=True} # Failed on <= v5.18 |
| 1725 |
\& chr(0x110000) =~ \ep{ASCII_Hex_Digit=False} # Failed! on <= v5.18 |
| 1726 |
.Ve |
| 1727 |
.PP |
| 1728 |
That is, it treated both matches as undefined, and converted that to |
| 1729 |
false (raising a warning on each). The first case is the expected |
| 1730 |
result, but the second is likely counterintuitive: "How could both be |
| 1731 |
false when they are complements?" Another problem was that the |
| 1732 |
implementation optimized many Unicode property matches down to already |
| 1733 |
existing simpler, faster operations, which don't raise the warning. We |
| 1734 |
chose to not forgo those optimizations, which help the vast majority of |
| 1735 |
matches, just to generate a warning for the unlikely event that an |
| 1736 |
above-Unicode code point is being matched against. |
| 1737 |
.PP |
| 1738 |
As a result of these problems, starting in v5.20, what Perl does is |
| 1739 |
to treat non-Unicode code points as just typical unassigned Unicode |
| 1740 |
characters, and matches accordingly. (Note: Unicode has atypical |
| 1741 |
unassigned code points. For example, it has noncharacter code points, |
| 1742 |
and ones that, when they do get assigned, are destined to be written |
| 1743 |
Right-to-left, as Arabic and Hebrew are. Perl assumes that no |
| 1744 |
non-Unicode code point has any atypical properties.) |
| 1745 |
.PP |
| 1746 |
Perl, in most cases, will raise a warning when matching an above-Unicode |
| 1747 |
code point against a Unicode property when the result is \f(CW\*(C`TRUE\*(C'\fR for |
| 1748 |
\&\f(CW\*(C`\ep{}\*(C'\fR, and \f(CW\*(C`FALSE\*(C'\fR for \f(CW\*(C`\eP{}\*(C'\fR. For example: |
| 1749 |
.PP |
| 1750 |
.Vb 2 |
| 1751 |
\& chr(0x110000) =~ \ep{ASCII_Hex_Digit=True} # Fails, no warning |
| 1752 |
\& chr(0x110000) =~ \ep{ASCII_Hex_Digit=False} # Succeeds, with warning |
| 1753 |
.Ve |
| 1754 |
.PP |
| 1755 |
In both these examples, the character being matched is non-Unicode, so |
| 1756 |
Unicode doesn't define how it should match. It clearly isn't an ASCII |
| 1757 |
hex digit, so the first example clearly should fail, and so it does, |
| 1758 |
with no warning. But it is arguable that the second example should have |
| 1759 |
an undefined, hence \f(CW\*(C`FALSE\*(C'\fR, result. So a warning is raised for it. |
| 1760 |
.PP |
| 1761 |
Thus the warning is raised for many fewer cases than in earlier Perls, |
| 1762 |
and only when what the result is could be arguable. It turns out that |
| 1763 |
none of the optimizations made by Perl (or are ever likely to be made) |
| 1764 |
cause the warning to be skipped, so it solves both problems of Perl's |
| 1765 |
earlier approach. The most commonly used property that is affected by |
| 1766 |
this change is \f(CW\*(C`\ep{Unassigned}\*(C'\fR which is a short form for |
| 1767 |
\&\f(CW\*(C`\ep{General_Category=Unassigned}\*(C'\fR. Starting in v5.20, all non-Unicode |
| 1768 |
code points are considered \f(CW\*(C`Unassigned\*(C'\fR. In earlier releases the |
| 1769 |
matches failed because the result was considered undefined. |
| 1770 |
.PP |
| 1771 |
The only place where the warning is not raised when it might ought to |
| 1772 |
have been is if optimizations cause the whole pattern match to not even |
| 1773 |
be attempted. For example, Perl may figure out that for a string to |
| 1774 |
match a certain regular expression pattern, the string has to contain |
| 1775 |
the substring \f(CW"foobar"\fR. Before attempting the match, Perl may look |
| 1776 |
for that substring, and if not found, immediately fail the match without |
| 1777 |
actually trying it; so no warning gets generated even if the string |
| 1778 |
contains an above-Unicode code point. |
| 1779 |
.PP |
| 1780 |
This behavior is more "Do what I mean" than in earlier Perls for most |
| 1781 |
applications. But it catches fewer issues for code that needs to be |
| 1782 |
strictly Unicode compliant. Therefore there is an additional mode of |
| 1783 |
operation available to accommodate such code. This mode is enabled if a |
| 1784 |
regular expression pattern is compiled within the lexical scope where |
| 1785 |
the \f(CW"non_unicode"\fR warning class has been made fatal, say by: |
| 1786 |
.PP |
| 1787 |
.Vb 1 |
| 1788 |
\& use warnings FATAL => "non_unicode" |
| 1789 |
.Ve |
| 1790 |
.PP |
| 1791 |
(see warnings). In this mode of operation, Perl will raise the |
| 1792 |
warning for all matches against a non-Unicode code point (not just the |
| 1793 |
arguable ones), and it skips the optimizations that might cause the |
| 1794 |
warning to not be output. (It currently still won't warn if the match |
| 1795 |
isn't even attempted, like in the \f(CW"foobar"\fR example above.) |
| 1796 |
.PP |
| 1797 |
In summary, Perl now normally treats non-Unicode code points as typical |
| 1798 |
Unicode unassigned code points for regular expression matches, raising a |
| 1799 |
warning only when it is arguable what the result should be. However, if |
| 1800 |
this warning has been made fatal, it isn't skipped. |
| 1801 |
.PP |
| 1802 |
There is one exception to all this. \f(CW\*(C`\ep{All}\*(C'\fR looks like a Unicode |
| 1803 |
property, but it is a Perl extension that is defined to be true for all |
| 1804 |
possible code points, Unicode or not, so no warning is ever generated |
| 1805 |
when matching this against a non-Unicode code point. (Prior to v5.20, |
| 1806 |
it was an exact synonym for \f(CW\*(C`\ep{Any}\*(C'\fR, matching code points \f(CW0\fR |
| 1807 |
through \f(CW0x10FFFF\fR.) |
| 1808 |
.SS "Security Implications of Unicode" |
| 1809 |
.IX Subsection "Security Implications of Unicode" |
| 1810 |
First, read |
| 1811 |
Unicode Security Considerations <https://www.unicode.org/reports/tr36>. |
| 1812 |
.PP |
| 1813 |
Also, note the following: |
| 1814 |
.IP \(bu 4 |
| 1815 |
Malformed UTF\-8 |
| 1816 |
.Sp |
| 1817 |
UTF\-8 is very structured, so many combinations of bytes are invalid. In |
| 1818 |
the past, Perl tried to soldier on and make some sense of invalid |
| 1819 |
combinations, but this can lead to security holes, so now, if the Perl |
| 1820 |
core needs to process an invalid combination, it will either raise a |
| 1821 |
fatal error, or will replace those bytes by the sequence that forms the |
| 1822 |
Unicode REPLACEMENT CHARACTER, for which purpose Unicode created it. |
| 1823 |
.Sp |
| 1824 |
Every code point can be represented by more than one possible |
| 1825 |
syntactically valid UTF\-8 sequence. Early on, both Unicode and Perl |
| 1826 |
considered any of these to be valid, but now, all sequences longer |
| 1827 |
than the shortest possible one are considered to be malformed. |
| 1828 |
.Sp |
| 1829 |
Unicode considers many code points to be illegal, or to be avoided. |
| 1830 |
Perl generally accepts them, once they have passed through any input |
| 1831 |
filters that may try to exclude them. These have been discussed above |
| 1832 |
(see "Surrogates" under UTF\-16 in "Unicode Encodings", |
| 1833 |
"Noncharacter code points", and "Beyond Unicode code points"). |
| 1834 |
.IP \(bu 4 |
| 1835 |
Regular expression pattern matching may surprise you if you're not |
| 1836 |
accustomed to Unicode. Starting in Perl 5.14, several pattern |
| 1837 |
modifiers are available to control this, called the character set |
| 1838 |
modifiers. Details are given in "Character set modifiers" in perlre. |
| 1839 |
.PP |
| 1840 |
As discussed elsewhere, Perl has one foot (two hooves?) planted in |
| 1841 |
each of two worlds: the old world of ASCII and single-byte locales, and |
| 1842 |
the new world of Unicode, upgrading when necessary. |
| 1843 |
If your legacy code does not explicitly use Unicode, no automatic |
| 1844 |
switch-over to Unicode should happen. |
| 1845 |
.SS "Unicode in Perl on EBCDIC" |
| 1846 |
.IX Subsection "Unicode in Perl on EBCDIC" |
| 1847 |
Unicode is supported on EBCDIC platforms. See perlebcdic. |
| 1848 |
.PP |
| 1849 |
Unless ASCII vs. EBCDIC issues are specifically being discussed, |
| 1850 |
references to UTF\-8 encoding in this document and elsewhere should be |
| 1851 |
read as meaning UTF-EBCDIC on EBCDIC platforms. |
| 1852 |
See "Unicode and UTF" in perlebcdic. |
| 1853 |
.PP |
| 1854 |
Because UTF-EBCDIC is so similar to UTF\-8, the differences are mostly |
| 1855 |
hidden from you; \f(CW\*(C`use\ utf8\*(C'\fR (and NOT something like |
| 1856 |
\&\f(CW\*(C`use\ utfebcdic\*(C'\fR) declares the script is in the platform's |
| 1857 |
"native" 8\-bit encoding of Unicode. (Similarly for the \f(CW":utf8"\fR |
| 1858 |
layer.) |
| 1859 |
.SS Locales |
| 1860 |
.IX Subsection "Locales" |
| 1861 |
See "Unicode and UTF\-8" in perllocale |
| 1862 |
.SS "When Unicode Does Not Happen" |
| 1863 |
.IX Subsection "When Unicode Does Not Happen" |
| 1864 |
There are still many places where Unicode (in some encoding or |
| 1865 |
another) could be given as arguments or received as results, or both in |
| 1866 |
Perl, but it is not, in spite of Perl having extensive ways to input and |
| 1867 |
output in Unicode, and a few other "entry points" like the \f(CW@ARGV\fR |
| 1868 |
array (which can sometimes be interpreted as UTF\-8). |
| 1869 |
.PP |
| 1870 |
The following are such interfaces. Also, see "The "Unicode Bug"". |
| 1871 |
For all of these interfaces Perl |
| 1872 |
currently (as of v5.16.0) simply assumes byte strings both as arguments |
| 1873 |
and results, or UTF\-8 strings if the (deprecated) \f(CW\*(C`encoding\*(C'\fR pragma has been used. |
| 1874 |
.PP |
| 1875 |
One reason that Perl does not attempt to resolve the role of Unicode in |
| 1876 |
these situations is that the answers are highly dependent on the operating |
| 1877 |
system and the file system(s). For example, whether filenames can be |
| 1878 |
in Unicode and in exactly what kind of encoding, is not exactly a |
| 1879 |
portable concept. Similarly for \f(CW\*(C`qx\*(C'\fR and \f(CW\*(C`system\*(C'\fR: how well will the |
| 1880 |
"command-line interface" (and which of them?) handle Unicode? |
| 1881 |
.IP \(bu 4 |
| 1882 |
\&\f(CW\*(C`chdir\*(C'\fR, \f(CW\*(C`chmod\*(C'\fR, \f(CW\*(C`chown\*(C'\fR, \f(CW\*(C`chroot\*(C'\fR, \f(CW\*(C`exec\*(C'\fR, \f(CW\*(C`link\*(C'\fR, \f(CW\*(C`lstat\*(C'\fR, \f(CW\*(C`mkdir\*(C'\fR, |
| 1883 |
\&\f(CW\*(C`rename\*(C'\fR, \f(CW\*(C`rmdir\*(C'\fR, \f(CW\*(C`stat\*(C'\fR, \f(CW\*(C`symlink\*(C'\fR, \f(CW\*(C`truncate\*(C'\fR, \f(CW\*(C`unlink\*(C'\fR, \f(CW\*(C`utime\*(C'\fR, \f(CW\*(C`\-X\*(C'\fR |
| 1884 |
.IP \(bu 4 |
| 1885 |
\&\f(CW%ENV\fR |
| 1886 |
.IP \(bu 4 |
| 1887 |
\&\f(CW\*(C`glob\*(C'\fR (aka the \f(CW\*(C`<*>\*(C'\fR) |
| 1888 |
.IP \(bu 4 |
| 1889 |
\&\f(CW\*(C`open\*(C'\fR, \f(CW\*(C`opendir\*(C'\fR, \f(CW\*(C`sysopen\*(C'\fR |
| 1890 |
.IP \(bu 4 |
| 1891 |
\&\f(CW\*(C`qx\*(C'\fR (aka the backtick operator), \f(CW\*(C`system\*(C'\fR |
| 1892 |
.IP \(bu 4 |
| 1893 |
\&\f(CW\*(C`readdir\*(C'\fR, \f(CW\*(C`readlink\*(C'\fR |
| 1894 |
.SS "The ""Unicode Bug""" |
| 1895 |
.IX Subsection "The ""Unicode Bug""" |
| 1896 |
The term, "Unicode bug" has been applied to an inconsistency with the |
| 1897 |
code points in the \f(CW\*(C`Latin\-1 Supplement\*(C'\fR block, that is, between |
| 1898 |
128 and 255. Without a locale specified, unlike all other characters or |
| 1899 |
code points, these characters can have very different semantics |
| 1900 |
depending on the rules in effect. (Characters whose code points are |
| 1901 |
above 255 force Unicode rules; whereas the rules for ASCII characters |
| 1902 |
are the same under both ASCII and Unicode rules.) |
| 1903 |
.PP |
| 1904 |
Under Unicode rules, these upper\-Latin1 characters are interpreted as |
| 1905 |
Unicode code points, which means they have the same semantics as Latin\-1 |
| 1906 |
(ISO\-8859\-1) and C1 controls. |
| 1907 |
.PP |
| 1908 |
As explained in "ASCII Rules versus Unicode Rules", under ASCII rules, |
| 1909 |
they are considered to be unassigned characters. |
| 1910 |
.PP |
| 1911 |
This can lead to unexpected results. For example, a string's |
| 1912 |
semantics can suddenly change if a code point above 255 is appended to |
| 1913 |
it, which changes the rules from ASCII to Unicode. As an |
| 1914 |
example, consider the following program and its output: |
| 1915 |
.PP |
| 1916 |
.Vb 11 |
| 1917 |
\& $ perl \-le\*(Aq |
| 1918 |
\& no feature "unicode_strings"; |
| 1919 |
\& $s1 = "\exC2"; |
| 1920 |
\& $s2 = "\ex{2660}"; |
| 1921 |
\& for ($s1, $s2, $s1.$s2) { |
| 1922 |
\& print /\ew/ || 0; |
| 1923 |
\& } |
| 1924 |
\& \*(Aq |
| 1925 |
\& 0 |
| 1926 |
\& 0 |
| 1927 |
\& 1 |
| 1928 |
.Ve |
| 1929 |
.PP |
| 1930 |
If there's no \f(CW\*(C`\ew\*(C'\fR in \f(CW\*(C`s1\*(C'\fR nor in \f(CW\*(C`s2\*(C'\fR, why does their concatenation |
| 1931 |
have one? |
| 1932 |
.PP |
| 1933 |
This anomaly stems from Perl's attempt to not disturb older programs that |
| 1934 |
didn't use Unicode, along with Perl's desire to add Unicode support |
| 1935 |
seamlessly. But the result turned out to not be seamless. (By the way, |
| 1936 |
you can choose to be warned when things like this happen. See |
| 1937 |
\&\f(CW\*(C`encoding::warnings\*(C'\fR.) |
| 1938 |
.PP |
| 1939 |
\&\f(CW\*(C`use\ feature\ \*(Aqunicode_strings\*(Aq\*(C'\fR |
| 1940 |
was added, starting in Perl v5.12, to address this problem. It affects |
| 1941 |
these things: |
| 1942 |
.IP \(bu 4 |
| 1943 |
Changing the case of a scalar, that is, using \f(CWuc()\fR, \f(CWucfirst()\fR, \f(CWlc()\fR, |
| 1944 |
and \f(CWlcfirst()\fR, or \f(CW\*(C`\eL\*(C'\fR, \f(CW\*(C`\eU\*(C'\fR, \f(CW\*(C`\eu\*(C'\fR and \f(CW\*(C`\el\*(C'\fR in double-quotish |
| 1945 |
contexts, such as regular expression substitutions. |
| 1946 |
.Sp |
| 1947 |
Under \f(CW\*(C`unicode_strings\*(C'\fR starting in Perl 5.12.0, Unicode rules are |
| 1948 |
generally used. See "lc" in perlfunc for details on how this works |
| 1949 |
in combination with various other pragmas. |
| 1950 |
.IP \(bu 4 |
| 1951 |
Using caseless (\f(CW\*(C`/i\*(C'\fR) regular expression matching. |
| 1952 |
.Sp |
| 1953 |
Starting in Perl 5.14.0, regular expressions compiled within |
| 1954 |
the scope of \f(CW\*(C`unicode_strings\*(C'\fR use Unicode rules |
| 1955 |
even when executed or compiled into larger |
| 1956 |
regular expressions outside the scope. |
| 1957 |
.IP \(bu 4 |
| 1958 |
Matching any of several properties in regular expressions. |
| 1959 |
.Sp |
| 1960 |
These properties are \f(CW\*(C`\eb\*(C'\fR (without braces), \f(CW\*(C`\eB\*(C'\fR (without braces), |
| 1961 |
\&\f(CW\*(C`\es\*(C'\fR, \f(CW\*(C`\eS\*(C'\fR, \f(CW\*(C`\ew\*(C'\fR, \f(CW\*(C`\eW\*(C'\fR, and all the Posix character classes |
| 1962 |
\&\fIexcept\fR \f(CW\*(C`[[:ascii:]]\*(C'\fR. |
| 1963 |
.Sp |
| 1964 |
Starting in Perl 5.14.0, regular expressions compiled within |
| 1965 |
the scope of \f(CW\*(C`unicode_strings\*(C'\fR use Unicode rules |
| 1966 |
even when executed or compiled into larger |
| 1967 |
regular expressions outside the scope. |
| 1968 |
.IP \(bu 4 |
| 1969 |
In \f(CW\*(C`quotemeta\*(C'\fR or its inline equivalent \f(CW\*(C`\eQ\*(C'\fR. |
| 1970 |
.Sp |
| 1971 |
Starting in Perl 5.16.0, consistent quoting rules are used within the |
| 1972 |
scope of \f(CW\*(C`unicode_strings\*(C'\fR, as described in "quotemeta" in perlfunc. |
| 1973 |
Prior to that, or outside its scope, no code points above 127 are quoted |
| 1974 |
in UTF\-8 encoded strings, but in byte encoded strings, code points |
| 1975 |
between 128\-255 are always quoted. |
| 1976 |
.IP \(bu 4 |
| 1977 |
In the \f(CW\*(C`..\*(C'\fR or range operator. |
| 1978 |
.Sp |
| 1979 |
Starting in Perl 5.26.0, the range operator on strings treats their lengths |
| 1980 |
consistently within the scope of \f(CW\*(C`unicode_strings\*(C'\fR. Prior to that, or |
| 1981 |
outside its scope, it could produce strings whose length in characters |
| 1982 |
exceeded that of the right-hand side, where the right-hand side took up more |
| 1983 |
bytes than the correct range endpoint. |
| 1984 |
.IP \(bu 4 |
| 1985 |
In \f(CW\*(C`split\*(C'\fR's special-case whitespace splitting. |
| 1986 |
.Sp |
| 1987 |
Starting in Perl 5.28.0, the \f(CW\*(C`split\*(C'\fR function with a pattern specified as |
| 1988 |
a string containing a single space handles whitespace characters consistently |
| 1989 |
within the scope of \f(CW\*(C`unicode_strings\*(C'\fR. Prior to that, or outside its scope, |
| 1990 |
characters that are whitespace according to Unicode rules but not according to |
| 1991 |
ASCII rules were treated as field contents rather than field separators when |
| 1992 |
they appear in byte-encoded strings. |
| 1993 |
.PP |
| 1994 |
You can see from the above that the effect of \f(CW\*(C`unicode_strings\*(C'\fR |
| 1995 |
increased over several Perl releases. (And Perl's support for Unicode |
| 1996 |
continues to improve; it's best to use the latest available release in |
| 1997 |
order to get the most complete and accurate results possible.) Note that |
| 1998 |
\&\f(CW\*(C`unicode_strings\*(C'\fR is automatically chosen if you \f(CW\*(C`use\ v5.12\*(C'\fR or |
| 1999 |
higher. |
| 2000 |
.PP |
| 2001 |
For Perls earlier than those described above, or when a string is passed |
| 2002 |
to a function outside the scope of \f(CW\*(C`unicode_strings\*(C'\fR, see the next section. |
| 2003 |
.SS "Forcing Unicode in Perl (Or Unforcing Unicode in Perl)" |
| 2004 |
.IX Subsection "Forcing Unicode in Perl (Or Unforcing Unicode in Perl)" |
| 2005 |
Sometimes (see "When Unicode Does Not Happen" or "The "Unicode Bug"") |
| 2006 |
there are situations where you simply need to force a byte |
| 2007 |
string into UTF\-8, or vice versa. The standard module Encode can be |
| 2008 |
used for this, or the low-level calls |
| 2009 |
\&\f(CWutf8::upgrade($bytestring)\fR and |
| 2010 |
\&\f(CW\*(C`utf8::downgrade($utf8string[, FAIL_OK])\*(C'\fR. |
| 2011 |
.PP |
| 2012 |
Note that \f(CWutf8::downgrade()\fR can fail if the string contains characters |
| 2013 |
that don't fit into a byte. |
| 2014 |
.PP |
| 2015 |
Calling either function on a string that already is in the desired state is a |
| 2016 |
no-op. |
| 2017 |
.PP |
| 2018 |
"ASCII Rules versus Unicode Rules" gives all the ways that a string is |
| 2019 |
made to use Unicode rules. |
| 2020 |
.SS "Using Unicode in XS" |
| 2021 |
.IX Subsection "Using Unicode in XS" |
| 2022 |
See "Unicode Support" in perlguts for an introduction to Unicode at |
| 2023 |
the XS level, and "Unicode Support" in perlapi for the API details. |
| 2024 |
.SS "Hacking Perl to work on earlier Unicode versions (for very serious hackers only)" |
| 2025 |
.IX Subsection "Hacking Perl to work on earlier Unicode versions (for very serious hackers only)" |
| 2026 |
Perl by default comes with the latest supported Unicode version built-in, but |
| 2027 |
the goal is to allow you to change to use any earlier one. In Perls |
| 2028 |
v5.20 and v5.22, however, the earliest usable version is Unicode 5.1. |
| 2029 |
Perl v5.18 and v5.24 are able to handle all earlier versions. |
| 2030 |
.PP |
| 2031 |
Download the files in the desired version of Unicode from the Unicode web |
| 2032 |
site <https://www.unicode.org>). These should replace the existing files in |
| 2033 |
\&\fIlib/unicore\fR in the Perl source tree. Follow the instructions in |
| 2034 |
\&\fIREADME.perl\fR in that directory to change some of their names, and then build |
| 2035 |
perl (see INSTALL). |
| 2036 |
.SS "Porting code from perl\-5.6.X" |
| 2037 |
.IX Subsection "Porting code from perl-5.6.X" |
| 2038 |
Perls starting in 5.8 have a different Unicode model from 5.6. In 5.6 the |
| 2039 |
programmer was required to use the \f(CW\*(C`utf8\*(C'\fR pragma to declare that a |
| 2040 |
given scope expected to deal with Unicode data and had to make sure that |
| 2041 |
only Unicode data were reaching that scope. If you have code that is |
| 2042 |
working with 5.6, you will need some of the following adjustments to |
| 2043 |
your code. The examples are written such that the code will continue to |
| 2044 |
work under 5.6, so you should be safe to try them out. |
| 2045 |
.IP \(bu 3 |
| 2046 |
A filehandle that should read or write UTF\-8 |
| 2047 |
.Sp |
| 2048 |
.Vb 3 |
| 2049 |
\& if ($] > 5.008) { |
| 2050 |
\& binmode $fh, ":encoding(UTF\-8)"; |
| 2051 |
\& } |
| 2052 |
.Ve |
| 2053 |
.IP \(bu 3 |
| 2054 |
A scalar that is going to be passed to some extension |
| 2055 |
.Sp |
| 2056 |
Be it \f(CW\*(C`Compress::Zlib\*(C'\fR, \f(CW\*(C`Apache::Request\*(C'\fR or any extension that has no |
| 2057 |
mention of Unicode in the manpage, you need to make sure that the |
| 2058 |
UTF8 flag is stripped off. Note that at the time of this writing |
| 2059 |
(January 2012) the mentioned modules are not UTF\-8\-aware. Please |
| 2060 |
check the documentation to verify if this is still true. |
| 2061 |
.Sp |
| 2062 |
.Vb 4 |
| 2063 |
\& if ($] > 5.008) { |
| 2064 |
\& require Encode; |
| 2065 |
\& $val = Encode::encode("UTF\-8", $val); # make octets |
| 2066 |
\& } |
| 2067 |
.Ve |
| 2068 |
.IP \(bu 3 |
| 2069 |
A scalar we got back from an extension |
| 2070 |
.Sp |
| 2071 |
If you believe the scalar comes back as UTF\-8, you will most likely |
| 2072 |
want the UTF8 flag restored: |
| 2073 |
.Sp |
| 2074 |
.Vb 4 |
| 2075 |
\& if ($] > 5.008) { |
| 2076 |
\& require Encode; |
| 2077 |
\& $val = Encode::decode("UTF\-8", $val); |
| 2078 |
\& } |
| 2079 |
.Ve |
| 2080 |
.IP \(bu 3 |
| 2081 |
Same thing, if you are really sure it is UTF\-8 |
| 2082 |
.Sp |
| 2083 |
.Vb 4 |
| 2084 |
\& if ($] > 5.008) { |
| 2085 |
\& require Encode; |
| 2086 |
\& Encode::_utf8_on($val); |
| 2087 |
\& } |
| 2088 |
.Ve |
| 2089 |
.IP \(bu 3 |
| 2090 |
A wrapper for DBI \f(CW\*(C`fetchrow_array\*(C'\fR and \f(CW\*(C`fetchrow_hashref\*(C'\fR |
| 2091 |
.Sp |
| 2092 |
When the database contains only UTF\-8, a wrapper function or method is |
| 2093 |
a convenient way to replace all your \f(CW\*(C`fetchrow_array\*(C'\fR and |
| 2094 |
\&\f(CW\*(C`fetchrow_hashref\*(C'\fR calls. A wrapper function will also make it easier to |
| 2095 |
adapt to future enhancements in your database driver. Note that at the |
| 2096 |
time of this writing (January 2012), the DBI has no standardized way |
| 2097 |
to deal with UTF\-8 data. Please check the DBI documentation to verify if |
| 2098 |
that is still true. |
| 2099 |
.Sp |
| 2100 |
.Vb 10 |
| 2101 |
\& sub fetchrow { |
| 2102 |
\& # $what is one of fetchrow_{array,hashref} |
| 2103 |
\& my($self, $sth, $what) = @_; |
| 2104 |
\& if ($] < 5.008) { |
| 2105 |
\& return $sth\->$what; |
| 2106 |
\& } else { |
| 2107 |
\& require Encode; |
| 2108 |
\& if (wantarray) { |
| 2109 |
\& my @arr = $sth\->$what; |
| 2110 |
\& for (@arr) { |
| 2111 |
\& defined && /[^\e000\-\e177]/ && Encode::_utf8_on($_); |
| 2112 |
\& } |
| 2113 |
\& return @arr; |
| 2114 |
\& } else { |
| 2115 |
\& my $ret = $sth\->$what; |
| 2116 |
\& if (ref $ret) { |
| 2117 |
\& for my $k (keys %$ret) { |
| 2118 |
\& defined |
| 2119 |
\& && /[^\e000\-\e177]/ |
| 2120 |
\& && Encode::_utf8_on($_) for $ret\->{$k}; |
| 2121 |
\& } |
| 2122 |
\& return $ret; |
| 2123 |
\& } else { |
| 2124 |
\& defined && /[^\e000\-\e177]/ && Encode::_utf8_on($_) for $ret; |
| 2125 |
\& return $ret; |
| 2126 |
\& } |
| 2127 |
\& } |
| 2128 |
\& } |
| 2129 |
\& } |
| 2130 |
.Ve |
| 2131 |
.IP \(bu 3 |
| 2132 |
A large scalar that you know can only contain ASCII |
| 2133 |
.Sp |
| 2134 |
Scalars that contain only ASCII and are marked as UTF\-8 are sometimes |
| 2135 |
a drag to your program. If you recognize such a situation, just remove |
| 2136 |
the UTF8 flag: |
| 2137 |
.Sp |
| 2138 |
.Vb 1 |
| 2139 |
\& utf8::downgrade($val) if $] > 5.008; |
| 2140 |
.Ve |
| 2141 |
.SH BUGS |
| 2142 |
.IX Header "BUGS" |
| 2143 |
See also "The "Unicode Bug"" above. |
| 2144 |
.SS "Interaction with Extensions" |
| 2145 |
.IX Subsection "Interaction with Extensions" |
| 2146 |
When Perl exchanges data with an extension, the extension should be |
| 2147 |
able to understand the UTF8 flag and act accordingly. If the |
| 2148 |
extension doesn't recognize that flag, it's likely that the extension |
| 2149 |
will return incorrectly-flagged data. |
| 2150 |
.PP |
| 2151 |
So if you're working with Unicode data, consult the documentation of |
| 2152 |
every module you're using if there are any issues with Unicode data |
| 2153 |
exchange. If the documentation does not talk about Unicode at all, |
| 2154 |
suspect the worst and probably look at the source to learn how the |
| 2155 |
module is implemented. Modules written completely in Perl shouldn't |
| 2156 |
cause problems. Modules that directly or indirectly access code written |
| 2157 |
in other programming languages are at risk. |
| 2158 |
.PP |
| 2159 |
For affected functions, the simple strategy to avoid data corruption is |
| 2160 |
to always make the encoding of the exchanged data explicit. Choose an |
| 2161 |
encoding that you know the extension can handle. Convert arguments passed |
| 2162 |
to the extensions to that encoding and convert results back from that |
| 2163 |
encoding. Write wrapper functions that do the conversions for you, so |
| 2164 |
you can later change the functions when the extension catches up. |
| 2165 |
.PP |
| 2166 |
To provide an example, let's say the popular \f(CW\*(C`Foo::Bar::escape_html\*(C'\fR |
| 2167 |
function doesn't deal with Unicode data yet. The wrapper function |
| 2168 |
would convert the argument to raw UTF\-8 and convert the result back to |
| 2169 |
Perl's internal representation like so: |
| 2170 |
.PP |
| 2171 |
.Vb 6 |
| 2172 |
\& sub my_escape_html ($) { |
| 2173 |
\& my($what) = shift; |
| 2174 |
\& return unless defined $what; |
| 2175 |
\& Encode::decode("UTF\-8", Foo::Bar::escape_html( |
| 2176 |
\& Encode::encode("UTF\-8", $what))); |
| 2177 |
\& } |
| 2178 |
.Ve |
| 2179 |
.PP |
| 2180 |
Sometimes, when the extension does not convert data but just stores |
| 2181 |
and retrieves it, you will be able to use the otherwise |
| 2182 |
dangerous \f(CWEncode::_utf8_on()\fR function. Let's say |
| 2183 |
the popular \f(CW\*(C`Foo::Bar\*(C'\fR extension, written in C, provides a \f(CW\*(C`param\*(C'\fR |
| 2184 |
method that lets you store and retrieve data according to these prototypes: |
| 2185 |
.PP |
| 2186 |
.Vb 2 |
| 2187 |
\& $self\->param($name, $value); # set a scalar |
| 2188 |
\& $value = $self\->param($name); # retrieve a scalar |
| 2189 |
.Ve |
| 2190 |
.PP |
| 2191 |
If it does not yet provide support for any encoding, one could write a |
| 2192 |
derived class with such a \f(CW\*(C`param\*(C'\fR method: |
| 2193 |
.PP |
| 2194 |
.Vb 12 |
| 2195 |
\& sub param { |
| 2196 |
\& my($self,$name,$value) = @_; |
| 2197 |
\& utf8::upgrade($name); # make sure it is UTF\-8 encoded |
| 2198 |
\& if (defined $value) { |
| 2199 |
\& utf8::upgrade($value); # make sure it is UTF\-8 encoded |
| 2200 |
\& return $self\->SUPER::param($name,$value); |
| 2201 |
\& } else { |
| 2202 |
\& my $ret = $self\->SUPER::param($name); |
| 2203 |
\& Encode::_utf8_on($ret); # we know, it is UTF\-8 encoded |
| 2204 |
\& return $ret; |
| 2205 |
\& } |
| 2206 |
\& } |
| 2207 |
.Ve |
| 2208 |
.PP |
| 2209 |
Some extensions provide filters on data entry/exit points, such as |
| 2210 |
\&\f(CW\*(C`DB_File::filter_store_key\*(C'\fR and family. Look out for such filters in |
| 2211 |
the documentation of your extensions; they can make the transition to |
| 2212 |
Unicode data much easier. |
| 2213 |
.SS Speed |
| 2214 |
.IX Subsection "Speed" |
| 2215 |
Some functions are slower when working on UTF\-8 encoded strings than |
| 2216 |
on byte encoded strings. All functions that need to hop over |
| 2217 |
characters such as \f(CWlength()\fR, \f(CWsubstr()\fR or \f(CWindex()\fR, or matching |
| 2218 |
regular expressions can work \fBmuch\fR faster when the underlying data are |
| 2219 |
byte-encoded. |
| 2220 |
.PP |
| 2221 |
In Perl 5.8.0 the slowness was often quite spectacular; in Perl 5.8.1 |
| 2222 |
a caching scheme was introduced which improved the situation. In general, |
| 2223 |
operations with UTF\-8 encoded strings are still slower. As an example, |
| 2224 |
the Unicode properties (character classes) like \f(CW\*(C`\ep{Nd}\*(C'\fR are known to |
| 2225 |
be quite a bit slower (5\-20 times) than their simpler counterparts |
| 2226 |
like \f(CW\*(C`[0\-9]\*(C'\fR (then again, there are hundreds of Unicode characters matching |
| 2227 |
\&\f(CW\*(C`Nd\*(C'\fR compared with the 10 ASCII characters matching \f(CW\*(C`[0\-9]\*(C'\fR). |
| 2228 |
.SH "SEE ALSO" |
| 2229 |
.IX Header "SEE ALSO" |
| 2230 |
perlunitut, perluniintro, perluniprops, Encode, open, utf8, bytes, |
| 2231 |
perlretut, "${^UNICODE}" in perlvar, |
| 2232 |
<https://www.unicode.org/reports/tr44>). |