Debian Codegraph @

upstream/fedora-44/man1/perlunicook.1

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man v6.0.2 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\"
56 .\" Required to disable full justification in groff 1.23.0.
57 .if n .ds AD l
58 .\" ========================================================================
59 .\"
60 .IX Title "PERLUNICOOK 1"
61 .TH PERLUNICOOK 1 2026-03-09 "perl v5.42.1" "Perl Programmers Reference Guide"
62 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
63 .\" way too many mistakes in technical documents.
64 .if n .ad l
65 .nh
66 .SH NAME
67 perlunicook \- cookbookish examples of handling Unicode in Perl
68 .SH DESCRIPTION
69 .IX Header "DESCRIPTION"
70 This manpage contains short recipes demonstrating how to handle common Unicode
71 operations in Perl, plus one complete program at the end. Any undeclared
72 variables in individual recipes are assumed to have a previous appropriate
73 value in them.
74 .SH EXAMPLES
75 .IX Header "EXAMPLES"
76 .SS "℞ 0: Standard preamble"
77 .IX Subsection "℞ 0: Standard preamble"
78 Unless otherwise noted, all examples below require this standard preamble
79 to work correctly, with the \f(CW\*(C`#!\*(C'\fR adjusted to work on your system:
80 .PP
81 .Vb 1
82 \& #!/usr/bin/env perl
83 \&
84 \& use v5.36; # or later to get "unicode_strings" feature,
85 \& # plus strict, warnings
86 \& use utf8; # so literals and identifiers can be in UTF\-8
87 \& use warnings qw(FATAL utf8); # fatalize encoding glitches
88 \& use open qw(:std :encoding(UTF\-8)); # undeclared streams in UTF\-8
89 \& use charnames qw(:full :short); # unneeded in v5.16
90 .Ve
91 .PP
92 This \fIdoes\fR make even Unix programmers \f(CW\*(C`binmode\*(C'\fR your binary streams,
93 or open them with \f(CW\*(C`:raw\*(C'\fR, but that\*(Aqs the only way to get at them
94 portably anyway.
95 .PP
96 \&\fBWARNING\fR: \f(CW\*(C`use autodie\*(C'\fR (pre 2.26) and \f(CW\*(C`use open\*(C'\fR do not get along with each
97 other.
98 .SS "℞ 1: Generic Unicode\-savvy filter"
99 .IX Subsection "℞ 1: Generic Unicode-savvy filter"
100 Always decompose on the way in, then recompose on the way out.
101 .PP
102 .Vb 1
103 \& use Unicode::Normalize;
104 \&
105 \& while (<>) {
106 \& $_ = NFD($_); # decompose + reorder canonically
107 \& ...
108 \& } continue {
109 \& print NFC($_); # recompose (where possible) + reorder canonically
110 \& }
111 .Ve
112 .SS "℞ 2: Fine\-tuning Unicode warnings"
113 .IX Subsection "℞ 2: Fine-tuning Unicode warnings"
114 As of v5.14, Perl distinguishes three subclasses of UTF‑8 warnings.
115 .PP
116 .Vb 4
117 \& use v5.14; # subwarnings unavailable any earlier
118 \& no warnings "nonchar"; # the 66 forbidden non\-characters
119 \& no warnings "surrogate"; # UTF\-16/CESU\-8 nonsense
120 \& no warnings "non_unicode"; # for codepoints over 0x10_FFFF
121 .Ve
122 .SS "℞ 3: Declare source in utf8 for identifiers and literals"
123 .IX Subsection "℞ 3: Declare source in utf8 for identifiers and literals"
124 Without the all\-critical \f(CW\*(C`use utf8\*(C'\fR declaration, putting UTF‑8 in your
125 literals and identifiers won’t work right. If you used the standard
126 preamble just given above, this already happened. If you did, you can
127 do things like this:
128 .PP
129 .Vb 1
130 \& use utf8;
131 \&
132 \& my $measure = "Ångström";
133 \& my @μsoft = qw( cp852 cp1251 cp1252 );
134 \& my @ὑπέρμεγας = qw( ὑπέρ μεγας );
135 \& my @鯉 = qw( koi8\-f koi8\-u koi8\-r );
136 \& my $motto = "👪 💗 🐪"; # FAMILY, GROWING HEART, DROMEDARY CAMEL
137 .Ve
138 .PP
139 If you forget \f(CW\*(C`use utf8\*(C'\fR, high bytes will be misunderstood as
140 separate characters, and nothing will work right.
141 .SS "℞ 4: Characters and their numbers"
142 .IX Subsection "℞ 4: Characters and their numbers"
143 The \f(CW\*(C`ord\*(C'\fR and \f(CW\*(C`chr\*(C'\fR functions work transparently on all codepoints,
144 not just on ASCII alone — nor in fact, not even just on Unicode alone.
145 .PP
146 .Vb 3
147 \& # ASCII characters
148 \& ord("A")
149 \& chr(65)
150 \&
151 \& # characters from the Basic Multilingual Plane
152 \& ord("Σ")
153 \& chr(0x3A3)
154 \&
155 \& # beyond the BMP
156 \& ord("𝑛") # MATHEMATICAL ITALIC SMALL N
157 \& chr(0x1D45B)
158 \&
159 \& # beyond Unicode! (up to MAXINT)
160 \& ord("\ex{20_0000}")
161 \& chr(0x20_0000)
162 .Ve
163 .SS "℞ 5: Unicode literals by character number"
164 .IX Subsection "℞ 5: Unicode literals by character number"
165 In an interpolated literal, whether a double\-quoted string or a
166 regex, you may specify a character by its number using the
167 \&\f(CW\*(C`\ex{\fR\f(CIHHHHHH\fR\f(CW}\*(C'\fR escape.
168 .PP
169 .Vb 2
170 \& String: "\ex{3a3}"
171 \& Regex: /\ex{3a3}/
172 \&
173 \& String: "\ex{1d45b}"
174 \& Regex: /\ex{1d45b}/
175 \&
176 \& # even non\-BMP ranges in regex work fine
177 \& /[\ex{1D434}\-\ex{1D467}]/
178 .Ve
179 .SS "℞ 6: Get character name by number"
180 .IX Subsection "℞ 6: Get character name by number"
181 .Vb 2
182 \& use charnames ();
183 \& my $name = charnames::viacode(0x03A3);
184 .Ve
185 .SS "℞ 7: Get character number by name"
186 .IX Subsection "℞ 7: Get character number by name"
187 .Vb 2
188 \& use charnames ();
189 \& my $number = charnames::vianame("GREEK CAPITAL LETTER SIGMA");
190 .Ve
191 .SS "℞ 8: Unicode named characters"
192 .IX Subsection "℞ 8: Unicode named characters"
193 Use the \f(CW\*(C`\eN{\fR\f(CIcharname\fR\f(CW}\*(C'\fR notation to get the character
194 by that name for use in interpolated literals (double\-quoted
195 strings and regexes). In v5.16, there is an implicit
196 .PP
197 .Vb 1
198 \& use charnames qw(:full :short);
199 .Ve
200 .PP
201 But prior to v5.16, you must be explicit about which set of charnames you
202 want. The \f(CW\*(C`:full\*(C'\fR names are the official Unicode character name, alias, or
203 sequence, which all share a namespace.
204 .PP
205 .Vb 1
206 \& use charnames qw(:full :short latin greek);
207 \&
208 \& "\eN{MATHEMATICAL ITALIC SMALL N}" # :full
209 \& "\eN{GREEK CAPITAL LETTER SIGMA}" # :full
210 .Ve
211 .PP
212 Anything else is a Perl\-specific convenience abbreviation. Specify one or
213 more scripts by names if you want short names that are script\-specific.
214 .PP
215 .Vb 3
216 \& "\eN{Greek:Sigma}" # :short
217 \& "\eN{ae}" # latin
218 \& "\eN{epsilon}" # greek
219 .Ve
220 .PP
221 The v5.16 release also supports a \f(CW\*(C`:loose\*(C'\fR import for loose matching of
222 character names, which works just like loose matching of property names:
223 that is, it disregards case, whitespace, and underscores:
224 .PP
225 .Vb 1
226 \& "\eN{euro sign}" # :loose (from v5.16)
227 .Ve
228 .PP
229 Starting in v5.32, you can also use
230 .PP
231 .Vb 1
232 \& qr/\ep{name=euro sign}/
233 .Ve
234 .PP
235 to get official Unicode named characters in regular expressions. Loose
236 matching is always done for these.
237 .SS "℞ 9: Unicode named sequences"
238 .IX Subsection "℞ 9: Unicode named sequences"
239 These look just like character names but return multiple codepoints.
240 Notice the \f(CW%vx\fR vector\-print functionality in \f(CW\*(C`printf\*(C'\fR.
241 .PP
242 .Vb 4
243 \& use charnames qw(:full);
244 \& my $seq = "\eN{LATIN CAPITAL LETTER A WITH MACRON AND GRAVE}";
245 \& printf "U+%v04X\en", $seq;
246 \& U+0100.0300
247 .Ve
248 .SS "℞ 10: Custom named characters"
249 .IX Subsection "℞ 10: Custom named characters"
250 Use \f(CW\*(C`:alias\*(C'\fR to give your own lexically scoped nicknames to existing
251 characters, or even to give unnamed private\-use characters useful names.
252 .PP
253 .Vb 4
254 \& use charnames ":full", ":alias" => {
255 \& ecute => "LATIN SMALL LETTER E WITH ACUTE",
256 \& "APPLE LOGO" => 0xF8FF, # private use character
257 \& };
258 \&
259 \& "\eN{ecute}"
260 \& "\eN{APPLE LOGO}"
261 .Ve
262 .SS "℞ 11: Names of CJK codepoints"
263 .IX Subsection "℞ 11: Names of CJK codepoints"
264 Sinograms like “東京” come back with character names of
265 \&\f(CW\*(C`CJK UNIFIED IDEOGRAPH\-6771\*(C'\fR and \f(CW\*(C`CJK UNIFIED IDEOGRAPH\-4EAC\*(C'\fR,
266 because their “names” vary. The CPAN \f(CW\*(C`Unicode::Unihan\*(C'\fR module
267 has a large database for decoding these (and a whole lot more), provided you
268 know how to understand its output.
269 .PP
270 .Vb 8
271 \& # cpan \-i Unicode::Unihan
272 \& use Unicode::Unihan;
273 \& my $str = "東京";
274 \& my $unhan = Unicode::Unihan\->new;
275 \& for my $lang (qw(Mandarin Cantonese Korean JapaneseOn JapaneseKun)) {
276 \& printf "CJK $str in %\-12s is ", $lang;
277 \& say $unhan\->$lang($str);
278 \& }
279 .Ve
280 .PP
281 prints:
282 .PP
283 .Vb 5
284 \& CJK 東京 in Mandarin is DONG1JING1
285 \& CJK 東京 in Cantonese is dung1ging1
286 \& CJK 東京 in Korean is TONGKYENG
287 \& CJK 東京 in JapaneseOn is TOUKYOU KEI KIN
288 \& CJK 東京 in JapaneseKun is HIGASHI AZUMAMIYAKO
289 .Ve
290 .PP
291 If you have a specific romanization scheme in mind,
292 use the specific module:
293 .PP
294 .Vb 5
295 \& # cpan \-i Lingua::JA::Romanize::Japanese
296 \& use Lingua::JA::Romanize::Japanese;
297 \& my $k2r = Lingua::JA::Romanize::Japanese\->new;
298 \& my $str = "東京";
299 \& say "Japanese for $str is ", $k2r\->chars($str);
300 .Ve
301 .PP
302 prints
303 .PP
304 .Vb 1
305 \& Japanese for 東京 is toukyou
306 .Ve
307 .SS "℞ 12: Explicit encode/decode"
308 .IX Subsection "℞ 12: Explicit encode/decode"
309 On rare occasion, such as a database read, you may be
310 given encoded text you need to decode.
311 .PP
312 .Vb 1
313 \& use Encode qw(encode decode);
314 \&
315 \& my $chars = decode("shiftjis", $bytes, 1);
316 \& # OR
317 \& my $bytes = encode("MIME\-Header\-ISO_2022_JP", $chars, 1);
318 .Ve
319 .PP
320 For streams all in the same encoding, don\*(Aqt use encode/decode; instead
321 set the file encoding when you open the file or immediately after with
322 \&\f(CW\*(C`binmode\*(C'\fR as described later below.
323 .SS "℞ 13: Decode program arguments as utf8"
324 .IX Subsection "℞ 13: Decode program arguments as utf8"
325 .Vb 6
326 \& $ perl \-CA ...
327 \& or
328 \& $ export PERL_UNICODE=A
329 \& or
330 \& use Encode qw(decode);
331 \& @ARGV = map { decode(\*(AqUTF\-8\*(Aq, $_, 1) } @ARGV;
332 .Ve
333 .SS "℞ 14: Decode program arguments as locale encoding"
334 .IX Subsection "℞ 14: Decode program arguments as locale encoding"
335 .Vb 3
336 \& # cpan \-i Encode::Locale
337 \& use Encode qw(locale);
338 \& use Encode::Locale;
339 \&
340 \& # use "locale" as an arg to encode/decode
341 \& @ARGV = map { decode(locale => $_, 1) } @ARGV;
342 .Ve
343 .SS "℞ 15: Declare STD{IN,OUT,ERR} to be utf8"
344 .IX Subsection "℞ 15: Declare STD{IN,OUT,ERR} to be utf8"
345 Use a command\-line option, an environment variable, or else
346 call \f(CW\*(C`binmode\*(C'\fR explicitly:
347 .PP
348 .Vb 9
349 \& $ perl \-CS ...
350 \& or
351 \& $ export PERL_UNICODE=S
352 \& or
353 \& use open qw(:std :encoding(UTF\-8));
354 \& or
355 \& binmode(STDIN, ":encoding(UTF\-8)");
356 \& binmode(STDOUT, ":utf8");
357 \& binmode(STDERR, ":utf8");
358 .Ve
359 .SS "℞ 16: Declare STD{IN,OUT,ERR} to be in locale encoding"
360 .IX Subsection "℞ 16: Declare STD{IN,OUT,ERR} to be in locale encoding"
361 .Vb 3
362 \& # cpan \-i Encode::Locale
363 \& use Encode;
364 \& use Encode::Locale;
365 \&
366 \& # or as a stream for binmode or open
367 \& binmode STDIN, ":encoding(console_in)" if \-t STDIN;
368 \& binmode STDOUT, ":encoding(console_out)" if \-t STDOUT;
369 \& binmode STDERR, ":encoding(console_out)" if \-t STDERR;
370 .Ve
371 .SS "℞ 17: Make file I/O default to utf8"
372 .IX Subsection "℞ 17: Make file I/O default to utf8"
373 Files opened without an encoding argument will be in UTF\-8:
374 .PP
375 .Vb 5
376 \& $ perl \-CD ...
377 \& or
378 \& $ export PERL_UNICODE=D
379 \& or
380 \& use open qw(:encoding(UTF\-8));
381 .Ve
382 .SS "℞ 18: Make all I/O and args default to utf8"
383 .IX Subsection "℞ 18: Make all I/O and args default to utf8"
384 .Vb 7
385 \& $ perl \-CSDA ...
386 \& or
387 \& $ export PERL_UNICODE=SDA
388 \& or
389 \& use open qw(:std :encoding(UTF\-8));
390 \& use Encode qw(decode);
391 \& @ARGV = map { decode(\*(AqUTF\-8\*(Aq, $_, 1) } @ARGV;
392 .Ve
393 .SS "℞ 19: Open file with specific encoding"
394 .IX Subsection "℞ 19: Open file with specific encoding"
395 Specify stream encoding. This is the normal way
396 to deal with encoded text, not by calling low\-level
397 functions.
398 .PP
399 .Vb 7
400 \& # input file
401 \& open(my $in_file, "< :encoding(UTF\-16)", "wintext");
402 \& OR
403 \& open(my $in_file, "<", "wintext");
404 \& binmode($in_file, ":encoding(UTF\-16)");
405 \& THEN
406 \& my $line = <$in_file>;
407 \&
408 \& # output file
409 \& open($out_file, "> :encoding(cp1252)", "wintext");
410 \& OR
411 \& open(my $out_file, ">", "wintext");
412 \& binmode($out_file, ":encoding(cp1252)");
413 \& THEN
414 \& print $out_file "some text\en";
415 .Ve
416 .PP
417 More layers than just the encoding can be specified here. For example,
418 the incantation \f(CW":raw :encoding(UTF\-16LE) :crlf"\fR includes implicit
419 CRLF handling.
420 .SS "℞ 20: Unicode casing"
421 .IX Subsection "℞ 20: Unicode casing"
422 Unicode casing is very different from ASCII casing.
423 .PP
424 .Vb 2
425 \& uc("henry ⅷ") # "HENRY Ⅷ"
426 \& uc("tschüß") # "TSCHÜSS" notice ß => SS
427 \&
428 \& # both are true:
429 \& "tschüß" =~ /TSCHÜSS/i # notice ß => SS
430 \& "Σίσυφος" =~ /ΣΊΣΥΦΟΣ/i # notice Σ,σ,ς sameness
431 .Ve
432 .SS "℞ 21: Unicode case\-insensitive comparisons"
433 .IX Subsection "℞ 21: Unicode case-insensitive comparisons"
434 Also available in the CPAN Unicode::CaseFold module,
435 the new \f(CW\*(C`fc\*(C'\fR “foldcase” function from v5.16 grants
436 access to the same Unicode casefolding as the \f(CW\*(C`/i\*(C'\fR
437 pattern modifier has always used:
438 .PP
439 .Vb 1
440 \& use feature "fc"; # fc() function is from v5.16
441 \&
442 \& # sort case\-insensitively
443 \& my @sorted = sort { fc($a) cmp fc($b) } @list;
444 \&
445 \& # both are true:
446 \& fc("tschüß") eq fc("TSCHÜSS")
447 \& fc("Σίσυφος") eq fc("ΣΊΣΥΦΟΣ")
448 .Ve
449 .SS "℞ 22: Match Unicode linebreak sequence in regex"
450 .IX Subsection "℞ 22: Match Unicode linebreak sequence in regex"
451 A Unicode linebreak matches the two\-character CRLF
452 grapheme or any of seven vertical whitespace characters.
453 Good for dealing with textfiles coming from different
454 operating systems.
455 .PP
456 .Vb 1
457 \& \eR
458 \&
459 \& s/\eR/\en/g; # normalize all linebreaks to \en
460 .Ve
461 .SS "℞ 23: Get character category"
462 .IX Subsection "℞ 23: Get character category"
463 Find the general category of a numeric codepoint.
464 .PP
465 .Vb 2
466 \& use Unicode::UCD qw(charinfo);
467 \& my $cat = charinfo(0x3A3)\->{category}; # "Lu"
468 .Ve
469 .SS "℞ 24: Disabling Unicode\-awareness in builtin charclasses"
470 .IX Subsection "℞ 24: Disabling Unicode-awareness in builtin charclasses"
471 Disable \f(CW\*(C`\ew\*(C'\fR, \f(CW\*(C`\eb\*(C'\fR, \f(CW\*(C`\es\*(C'\fR, \f(CW\*(C`\ed\*(C'\fR, and the POSIX
472 classes from working correctly on Unicode either in this
473 scope, or in just one regex.
474 .PP
475 .Vb 2
476 \& use v5.14;
477 \& use re "/a";
478 \&
479 \& # OR
480 \&
481 \& my($num) = $str =~ /(\ed+)/a;
482 .Ve
483 .PP
484 Or use specific un\-Unicode properties, like \f(CW\*(C`\ep{ahex}\*(C'\fR
485 and \f(CW\*(C`\ep{POSIX_Digit}\*(C'\fR. Properties still work normally
486 no matter what charset modifiers (\f(CW\*(C`/d /u /l /a /aa\*(C'\fR)
487 are in effect.
488 .SS "℞ 25: Match Unicode properties in regex with \ep, \eP"
489 .IX Subsection "℞ 25: Match Unicode properties in regex with p, P"
490 These all match a single codepoint with the given
491 property. Use \f(CW\*(C`\eP\*(C'\fR in place of \f(CW\*(C`\ep\*(C'\fR to match
492 one codepoint lacking that property.
493 .PP
494 .Vb 8
495 \& \epL, \epN, \epS, \epP, \epM, \epZ, \epC
496 \& \ep{Sk}, \ep{Ps}, \ep{Lt}
497 \& \ep{alpha}, \ep{upper}, \ep{lower}
498 \& \ep{Latin}, \ep{Greek}
499 \& \ep{script_extensions=Latin}, \ep{scx=Greek}
500 \& \ep{East_Asian_Width=Wide}, \ep{EA=W}
501 \& \ep{Line_Break=Hyphen}, \ep{LB=HY}
502 \& \ep{Numeric_Value=4}, \ep{NV=4}
503 .Ve
504 .SS "℞ 26: Custom character properties"
505 .IX Subsection "℞ 26: Custom character properties"
506 Define at compile\-time your own custom character
507 properties for use in regexes.
508 .PP
509 .Vb 2
510 \& # using private\-use characters
511 \& sub In_Tengwar { "E000\etE07F\en" }
512 \&
513 \& if (/\ep{In_Tengwar}/) { ... }
514 \&
515 \& # blending existing properties
516 \& sub Is_GraecoRoman_Title {<<\*(AqEND_OF_SET\*(Aq}
517 \& +utf8::IsLatin
518 \& +utf8::IsGreek
519 \& &utf8::IsTitle
520 \& END_OF_SET
521 \&
522 \& if (/\ep{Is_GraecoRoman_Title}/) { ... }
523 .Ve
524 .SS "℞ 27: Unicode normalization"
525 .IX Subsection "℞ 27: Unicode normalization"
526 Typically render into NFD on input and NFC on output. Using NFKC or NFKD
527 functions improves recall on searches, assuming you\*(Aqve already done to the
528 same text to be searched. Note that this is about much more than just pre\-
529 combined compatibility glyphs; it also reorders marks according to their
530 canonical combining classes and weeds out singletons.
531 .PP
532 .Vb 5
533 \& use Unicode::Normalize;
534 \& my $nfd = NFD($orig);
535 \& my $nfc = NFC($orig);
536 \& my $nfkd = NFKD($orig);
537 \& my $nfkc = NFKC($orig);
538 .Ve
539 .SS "℞ 28: Convert non\-ASCII Unicode numerics"
540 .IX Subsection "℞ 28: Convert non-ASCII Unicode numerics"
541 Unless you’ve used \f(CW\*(C`/a\*(C'\fR or \f(CW\*(C`/aa\*(C'\fR, \f(CW\*(C`\ed\*(C'\fR matches more than
542 ASCII digits only, but Perl’s implicit string\-to\-number
543 conversion does not currently recognize these. Here’s how to
544 convert such strings manually.
545 .PP
546 .Vb 9
547 \& use utf8;
548 \& use v5.14; # needed for num() function
549 \& use Unicode::UCD qw(num);
550 \& my $str = "got Ⅻ and ४५६७ and ⅞ and 兆 here";
551 \& my @nums = ();
552 \& while ($str =~ /(\ed+|\eN)/g) { # not just ASCII!
553 \& push @nums, num($1);
554 \& }
555 \& say "@nums"; # 12 4567 0.875 1000000000000
556 \&
557 \& use charnames qw(:full);
558 \& my $nv = num("\eN{VAI DIGIT ONE}\eN{VAI DIGIT TWO}");
559 \& say $nv; # 12
560 .Ve
561 .SS "℞ 29: Match Unicode grapheme cluster in regex"
562 .IX Subsection "℞ 29: Match Unicode grapheme cluster in regex"
563 Programmer\-visible “characters” are codepoints matched by \f(CW\*(C`/./s\*(C'\fR,
564 but user\-visible “characters” are graphemes matched by \f(CW\*(C`/\eX/\*(C'\fR.
565 .PP
566 .Vb 3
567 \& # Find vowel *plus* any combining diacritics, underlining, etc.
568 \& my $nfd = NFD($orig);
569 \& $nfd =~ / (?=[aeiou]) \eX /xi
570 .Ve
571 .SS "℞ 30: Extract by grapheme instead of by codepoint (regex)"
572 .IX Subsection "℞ 30: Extract by grapheme instead of by codepoint (regex)"
573 .Vb 2
574 \& # match and grab five first graphemes
575 \& my($first_five) = $str =~ /^ ( \eX{5} ) /x;
576 .Ve
577 .SS "℞ 31: Extract by grapheme instead of by codepoint (substr)"
578 .IX Subsection "℞ 31: Extract by grapheme instead of by codepoint (substr)"
579 .Vb 4
580 \& # cpan \-i Unicode::GCString
581 \& use Unicode::GCString;
582 \& my $gcs = Unicode::GCString\->new($str);
583 \& my $first_five = $gcs\->substr(0, 5);
584 .Ve
585 .SS "℞ 32: Reverse string by grapheme"
586 .IX Subsection "℞ 32: Reverse string by grapheme"
587 Reversing by codepoint messes up diacritics, mistakenly converting
588 \&\f(CW\*(C`crème brûlée\*(C'\fR into \f(CW\*(C`éel̂urb em̀erc\*(C'\fR instead of into \f(CW\*(C`eélûrb emèrc\*(C'\fR;
589 so reverse by grapheme instead. Both these approaches work
590 right no matter what normalization the string is in:
591 .PP
592 .Vb 1
593 \& $str = join("", reverse $str =~ /\eX/g);
594 \&
595 \& # OR: cpan \-i Unicode::GCString
596 \& use Unicode::GCString;
597 \& $str = reverse Unicode::GCString\->new($str);
598 .Ve
599 .SS "℞ 33: String length in graphemes"
600 .IX Subsection "℞ 33: String length in graphemes"
601 The string \f(CW\*(C`brûlée\*(C'\fR has six graphemes but up to eight codepoints.
602 This counts by grapheme, not by codepoint:
603 .PP
604 .Vb 3
605 \& my $str = "brûlée";
606 \& my $count = 0;
607 \& while ($str =~ /\eX/g) { $count++ }
608 \&
609 \& # OR: cpan \-i Unicode::GCString
610 \& use Unicode::GCString;
611 \& my $gcs = Unicode::GCString\->new($str);
612 \& my $count = $gcs\->length;
613 .Ve
614 .SS "℞ 34: Unicode column\-width for printing"
615 .IX Subsection "℞ 34: Unicode column-width for printing"
616 Perl’s \f(CW\*(C`printf\*(C'\fR, \f(CW\*(C`sprintf\*(C'\fR, and \f(CW\*(C`format\*(C'\fR think all
617 codepoints take up 1 print column, but many take 0 or 2.
618 Here to show that normalization makes no difference,
619 we print out both forms:
620 .PP
621 .Vb 2
622 \& use Unicode::GCString;
623 \& use Unicode::Normalize;
624 \&
625 \& my @words = qw/crème brûlée/;
626 \& @words = map { NFC($_), NFD($_) } @words;
627 \&
628 \& for my $str (@words) {
629 \& my $gcs = Unicode::GCString\->new($str);
630 \& my $cols = $gcs\->columns;
631 \& my $pad = " " x (10 \- $cols);
632 \& say $str, $pad, " |";
633 \& }
634 .Ve
635 .PP
636 generates this to show that it pads correctly no matter
637 the normalization:
638 .PP
639 .Vb 4
640 \& crème |
641 \& crème |
642 \& brûlée |
643 \& brûlée |
644 .Ve
645 .SS "℞ 35: Unicode collation"
646 .IX Subsection "℞ 35: Unicode collation"
647 Text sorted by numeric codepoint follows no reasonable alphabetic order;
648 use the UCA for sorting text.
649 .PP
650 .Vb 3
651 \& use Unicode::Collate;
652 \& my $col = Unicode::Collate\->new();
653 \& my @list = $col\->sort(@old_list);
654 .Ve
655 .PP
656 See the \fIucsort\fR program from the Unicode::Tussle CPAN module
657 for a convenient command\-line interface to this module.
658 .SS "℞ 36: Case\- \fIand\fP accent\-insensitive Unicode sort"
659 .IX Subsection "℞ 36: Case- and accent-insensitive Unicode sort"
660 Specify a collation strength of level 1 to ignore case and
661 diacritics, only looking at the basic character.
662 .PP
663 .Vb 3
664 \& use Unicode::Collate;
665 \& my $col = Unicode::Collate\->new(level => 1);
666 \& my @list = $col\->sort(@old_list);
667 .Ve
668 .SS "℞ 37: Unicode locale collation"
669 .IX Subsection "℞ 37: Unicode locale collation"
670 Some locales have special sorting rules.
671 .PP
672 .Vb 4
673 \& # either use v5.12, OR: cpan \-i Unicode::Collate::Locale
674 \& use Unicode::Collate::Locale;
675 \& my $col = Unicode::Collate::Locale\->new(locale => "de_\|_phonebook");
676 \& my @list = $col\->sort(@old_list);
677 .Ve
678 .PP
679 The \fIucsort\fR program mentioned above accepts a \f(CW\*(C`\-\-locale\*(C'\fR parameter.
680 .ie n .SS "℞ 38: Making ""cmp"" work on text instead of codepoints"
681 .el .SS "℞ 38: Making \f(CWcmp\fP work on text instead of codepoints"
682 .IX Subsection "℞ 38: Making cmp work on text instead of codepoints"
683 Instead of this:
684 .PP
685 .Vb 5
686 \& @srecs = sort {
687 \& $b\->{AGE} <=> $a\->{AGE}
688 \& ||
689 \& $a\->{NAME} cmp $b\->{NAME}
690 \& } @recs;
691 .Ve
692 .PP
693 Use this:
694 .PP
695 .Vb 9
696 \& my $coll = Unicode::Collate\->new();
697 \& for my $rec (@recs) {
698 \& $rec\->{NAME_key} = $coll\->getSortKey( $rec\->{NAME} );
699 \& }
700 \& @srecs = sort {
701 \& $b\->{AGE} <=> $a\->{AGE}
702 \& ||
703 \& $a\->{NAME_key} cmp $b\->{NAME_key}
704 \& } @recs;
705 .Ve
706 .SS "℞ 39: Case\- \fIand\fP accent\-insensitive comparisons"
707 .IX Subsection "℞ 39: Case- and accent-insensitive comparisons"
708 Use a collator object to compare Unicode text by character
709 instead of by codepoint.
710 .PP
711 .Vb 5
712 \& use Unicode::Collate;
713 \& my $es = Unicode::Collate\->new(
714 \& level => 1,
715 \& normalization => undef
716 \& );
717 \&
718 \& # now both are true:
719 \& $es\->eq("García", "GARCIA" );
720 \& $es\->eq("Márquez", "MARQUEZ");
721 .Ve
722 .SS "℞ 40: Case\- \fIand\fP accent\-insensitive locale comparisons"
723 .IX Subsection "℞ 40: Case- and accent-insensitive locale comparisons"
724 Same, but in a specific locale.
725 .PP
726 .Vb 3
727 \& my $de = Unicode::Collate::Locale\->new(
728 \& locale => "de_\|_phonebook",
729 \& );
730 \&
731 \& # now this is true:
732 \& $de\->eq("tschüß", "TSCHUESS"); # notice ü => UE, ß => SS
733 .Ve
734 .SS "℞ 41: Unicode linebreaking"
735 .IX Subsection "℞ 41: Unicode linebreaking"
736 Break up text into lines according to Unicode rules.
737 .PP
738 .Vb 3
739 \& # cpan \-i Unicode::LineBreak
740 \& use Unicode::LineBreak;
741 \& use charnames qw(:full);
742 \&
743 \& my $para = "This is a super\eN{HYPHEN}long string. " x 20;
744 \& my $fmt = Unicode::LineBreak\->new;
745 \& print $fmt\->break($para), "\en";
746 .Ve
747 .SS "℞ 42: Unicode text in DBM hashes, the tedious way"
748 .IX Subsection "℞ 42: Unicode text in DBM hashes, the tedious way"
749 Using a regular Perl string as a key or value for a DBM
750 hash will trigger a wide character exception if any codepoints
751 won’t fit into a byte. Here’s how to manually manage the translation:
752 .PP
753 .Vb 3
754 \& use DB_File;
755 \& use Encode qw(encode decode);
756 \& tie %dbhash, "DB_File", "pathname";
757 \&
758 \& # STORE
759 \&
760 \& # assume $uni_key and $uni_value are abstract Unicode strings
761 \& my $enc_key = encode("UTF\-8", $uni_key, 1);
762 \& my $enc_value = encode("UTF\-8", $uni_value, 1);
763 \& $dbhash{$enc_key} = $enc_value;
764 \&
765 \& # FETCH
766 \&
767 \& # assume $uni_key holds a normal Perl string (abstract Unicode)
768 \& my $enc_key = encode("UTF\-8", $uni_key, 1);
769 \& my $enc_value = $dbhash{$enc_key};
770 \& my $uni_value = decode("UTF\-8", $enc_value, 1);
771 .Ve
772 .SS "℞ 43: Unicode text in DBM hashes, the easy way"
773 .IX Subsection "℞ 43: Unicode text in DBM hashes, the easy way"
774 Here’s how to implicitly manage the translation; all encoding
775 and decoding is done automatically, just as with streams that
776 have a particular encoding attached to them:
777 .PP
778 .Vb 2
779 \& use DB_File;
780 \& use DBM_Filter;
781 \&
782 \& my $dbobj = tie %dbhash, "DB_File", "pathname";
783 \& $dbobj\->Filter_Value("utf8"); # this is the magic bit
784 \&
785 \& # STORE
786 \&
787 \& # assume $uni_key and $uni_value are abstract Unicode strings
788 \& $dbhash{$uni_key} = $uni_value;
789 \&
790 \& # FETCH
791 \&
792 \& # $uni_key holds a normal Perl string (abstract Unicode)
793 \& my $uni_value = $dbhash{$uni_key};
794 .Ve
795 .SS "℞ 44: PROGRAM: Demo of Unicode collation and printing"
796 .IX Subsection "℞ 44: PROGRAM: Demo of Unicode collation and printing"
797 Here’s a full program showing how to make use of locale\-sensitive
798 sorting, Unicode casing, and managing print widths when some of the
799 characters take up zero or two columns, not just one column each time.
800 When run, the following program produces this nicely aligned output:
801 .PP
802 .Vb 10
803 \& Crème Brûlée....... €2.00
804 \& Éclair............. €1.60
805 \& Fideuà............. €4.20
806 \& Hamburger.......... €6.00
807 \& Jamón Serrano...... €4.45
808 \& Linguiça........... €7.00
809 \& Pâté............... €4.15
810 \& Pears.............. €2.00
811 \& Pêches............. €2.25
812 \& Smørbrød........... €5.75
813 \& Spätzle............ €5.50
814 \& Xoriço............. €3.00
815 \& Γύρος.............. €6.50
816 \& 막걸리............. €4.00
817 \& おもち............. €2.65
818 \& お好み焼き......... €8.00
819 \& シュークリーム..... €1.85
820 \& 寿司............... €9.99
821 \& 包子............... €7.50
822 .Ve
823 .PP
824 Here\*(Aqs that program.
825 .PP
826 .Vb 10
827 \& #!/usr/bin/env perl
828 \& # umenu \- demo sorting and printing of Unicode food
829 \& #
830 \& # (obligatory and increasingly long preamble)
831 \& #
832 \& use v5.36;
833 \& use utf8;
834 \& use warnings qw(FATAL utf8); # fatalize encoding faults
835 \& use open qw(:std :encoding(UTF\-8)); # undeclared streams in UTF\-8
836 \& use charnames qw(:full :short); # unneeded in v5.16
837 \&
838 \& # std modules
839 \& use Unicode::Normalize; # std perl distro as of v5.8
840 \& use List::Util qw(max); # std perl distro as of v5.10
841 \& use Unicode::Collate::Locale; # std perl distro as of v5.14
842 \&
843 \& # cpan modules
844 \& use Unicode::GCString; # from CPAN
845 \&
846 \& my %price = (
847 \& "γύρος" => 6.50, # gyros
848 \& "pears" => 2.00, # like um, pears
849 \& "linguiça" => 7.00, # spicy sausage, Portuguese
850 \& "xoriço" => 3.00, # chorizo sausage, Catalan
851 \& "hamburger" => 6.00, # burgermeister meisterburger
852 \& "éclair" => 1.60, # dessert, French
853 \& "smørbrød" => 5.75, # sandwiches, Norwegian
854 \& "spätzle" => 5.50, # Bayerisch noodles, little sparrows
855 \& "包子" => 7.50, # bao1 zi5, steamed pork buns, Mandarin
856 \& "jamón serrano" => 4.45, # country ham, Spanish
857 \& "pêches" => 2.25, # peaches, French
858 \& "シュークリーム" => 1.85, # cream\-filled pastry like eclair
859 \& "막걸리" => 4.00, # makgeolli, Korean rice wine
860 \& "寿司" => 9.99, # sushi, Japanese
861 \& "おもち" => 2.65, # omochi, rice cakes, Japanese
862 \& "crème brûlée" => 2.00, # crema catalana
863 \& "fideuà" => 4.20, # more noodles, Valencian
864 \& # (Catalan=fideuada)
865 \& "pâté" => 4.15, # goose liver paste, French
866 \& "お好み焼き" => 8.00, # okonomiyaki, Japanese
867 \& );
868 \&
869 \& my $width = 5 + max map { colwidth($_) } keys %price;
870 \&
871 \& # So the Asian stuff comes out in an order that someone
872 \& # who reads those scripts won\*(Aqt freak out over; the
873 \& # CJK stuff will be in JIS X 0208 order that way.
874 \& my $coll = Unicode::Collate::Locale\->new(locale => "ja");
875 \&
876 \& for my $item ($coll\->sort(keys %price)) {
877 \& print pad(entitle($item), $width, ".");
878 \& printf " €%.2f\en", $price{$item};
879 \& }
880 \&
881 \& sub pad ($str, $width, $padchar) {
882 \& return $str . ($padchar x ($width \- colwidth($str)));
883 \& }
884 \&
885 \& sub colwidth ($str) {
886 \& return Unicode::GCString\->new($str)\->columns;
887 \& }
888 \&
889 \& sub entitle ($str) {
890 \& $str =~ s{ (?=\epL)(\eS) (\eS*) }
891 \& { ucfirst($1) . lc($2) }xge;
892 \& return $str;
893 \& }
894 .Ve
895 .SH "SEE ALSO"
896 .IX Header "SEE ALSO"
897 See these manpages, some of which are CPAN modules:
898 perlunicode, perluniprops,
899 perlre, perlrecharclass,
900 perluniintro, perlunitut, perlunifaq,
901 PerlIO, DB_File, DBM_Filter, DBM_Filter::utf8,
902 Encode, Encode::Locale,
903 Unicode::UCD,
904 Unicode::Normalize,
905 Unicode::GCString, Unicode::LineBreak,
906 Unicode::Collate, Unicode::Collate::Locale,
907 Unicode::Unihan,
908 Unicode::CaseFold,
909 Unicode::Tussle,
910 Lingua::JA::Romanize::Japanese,
911 Lingua::ZH::Romanize::Pinyin,
912 Lingua::KO::Romanize::Hangul.
913 .PP
914 The Unicode::Tussle CPAN module includes many programs
915 to help with working with Unicode, including
916 these programs to fully or partly replace standard utilities:
917 \&\fItcgrep\fR instead of \fIegrep\fR,
918 \&\fIuniquote\fR instead of \fIcat \-v\fR or \fIhexdump\fR,
919 \&\fIuniwc\fR instead of \fIwc\fR,
920 \&\fIunilook\fR instead of \fIlook\fR,
921 \&\fIunifmt\fR instead of \fIfmt\fR,
922 and
923 \&\fIucsort\fR instead of \fIsort\fR.
924 For exploring Unicode character names and character properties,
925 see its \fIuniprops\fR, \fIunichars\fR, and \fIuninames\fR programs.
926 It also supplies these programs, all of which are general filters that do Unicode\-y things:
927 \&\fIunititle\fR and \fIunicaps\fR;
928 \&\fIuniwide\fR and \fIuninarrow\fR;
929 \&\fIunisupers\fR and \fIunisubs\fR;
930 \&\fInfd\fR, \fInfc\fR, \fInfkd\fR, and \fInfkc\fR;
931 and \fIuc\fR, \fIlc\fR, and \fItc\fR.
932 .PP
933 Finally, see the published Unicode Standard (page numbers are from version
934 6.0.0), including these specific annexes and technical reports:
935 .IP "§3.13 Default Case Algorithms, page 113; §4.2 Case, pages 120–122; Case Mappings, page 166–172, especially Caseless Matching starting on page 170." 4
936 .IX Item "§3.13 Default Case Algorithms, page 113; §4.2 Case, pages 120–122; Case Mappings, page 166–172, especially Caseless Matching starting on page 170."
937 .PD 0
938 .IP "UAX #44: Unicode Character Database" 4
939 .IX Item "UAX #44: Unicode Character Database"
940 .IP "UTS #18: Unicode Regular Expressions" 4
941 .IX Item "UTS #18: Unicode Regular Expressions"
942 .IP "UAX #15: Unicode Normalization Forms" 4
943 .IX Item "UAX #15: Unicode Normalization Forms"
944 .IP "UTS #10: Unicode Collation Algorithm" 4
945 .IX Item "UTS #10: Unicode Collation Algorithm"
946 .IP "UAX #29: Unicode Text Segmentation" 4
947 .IX Item "UAX #29: Unicode Text Segmentation"
948 .IP "UAX #14: Unicode Line Breaking Algorithm" 4
949 .IX Item "UAX #14: Unicode Line Breaking Algorithm"
950 .IP "UAX #11: East Asian Width" 4
951 .IX Item "UAX #11: East Asian Width"
952 .PD
953 .SH AUTHOR
954 .IX Header "AUTHOR"
955 Tom Christiansen <[email protected]> wrote this, with occasional
956 kibbitzing from Larry Wall and Jeffrey Friedl in the background.
957 .SH "COPYRIGHT AND LICENCE"
958 .IX Header "COPYRIGHT AND LICENCE"
959 Copyright © 2012 Tom Christiansen.
960 .PP
961 This program is free software; you may redistribute it and/or modify it
962 under the same terms as Perl itself.
963 .PP
964 Most of these examples taken from the current edition of the “Camel Book”;
965 that is, from the 4ᵗʰ Edition of \fIProgramming Perl\fR, Copyright © 2012 Tom
966 Christiansen <et al.>, 2012\-02\-13 by O’Reilly Media. The code itself is
967 freely redistributable, and you are encouraged to transplant, fold,
968 spindle, and mutilate any of the examples in this manpage however you please
969 for inclusion into your own programs without any encumbrance whatsoever.
970 Acknowledgement via code comment is polite but not required.
971 .SH "REVISION HISTORY"
972 .IX Header "REVISION HISTORY"
973 v1.0.0 – first public release, 2012\-02\-27