Debian Codegraph @

upstream/mageia-cauldron/man1/perluniintro.1

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man v6.0.2 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\"
56 .\" Required to disable full justification in groff 1.23.0.
57 .if n .ds AD l
58 .\" ========================================================================
59 .\"
60 .IX Title "PERLUNIINTRO 1"
61 .TH PERLUNIINTRO 1 2023-12-22 "perl v5.42.0" "Perl Programmers Reference Guide"
62 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
63 .\" way too many mistakes in technical documents.
64 .if n .ad l
65 .nh
66 .SH NAME
67 perluniintro \- Perl Unicode introduction
68 .SH DESCRIPTION
69 .IX Header "DESCRIPTION"
70 This document gives a general idea of Unicode and how to use Unicode
71 in Perl. See "Further Resources" for references to more in\-depth
72 treatments of Unicode.
73 .SS Unicode
74 .IX Subsection "Unicode"
75 Unicode is a character set standard which plans to codify all of the
76 writing systems of the world, plus many other symbols.
77 .PP
78 Unicode and ISO/IEC 10646 are coordinated standards that unify
79 almost all other modern character set standards,
80 covering more than 80 writing systems and hundreds of languages,
81 including all commercially\-important modern languages. All characters
82 in the largest Chinese, Japanese, and Korean dictionaries are also
83 encoded. The standards will eventually cover almost all characters in
84 more than 250 writing systems and thousands of languages.
85 Unicode 1.0 was released in October 1991, and 6.0 in October 2010.
86 .PP
87 A Unicode \fIcharacter\fR is an abstract entity. It is not bound to any
88 particular integer width, especially not to the C language \f(CW\*(C`char\*(C'\fR.
89 Unicode is language\-neutral and display\-neutral: it does not encode the
90 language of the text, and it does not generally define fonts or other graphical
91 layout details. Unicode operates on characters and on text built from
92 those characters.
93 .PP
94 Unicode defines characters like \f(CW\*(C`LATIN CAPITAL LETTER A\*(C'\fR or \f(CW\*(C`GREEK
95 SMALL LETTER ALPHA\*(C'\fR and unique numbers for the characters, in this
96 case 0x0041 and 0x03B1, respectively. These unique numbers are called
97 \&\fIcode points\fR. A code point is essentially the position of the
98 character within the set of all possible Unicode characters, and thus in
99 Perl, the term \fIordinal\fR is often used interchangeably with it.
100 .PP
101 The Unicode standard prefers using hexadecimal notation for the code
102 points. If numbers like \f(CW0x0041\fR are unfamiliar to you, take a peek
103 at a later section, "Hexadecimal Notation". The Unicode standard
104 uses the notation \f(CW\*(C`U+0041 LATIN CAPITAL LETTER A\*(C'\fR, to give the
105 hexadecimal code point and the normative name of the character.
106 .PP
107 Unicode also defines various \fIproperties\fR for the characters, like
108 "uppercase" or "lowercase", "decimal digit", or "punctuation";
109 these properties are independent of the names of the characters.
110 Furthermore, various operations on the characters like uppercasing,
111 lowercasing, and collating (sorting) are defined.
112 .PP
113 A Unicode \fIlogical\fR "character" can actually consist of more than one internal
114 \&\fIactual\fR "character" or code point. For Western languages, this is adequately
115 modelled by a \fIbase character\fR (like \f(CW\*(C`LATIN CAPITAL LETTER A\*(C'\fR) followed
116 by one or more \fImodifiers\fR (like \f(CW\*(C`COMBINING ACUTE ACCENT\*(C'\fR). This sequence of
117 base character and modifiers is called a \fIcombining character
118 sequence\fR. Some non\-western languages require more complicated
119 models, so Unicode created the \fIgrapheme cluster\fR concept, which was
120 later further refined into the \fIextended grapheme cluster\fR. For
121 example, a Korean Hangul syllable is considered a single logical
122 character, but most often consists of three actual
123 Unicode characters: a leading consonant followed by an interior vowel followed
124 by a trailing consonant.
125 .PP
126 Whether to call these extended grapheme clusters "characters" depends on your
127 point of view. If you are a programmer, you probably would tend towards seeing
128 each element in the sequences as one unit, or "character". However from
129 the user\*(Aqs point of view, the whole sequence could be seen as one
130 "character" since that\*(Aqs probably what it looks like in the context of the
131 user\*(Aqs language. In this document, we take the programmer\*(Aqs point of
132 view: one "character" is one Unicode code point.
133 .PP
134 For some combinations of base character and modifiers, there are
135 \&\fIprecomposed\fR characters. There is a single character equivalent, for
136 example, for the sequence \f(CW\*(C`LATIN CAPITAL LETTER A\*(C'\fR followed by
137 \&\f(CW\*(C`COMBINING ACUTE ACCENT\*(C'\fR. It is called \f(CW\*(C`LATIN CAPITAL LETTER A WITH
138 ACUTE\*(C'\fR. These precomposed characters are, however, only available for
139 some combinations, and are mainly meant to support round\-trip
140 conversions between Unicode and legacy standards (like ISO 8859). Using
141 sequences, as Unicode does, allows for needing fewer basic building blocks
142 (code points) to express many more potential grapheme clusters. To
143 support conversion between equivalent forms, various \fInormalization
144 forms\fR are also defined. Thus, \f(CW\*(C`LATIN CAPITAL LETTER A WITH ACUTE\*(C'\fR is
145 in \fINormalization Form Composed\fR, (abbreviated NFC), and the sequence
146 \&\f(CW\*(C`LATIN CAPITAL LETTER A\*(C'\fR followed by \f(CW\*(C`COMBINING ACUTE ACCENT\*(C'\fR
147 represents the same character in \fINormalization Form Decomposed\fR (NFD).
148 .PP
149 Because of backward compatibility with legacy encodings, the "a unique
150 number for every character" idea breaks down a bit: instead, there is
151 "at least one number for every character". The same character could
152 be represented differently in several legacy encodings. The
153 converse is not true: some code points do not have an assigned
154 character. Firstly, there are unallocated code points within
155 otherwise used blocks. Secondly, there are special Unicode control
156 characters that do not represent true characters.
157 .PP
158 When Unicode was first conceived, it was thought that all the world\*(Aqs
159 characters could be represented using a 16\-bit word; that is a maximum of
160 \&\f(CW0x10000\fR (or 65,536) characters would be needed, from \f(CW0x0000\fR to
161 \&\f(CW0xFFFF\fR. This soon proved to be wrong, and since Unicode 2.0 (July
162 1996), Unicode has been defined all the way up to 21 bits (\f(CW0x10FFFF\fR),
163 and Unicode 3.1 (March 2001) defined the first characters above \f(CW0xFFFF\fR.
164 The first \f(CW0x10000\fR characters are called the \fIPlane 0\fR, or the
165 \&\fIBasic Multilingual Plane\fR (BMP). With Unicode 3.1, 17 (yes,
166 seventeen) planes in all were defined\-\-but they are nowhere near full of
167 defined characters, yet.
168 .PP
169 When a new language is being encoded, Unicode generally will choose a
170 \&\f(CW\*(C`block\*(C'\fR of consecutive unallocated code points for its characters. So
171 far, the number of code points in these blocks has always been evenly
172 divisible by 16. Extras in a block, not currently needed, are left
173 unallocated, for future growth. But there have been occasions when
174 a later release needed more code points than the available extras, and a
175 new block had to allocated somewhere else, not contiguous to the initial
176 one, to handle the overflow. Thus, it became apparent early on that
177 "block" wasn\*(Aqt an adequate organizing principle, and so the \f(CW\*(C`Script\*(C'\fR
178 property was created. (Later an improved script property was added as
179 well, the \f(CW\*(C`Script_Extensions\*(C'\fR property.) Those code points that are in
180 overflow blocks can still
181 have the same script as the original ones. The script concept fits more
182 closely with natural language: there is \f(CW\*(C`Latin\*(C'\fR script, \f(CW\*(C`Greek\*(C'\fR
183 script, and so on; and there are several artificial scripts, like
184 \&\f(CW\*(C`Common\*(C'\fR for characters that are used in multiple scripts, such as
185 mathematical symbols. Scripts usually span varied parts of several
186 blocks. For more information about scripts, see "Scripts" in perlunicode.
187 The division into blocks exists, but it is almost completely
188 accidental\-\-an artifact of how the characters have been and still are
189 allocated. (Note that this paragraph has oversimplified things for the
190 sake of this being an introduction. Unicode doesn\*(Aqt really encode
191 languages, but the writing systems for them\-\-their scripts; and one
192 script can be used by many languages. Unicode also encodes things that
193 aren\*(Aqt really about languages, such as symbols like \f(CW\*(C`BAGGAGE CLAIM\*(C'\fR.)
194 .PP
195 The Unicode code points are just abstract numbers. To input and
196 output these abstract numbers, the numbers must be \fIencoded\fR or
197 \&\fIserialised\fR somehow. Unicode defines several \fIcharacter encoding
198 forms\fR, of which \fIUTF\-8\fR is the most popular. UTF\-8 is a
199 variable length encoding that encodes Unicode characters as 1 to 4
200 bytes. Other encodings
201 include UTF\-16 and UTF\-32 and their big\- and little\-endian variants
202 (UTF\-8 is byte\-order independent). The ISO/IEC 10646 defines the UCS\-2
203 and UCS\-4 encoding forms.
204 .PP
205 For more information about encodings\-\-for instance, to learn what
206 \&\fIsurrogates\fR and \fIbyte order marks\fR (BOMs) are\-\-see perlunicode.
207 .SS "Perl\*(Aqs Unicode Support"
208 .IX Subsection "Perl's Unicode Support"
209 Starting from Perl v5.6.0, Perl has had the capacity to handle Unicode
210 natively. Perl v5.8.0, however, is the first recommended release for
211 serious Unicode work. The maintenance release 5.6.1 fixed many of the
212 problems of the initial Unicode implementation, but for example
213 regular expressions still do not work with Unicode in 5.6.1.
214 Perl v5.14.0 is the first release where Unicode support is
215 (almost) seamlessly integratable without some gotchas. (There are a few
216 exceptions. Firstly, some differences in quotemeta
217 were fixed starting in Perl 5.16.0. Secondly, some differences in
218 the range operator were fixed starting in
219 Perl 5.26.0. Thirdly, some differences in split were fixed
220 started in Perl 5.28.0.)
221 .PP
222 To enable this
223 seamless support, you should \f(CW\*(C`use feature \*(Aqunicode_strings\*(Aq\*(C'\fR (which is
224 automatically selected if you \f(CW\*(C`use v5.12\*(C'\fR or higher). See feature.
225 (5.14 also fixes a number of bugs and departures from the Unicode
226 standard.)
227 .PP
228 Before Perl v5.8.0, the use of \f(CW\*(C`use utf8\*(C'\fR was used to declare
229 that operations in the current block or file would be Unicode\-aware.
230 This model was found to be wrong, or at least clumsy: the "Unicodeness"
231 is now carried with the data, instead of being attached to the
232 operations.
233 Starting with Perl v5.8.0, only one case remains where an explicit \f(CW\*(C`use
234 utf8\*(C'\fR is needed: if your Perl script itself is encoded in UTF\-8, you can
235 use UTF\-8 in your identifier names, and in string and regular expression
236 literals, by saying \f(CW\*(C`use utf8\*(C'\fR. This is not the default because
237 scripts with legacy 8\-bit data in them would break. See utf8.
238 .SS "Perl\*(Aqs Unicode Model"
239 .IX Subsection "Perl's Unicode Model"
240 Perl supports both pre\-5.6 strings of eight\-bit native bytes, and
241 strings of Unicode characters. The general principle is that Perl tries
242 to keep its data as eight\-bit bytes for as long as possible, but as soon
243 as Unicodeness cannot be avoided, the data is transparently upgraded
244 to Unicode. Prior to Perl v5.14.0, the upgrade was not completely
245 transparent (see "The "Unicode Bug"" in perlunicode), and for backwards
246 compatibility, full transparency is not gained unless \f(CWuse feature
247 \&\*(Aqunicode_strings\*(Aq\fR (see feature) or \f(CW\*(C`use v5.12\*(C'\fR (or higher) is
248 selected.
249 .PP
250 Internally, Perl currently uses either whatever the native eight\-bit
251 character set of the platform (for example Latin\-1) is, defaulting to
252 UTF\-8, to encode Unicode strings. Specifically, if all code points in
253 the string are \f(CW0xFF\fR or less, Perl uses the native eight\-bit
254 character set. Otherwise, it uses UTF\-8.
255 .PP
256 A user of Perl does not normally need to know nor care how Perl
257 happens to encode its internal strings, but it becomes relevant when
258 outputting Unicode strings to a stream without a PerlIO layer (one with
259 the "default" encoding). In such a case, the raw bytes used internally
260 (the native character set or UTF\-8, as appropriate for each string)
261 will be used, and a "Wide character" warning will be issued if those
262 strings contain a character beyond 0x00FF.
263 .PP
264 For example,
265 .PP
266 .Vb 1
267 \& perl \-e \*(Aqprint "\ex{DF}\en", "\ex{0100}\ex{DF}\en"\*(Aq
268 .Ve
269 .PP
270 produces a fairly useless mixture of native bytes and UTF\-8, as well
271 as a warning:
272 .PP
273 .Vb 1
274 \& Wide character in print at ...
275 .Ve
276 .PP
277 To output UTF\-8, use the \f(CW\*(C`:encoding\*(C'\fR or \f(CW\*(C`:utf8\*(C'\fR output layer. Prepending
278 .PP
279 .Vb 1
280 \& binmode(STDOUT, ":utf8");
281 .Ve
282 .PP
283 to this sample program ensures that the output is completely UTF\-8,
284 and removes the program\*(Aqs warning.
285 .PP
286 You can enable automatic UTF\-8\-ification of your standard file
287 handles, default \f(CWopen()\fR layer, and \f(CW@ARGV\fR by using either
288 the \f(CW\*(C`\-C\*(C'\fR command line switch or the \f(CW\*(C`PERL_UNICODE\*(C'\fR environment
289 variable, see perlrun for the
290 documentation of the \f(CW\*(C`\-C\*(C'\fR switch.
291 .PP
292 Note that this means that Perl expects other software to work the same
293 way:
294 if Perl has been led to believe that STDIN should be UTF\-8, but then
295 STDIN coming in from another command is not UTF\-8, Perl will likely
296 complain about the malformed UTF\-8.
297 .PP
298 All features that combine Unicode and I/O also require using the new
299 PerlIO feature. Almost all Perl 5.8 platforms do use PerlIO, though:
300 you can see whether yours is by running "perl \-V" and looking for
301 \&\f(CW\*(C`useperlio=define\*(C'\fR.
302 .SS "Unicode and EBCDIC"
303 .IX Subsection "Unicode and EBCDIC"
304 Perl 5.8.0 added support for Unicode on EBCDIC platforms. This support
305 was allowed to lapse in later releases, but was revived in 5.22.
306 Unicode support is somewhat more complex to implement since additional
307 conversions are needed. See perlebcdic for more information.
308 .PP
309 On EBCDIC platforms, the internal Unicode encoding form is UTF\-EBCDIC
310 instead of UTF\-8. The difference is that as UTF\-8 is "ASCII\-safe" in
311 that ASCII characters encode to UTF\-8 as\-is, while UTF\-EBCDIC is
312 "EBCDIC\-safe", in that all the basic characters (which includes all
313 those that have ASCII equivalents (like \f(CW"A"\fR, \f(CW"0"\fR, \f(CW"%"\fR, \fIetc.\fR)
314 are the same in both EBCDIC and UTF\-EBCDIC. Often, documentation
315 will use the term "UTF\-8" to mean UTF\-EBCDIC as well. This is the case
316 in this document.
317 .SS "Creating Unicode"
318 .IX Subsection "Creating Unicode"
319 This section applies fully to Perls starting with v5.22. Various
320 caveats for earlier releases are in the "Earlier releases caveats"
321 subsection below.
322 .PP
323 To create Unicode characters in literals,
324 use the \f(CW\*(C`\eN{...}\*(C'\fR notation in double\-quoted strings:
325 .PP
326 .Vb 2
327 \& my $smiley_from_name = "\eN{WHITE SMILING FACE}";
328 \& my $smiley_from_code_point = "\eN{U+263a}";
329 .Ve
330 .PP
331 Similarly, they can be used in regular expression literals
332 .PP
333 .Vb 2
334 \& $smiley =~ /\eN{WHITE SMILING FACE}/;
335 \& $smiley =~ /\eN{U+263a}/;
336 .Ve
337 .PP
338 or, starting in v5.32:
339 .PP
340 .Vb 2
341 \& $smiley =~ /\ep{Name=WHITE SMILING FACE}/;
342 \& $smiley =~ /\ep{Name=whitesmilingface}/;
343 .Ve
344 .PP
345 At run\-time you can use:
346 .PP
347 .Vb 4
348 \& use charnames ();
349 \& my $hebrew_alef_from_name
350 \& = charnames::string_vianame("HEBREW LETTER ALEF");
351 \& my $hebrew_alef_from_code_point = charnames::string_vianame("U+05D0");
352 .Ve
353 .PP
354 Naturally, \f(CWord()\fR will do the reverse: it turns a character into
355 a code point.
356 .PP
357 There are other runtime options as well. You can use \f(CWpack()\fR:
358 .PP
359 .Vb 1
360 \& my $hebrew_alef_from_code_point = pack("U", 0x05d0);
361 .Ve
362 .PP
363 Or you can use \f(CWchr()\fR, though it is less convenient in the general
364 case:
365 .PP
366 .Vb 2
367 \& $hebrew_alef_from_code_point = chr(utf8::unicode_to_native(0x05d0));
368 \& utf8::upgrade($hebrew_alef_from_code_point);
369 .Ve
370 .PP
371 The \f(CWutf8::unicode_to_native()\fR and \f(CWutf8::upgrade()\fR aren\*(Aqt needed if
372 the argument is above 0xFF, so the above could have been written as
373 .PP
374 .Vb 1
375 \& $hebrew_alef_from_code_point = chr(0x05d0);
376 .Ve
377 .PP
378 since 0x5d0 is above 255.
379 .PP
380 \&\f(CW\*(C`\ex{}\*(C'\fR and \f(CW\*(C`\eo{}\*(C'\fR can also be used to specify code points at compile
381 time in double\-quotish strings, but, for backward compatibility with
382 older Perls, the same rules apply as with \f(CWchr()\fR for code points less
383 than 256.
384 .PP
385 \&\f(CWutf8::unicode_to_native()\fR is used so that the Perl code is portable
386 to EBCDIC platforms. You can omit it if you\*(Aqre \fIreally\fR sure no one
387 will ever want to use your code on a non\-ASCII platform. Starting in
388 Perl v5.22, calls to it on ASCII platforms are optimized out, so there\*(Aqs
389 no performance penalty at all in adding it. Or you can simply use the
390 other constructs that don\*(Aqt require it.
391 .PP
392 See "Further Resources" for how to find all these names and numeric
393 codes.
394 .PP
395 \fIEarlier releases caveats\fR
396 .IX Subsection "Earlier releases caveats"
397 .PP
398 On EBCDIC platforms, prior to v5.22, using \f(CW\*(C`\eN{U+...}\*(C'\fR doesn\*(Aqt work
399 properly.
400 .PP
401 Prior to v5.16, using \f(CW\*(C`\eN{...}\*(C'\fR with a character name (as opposed to a
402 \&\f(CW\*(C`U+...\*(C'\fR code point) required a \f(CW\*(C`use\ charnames\ :full\*(C'\fR.
403 .PP
404 Prior to v5.14, there were some bugs in \f(CW\*(C`\eN{...}\*(C'\fR with a character name
405 (as opposed to a \f(CW\*(C`U+...\*(C'\fR code point).
406 .PP
407 \&\f(CWcharnames::string_vianame()\fR was introduced in v5.14. Prior to that,
408 \&\f(CWcharnames::vianame()\fR should work, but only if the argument is of the
409 form \f(CW"U+..."\fR. Your best bet there for runtime Unicode by character
410 name is probably:
411 .PP
412 .Vb 3
413 \& use charnames ();
414 \& my $hebrew_alef_from_name
415 \& = pack("U", charnames::vianame("HEBREW LETTER ALEF"));
416 .Ve
417 .SS "Handling Unicode"
418 .IX Subsection "Handling Unicode"
419 Handling Unicode is for the most part transparent: just use the
420 strings as usual. Functions like \f(CWindex()\fR, \f(CWlength()\fR, and
421 \&\f(CWsubstr()\fR will work on the Unicode characters; regular expressions
422 will work on the Unicode characters (see perlunicode and perlretut).
423 .PP
424 Note that Perl considers grapheme clusters to be separate characters, so for
425 example
426 .PP
427 .Vb 2
428 \& print length("\eN{LATIN CAPITAL LETTER A}\eN{COMBINING ACUTE ACCENT}"),
429 \& "\en";
430 .Ve
431 .PP
432 will print 2, not 1. The only exception is that regular expressions
433 have \f(CW\*(C`\eX\*(C'\fR for matching an extended grapheme cluster. (Thus \f(CW\*(C`\eX\*(C'\fR in a
434 regular expression would match the entire sequence of both the example
435 characters.)
436 .PP
437 Life is not quite so transparent, however, when working with legacy
438 encodings, I/O, and certain special cases:
439 .SS "Legacy Encodings"
440 .IX Subsection "Legacy Encodings"
441 When you combine legacy data and Unicode, the legacy data needs
442 to be upgraded to Unicode. Normally the legacy data is assumed to be
443 ISO 8859\-1 (or EBCDIC, if applicable).
444 .PP
445 The \f(CW\*(C`Encode\*(C'\fR module knows about many encodings and has interfaces
446 for doing conversions between those encodings:
447 .PP
448 .Vb 2
449 \& use Encode \*(Aqdecode\*(Aq;
450 \& $data = decode("iso\-8859\-3", $data); # convert from legacy
451 .Ve
452 .SS "Unicode I/O"
453 .IX Subsection "Unicode I/O"
454 Normally, writing out Unicode data
455 .PP
456 .Vb 1
457 \& print FH $some_string_with_unicode, "\en";
458 .Ve
459 .PP
460 produces raw bytes that Perl happens to use to internally encode the
461 Unicode string. Perl\*(Aqs internal encoding depends on the system as
462 well as what characters happen to be in the string at the time. If
463 any of the characters are at code points \f(CW0x100\fR or above, you will get
464 a warning. To ensure that the output is explicitly rendered in the
465 encoding you desire\-\-and to avoid the warning\-\-open the stream with
466 the desired encoding. Some examples:
467 .PP
468 .Vb 1
469 \& open FH, ">:utf8", "file";
470 \&
471 \& open FH, ">:encoding(ucs2)", "file";
472 \& open FH, ">:encoding(UTF\-8)", "file";
473 \& open FH, ">:encoding(shift_jis)", "file";
474 .Ve
475 .PP
476 and on already open streams, use \f(CWbinmode()\fR:
477 .PP
478 .Vb 1
479 \& binmode(STDOUT, ":utf8");
480 \&
481 \& binmode(STDOUT, ":encoding(ucs2)");
482 \& binmode(STDOUT, ":encoding(UTF\-8)");
483 \& binmode(STDOUT, ":encoding(shift_jis)");
484 .Ve
485 .PP
486 The matching of encoding names is loose: case does not matter, and
487 many encodings have several aliases. Note that the \f(CW\*(C`:utf8\*(C'\fR layer
488 must always be specified exactly like that; it is \fInot\fR subject to
489 the loose matching of encoding names. Also note that currently \f(CW\*(C`:utf8\*(C'\fR is unsafe for
490 input, because it accepts the data without validating that it is indeed valid
491 UTF\-8; you should instead use \f(CW:encoding(UTF\-8)\fR (with or without a
492 hyphen).
493 .PP
494 See PerlIO for the \f(CW\*(C`:utf8\*(C'\fR layer, PerlIO::encoding and
495 Encode::PerlIO for the \f(CW:encoding()\fR layer, and
496 Encode::Supported for many encodings supported by the \f(CW\*(C`Encode\*(C'\fR
497 module.
498 .PP
499 Reading in a file that you know happens to be encoded in one of the
500 Unicode or legacy encodings does not magically turn the data into
501 Unicode in Perl\*(Aqs eyes. To do that, specify the appropriate
502 layer when opening files
503 .PP
504 .Vb 2
505 \& open(my $fh,\*(Aq<:encoding(UTF\-8)\*(Aq, \*(Aqanything\*(Aq);
506 \& my $line_of_unicode = <$fh>;
507 \&
508 \& open(my $fh,\*(Aq<:encoding(Big5)\*(Aq, \*(Aqanything\*(Aq);
509 \& my $line_of_unicode = <$fh>;
510 .Ve
511 .PP
512 The I/O layers can also be specified more flexibly with
513 the \f(CW\*(C`open\*(C'\fR pragma. See open, or look at the following example.
514 .PP
515 .Vb 8
516 \& use open \*(Aq:encoding(UTF\-8)\*(Aq; # input/output default encoding will be
517 \& # UTF\-8
518 \& open X, ">file";
519 \& print X chr(0x100), "\en";
520 \& close X;
521 \& open Y, "<file";
522 \& printf "%#x\en", ord(<Y>); # this should print 0x100
523 \& close Y;
524 .Ve
525 .PP
526 With the \f(CW\*(C`open\*(C'\fR pragma you can use the \f(CW\*(C`:locale\*(C'\fR layer
527 .PP
528 .Vb 10
529 \& BEGIN { $ENV{LC_ALL} = $ENV{LANG} = \*(Aqru_RU.KOI8\-R\*(Aq }
530 \& # the :locale will probe the locale environment variables like
531 \& # LC_ALL
532 \& use open OUT => \*(Aq:locale\*(Aq; # russki parusski
533 \& open(O, ">koi8");
534 \& print O chr(0x430); # Unicode CYRILLIC SMALL LETTER A = KOI8\-R 0xc1
535 \& close O;
536 \& open(I, "<koi8");
537 \& printf "%#x\en", ord(<I>), "\en"; # this should print 0xc1
538 \& close I;
539 .Ve
540 .PP
541 These methods install a transparent filter on the I/O stream that
542 converts data from the specified encoding when it is read in from the
543 stream. The result is always Unicode.
544 .PP
545 The open pragma affects all the \f(CWopen()\fR calls after the pragma by
546 setting default layers. If you want to affect only certain
547 streams, use explicit layers directly in the \f(CWopen()\fR call.
548 .PP
549 You can switch encodings on an already opened stream by using
550 \&\f(CWbinmode()\fR; see "binmode" in perlfunc.
551 .PP
552 The \f(CW\*(C`:locale\*(C'\fR does not currently work with
553 \&\f(CWopen()\fR and \f(CWbinmode()\fR, only with the \f(CW\*(C`open\*(C'\fR pragma. The
554 \&\f(CW\*(C`:utf8\*(C'\fR and \f(CW:encoding(...)\fR methods do work with all of \f(CWopen()\fR,
555 \&\f(CWbinmode()\fR, and the \f(CW\*(C`open\*(C'\fR pragma.
556 .PP
557 Similarly, you may use these I/O layers on output streams to
558 automatically convert Unicode to the specified encoding when it is
559 written to the stream. For example, the following snippet copies the
560 contents of the file "text.jis" (encoded as ISO\-2022\-JP, aka JIS) to
561 the file "text.utf8", encoded as UTF\-8:
562 .PP
563 .Vb 3
564 \& open(my $nihongo, \*(Aq<:encoding(iso\-2022\-jp)\*(Aq, \*(Aqtext.jis\*(Aq);
565 \& open(my $unicode, \*(Aq>:utf8\*(Aq, \*(Aqtext.utf8\*(Aq);
566 \& while (<$nihongo>) { print $unicode $_ }
567 .Ve
568 .PP
569 The naming of encodings, both by the \f(CWopen()\fR and by the \f(CW\*(C`open\*(C'\fR
570 pragma allows for flexible names: \f(CW\*(C`koi8\-r\*(C'\fR and \f(CW\*(C`KOI8R\*(C'\fR will both be
571 understood.
572 .PP
573 Common encodings recognized by ISO, MIME, IANA, and various other
574 standardisation organisations are recognised; for a more detailed
575 list see Encode::Supported.
576 .PP
577 \&\f(CWread()\fR reads characters and returns the number of characters.
578 \&\f(CWseek()\fR and \f(CWtell()\fR operate on byte counts, as does \f(CWsysseek()\fR.
579 .PP
580 \&\f(CWsysread()\fR and \f(CWsyswrite()\fR should not be used on file handles with
581 character encoding layers, they behave badly, and that behaviour has
582 been deprecated since perl 5.24.
583 .PP
584 Notice that because of the default behaviour of not doing any
585 conversion upon input if there is no default layer,
586 it is easy to mistakenly write code that keeps on expanding a file
587 by repeatedly encoding the data:
588 .PP
589 .Vb 8
590 \& # BAD CODE WARNING
591 \& open F, "file";
592 \& local $/; ## read in the whole file of 8\-bit characters
593 \& $t = <F>;
594 \& close F;
595 \& open F, ">:encoding(UTF\-8)", "file";
596 \& print F $t; ## convert to UTF\-8 on output
597 \& close F;
598 .Ve
599 .PP
600 If you run this code twice, the contents of the \fIfile\fR will be twice
601 UTF\-8 encoded. A \f(CW\*(C`use open \*(Aq:encoding(UTF\-8)\*(Aq\*(C'\fR would have avoided the
602 bug, or explicitly opening also the \fIfile\fR for input as UTF\-8.
603 .PP
604 \&\fBNOTE\fR: the \f(CW\*(C`:utf8\*(C'\fR and \f(CW\*(C`:encoding\*(C'\fR features work only if your
605 Perl has been built with PerlIO, which is the default
606 on most systems.
607 .SS "Displaying Unicode As Text"
608 .IX Subsection "Displaying Unicode As Text"
609 Sometimes you might want to display Perl scalars containing Unicode as
610 simple ASCII (or EBCDIC) text. The following subroutine converts
611 its argument so that Unicode characters with code points greater than
612 255 are displayed as \f(CW\*(C`\ex{...}\*(C'\fR, control characters (like \f(CW\*(C`\en\*(C'\fR) are
613 displayed as \f(CW\*(C`\ex..\*(C'\fR, and the rest of the characters as themselves:
614 .PP
615 .Vb 9
616 \& sub nice_string {
617 \& join("",
618 \& map { $_ > 255 # if wide character...
619 \& ? sprintf("\e\ex{%04X}", $_) # \ex{...}
620 \& : chr($_) =~ /[[:cntrl:]]/ # else if control character...
621 \& ? sprintf("\e\ex%02X", $_) # \ex..
622 \& : quotemeta(chr($_)) # else quoted or as themselves
623 \& } unpack("W*", $_[0])); # unpack Unicode characters
624 \& }
625 .Ve
626 .PP
627 For example,
628 .PP
629 .Vb 1
630 \& nice_string("foo\ex{100}bar\en")
631 .Ve
632 .PP
633 returns the string
634 .PP
635 .Vb 1
636 \& \*(Aqfoo\ex{0100}bar\ex0A\*(Aq
637 .Ve
638 .PP
639 which is ready to be printed.
640 .PP
641 (\f(CW\*(C`\e\ex{}\*(C'\fR is used here instead of \f(CW\*(C`\e\eN{}\*(C'\fR, since it\*(Aqs most likely that
642 you want to see what the native values are.)
643 .SS "Special Cases"
644 .IX Subsection "Special Cases"
645 .IP \(bu 4
646 Starting in Perl 5.28, it is illegal for bit operators, like \f(CW\*(C`~\*(C'\fR, to
647 operate on strings containing code points above 255.
648 .IP \(bu 4
649 The \fBvec()\fR function may produce surprising results if
650 used on strings containing characters with ordinal values above
651 255. In such a case, the results are consistent with the internal
652 encoding of the characters, but not with much else. So don\*(Aqt do
653 that, and starting in Perl 5.28, a deprecation message is issued if you
654 do so, becoming illegal in Perl 5.32.
655 .IP \(bu 4
656 Peeking At Perl\*(Aqs Internal Encoding
657 .Sp
658 Normal users of Perl should never care how Perl encodes any particular
659 Unicode string (because the normal ways to get at the contents of a
660 string with Unicode\-\-via input and output\-\-should always be via
661 explicitly\-defined I/O layers). But if you must, there are two
662 ways of looking behind the scenes.
663 .Sp
664 One way of peeking inside the internal encoding of Unicode characters
665 is to use \f(CW\*(C`unpack("C*", ...\*(C'\fR to get the bytes of whatever the string
666 encoding happens to be, or \f(CW\*(C`unpack("U0..", ...)\*(C'\fR to get the bytes of the
667 UTF\-8 encoding:
668 .Sp
669 .Vb 2
670 \& # this prints c4 80 for the UTF\-8 bytes 0xc4 0x80
671 \& print join(" ", unpack("U0(H2)*", pack("U", 0x100))), "\en";
672 .Ve
673 .Sp
674 Yet another way would be to use the Devel::Peek module:
675 .Sp
676 .Vb 1
677 \& perl \-MDevel::Peek \-e \*(AqDump(chr(0x100))\*(Aq
678 .Ve
679 .Sp
680 That shows the \f(CW\*(C`UTF8\*(C'\fR flag in FLAGS and both the UTF\-8 bytes
681 and Unicode characters in \f(CW\*(C`PV\*(C'\fR. See also later in this document
682 the discussion about the \f(CWutf8::is_utf8()\fR function.
683 .SS "Advanced Topics"
684 .IX Subsection "Advanced Topics"
685 .IP \(bu 4
686 String Equivalence
687 .Sp
688 The question of string equivalence turns somewhat complicated
689 in Unicode: what do you mean by "equal"?
690 .Sp
691 (Is \f(CW\*(C`LATIN CAPITAL LETTER A WITH ACUTE\*(C'\fR equal to
692 \&\f(CW\*(C`LATIN CAPITAL LETTER A\*(C'\fR?)
693 .Sp
694 The short answer is that by default Perl compares equivalence (\f(CW\*(C`eq\*(C'\fR,
695 \&\f(CW\*(C`ne\*(C'\fR) based only on code points of the characters. In the above
696 case, the answer is no (because 0x00C1 != 0x0041). But sometimes, any
697 CAPITAL LETTER A\*(Aqs should be considered equal, or even A\*(Aqs of any case.
698 .Sp
699 The long answer is that you need to consider character normalization
700 and casing issues: see Unicode::Normalize, Unicode Technical Report #15,
701 Unicode Normalization Forms <https://www.unicode.org/reports/tr15> and
702 sections on case mapping in the Unicode Standard <https://www.unicode.org>.
703 .Sp
704 As of Perl 5.8.0, the "Full" case\-folding of \fICase
705 Mappings/SpecialCasing\fR is implemented, but bugs remain in \f(CW\*(C`qr//i\*(C'\fR with them,
706 mostly fixed by 5.14, and essentially entirely by 5.18.
707 .IP \(bu 4
708 String Collation
709 .Sp
710 People like to see their strings nicely sorted\-\-or as Unicode
711 parlance goes, collated. But again, what do you mean by collate?
712 .Sp
713 (Does \f(CW\*(C`LATIN CAPITAL LETTER A WITH ACUTE\*(C'\fR come before or after
714 \&\f(CW\*(C`LATIN CAPITAL LETTER A WITH GRAVE\*(C'\fR?)
715 .Sp
716 The short answer is that by default, Perl compares strings (\f(CW\*(C`lt\*(C'\fR,
717 \&\f(CW\*(C`le\*(C'\fR, \f(CW\*(C`cmp\*(C'\fR, \f(CW\*(C`ge\*(C'\fR, \f(CW\*(C`gt\*(C'\fR) based only on the code points of the
718 characters. In the above case, the answer is "after", since
719 \&\f(CW0x00C1\fR > \f(CW0x00C0\fR.
720 .Sp
721 The long answer is that "it depends", and a good answer cannot be
722 given without knowing (at the very least) the language context.
723 See Unicode::Collate, and \fIUnicode Collation Algorithm\fR
724 <https://www.unicode.org/reports/tr10/>
725 .SS Miscellaneous
726 .IX Subsection "Miscellaneous"
727 .IP \(bu 4
728 Character Ranges and Classes
729 .Sp
730 Character ranges in regular expression bracketed character classes ( e.g.,
731 \&\f(CW\*(C`/[a\-z]/\*(C'\fR) and in the \f(CW\*(C`tr///\*(C'\fR (also known as \f(CW\*(C`y///\*(C'\fR) operator are not
732 magically Unicode\-aware. What this means is that \f(CW\*(C`[A\-Za\-z]\*(C'\fR will not
733 magically start to mean "all alphabetic letters" (not that it does mean that
734 even for 8\-bit characters; for those, if you are using locales (perllocale),
735 use \f(CW\*(C`/[[:alpha:]]/\*(C'\fR; and if not, use the 8\-bit\-aware property \f(CW\*(C`\ep{alpha}\*(C'\fR).
736 .Sp
737 All the properties that begin with \f(CW\*(C`\ep\*(C'\fR (and its inverse \f(CW\*(C`\eP\*(C'\fR) are actually
738 character classes that are Unicode\-aware. There are dozens of them, see
739 perluniprops.
740 .Sp
741 Starting in v5.22, you can use Unicode code points as the end points of
742 regular expression pattern character ranges, and the range will include
743 all Unicode code points that lie between those end points, inclusive.
744 .Sp
745 .Vb 1
746 \& qr/ [ \eN{U+03} \- \eN{U+20} ] /xx
747 .Ve
748 .Sp
749 includes the code points
750 \&\f(CW\*(C`\eN{U+03}\*(C'\fR, \f(CW\*(C`\eN{U+04}\*(C'\fR, ..., \f(CW\*(C`\eN{U+20}\*(C'\fR.
751 .Sp
752 This also works for ranges in \f(CW\*(C`tr///\*(C'\fR starting in Perl v5.24.
753 .IP \(bu 4
754 String\-To\-Number Conversions
755 .Sp
756 Unicode does define several other decimal\-\-and numeric\-\-characters
757 besides the familiar 0 to 9, such as the Arabic and Indic digits.
758 Perl does not support string\-to\-number conversion for digits other
759 than ASCII \f(CW0\fR to \f(CW9\fR (and ASCII \f(CW\*(C`a\*(C'\fR to \f(CW\*(C`f\*(C'\fR for hexadecimal).
760 To get safe conversions from any Unicode string, use
761 "\fBnum()\fR" in Unicode::UCD.
762 .SS "Questions With Answers"
763 .IX Subsection "Questions With Answers"
764 .IP \(bu 4
765 Will My Old Scripts Break?
766 .Sp
767 Very probably not. Unless you are generating Unicode characters
768 somehow, old behaviour should be preserved. About the only behaviour
769 that has changed and which could start generating Unicode is the old
770 behaviour of \f(CWchr()\fR where supplying an argument more than 255
771 produced a character modulo 255. \f(CWchr(300)\fR, for example, was equal
772 to \f(CWchr(45)\fR or "\-" (in ASCII), now it is LATIN CAPITAL LETTER I WITH
773 BREVE.
774 .IP \(bu 4
775 How Do I Make My Scripts Work With Unicode?
776 .Sp
777 Very little work should be needed since nothing changes until you
778 generate Unicode data. The most important thing is getting input as
779 Unicode; for that, see the earlier I/O discussion.
780 To get full seamless Unicode support, add
781 \&\f(CW\*(C`use feature \*(Aqunicode_strings\*(Aq\*(C'\fR (or \f(CW\*(C`use v5.12\*(C'\fR or higher) to your
782 script.
783 .IP \(bu 4
784 How Do I Know Whether My String Is In Unicode?
785 .Sp
786 You shouldn\*(Aqt have to care. But you may if your Perl is before 5.14.0
787 or you haven\*(Aqt specified \f(CW\*(C`use feature \*(Aqunicode_strings\*(Aq\*(C'\fR or \f(CWuse
788 5.012\fR (or higher) because otherwise the rules for the code points
789 in the range 128 to 255 are different depending on
790 whether the string they are contained within is in Unicode or not.
791 (See "When Unicode Does Not Happen" in perlunicode.)
792 .Sp
793 To determine if a string is in Unicode, use:
794 .Sp
795 .Vb 1
796 \& print utf8::is_utf8($string) ? 1 : 0, "\en";
797 .Ve
798 .Sp
799 But note that this doesn\*(Aqt mean that any of the characters in the
800 string are necessary UTF\-8 encoded, or that any of the characters have
801 code points greater than 0xFF (255) or even 0x80 (128), or that the
802 string has any characters at all. All the \f(CWis_utf8()\fR does is to
803 return the value of the internal "utf8ness" flag attached to the
804 \&\f(CW$string\fR. If the flag is off, the bytes in the scalar are interpreted
805 as a single byte encoding. If the flag is on, the bytes in the scalar
806 are interpreted as the (variable\-length, potentially multi\-byte) UTF\-8 encoded
807 code points of the characters. Bytes added to a UTF\-8 encoded string are
808 automatically upgraded to UTF\-8. If mixed non\-UTF\-8 and UTF\-8 scalars
809 are merged (double\-quoted interpolation, explicit concatenation, or
810 printf/sprintf parameter substitution), the result will be UTF\-8 encoded
811 as if copies of the byte strings were upgraded to UTF\-8: for example,
812 .Sp
813 .Vb 3
814 \& $x = "ab\ex80c";
815 \& $y = "\ex{100}";
816 \& print "$x = $y\en";
817 .Ve
818 .Sp
819 the output string will be UTF\-8\-encoded \f(CW\*(C`ab\ex80c = \ex{100}\en\*(C'\fR, but
820 \&\f(CW$x\fR will stay byte\-encoded.
821 .Sp
822 Sometimes you might really need to know the byte length of a string
823 instead of the character length. For that use the \f(CW\*(C`bytes\*(C'\fR pragma
824 and the \f(CWlength()\fR function:
825 .Sp
826 .Vb 6
827 \& my $unicode = chr(0x100);
828 \& print length($unicode), "\en"; # will print 1
829 \& use bytes;
830 \& print length($unicode), "\en"; # will print 2
831 \& # (the 0xC4 0x80 of the UTF\-8)
832 \& no bytes;
833 .Ve
834 .IP \(bu 4
835 How Do I Find Out What Encoding a File Has?
836 .Sp
837 You might try Encode::Guess, but it has a number of limitations.
838 .IP \(bu 4
839 How Do I Detect Data That\*(Aqs Not Valid In a Particular Encoding?
840 .Sp
841 Use the \f(CW\*(C`Encode\*(C'\fR package to try converting it.
842 For example,
843 .Sp
844 .Vb 1
845 \& use Encode \*(Aqdecode\*(Aq;
846 \&
847 \& if (eval { decode(\*(AqUTF\-8\*(Aq, $string, Encode::FB_CROAK); 1 }) {
848 \& # $string is valid UTF\-8
849 \& } else {
850 \& # $string is not valid UTF\-8
851 \& }
852 .Ve
853 .Sp
854 Or use \f(CW\*(C`unpack\*(C'\fR to try decoding it:
855 .Sp
856 .Vb 2
857 \& use warnings;
858 \& @chars = unpack("C0U*", $string_of_bytes_that_I_think_is_utf8);
859 .Ve
860 .Sp
861 If invalid, a \f(CW\*(C`Malformed UTF\-8 character\*(C'\fR warning is produced. The "C0" means
862 "process the string character per character". Without that, the
863 \&\f(CW\*(C`unpack("U*", ...)\*(C'\fR would work in \f(CW\*(C`U0\*(C'\fR mode (the default if the format
864 string starts with \f(CW\*(C`U\*(C'\fR) and it would return the bytes making up the UTF\-8
865 encoding of the target string, something that will always work.
866 .IP \(bu 4
867 How Do I Convert Binary Data Into a Particular Encoding, Or Vice Versa?
868 .Sp
869 This probably isn\*(Aqt as useful as you might think.
870 Normally, you shouldn\*(Aqt need to.
871 .Sp
872 In one sense, what you are asking doesn\*(Aqt make much sense: encodings
873 are for characters, and binary data are not "characters", so converting
874 "data" into some encoding isn\*(Aqt meaningful unless you know in what
875 character set and encoding the binary data is in, in which case it\*(Aqs
876 not just binary data, now is it?
877 .Sp
878 If you have a raw sequence of bytes that you know should be
879 interpreted via a particular encoding, you can use \f(CW\*(C`Encode\*(C'\fR:
880 .Sp
881 .Vb 2
882 \& use Encode \*(Aqfrom_to\*(Aq;
883 \& from_to($data, "iso\-8859\-1", "UTF\-8"); # from latin\-1 to UTF\-8
884 .Ve
885 .Sp
886 The call to \f(CWfrom_to()\fR changes the bytes in \f(CW$data\fR, but nothing
887 material about the nature of the string has changed as far as Perl is
888 concerned. Both before and after the call, the string \f(CW$data\fR
889 contains just a bunch of 8\-bit bytes. As far as Perl is concerned,
890 the encoding of the string remains as "system\-native 8\-bit bytes".
891 .Sp
892 You might relate this to a fictional \*(AqTranslate\*(Aq module:
893 .Sp
894 .Vb 4
895 \& use Translate;
896 \& my $phrase = "Yes";
897 \& Translate::from_to($phrase, \*(Aqenglish\*(Aq, \*(Aqdeutsch\*(Aq);
898 \& ## phrase now contains "Ja"
899 .Ve
900 .Sp
901 The contents of the string changes, but not the nature of the string.
902 Perl doesn\*(Aqt know any more after the call than before that the
903 contents of the string indicates the affirmative.
904 .Sp
905 Back to converting data. If you have (or want) data in your system\*(Aqs
906 native 8\-bit encoding (e.g. Latin\-1, EBCDIC, etc.), you can use
907 pack/unpack to convert to/from Unicode.
908 .Sp
909 .Vb 2
910 \& $native_string = pack("W*", unpack("U*", $Unicode_string));
911 \& $Unicode_string = pack("U*", unpack("W*", $native_string));
912 .Ve
913 .Sp
914 If you have a sequence of bytes you \fBknow\fR is valid UTF\-8,
915 but Perl doesn\*(Aqt know it yet, you can make Perl a believer, too:
916 .Sp
917 .Vb 2
918 \& $Unicode = $bytes;
919 \& utf8::decode($Unicode);
920 .Ve
921 .Sp
922 or:
923 .Sp
924 .Vb 1
925 \& $Unicode = pack("U0a*", $bytes);
926 .Ve
927 .Sp
928 You can find the bytes that make up a UTF\-8 sequence with
929 .Sp
930 .Vb 1
931 \& @bytes = unpack("C*", $Unicode_string)
932 .Ve
933 .Sp
934 and you can create well\-formed Unicode with
935 .Sp
936 .Vb 1
937 \& $Unicode_string = pack("U*", 0xff, ...)
938 .Ve
939 .IP \(bu 4
940 How Do I Display Unicode? How Do I Input Unicode?
941 .Sp
942 See <http://www.alanwood.net/unicode/> and
943 <http://www.cl.cam.ac.uk/~mgk25/unicode.html>
944 .IP \(bu 4
945 How Does Unicode Work With Traditional Locales?
946 .Sp
947 If your locale is a UTF\-8 locale, starting in Perl v5.26, Perl works
948 well for all categories; before this, starting with Perl v5.20, it works
949 for all categories but \f(CW\*(C`LC_COLLATE\*(C'\fR, which deals with
950 sorting and the \f(CW\*(C`cmp\*(C'\fR operator. But note that the standard
951 \&\f(CW\*(C`Unicode::Collate\*(C'\fR and \f(CW\*(C`Unicode::Collate::Locale\*(C'\fR modules offer
952 much more powerful solutions to collation issues, and work on earlier
953 releases.
954 .Sp
955 For other locales, starting in Perl 5.16, you can specify
956 .Sp
957 .Vb 1
958 \& use locale \*(Aq:not_characters\*(Aq;
959 .Ve
960 .Sp
961 to get Perl to work well with them. The catch is that you
962 have to translate from the locale character set to/from Unicode
963 yourself. See "Unicode I/O" above for how to
964 .Sp
965 .Vb 1
966 \& use open \*(Aq:locale\*(Aq;
967 .Ve
968 .Sp
969 to accomplish this, but full details are in "Unicode and
970 UTF\-8" in perllocale, including gotchas that happen if you don\*(Aqt specify
971 \&\f(CW\*(C`:not_characters\*(C'\fR.
972 .SS "Hexadecimal Notation"
973 .IX Subsection "Hexadecimal Notation"
974 The Unicode standard prefers using hexadecimal notation because
975 that more clearly shows the division of Unicode into blocks of 256 characters.
976 Hexadecimal is also simply shorter than decimal. You can use decimal
977 notation, too, but learning to use hexadecimal just makes life easier
978 with the Unicode standard. The \f(CW\*(C`U+HHHH\*(C'\fR notation uses hexadecimal,
979 for example.
980 .PP
981 The \f(CW\*(C`0x\*(C'\fR prefix means a hexadecimal number, the digits are 0\-9 \fIand\fR
982 a\-f (or A\-F, case doesn\*(Aqt matter). Each hexadecimal digit represents
983 four bits, or half a byte. \f(CW\*(C`print 0x..., "\en"\*(C'\fR will show a
984 hexadecimal number in decimal, and \f(CW\*(C`printf "%x\en", $decimal\*(C'\fR will
985 show a decimal number in hexadecimal. If you have just the
986 "hex digits" of a hexadecimal number, you can use the \f(CWhex()\fR function.
987 .PP
988 .Vb 6
989 \& print 0x0009, "\en"; # 9
990 \& print 0x000a, "\en"; # 10
991 \& print 0x000f, "\en"; # 15
992 \& print 0x0010, "\en"; # 16
993 \& print 0x0011, "\en"; # 17
994 \& print 0x0100, "\en"; # 256
995 \&
996 \& print 0x0041, "\en"; # 65
997 \&
998 \& printf "%x\en", 65; # 41
999 \& printf "%#x\en", 65; # 0x41
1000 \&
1001 \& print hex("41"), "\en"; # 65
1002 .Ve
1003 .SS "Further Resources"
1004 .IX Subsection "Further Resources"
1005 .IP \(bu 4
1006 Unicode Consortium
1007 .Sp
1008 <https://www.unicode.org/>
1009 .IP \(bu 4
1010 Unicode FAQ
1011 .Sp
1012 <https://www.unicode.org/faq/>
1013 .IP \(bu 4
1014 Unicode Glossary
1015 .Sp
1016 <https://www.unicode.org/glossary/>
1017 .IP \(bu 4
1018 Unicode Recommended Reading List
1019 .Sp
1020 The Unicode Consortium has a list of articles and books, some of which
1021 give a much more in depth treatment of Unicode:
1022 <http://unicode.org/resources/readinglist.html>
1023 .IP \(bu 4
1024 Unicode Useful Resources
1025 .Sp
1026 <https://www.unicode.org/unicode/onlinedat/resources.html>
1027 .IP \(bu 4
1028 Unicode and Multilingual Support in HTML, Fonts, Web Browsers and Other Applications
1029 .Sp
1030 <http://www.alanwood.net/unicode/>
1031 .IP \(bu 4
1032 UTF\-8 and Unicode FAQ for Unix/Linux
1033 .Sp
1034 <http://www.cl.cam.ac.uk/~mgk25/unicode.html>
1035 .IP \(bu 4
1036 Legacy Character Sets
1037 .Sp
1038 <http://www.czyborra.com/>
1039 <http://www.eki.ee/letter/>
1040 .IP \(bu 4
1041 You can explore various information from the Unicode data files using
1042 the \f(CW\*(C`Unicode::UCD\*(C'\fR module.
1043 .SH "UNICODE IN OLDER PERLS"
1044 .IX Header "UNICODE IN OLDER PERLS"
1045 If you cannot upgrade your Perl to 5.8.0 or later, you can still
1046 do some Unicode processing by using the modules \f(CW\*(C`Unicode::String\*(C'\fR,
1047 \&\f(CW\*(C`Unicode::Map8\*(C'\fR, and \f(CW\*(C`Unicode::Map\*(C'\fR, available from CPAN.
1048 If you have the GNU recode installed, you can also use the
1049 Perl front\-end \f(CW\*(C`Convert::Recode\*(C'\fR for character conversions.
1050 .PP
1051 The following are fast conversions from ISO 8859\-1 (Latin\-1) bytes
1052 to UTF\-8 bytes and back, the code works even with older Perl 5 versions.
1053 .PP
1054 .Vb 2
1055 \& # ISO 8859\-1 to UTF\-8
1056 \& s/([\ex80\-\exFF])/chr(0xC0|ord($1)>>6).chr(0x80|ord($1)&0x3F)/eg;
1057 \&
1058 \& # UTF\-8 to ISO 8859\-1
1059 \& s/([\exC2\exC3])([\ex80\-\exBF])/chr(ord($1)<<6&0xC0|ord($2)&0x3F)/eg;
1060 .Ve
1061 .SH "SEE ALSO"
1062 .IX Header "SEE ALSO"
1063 perlunitut, perlunicode, Encode, open, utf8, bytes,
1064 perlretut, perlrun, Unicode::Collate, Unicode::Normalize,
1065 Unicode::UCD
1066 .SH ACKNOWLEDGMENTS
1067 .IX Header "ACKNOWLEDGMENTS"
1068 Thanks to the kind readers of the perl5\[email protected],
1069 perl\[email protected], linux\[email protected], and [email protected]
1070 mailing lists for their valuable feedback.
1071 .SH "AUTHOR, COPYRIGHT, AND LICENSE"
1072 .IX Header "AUTHOR, COPYRIGHT, AND LICENSE"
1073 Copyright 2001\-2011 Jarkko Hietaniemi <[email protected]>.
1074 Now maintained by Perl 5 Porters.
1075 .PP
1076 This document may be distributed under the same terms as Perl itself.