Debian Codegraph @

upstream/debian-unstable/man1/perluniintro.1

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man 5.0102 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\" ========================================================================
56 .\"
57 .IX Title "PERLUNIINTRO 1"
58 .TH PERLUNIINTRO 1 2025-11-16 "perl v5.40.1" "Perl Programmers Reference Guide"
59 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
60 .\" way too many mistakes in technical documents.
61 .if n .ad l
62 .nh
63 .SH NAME
64 perluniintro \- Perl Unicode introduction
65 .SH DESCRIPTION
66 .IX Header "DESCRIPTION"
67 This document gives a general idea of Unicode and how to use Unicode
68 in Perl. See "Further Resources" for references to more in-depth
69 treatments of Unicode.
70 .SS Unicode
71 .IX Subsection "Unicode"
72 Unicode is a character set standard which plans to codify all of the
73 writing systems of the world, plus many other symbols.
74 .PP
75 Unicode and ISO/IEC 10646 are coordinated standards that unify
76 almost all other modern character set standards,
77 covering more than 80 writing systems and hundreds of languages,
78 including all commercially-important modern languages. All characters
79 in the largest Chinese, Japanese, and Korean dictionaries are also
80 encoded. The standards will eventually cover almost all characters in
81 more than 250 writing systems and thousands of languages.
82 Unicode 1.0 was released in October 1991, and 6.0 in October 2010.
83 .PP
84 A Unicode \fIcharacter\fR is an abstract entity. It is not bound to any
85 particular integer width, especially not to the C language \f(CW\*(C`char\*(C'\fR.
86 Unicode is language-neutral and display-neutral: it does not encode the
87 language of the text, and it does not generally define fonts or other graphical
88 layout details. Unicode operates on characters and on text built from
89 those characters.
90 .PP
91 Unicode defines characters like \f(CW\*(C`LATIN CAPITAL LETTER A\*(C'\fR or \f(CW\*(C`GREEK
92 SMALL LETTER ALPHA\*(C'\fR and unique numbers for the characters, in this
93 case 0x0041 and 0x03B1, respectively. These unique numbers are called
94 \&\fIcode points\fR. A code point is essentially the position of the
95 character within the set of all possible Unicode characters, and thus in
96 Perl, the term \fIordinal\fR is often used interchangeably with it.
97 .PP
98 The Unicode standard prefers using hexadecimal notation for the code
99 points. If numbers like \f(CW0x0041\fR are unfamiliar to you, take a peek
100 at a later section, "Hexadecimal Notation". The Unicode standard
101 uses the notation \f(CW\*(C`U+0041 LATIN CAPITAL LETTER A\*(C'\fR, to give the
102 hexadecimal code point and the normative name of the character.
103 .PP
104 Unicode also defines various \fIproperties\fR for the characters, like
105 "uppercase" or "lowercase", "decimal digit", or "punctuation";
106 these properties are independent of the names of the characters.
107 Furthermore, various operations on the characters like uppercasing,
108 lowercasing, and collating (sorting) are defined.
109 .PP
110 A Unicode \fIlogical\fR "character" can actually consist of more than one internal
111 \&\fIactual\fR "character" or code point. For Western languages, this is adequately
112 modelled by a \fIbase character\fR (like \f(CW\*(C`LATIN CAPITAL LETTER A\*(C'\fR) followed
113 by one or more \fImodifiers\fR (like \f(CW\*(C`COMBINING ACUTE ACCENT\*(C'\fR). This sequence of
114 base character and modifiers is called a \fIcombining character
115 sequence\fR. Some non-western languages require more complicated
116 models, so Unicode created the \fIgrapheme cluster\fR concept, which was
117 later further refined into the \fIextended grapheme cluster\fR. For
118 example, a Korean Hangul syllable is considered a single logical
119 character, but most often consists of three actual
120 Unicode characters: a leading consonant followed by an interior vowel followed
121 by a trailing consonant.
122 .PP
123 Whether to call these extended grapheme clusters "characters" depends on your
124 point of view. If you are a programmer, you probably would tend towards seeing
125 each element in the sequences as one unit, or "character". However from
126 the user's point of view, the whole sequence could be seen as one
127 "character" since that's probably what it looks like in the context of the
128 user's language. In this document, we take the programmer's point of
129 view: one "character" is one Unicode code point.
130 .PP
131 For some combinations of base character and modifiers, there are
132 \&\fIprecomposed\fR characters. There is a single character equivalent, for
133 example, for the sequence \f(CW\*(C`LATIN CAPITAL LETTER A\*(C'\fR followed by
134 \&\f(CW\*(C`COMBINING ACUTE ACCENT\*(C'\fR. It is called \f(CW\*(C`LATIN CAPITAL LETTER A WITH
135 ACUTE\*(C'\fR. These precomposed characters are, however, only available for
136 some combinations, and are mainly meant to support round-trip
137 conversions between Unicode and legacy standards (like ISO 8859). Using
138 sequences, as Unicode does, allows for needing fewer basic building blocks
139 (code points) to express many more potential grapheme clusters. To
140 support conversion between equivalent forms, various \fInormalization
141 forms\fR are also defined. Thus, \f(CW\*(C`LATIN CAPITAL LETTER A WITH ACUTE\*(C'\fR is
142 in \fINormalization Form Composed\fR, (abbreviated NFC), and the sequence
143 \&\f(CW\*(C`LATIN CAPITAL LETTER A\*(C'\fR followed by \f(CW\*(C`COMBINING ACUTE ACCENT\*(C'\fR
144 represents the same character in \fINormalization Form Decomposed\fR (NFD).
145 .PP
146 Because of backward compatibility with legacy encodings, the "a unique
147 number for every character" idea breaks down a bit: instead, there is
148 "at least one number for every character". The same character could
149 be represented differently in several legacy encodings. The
150 converse is not true: some code points do not have an assigned
151 character. Firstly, there are unallocated code points within
152 otherwise used blocks. Secondly, there are special Unicode control
153 characters that do not represent true characters.
154 .PP
155 When Unicode was first conceived, it was thought that all the world's
156 characters could be represented using a 16\-bit word; that is a maximum of
157 \&\f(CW0x10000\fR (or 65,536) characters would be needed, from \f(CW0x0000\fR to
158 \&\f(CW0xFFFF\fR. This soon proved to be wrong, and since Unicode 2.0 (July
159 1996), Unicode has been defined all the way up to 21 bits (\f(CW0x10FFFF\fR),
160 and Unicode 3.1 (March 2001) defined the first characters above \f(CW0xFFFF\fR.
161 The first \f(CW0x10000\fR characters are called the \fIPlane 0\fR, or the
162 \&\fIBasic Multilingual Plane\fR (BMP). With Unicode 3.1, 17 (yes,
163 seventeen) planes in all were defined\-\-but they are nowhere near full of
164 defined characters, yet.
165 .PP
166 When a new language is being encoded, Unicode generally will choose a
167 \&\f(CW\*(C`block\*(C'\fR of consecutive unallocated code points for its characters. So
168 far, the number of code points in these blocks has always been evenly
169 divisible by 16. Extras in a block, not currently needed, are left
170 unallocated, for future growth. But there have been occasions when
171 a later release needed more code points than the available extras, and a
172 new block had to allocated somewhere else, not contiguous to the initial
173 one, to handle the overflow. Thus, it became apparent early on that
174 "block" wasn't an adequate organizing principle, and so the \f(CW\*(C`Script\*(C'\fR
175 property was created. (Later an improved script property was added as
176 well, the \f(CW\*(C`Script_Extensions\*(C'\fR property.) Those code points that are in
177 overflow blocks can still
178 have the same script as the original ones. The script concept fits more
179 closely with natural language: there is \f(CW\*(C`Latin\*(C'\fR script, \f(CW\*(C`Greek\*(C'\fR
180 script, and so on; and there are several artificial scripts, like
181 \&\f(CW\*(C`Common\*(C'\fR for characters that are used in multiple scripts, such as
182 mathematical symbols. Scripts usually span varied parts of several
183 blocks. For more information about scripts, see "Scripts" in perlunicode.
184 The division into blocks exists, but it is almost completely
185 accidental\-\-an artifact of how the characters have been and still are
186 allocated. (Note that this paragraph has oversimplified things for the
187 sake of this being an introduction. Unicode doesn't really encode
188 languages, but the writing systems for them\-\-their scripts; and one
189 script can be used by many languages. Unicode also encodes things that
190 aren't really about languages, such as symbols like \f(CW\*(C`BAGGAGE CLAIM\*(C'\fR.)
191 .PP
192 The Unicode code points are just abstract numbers. To input and
193 output these abstract numbers, the numbers must be \fIencoded\fR or
194 \&\fIserialised\fR somehow. Unicode defines several \fIcharacter encoding
195 forms\fR, of which \fIUTF\-8\fR is the most popular. UTF\-8 is a
196 variable length encoding that encodes Unicode characters as 1 to 4
197 bytes. Other encodings
198 include UTF\-16 and UTF\-32 and their big\- and little-endian variants
199 (UTF\-8 is byte-order independent). The ISO/IEC 10646 defines the UCS\-2
200 and UCS\-4 encoding forms.
201 .PP
202 For more information about encodings\-\-for instance, to learn what
203 \&\fIsurrogates\fR and \fIbyte order marks\fR (BOMs) are\-\-see perlunicode.
204 .SS "Perl's Unicode Support"
205 .IX Subsection "Perl's Unicode Support"
206 Starting from Perl v5.6.0, Perl has had the capacity to handle Unicode
207 natively. Perl v5.8.0, however, is the first recommended release for
208 serious Unicode work. The maintenance release 5.6.1 fixed many of the
209 problems of the initial Unicode implementation, but for example
210 regular expressions still do not work with Unicode in 5.6.1.
211 Perl v5.14.0 is the first release where Unicode support is
212 (almost) seamlessly integratable without some gotchas. (There are a few
213 exceptions. Firstly, some differences in quotemeta
214 were fixed starting in Perl 5.16.0. Secondly, some differences in
215 the range operator were fixed starting in
216 Perl 5.26.0. Thirdly, some differences in split were fixed
217 started in Perl 5.28.0.)
218 .PP
219 To enable this
220 seamless support, you should \f(CW\*(C`use feature \*(Aqunicode_strings\*(Aq\*(C'\fR (which is
221 automatically selected if you \f(CW\*(C`use v5.12\*(C'\fR or higher). See feature.
222 (5.14 also fixes a number of bugs and departures from the Unicode
223 standard.)
224 .PP
225 Before Perl v5.8.0, the use of \f(CW\*(C`use utf8\*(C'\fR was used to declare
226 that operations in the current block or file would be Unicode-aware.
227 This model was found to be wrong, or at least clumsy: the "Unicodeness"
228 is now carried with the data, instead of being attached to the
229 operations.
230 Starting with Perl v5.8.0, only one case remains where an explicit \f(CW\*(C`use
231 utf8\*(C'\fR is needed: if your Perl script itself is encoded in UTF\-8, you can
232 use UTF\-8 in your identifier names, and in string and regular expression
233 literals, by saying \f(CW\*(C`use utf8\*(C'\fR. This is not the default because
234 scripts with legacy 8\-bit data in them would break. See utf8.
235 .SS "Perl's Unicode Model"
236 .IX Subsection "Perl's Unicode Model"
237 Perl supports both pre\-5.6 strings of eight-bit native bytes, and
238 strings of Unicode characters. The general principle is that Perl tries
239 to keep its data as eight-bit bytes for as long as possible, but as soon
240 as Unicodeness cannot be avoided, the data is transparently upgraded
241 to Unicode. Prior to Perl v5.14.0, the upgrade was not completely
242 transparent (see "The "Unicode Bug"" in perlunicode), and for backwards
243 compatibility, full transparency is not gained unless \f(CWuse feature
244 \&\*(Aqunicode_strings\*(Aq\fR (see feature) or \f(CW\*(C`use v5.12\*(C'\fR (or higher) is
245 selected.
246 .PP
247 Internally, Perl currently uses either whatever the native eight-bit
248 character set of the platform (for example Latin\-1) is, defaulting to
249 UTF\-8, to encode Unicode strings. Specifically, if all code points in
250 the string are \f(CW0xFF\fR or less, Perl uses the native eight-bit
251 character set. Otherwise, it uses UTF\-8.
252 .PP
253 A user of Perl does not normally need to know nor care how Perl
254 happens to encode its internal strings, but it becomes relevant when
255 outputting Unicode strings to a stream without a PerlIO layer (one with
256 the "default" encoding). In such a case, the raw bytes used internally
257 (the native character set or UTF\-8, as appropriate for each string)
258 will be used, and a "Wide character" warning will be issued if those
259 strings contain a character beyond 0x00FF.
260 .PP
261 For example,
262 .PP
263 .Vb 1
264 \& perl \-e \*(Aqprint "\ex{DF}\en", "\ex{0100}\ex{DF}\en"\*(Aq
265 .Ve
266 .PP
267 produces a fairly useless mixture of native bytes and UTF\-8, as well
268 as a warning:
269 .PP
270 .Vb 1
271 \& Wide character in print at ...
272 .Ve
273 .PP
274 To output UTF\-8, use the \f(CW\*(C`:encoding\*(C'\fR or \f(CW\*(C`:utf8\*(C'\fR output layer. Prepending
275 .PP
276 .Vb 1
277 \& binmode(STDOUT, ":utf8");
278 .Ve
279 .PP
280 to this sample program ensures that the output is completely UTF\-8,
281 and removes the program's warning.
282 .PP
283 You can enable automatic UTF\-8\-ification of your standard file
284 handles, default \f(CWopen()\fR layer, and \f(CW@ARGV\fR by using either
285 the \f(CW\*(C`\-C\*(C'\fR command line switch or the \f(CW\*(C`PERL_UNICODE\*(C'\fR environment
286 variable, see perlrun for the
287 documentation of the \f(CW\*(C`\-C\*(C'\fR switch.
288 .PP
289 Note that this means that Perl expects other software to work the same
290 way:
291 if Perl has been led to believe that STDIN should be UTF\-8, but then
292 STDIN coming in from another command is not UTF\-8, Perl will likely
293 complain about the malformed UTF\-8.
294 .PP
295 All features that combine Unicode and I/O also require using the new
296 PerlIO feature. Almost all Perl 5.8 platforms do use PerlIO, though:
297 you can see whether yours is by running "perl \-V" and looking for
298 \&\f(CW\*(C`useperlio=define\*(C'\fR.
299 .SS "Unicode and EBCDIC"
300 .IX Subsection "Unicode and EBCDIC"
301 Perl 5.8.0 added support for Unicode on EBCDIC platforms. This support
302 was allowed to lapse in later releases, but was revived in 5.22.
303 Unicode support is somewhat more complex to implement since additional
304 conversions are needed. See perlebcdic for more information.
305 .PP
306 On EBCDIC platforms, the internal Unicode encoding form is UTF-EBCDIC
307 instead of UTF\-8. The difference is that as UTF\-8 is "ASCII-safe" in
308 that ASCII characters encode to UTF\-8 as-is, while UTF-EBCDIC is
309 "EBCDIC-safe", in that all the basic characters (which includes all
310 those that have ASCII equivalents (like \f(CW"A"\fR, \f(CW"0"\fR, \f(CW"%"\fR, \fIetc.\fR)
311 are the same in both EBCDIC and UTF-EBCDIC. Often, documentation
312 will use the term "UTF\-8" to mean UTF-EBCDIC as well. This is the case
313 in this document.
314 .SS "Creating Unicode"
315 .IX Subsection "Creating Unicode"
316 This section applies fully to Perls starting with v5.22. Various
317 caveats for earlier releases are in the "Earlier releases caveats"
318 subsection below.
319 .PP
320 To create Unicode characters in literals,
321 use the \f(CW\*(C`\eN{...}\*(C'\fR notation in double-quoted strings:
322 .PP
323 .Vb 2
324 \& my $smiley_from_name = "\eN{WHITE SMILING FACE}";
325 \& my $smiley_from_code_point = "\eN{U+263a}";
326 .Ve
327 .PP
328 Similarly, they can be used in regular expression literals
329 .PP
330 .Vb 2
331 \& $smiley =~ /\eN{WHITE SMILING FACE}/;
332 \& $smiley =~ /\eN{U+263a}/;
333 .Ve
334 .PP
335 or, starting in v5.32:
336 .PP
337 .Vb 2
338 \& $smiley =~ /\ep{Name=WHITE SMILING FACE}/;
339 \& $smiley =~ /\ep{Name=whitesmilingface}/;
340 .Ve
341 .PP
342 At run-time you can use:
343 .PP
344 .Vb 4
345 \& use charnames ();
346 \& my $hebrew_alef_from_name
347 \& = charnames::string_vianame("HEBREW LETTER ALEF");
348 \& my $hebrew_alef_from_code_point = charnames::string_vianame("U+05D0");
349 .Ve
350 .PP
351 Naturally, \f(CWord()\fR will do the reverse: it turns a character into
352 a code point.
353 .PP
354 There are other runtime options as well. You can use \f(CWpack()\fR:
355 .PP
356 .Vb 1
357 \& my $hebrew_alef_from_code_point = pack("U", 0x05d0);
358 .Ve
359 .PP
360 Or you can use \f(CWchr()\fR, though it is less convenient in the general
361 case:
362 .PP
363 .Vb 2
364 \& $hebrew_alef_from_code_point = chr(utf8::unicode_to_native(0x05d0));
365 \& utf8::upgrade($hebrew_alef_from_code_point);
366 .Ve
367 .PP
368 The \f(CWutf8::unicode_to_native()\fR and \f(CWutf8::upgrade()\fR aren't needed if
369 the argument is above 0xFF, so the above could have been written as
370 .PP
371 .Vb 1
372 \& $hebrew_alef_from_code_point = chr(0x05d0);
373 .Ve
374 .PP
375 since 0x5d0 is above 255.
376 .PP
377 \&\f(CW\*(C`\ex{}\*(C'\fR and \f(CW\*(C`\eo{}\*(C'\fR can also be used to specify code points at compile
378 time in double-quotish strings, but, for backward compatibility with
379 older Perls, the same rules apply as with \f(CWchr()\fR for code points less
380 than 256.
381 .PP
382 \&\f(CWutf8::unicode_to_native()\fR is used so that the Perl code is portable
383 to EBCDIC platforms. You can omit it if you're \fIreally\fR sure no one
384 will ever want to use your code on a non-ASCII platform. Starting in
385 Perl v5.22, calls to it on ASCII platforms are optimized out, so there's
386 no performance penalty at all in adding it. Or you can simply use the
387 other constructs that don't require it.
388 .PP
389 See "Further Resources" for how to find all these names and numeric
390 codes.
391 .PP
392 \fIEarlier releases caveats\fR
393 .IX Subsection "Earlier releases caveats"
394 .PP
395 On EBCDIC platforms, prior to v5.22, using \f(CW\*(C`\eN{U+...}\*(C'\fR doesn't work
396 properly.
397 .PP
398 Prior to v5.16, using \f(CW\*(C`\eN{...}\*(C'\fR with a character name (as opposed to a
399 \&\f(CW\*(C`U+...\*(C'\fR code point) required a \f(CW\*(C`use\ charnames\ :full\*(C'\fR.
400 .PP
401 Prior to v5.14, there were some bugs in \f(CW\*(C`\eN{...}\*(C'\fR with a character name
402 (as opposed to a \f(CW\*(C`U+...\*(C'\fR code point).
403 .PP
404 \&\f(CWcharnames::string_vianame()\fR was introduced in v5.14. Prior to that,
405 \&\f(CWcharnames::vianame()\fR should work, but only if the argument is of the
406 form \f(CW"U+..."\fR. Your best bet there for runtime Unicode by character
407 name is probably:
408 .PP
409 .Vb 3
410 \& use charnames ();
411 \& my $hebrew_alef_from_name
412 \& = pack("U", charnames::vianame("HEBREW LETTER ALEF"));
413 .Ve
414 .SS "Handling Unicode"
415 .IX Subsection "Handling Unicode"
416 Handling Unicode is for the most part transparent: just use the
417 strings as usual. Functions like \f(CWindex()\fR, \f(CWlength()\fR, and
418 \&\f(CWsubstr()\fR will work on the Unicode characters; regular expressions
419 will work on the Unicode characters (see perlunicode and perlretut).
420 .PP
421 Note that Perl considers grapheme clusters to be separate characters, so for
422 example
423 .PP
424 .Vb 2
425 \& print length("\eN{LATIN CAPITAL LETTER A}\eN{COMBINING ACUTE ACCENT}"),
426 \& "\en";
427 .Ve
428 .PP
429 will print 2, not 1. The only exception is that regular expressions
430 have \f(CW\*(C`\eX\*(C'\fR for matching an extended grapheme cluster. (Thus \f(CW\*(C`\eX\*(C'\fR in a
431 regular expression would match the entire sequence of both the example
432 characters.)
433 .PP
434 Life is not quite so transparent, however, when working with legacy
435 encodings, I/O, and certain special cases:
436 .SS "Legacy Encodings"
437 .IX Subsection "Legacy Encodings"
438 When you combine legacy data and Unicode, the legacy data needs
439 to be upgraded to Unicode. Normally the legacy data is assumed to be
440 ISO 8859\-1 (or EBCDIC, if applicable).
441 .PP
442 The \f(CW\*(C`Encode\*(C'\fR module knows about many encodings and has interfaces
443 for doing conversions between those encodings:
444 .PP
445 .Vb 2
446 \& use Encode \*(Aqdecode\*(Aq;
447 \& $data = decode("iso\-8859\-3", $data); # convert from legacy
448 .Ve
449 .SS "Unicode I/O"
450 .IX Subsection "Unicode I/O"
451 Normally, writing out Unicode data
452 .PP
453 .Vb 1
454 \& print FH $some_string_with_unicode, "\en";
455 .Ve
456 .PP
457 produces raw bytes that Perl happens to use to internally encode the
458 Unicode string. Perl's internal encoding depends on the system as
459 well as what characters happen to be in the string at the time. If
460 any of the characters are at code points \f(CW0x100\fR or above, you will get
461 a warning. To ensure that the output is explicitly rendered in the
462 encoding you desire\-\-and to avoid the warning\-\-open the stream with
463 the desired encoding. Some examples:
464 .PP
465 .Vb 1
466 \& open FH, ">:utf8", "file";
467 \&
468 \& open FH, ">:encoding(ucs2)", "file";
469 \& open FH, ">:encoding(UTF\-8)", "file";
470 \& open FH, ">:encoding(shift_jis)", "file";
471 .Ve
472 .PP
473 and on already open streams, use \f(CWbinmode()\fR:
474 .PP
475 .Vb 1
476 \& binmode(STDOUT, ":utf8");
477 \&
478 \& binmode(STDOUT, ":encoding(ucs2)");
479 \& binmode(STDOUT, ":encoding(UTF\-8)");
480 \& binmode(STDOUT, ":encoding(shift_jis)");
481 .Ve
482 .PP
483 The matching of encoding names is loose: case does not matter, and
484 many encodings have several aliases. Note that the \f(CW\*(C`:utf8\*(C'\fR layer
485 must always be specified exactly like that; it is \fInot\fR subject to
486 the loose matching of encoding names. Also note that currently \f(CW\*(C`:utf8\*(C'\fR is unsafe for
487 input, because it accepts the data without validating that it is indeed valid
488 UTF\-8; you should instead use \f(CW:encoding(UTF\-8)\fR (with or without a
489 hyphen).
490 .PP
491 See PerlIO for the \f(CW\*(C`:utf8\*(C'\fR layer, PerlIO::encoding and
492 Encode::PerlIO for the \f(CW:encoding()\fR layer, and
493 Encode::Supported for many encodings supported by the \f(CW\*(C`Encode\*(C'\fR
494 module.
495 .PP
496 Reading in a file that you know happens to be encoded in one of the
497 Unicode or legacy encodings does not magically turn the data into
498 Unicode in Perl's eyes. To do that, specify the appropriate
499 layer when opening files
500 .PP
501 .Vb 2
502 \& open(my $fh,\*(Aq<:encoding(UTF\-8)\*(Aq, \*(Aqanything\*(Aq);
503 \& my $line_of_unicode = <$fh>;
504 \&
505 \& open(my $fh,\*(Aq<:encoding(Big5)\*(Aq, \*(Aqanything\*(Aq);
506 \& my $line_of_unicode = <$fh>;
507 .Ve
508 .PP
509 The I/O layers can also be specified more flexibly with
510 the \f(CW\*(C`open\*(C'\fR pragma. See open, or look at the following example.
511 .PP
512 .Vb 8
513 \& use open \*(Aq:encoding(UTF\-8)\*(Aq; # input/output default encoding will be
514 \& # UTF\-8
515 \& open X, ">file";
516 \& print X chr(0x100), "\en";
517 \& close X;
518 \& open Y, "<file";
519 \& printf "%#x\en", ord(<Y>); # this should print 0x100
520 \& close Y;
521 .Ve
522 .PP
523 With the \f(CW\*(C`open\*(C'\fR pragma you can use the \f(CW\*(C`:locale\*(C'\fR layer
524 .PP
525 .Vb 10
526 \& BEGIN { $ENV{LC_ALL} = $ENV{LANG} = \*(Aqru_RU.KOI8\-R\*(Aq }
527 \& # the :locale will probe the locale environment variables like
528 \& # LC_ALL
529 \& use open OUT => \*(Aq:locale\*(Aq; # russki parusski
530 \& open(O, ">koi8");
531 \& print O chr(0x430); # Unicode CYRILLIC SMALL LETTER A = KOI8\-R 0xc1
532 \& close O;
533 \& open(I, "<koi8");
534 \& printf "%#x\en", ord(<I>), "\en"; # this should print 0xc1
535 \& close I;
536 .Ve
537 .PP
538 These methods install a transparent filter on the I/O stream that
539 converts data from the specified encoding when it is read in from the
540 stream. The result is always Unicode.
541 .PP
542 The open pragma affects all the \f(CWopen()\fR calls after the pragma by
543 setting default layers. If you want to affect only certain
544 streams, use explicit layers directly in the \f(CWopen()\fR call.
545 .PP
546 You can switch encodings on an already opened stream by using
547 \&\f(CWbinmode()\fR; see "binmode" in perlfunc.
548 .PP
549 The \f(CW\*(C`:locale\*(C'\fR does not currently work with
550 \&\f(CWopen()\fR and \f(CWbinmode()\fR, only with the \f(CW\*(C`open\*(C'\fR pragma. The
551 \&\f(CW\*(C`:utf8\*(C'\fR and \f(CW:encoding(...)\fR methods do work with all of \f(CWopen()\fR,
552 \&\f(CWbinmode()\fR, and the \f(CW\*(C`open\*(C'\fR pragma.
553 .PP
554 Similarly, you may use these I/O layers on output streams to
555 automatically convert Unicode to the specified encoding when it is
556 written to the stream. For example, the following snippet copies the
557 contents of the file "text.jis" (encoded as ISO\-2022\-JP, aka JIS) to
558 the file "text.utf8", encoded as UTF\-8:
559 .PP
560 .Vb 3
561 \& open(my $nihongo, \*(Aq<:encoding(iso\-2022\-jp)\*(Aq, \*(Aqtext.jis\*(Aq);
562 \& open(my $unicode, \*(Aq>:utf8\*(Aq, \*(Aqtext.utf8\*(Aq);
563 \& while (<$nihongo>) { print $unicode $_ }
564 .Ve
565 .PP
566 The naming of encodings, both by the \f(CWopen()\fR and by the \f(CW\*(C`open\*(C'\fR
567 pragma allows for flexible names: \f(CW\*(C`koi8\-r\*(C'\fR and \f(CW\*(C`KOI8R\*(C'\fR will both be
568 understood.
569 .PP
570 Common encodings recognized by ISO, MIME, IANA, and various other
571 standardisation organisations are recognised; for a more detailed
572 list see Encode::Supported.
573 .PP
574 \&\f(CWread()\fR reads characters and returns the number of characters.
575 \&\f(CWseek()\fR and \f(CWtell()\fR operate on byte counts, as does \f(CWsysseek()\fR.
576 .PP
577 \&\f(CWsysread()\fR and \f(CWsyswrite()\fR should not be used on file handles with
578 character encoding layers, they behave badly, and that behaviour has
579 been deprecated since perl 5.24.
580 .PP
581 Notice that because of the default behaviour of not doing any
582 conversion upon input if there is no default layer,
583 it is easy to mistakenly write code that keeps on expanding a file
584 by repeatedly encoding the data:
585 .PP
586 .Vb 8
587 \& # BAD CODE WARNING
588 \& open F, "file";
589 \& local $/; ## read in the whole file of 8\-bit characters
590 \& $t = <F>;
591 \& close F;
592 \& open F, ">:encoding(UTF\-8)", "file";
593 \& print F $t; ## convert to UTF\-8 on output
594 \& close F;
595 .Ve
596 .PP
597 If you run this code twice, the contents of the \fIfile\fR will be twice
598 UTF\-8 encoded. A \f(CW\*(C`use open \*(Aq:encoding(UTF\-8)\*(Aq\*(C'\fR would have avoided the
599 bug, or explicitly opening also the \fIfile\fR for input as UTF\-8.
600 .PP
601 \&\fBNOTE\fR: the \f(CW\*(C`:utf8\*(C'\fR and \f(CW\*(C`:encoding\*(C'\fR features work only if your
602 Perl has been built with PerlIO, which is the default
603 on most systems.
604 .SS "Displaying Unicode As Text"
605 .IX Subsection "Displaying Unicode As Text"
606 Sometimes you might want to display Perl scalars containing Unicode as
607 simple ASCII (or EBCDIC) text. The following subroutine converts
608 its argument so that Unicode characters with code points greater than
609 255 are displayed as \f(CW\*(C`\ex{...}\*(C'\fR, control characters (like \f(CW\*(C`\en\*(C'\fR) are
610 displayed as \f(CW\*(C`\ex..\*(C'\fR, and the rest of the characters as themselves:
611 .PP
612 .Vb 9
613 \& sub nice_string {
614 \& join("",
615 \& map { $_ > 255 # if wide character...
616 \& ? sprintf("\e\ex{%04X}", $_) # \ex{...}
617 \& : chr($_) =~ /[[:cntrl:]]/ # else if control character...
618 \& ? sprintf("\e\ex%02X", $_) # \ex..
619 \& : quotemeta(chr($_)) # else quoted or as themselves
620 \& } unpack("W*", $_[0])); # unpack Unicode characters
621 \& }
622 .Ve
623 .PP
624 For example,
625 .PP
626 .Vb 1
627 \& nice_string("foo\ex{100}bar\en")
628 .Ve
629 .PP
630 returns the string
631 .PP
632 .Vb 1
633 \& \*(Aqfoo\ex{0100}bar\ex0A\*(Aq
634 .Ve
635 .PP
636 which is ready to be printed.
637 .PP
638 (\f(CW\*(C`\e\ex{}\*(C'\fR is used here instead of \f(CW\*(C`\e\eN{}\*(C'\fR, since it's most likely that
639 you want to see what the native values are.)
640 .SS "Special Cases"
641 .IX Subsection "Special Cases"
642 .IP \(bu 4
643 Starting in Perl 5.28, it is illegal for bit operators, like \f(CW\*(C`~\*(C'\fR, to
644 operate on strings containing code points above 255.
645 .IP \(bu 4
646 The \fBvec()\fR function may produce surprising results if
647 used on strings containing characters with ordinal values above
648 255. In such a case, the results are consistent with the internal
649 encoding of the characters, but not with much else. So don't do
650 that, and starting in Perl 5.28, a deprecation message is issued if you
651 do so, becoming illegal in Perl 5.32.
652 .IP \(bu 4
653 Peeking At Perl's Internal Encoding
654 .Sp
655 Normal users of Perl should never care how Perl encodes any particular
656 Unicode string (because the normal ways to get at the contents of a
657 string with Unicode\-\-via input and output\-\-should always be via
658 explicitly-defined I/O layers). But if you must, there are two
659 ways of looking behind the scenes.
660 .Sp
661 One way of peeking inside the internal encoding of Unicode characters
662 is to use \f(CW\*(C`unpack("C*", ...\*(C'\fR to get the bytes of whatever the string
663 encoding happens to be, or \f(CW\*(C`unpack("U0..", ...)\*(C'\fR to get the bytes of the
664 UTF\-8 encoding:
665 .Sp
666 .Vb 2
667 \& # this prints c4 80 for the UTF\-8 bytes 0xc4 0x80
668 \& print join(" ", unpack("U0(H2)*", pack("U", 0x100))), "\en";
669 .Ve
670 .Sp
671 Yet another way would be to use the Devel::Peek module:
672 .Sp
673 .Vb 1
674 \& perl \-MDevel::Peek \-e \*(AqDump(chr(0x100))\*(Aq
675 .Ve
676 .Sp
677 That shows the \f(CW\*(C`UTF8\*(C'\fR flag in FLAGS and both the UTF\-8 bytes
678 and Unicode characters in \f(CW\*(C`PV\*(C'\fR. See also later in this document
679 the discussion about the \f(CWutf8::is_utf8()\fR function.
680 .SS "Advanced Topics"
681 .IX Subsection "Advanced Topics"
682 .IP \(bu 4
683 String Equivalence
684 .Sp
685 The question of string equivalence turns somewhat complicated
686 in Unicode: what do you mean by "equal"?
687 .Sp
688 (Is \f(CW\*(C`LATIN CAPITAL LETTER A WITH ACUTE\*(C'\fR equal to
689 \&\f(CW\*(C`LATIN CAPITAL LETTER A\*(C'\fR?)
690 .Sp
691 The short answer is that by default Perl compares equivalence (\f(CW\*(C`eq\*(C'\fR,
692 \&\f(CW\*(C`ne\*(C'\fR) based only on code points of the characters. In the above
693 case, the answer is no (because 0x00C1 != 0x0041). But sometimes, any
694 CAPITAL LETTER A's should be considered equal, or even A's of any case.
695 .Sp
696 The long answer is that you need to consider character normalization
697 and casing issues: see Unicode::Normalize, Unicode Technical Report #15,
698 Unicode Normalization Forms <https://www.unicode.org/reports/tr15> and
699 sections on case mapping in the Unicode Standard <https://www.unicode.org>.
700 .Sp
701 As of Perl 5.8.0, the "Full" case-folding of \fICase
702 Mappings/SpecialCasing\fR is implemented, but bugs remain in \f(CW\*(C`qr//i\*(C'\fR with them,
703 mostly fixed by 5.14, and essentially entirely by 5.18.
704 .IP \(bu 4
705 String Collation
706 .Sp
707 People like to see their strings nicely sorted\-\-or as Unicode
708 parlance goes, collated. But again, what do you mean by collate?
709 .Sp
710 (Does \f(CW\*(C`LATIN CAPITAL LETTER A WITH ACUTE\*(C'\fR come before or after
711 \&\f(CW\*(C`LATIN CAPITAL LETTER A WITH GRAVE\*(C'\fR?)
712 .Sp
713 The short answer is that by default, Perl compares strings (\f(CW\*(C`lt\*(C'\fR,
714 \&\f(CW\*(C`le\*(C'\fR, \f(CW\*(C`cmp\*(C'\fR, \f(CW\*(C`ge\*(C'\fR, \f(CW\*(C`gt\*(C'\fR) based only on the code points of the
715 characters. In the above case, the answer is "after", since
716 \&\f(CW0x00C1\fR > \f(CW0x00C0\fR.
717 .Sp
718 The long answer is that "it depends", and a good answer cannot be
719 given without knowing (at the very least) the language context.
720 See Unicode::Collate, and \fIUnicode Collation Algorithm\fR
721 <https://www.unicode.org/reports/tr10/>
722 .SS Miscellaneous
723 .IX Subsection "Miscellaneous"
724 .IP \(bu 4
725 Character Ranges and Classes
726 .Sp
727 Character ranges in regular expression bracketed character classes ( e.g.,
728 \&\f(CW\*(C`/[a\-z]/\*(C'\fR) and in the \f(CW\*(C`tr///\*(C'\fR (also known as \f(CW\*(C`y///\*(C'\fR) operator are not
729 magically Unicode-aware. What this means is that \f(CW\*(C`[A\-Za\-z]\*(C'\fR will not
730 magically start to mean "all alphabetic letters" (not that it does mean that
731 even for 8\-bit characters; for those, if you are using locales (perllocale),
732 use \f(CW\*(C`/[[:alpha:]]/\*(C'\fR; and if not, use the 8\-bit\-aware property \f(CW\*(C`\ep{alpha}\*(C'\fR).
733 .Sp
734 All the properties that begin with \f(CW\*(C`\ep\*(C'\fR (and its inverse \f(CW\*(C`\eP\*(C'\fR) are actually
735 character classes that are Unicode-aware. There are dozens of them, see
736 perluniprops.
737 .Sp
738 Starting in v5.22, you can use Unicode code points as the end points of
739 regular expression pattern character ranges, and the range will include
740 all Unicode code points that lie between those end points, inclusive.
741 .Sp
742 .Vb 1
743 \& qr/ [ \eN{U+03} \- \eN{U+20} ] /xx
744 .Ve
745 .Sp
746 includes the code points
747 \&\f(CW\*(C`\eN{U+03}\*(C'\fR, \f(CW\*(C`\eN{U+04}\*(C'\fR, ..., \f(CW\*(C`\eN{U+20}\*(C'\fR.
748 .Sp
749 This also works for ranges in \f(CW\*(C`tr///\*(C'\fR starting in Perl v5.24.
750 .IP \(bu 4
751 String-To-Number Conversions
752 .Sp
753 Unicode does define several other decimal\-\-and numeric\-\-characters
754 besides the familiar 0 to 9, such as the Arabic and Indic digits.
755 Perl does not support string-to-number conversion for digits other
756 than ASCII \f(CW0\fR to \f(CW9\fR (and ASCII \f(CW\*(C`a\*(C'\fR to \f(CW\*(C`f\*(C'\fR for hexadecimal).
757 To get safe conversions from any Unicode string, use
758 "\fBnum()\fR" in Unicode::UCD.
759 .SS "Questions With Answers"
760 .IX Subsection "Questions With Answers"
761 .IP \(bu 4
762 Will My Old Scripts Break?
763 .Sp
764 Very probably not. Unless you are generating Unicode characters
765 somehow, old behaviour should be preserved. About the only behaviour
766 that has changed and which could start generating Unicode is the old
767 behaviour of \f(CWchr()\fR where supplying an argument more than 255
768 produced a character modulo 255. \f(CWchr(300)\fR, for example, was equal
769 to \f(CWchr(45)\fR or "\-" (in ASCII), now it is LATIN CAPITAL LETTER I WITH
770 BREVE.
771 .IP \(bu 4
772 How Do I Make My Scripts Work With Unicode?
773 .Sp
774 Very little work should be needed since nothing changes until you
775 generate Unicode data. The most important thing is getting input as
776 Unicode; for that, see the earlier I/O discussion.
777 To get full seamless Unicode support, add
778 \&\f(CW\*(C`use feature \*(Aqunicode_strings\*(Aq\*(C'\fR (or \f(CW\*(C`use v5.12\*(C'\fR or higher) to your
779 script.
780 .IP \(bu 4
781 How Do I Know Whether My String Is In Unicode?
782 .Sp
783 You shouldn't have to care. But you may if your Perl is before 5.14.0
784 or you haven't specified \f(CW\*(C`use feature \*(Aqunicode_strings\*(Aq\*(C'\fR or \f(CWuse
785 5.012\fR (or higher) because otherwise the rules for the code points
786 in the range 128 to 255 are different depending on
787 whether the string they are contained within is in Unicode or not.
788 (See "When Unicode Does Not Happen" in perlunicode.)
789 .Sp
790 To determine if a string is in Unicode, use:
791 .Sp
792 .Vb 1
793 \& print utf8::is_utf8($string) ? 1 : 0, "\en";
794 .Ve
795 .Sp
796 But note that this doesn't mean that any of the characters in the
797 string are necessary UTF\-8 encoded, or that any of the characters have
798 code points greater than 0xFF (255) or even 0x80 (128), or that the
799 string has any characters at all. All the \f(CWis_utf8()\fR does is to
800 return the value of the internal "utf8ness" flag attached to the
801 \&\f(CW$string\fR. If the flag is off, the bytes in the scalar are interpreted
802 as a single byte encoding. If the flag is on, the bytes in the scalar
803 are interpreted as the (variable-length, potentially multi-byte) UTF\-8 encoded
804 code points of the characters. Bytes added to a UTF\-8 encoded string are
805 automatically upgraded to UTF\-8. If mixed non\-UTF\-8 and UTF\-8 scalars
806 are merged (double-quoted interpolation, explicit concatenation, or
807 printf/sprintf parameter substitution), the result will be UTF\-8 encoded
808 as if copies of the byte strings were upgraded to UTF\-8: for example,
809 .Sp
810 .Vb 3
811 \& $x = "ab\ex80c";
812 \& $y = "\ex{100}";
813 \& print "$x = $y\en";
814 .Ve
815 .Sp
816 the output string will be UTF\-8\-encoded \f(CW\*(C`ab\ex80c = \ex{100}\en\*(C'\fR, but
817 \&\f(CW$x\fR will stay byte-encoded.
818 .Sp
819 Sometimes you might really need to know the byte length of a string
820 instead of the character length. For that use the \f(CW\*(C`bytes\*(C'\fR pragma
821 and the \f(CWlength()\fR function:
822 .Sp
823 .Vb 6
824 \& my $unicode = chr(0x100);
825 \& print length($unicode), "\en"; # will print 1
826 \& use bytes;
827 \& print length($unicode), "\en"; # will print 2
828 \& # (the 0xC4 0x80 of the UTF\-8)
829 \& no bytes;
830 .Ve
831 .IP \(bu 4
832 How Do I Find Out What Encoding a File Has?
833 .Sp
834 You might try Encode::Guess, but it has a number of limitations.
835 .IP \(bu 4
836 How Do I Detect Data That's Not Valid In a Particular Encoding?
837 .Sp
838 Use the \f(CW\*(C`Encode\*(C'\fR package to try converting it.
839 For example,
840 .Sp
841 .Vb 1
842 \& use Encode \*(Aqdecode\*(Aq;
843 \&
844 \& if (eval { decode(\*(AqUTF\-8\*(Aq, $string, Encode::FB_CROAK); 1 }) {
845 \& # $string is valid UTF\-8
846 \& } else {
847 \& # $string is not valid UTF\-8
848 \& }
849 .Ve
850 .Sp
851 Or use \f(CW\*(C`unpack\*(C'\fR to try decoding it:
852 .Sp
853 .Vb 2
854 \& use warnings;
855 \& @chars = unpack("C0U*", $string_of_bytes_that_I_think_is_utf8);
856 .Ve
857 .Sp
858 If invalid, a \f(CW\*(C`Malformed UTF\-8 character\*(C'\fR warning is produced. The "C0" means
859 "process the string character per character". Without that, the
860 \&\f(CW\*(C`unpack("U*", ...)\*(C'\fR would work in \f(CW\*(C`U0\*(C'\fR mode (the default if the format
861 string starts with \f(CW\*(C`U\*(C'\fR) and it would return the bytes making up the UTF\-8
862 encoding of the target string, something that will always work.
863 .IP \(bu 4
864 How Do I Convert Binary Data Into a Particular Encoding, Or Vice Versa?
865 .Sp
866 This probably isn't as useful as you might think.
867 Normally, you shouldn't need to.
868 .Sp
869 In one sense, what you are asking doesn't make much sense: encodings
870 are for characters, and binary data are not "characters", so converting
871 "data" into some encoding isn't meaningful unless you know in what
872 character set and encoding the binary data is in, in which case it's
873 not just binary data, now is it?
874 .Sp
875 If you have a raw sequence of bytes that you know should be
876 interpreted via a particular encoding, you can use \f(CW\*(C`Encode\*(C'\fR:
877 .Sp
878 .Vb 2
879 \& use Encode \*(Aqfrom_to\*(Aq;
880 \& from_to($data, "iso\-8859\-1", "UTF\-8"); # from latin\-1 to UTF\-8
881 .Ve
882 .Sp
883 The call to \f(CWfrom_to()\fR changes the bytes in \f(CW$data\fR, but nothing
884 material about the nature of the string has changed as far as Perl is
885 concerned. Both before and after the call, the string \f(CW$data\fR
886 contains just a bunch of 8\-bit bytes. As far as Perl is concerned,
887 the encoding of the string remains as "system-native 8\-bit bytes".
888 .Sp
889 You might relate this to a fictional 'Translate' module:
890 .Sp
891 .Vb 4
892 \& use Translate;
893 \& my $phrase = "Yes";
894 \& Translate::from_to($phrase, \*(Aqenglish\*(Aq, \*(Aqdeutsch\*(Aq);
895 \& ## phrase now contains "Ja"
896 .Ve
897 .Sp
898 The contents of the string changes, but not the nature of the string.
899 Perl doesn't know any more after the call than before that the
900 contents of the string indicates the affirmative.
901 .Sp
902 Back to converting data. If you have (or want) data in your system's
903 native 8\-bit encoding (e.g. Latin\-1, EBCDIC, etc.), you can use
904 pack/unpack to convert to/from Unicode.
905 .Sp
906 .Vb 2
907 \& $native_string = pack("W*", unpack("U*", $Unicode_string));
908 \& $Unicode_string = pack("U*", unpack("W*", $native_string));
909 .Ve
910 .Sp
911 If you have a sequence of bytes you \fBknow\fR is valid UTF\-8,
912 but Perl doesn't know it yet, you can make Perl a believer, too:
913 .Sp
914 .Vb 2
915 \& $Unicode = $bytes;
916 \& utf8::decode($Unicode);
917 .Ve
918 .Sp
919 or:
920 .Sp
921 .Vb 1
922 \& $Unicode = pack("U0a*", $bytes);
923 .Ve
924 .Sp
925 You can find the bytes that make up a UTF\-8 sequence with
926 .Sp
927 .Vb 1
928 \& @bytes = unpack("C*", $Unicode_string)
929 .Ve
930 .Sp
931 and you can create well-formed Unicode with
932 .Sp
933 .Vb 1
934 \& $Unicode_string = pack("U*", 0xff, ...)
935 .Ve
936 .IP \(bu 4
937 How Do I Display Unicode? How Do I Input Unicode?
938 .Sp
939 See <http://www.alanwood.net/unicode/> and
940 <http://www.cl.cam.ac.uk/~mgk25/unicode.html>
941 .IP \(bu 4
942 How Does Unicode Work With Traditional Locales?
943 .Sp
944 If your locale is a UTF\-8 locale, starting in Perl v5.26, Perl works
945 well for all categories; before this, starting with Perl v5.20, it works
946 for all categories but \f(CW\*(C`LC_COLLATE\*(C'\fR, which deals with
947 sorting and the \f(CW\*(C`cmp\*(C'\fR operator. But note that the standard
948 \&\f(CW\*(C`Unicode::Collate\*(C'\fR and \f(CW\*(C`Unicode::Collate::Locale\*(C'\fR modules offer
949 much more powerful solutions to collation issues, and work on earlier
950 releases.
951 .Sp
952 For other locales, starting in Perl 5.16, you can specify
953 .Sp
954 .Vb 1
955 \& use locale \*(Aq:not_characters\*(Aq;
956 .Ve
957 .Sp
958 to get Perl to work well with them. The catch is that you
959 have to translate from the locale character set to/from Unicode
960 yourself. See "Unicode I/O" above for how to
961 .Sp
962 .Vb 1
963 \& use open \*(Aq:locale\*(Aq;
964 .Ve
965 .Sp
966 to accomplish this, but full details are in "Unicode and
967 UTF\-8" in perllocale, including gotchas that happen if you don't specify
968 \&\f(CW\*(C`:not_characters\*(C'\fR.
969 .SS "Hexadecimal Notation"
970 .IX Subsection "Hexadecimal Notation"
971 The Unicode standard prefers using hexadecimal notation because
972 that more clearly shows the division of Unicode into blocks of 256 characters.
973 Hexadecimal is also simply shorter than decimal. You can use decimal
974 notation, too, but learning to use hexadecimal just makes life easier
975 with the Unicode standard. The \f(CW\*(C`U+HHHH\*(C'\fR notation uses hexadecimal,
976 for example.
977 .PP
978 The \f(CW\*(C`0x\*(C'\fR prefix means a hexadecimal number, the digits are 0\-9 \fIand\fR
979 a\-f (or A\-F, case doesn't matter). Each hexadecimal digit represents
980 four bits, or half a byte. \f(CW\*(C`print 0x..., "\en"\*(C'\fR will show a
981 hexadecimal number in decimal, and \f(CW\*(C`printf "%x\en", $decimal\*(C'\fR will
982 show a decimal number in hexadecimal. If you have just the
983 "hex digits" of a hexadecimal number, you can use the \f(CWhex()\fR function.
984 .PP
985 .Vb 6
986 \& print 0x0009, "\en"; # 9
987 \& print 0x000a, "\en"; # 10
988 \& print 0x000f, "\en"; # 15
989 \& print 0x0010, "\en"; # 16
990 \& print 0x0011, "\en"; # 17
991 \& print 0x0100, "\en"; # 256
992 \&
993 \& print 0x0041, "\en"; # 65
994 \&
995 \& printf "%x\en", 65; # 41
996 \& printf "%#x\en", 65; # 0x41
997 \&
998 \& print hex("41"), "\en"; # 65
999 .Ve
1000 .SS "Further Resources"
1001 .IX Subsection "Further Resources"
1002 .IP \(bu 4
1003 Unicode Consortium
1004 .Sp
1005 <https://www.unicode.org/>
1006 .IP \(bu 4
1007 Unicode FAQ
1008 .Sp
1009 <https://www.unicode.org/faq/>
1010 .IP \(bu 4
1011 Unicode Glossary
1012 .Sp
1013 <https://www.unicode.org/glossary/>
1014 .IP \(bu 4
1015 Unicode Recommended Reading List
1016 .Sp
1017 The Unicode Consortium has a list of articles and books, some of which
1018 give a much more in depth treatment of Unicode:
1019 <http://unicode.org/resources/readinglist.html>
1020 .IP \(bu 4
1021 Unicode Useful Resources
1022 .Sp
1023 <https://www.unicode.org/unicode/onlinedat/resources.html>
1024 .IP \(bu 4
1025 Unicode and Multilingual Support in HTML, Fonts, Web Browsers and Other Applications
1026 .Sp
1027 <http://www.alanwood.net/unicode/>
1028 .IP \(bu 4
1029 UTF\-8 and Unicode FAQ for Unix/Linux
1030 .Sp
1031 <http://www.cl.cam.ac.uk/~mgk25/unicode.html>
1032 .IP \(bu 4
1033 Legacy Character Sets
1034 .Sp
1035 <http://www.czyborra.com/>
1036 <http://www.eki.ee/letter/>
1037 .IP \(bu 4
1038 You can explore various information from the Unicode data files using
1039 the \f(CW\*(C`Unicode::UCD\*(C'\fR module.
1040 .SH "UNICODE IN OLDER PERLS"
1041 .IX Header "UNICODE IN OLDER PERLS"
1042 If you cannot upgrade your Perl to 5.8.0 or later, you can still
1043 do some Unicode processing by using the modules \f(CW\*(C`Unicode::String\*(C'\fR,
1044 \&\f(CW\*(C`Unicode::Map8\*(C'\fR, and \f(CW\*(C`Unicode::Map\*(C'\fR, available from CPAN.
1045 If you have the GNU recode installed, you can also use the
1046 Perl front-end \f(CW\*(C`Convert::Recode\*(C'\fR for character conversions.
1047 .PP
1048 The following are fast conversions from ISO 8859\-1 (Latin\-1) bytes
1049 to UTF\-8 bytes and back, the code works even with older Perl 5 versions.
1050 .PP
1051 .Vb 2
1052 \& # ISO 8859\-1 to UTF\-8
1053 \& s/([\ex80\-\exFF])/chr(0xC0|ord($1)>>6).chr(0x80|ord($1)&0x3F)/eg;
1054 \&
1055 \& # UTF\-8 to ISO 8859\-1
1056 \& s/([\exC2\exC3])([\ex80\-\exBF])/chr(ord($1)<<6&0xC0|ord($2)&0x3F)/eg;
1057 .Ve
1058 .SH "SEE ALSO"
1059 .IX Header "SEE ALSO"
1060 perlunitut, perlunicode, Encode, open, utf8, bytes,
1061 perlretut, perlrun, Unicode::Collate, Unicode::Normalize,
1062 Unicode::UCD
1063 .SH ACKNOWLEDGMENTS
1064 .IX Header "ACKNOWLEDGMENTS"
1065 Thanks to the kind readers of the perl5\[email protected],
1066 perl\[email protected], linux\[email protected], and [email protected]
1067 mailing lists for their valuable feedback.
1068 .SH "AUTHOR, COPYRIGHT, AND LICENSE"
1069 .IX Header "AUTHOR, COPYRIGHT, AND LICENSE"
1070 Copyright 2001\-2011 Jarkko Hietaniemi <[email protected]>.
1071 Now maintained by Perl 5 Porters.
1072 .PP
1073 This document may be distributed under the same terms as Perl itself.