Debian Codegraph @

upstream/opensuse-tumbleweed/man3/Encode.3pm

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man v6.0.2 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\"
56 .\" Required to disable full justification in groff 1.23.0.
57 .if n .ds AD l
58 .\" ========================================================================
59 .\"
60 .IX Title "Encode 3pm"
61 .TH Encode 3pm 2026-03-09 "perl v5.42.1" "Perl Programmers Reference Guide"
62 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
63 .\" way too many mistakes in technical documents.
64 .if n .ad l
65 .nh
66 .SH NAME
67 Encode \- character encodings in Perl
68 .SH SYNOPSIS
69 .IX Header "SYNOPSIS"
70 .Vb 3
71 \& use Encode qw(decode encode);
72 \& $characters = decode(\*(AqUTF\-8\*(Aq, $octets, Encode::FB_CROAK);
73 \& $octets = encode(\*(AqUTF\-8\*(Aq, $characters, Encode::FB_CROAK);
74 .Ve
75 .SS "Table of Contents"
76 .IX Subsection "Table of Contents"
77 Encode consists of a collection of modules whose details are too extensive
78 to fit in one document. This one itself explains the top\-level APIs
79 and general topics at a glance. For other topics and more details,
80 see the documentation for these modules:
81 .IP "Encode::Alias \- Alias definitions to encodings" 2
82 .IX Item "Encode::Alias - Alias definitions to encodings"
83 .PD 0
84 .IP "Encode::Encoding \- Encode Implementation Base Class" 2
85 .IX Item "Encode::Encoding - Encode Implementation Base Class"
86 .IP "Encode::Supported \- List of Supported Encodings" 2
87 .IX Item "Encode::Supported - List of Supported Encodings"
88 .IP "Encode::CN \- Simplified Chinese Encodings" 2
89 .IX Item "Encode::CN - Simplified Chinese Encodings"
90 .IP "Encode::JP \- Japanese Encodings" 2
91 .IX Item "Encode::JP - Japanese Encodings"
92 .IP "Encode::KR \- Korean Encodings" 2
93 .IX Item "Encode::KR - Korean Encodings"
94 .IP "Encode::TW \- Traditional Chinese Encodings" 2
95 .IX Item "Encode::TW - Traditional Chinese Encodings"
96 .PD
97 .SH DESCRIPTION
98 .IX Header "DESCRIPTION"
99 The \f(CW\*(C`Encode\*(C'\fR module provides the interface between Perl strings
100 and the rest of the system. Perl strings are sequences of
101 \&\fIcharacters\fR.
102 .PP
103 The repertoire of characters that Perl can represent is a superset of those
104 defined by the Unicode Consortium. On most platforms the ordinal
105 values of a character as returned by \f(CWord(\fR\f(CIS\fR\f(CW)\fR is the \fIUnicode
106 codepoint\fR for that character. The exceptions are platforms where
107 the legacy encoding is some variant of EBCDIC rather than a superset
108 of ASCII; see perlebcdic.
109 .PP
110 During recent history, data is moved around a computer in 8\-bit chunks,
111 often called "bytes" but also known as "octets" in standards documents.
112 Perl is widely used to manipulate data of many types: not only strings of
113 characters representing human or computer languages, but also "binary"
114 data, being the machine\*(Aqs representation of numbers, pixels in an image, or
115 just about anything.
116 .PP
117 When Perl is processing "binary data", the programmer wants Perl to
118 process "sequences of bytes". This is not a problem for Perl: because a
119 byte has 256 possible values, it easily fits in Perl\*(Aqs much larger
120 "logical character".
121 .PP
122 This document mostly explains the \fIhow\fR. perlunitut and perlunifaq
123 explain the \fIwhy\fR.
124 .SS TERMINOLOGY
125 .IX Subsection "TERMINOLOGY"
126 \fIcharacter\fR
127 .IX Subsection "character"
128 .PP
129 A character in the range 0 .. 2**32\-1 (or more);
130 what Perl\*(Aqs strings are made of.
131 .PP
132 \fIbyte\fR
133 .IX Subsection "byte"
134 .PP
135 A character in the range 0..255;
136 a special case of a Perl character.
137 .PP
138 \fIoctet\fR
139 .IX Subsection "octet"
140 .PP
141 8 bits of data, with ordinal values 0..255;
142 term for bytes passed to or from a non\-Perl context, such as a disk file,
143 standard I/O stream, database, command\-line argument, environment variable,
144 socket etc.
145 .SH "THE PERL ENCODING API"
146 .IX Header "THE PERL ENCODING API"
147 .SS "Basic methods"
148 .IX Subsection "Basic methods"
149 \fIencode\fR
150 .IX Subsection "encode"
151 .PP
152 .Vb 1
153 \& $octets = encode(ENCODING, STRING[, CHECK])
154 .Ve
155 .PP
156 Encodes the scalar value \fISTRING\fR from Perl\*(Aqs internal form into
157 \&\fIENCODING\fR and returns a sequence of octets. \fIENCODING\fR can be either a
158 canonical name or an alias. For encoding names and aliases, see
159 "Defining Aliases". For CHECK, see "Handling Malformed Data".
160 .PP
161 \&\fBCAVEAT\fR: the input scalar \fISTRING\fR might be modified in\-place depending
162 on what is set in CHECK. See "LEAVE_SRC" if you want your inputs to be
163 left unchanged.
164 .PP
165 For example, to convert a string from Perl\*(Aqs internal format into
166 ISO\-8859\-1, also known as Latin1:
167 .PP
168 .Vb 1
169 \& $octets = encode("iso\-8859\-1", $string);
170 .Ve
171 .PP
172 \&\fBCAVEAT\fR: When you run \f(CW\*(C`$octets = encode("UTF\-8", $string)\*(C'\fR, then
173 \&\f(CW$octets\fR \fImight not be equal to\fR \f(CW$string\fR. Though both contain the
174 same data, the UTF8 flag for \f(CW$octets\fR is \fIalways\fR off. When you
175 encode anything, the UTF8 flag on the result is always off, even when it
176 contains a completely valid UTF\-8 string. See "The UTF8 flag" below.
177 .PP
178 If the \f(CW$string\fR is \f(CW\*(C`undef\*(C'\fR, then \f(CW\*(C`undef\*(C'\fR is returned.
179 .PP
180 \&\f(CW\*(C`str2bytes\*(C'\fR may be used as an alias for \f(CW\*(C`encode\*(C'\fR.
181 .PP
182 \fIdecode\fR
183 .IX Subsection "decode"
184 .PP
185 .Vb 1
186 \& $string = decode(ENCODING, OCTETS[, CHECK])
187 .Ve
188 .PP
189 This function returns the string that results from decoding the scalar
190 value \fIOCTETS\fR, assumed to be a sequence of octets in \fIENCODING\fR, into
191 Perl\*(Aqs internal form. As with \fBencode()\fR,
192 \&\fIENCODING\fR can be either a canonical name or an alias. For encoding names
193 and aliases, see "Defining Aliases"; for \fICHECK\fR, see "Handling
194 Malformed Data".
195 .PP
196 \&\fBCAVEAT\fR: the input scalar \fIOCTETS\fR might be modified in\-place depending
197 on what is set in CHECK. See "LEAVE_SRC" if you want your inputs to be
198 left unchanged.
199 .PP
200 For example, to convert ISO\-8859\-1 data into a string in Perl\*(Aqs
201 internal format:
202 .PP
203 .Vb 1
204 \& $string = decode("iso\-8859\-1", $octets);
205 .Ve
206 .PP
207 \&\fBCAVEAT\fR: When you run \f(CW\*(C`$string = decode("UTF\-8", $octets)\*(C'\fR, then \f(CW$string\fR
208 \&\fImight not be equal to\fR \f(CW$octets\fR. Though both contain the same data, the
209 UTF8 flag for \f(CW$string\fR is on. See "The UTF8 flag"
210 below.
211 .PP
212 If the \f(CW$string\fR is \f(CW\*(C`undef\*(C'\fR, then \f(CW\*(C`undef\*(C'\fR is returned.
213 .PP
214 \&\f(CW\*(C`bytes2str\*(C'\fR may be used as an alias for \f(CW\*(C`decode\*(C'\fR.
215 .PP
216 \fIfind_encoding\fR
217 .IX Subsection "find_encoding"
218 .PP
219 .Vb 1
220 \& [$obj =] find_encoding(ENCODING)
221 .Ve
222 .PP
223 Returns the \fIencoding object\fR corresponding to \fIENCODING\fR. Returns
224 \&\f(CW\*(C`undef\*(C'\fR if no matching \fIENCODING\fR is find. The returned object is
225 what does the actual encoding or decoding.
226 .PP
227 .Vb 1
228 \& $string = decode($name, $bytes);
229 .Ve
230 .PP
231 is in fact
232 .PP
233 .Vb 5
234 \& $string = do {
235 \& $obj = find_encoding($name);
236 \& croak qq(encoding "$name" not found) unless ref $obj;
237 \& $obj\->decode($bytes);
238 \& };
239 .Ve
240 .PP
241 with more error checking.
242 .PP
243 You can therefore save time by reusing this object as follows;
244 .PP
245 .Vb 5
246 \& my $enc = find_encoding("iso\-8859\-1");
247 \& while(<>) {
248 \& my $string = $enc\->decode($_);
249 \& ... # now do something with $string;
250 \& }
251 .Ve
252 .PP
253 Besides "decode" and "encode", other methods are
254 available as well. For instance, \f(CWname()\fR returns the canonical
255 name of the encoding object.
256 .PP
257 .Vb 1
258 \& find_encoding("latin1")\->name; # iso\-8859\-1
259 .Ve
260 .PP
261 See Encode::Encoding for details.
262 .PP
263 \fIfind_mime_encoding\fR
264 .IX Subsection "find_mime_encoding"
265 .PP
266 .Vb 1
267 \& [$obj =] find_mime_encoding(MIME_ENCODING)
268 .Ve
269 .PP
270 Returns the \fIencoding object\fR corresponding to \fIMIME_ENCODING\fR. Acts
271 same as \f(CWfind_encoding()\fR but \f(CWmime_name()\fR of returned object must
272 match to \fIMIME_ENCODING\fR. So as opposite of \f(CWfind_encoding()\fR
273 canonical names and aliases are not used when searching for object.
274 .PP
275 .Vb 4
276 \& find_mime_encoding("utf8"); # returns undef because "utf8" is not a valid MIME_ENCODING
277 \& find_mime_encoding("utf\-8"); # returns encode object "utf\-8\-strict"
278 \& find_mime_encoding("UTF\-8"); # same as "utf\-8" because MIME_ENCODING is case insensitive
279 \& find_mime_encoding("utf\-8\-strict"); returns undef because "utf\-8\-strict" is not a valid MIME_ENCODING
280 .Ve
281 .PP
282 \fIfrom_to\fR
283 .IX Subsection "from_to"
284 .PP
285 .Vb 1
286 \& [$length =] from_to($octets, FROM_ENC, TO_ENC [, CHECK])
287 .Ve
288 .PP
289 Converts \fIin\-place\fR data between two encodings. The data in \f(CW$octets\fR
290 must be encoded as octets and \fInot\fR as characters in Perl\*(Aqs internal
291 format. For example, to convert ISO\-8859\-1 data into Microsoft\*(Aqs CP1250
292 encoding:
293 .PP
294 .Vb 1
295 \& from_to($octets, "iso\-8859\-1", "cp1250");
296 .Ve
297 .PP
298 and to convert it back:
299 .PP
300 .Vb 1
301 \& from_to($octets, "cp1250", "iso\-8859\-1");
302 .Ve
303 .PP
304 Because the conversion happens in place, the data to be
305 converted cannot be a string constant: it must be a scalar variable.
306 .PP
307 \&\f(CWfrom_to()\fR returns the length of the converted string in octets on success,
308 and \f(CW\*(C`undef\*(C'\fR on error.
309 .PP
310 \&\fBCAVEAT\fR: The following operations may look the same, but are not:
311 .PP
312 .Vb 2
313 \& from_to($data, "iso\-8859\-1", "UTF\-8"); #1
314 \& $data = decode("iso\-8859\-1", $data); #2
315 .Ve
316 .PP
317 Both #1 and #2 make \f(CW$data\fR consist of a completely valid UTF\-8 string,
318 but only #2 turns the UTF8 flag on. #1 is equivalent to:
319 .PP
320 .Vb 1
321 \& $data = encode("UTF\-8", decode("iso\-8859\-1", $data));
322 .Ve
323 .PP
324 See "The UTF8 flag" below.
325 .PP
326 Also note that:
327 .PP
328 .Vb 1
329 \& from_to($octets, $from, $to, $check);
330 .Ve
331 .PP
332 is equivalent to:
333 .PP
334 .Vb 1
335 \& $octets = encode($to, decode($from, $octets), $check);
336 .Ve
337 .PP
338 Yes, it does \fInot\fR respect the \f(CW$check\fR during decoding. It is
339 deliberately done that way. If you need minute control, use \f(CW\*(C`decode\*(C'\fR
340 followed by \f(CW\*(C`encode\*(C'\fR as follows:
341 .PP
342 .Vb 1
343 \& $octets = encode($to, decode($from, $octets, $check_from), $check_to);
344 .Ve
345 .PP
346 \fIencode_utf8\fR
347 .IX Subsection "encode_utf8"
348 .PP
349 .Vb 1
350 \& $octets = encode_utf8($string);
351 .Ve
352 .PP
353 \&\fBWARNING\fR: This function can produce invalid UTF\-8!
354 Do not use it for data exchange.
355 Unless you want Perl\*(Aqs older "lax" mode, prefer
356 \&\f(CW\*(C`$octets = encode("UTF\-8", $string)\*(C'\fR.
357 .PP
358 Equivalent to \f(CW\*(C`$octets = encode("utf8", $string)\*(C'\fR. The characters in
359 \&\f(CW$string\fR are encoded in Perl\*(Aqs internal format, and the result is returned
360 as a sequence of octets. Because all possible characters in Perl have a
361 (loose, not strict) utf8 representation, this function cannot fail.
362 .PP
363 \fIdecode_utf8\fR
364 .IX Subsection "decode_utf8"
365 .PP
366 .Vb 1
367 \& $string = decode_utf8($octets [, CHECK]);
368 .Ve
369 .PP
370 \&\fBWARNING\fR: This function accepts invalid UTF\-8!
371 Do not use it for data exchange.
372 Unless you want Perl\*(Aqs older "lax" mode, prefer
373 \&\f(CW\*(C`$string = decode("UTF\-8", $octets [, CHECK])\*(C'\fR.
374 .PP
375 Equivalent to \f(CW\*(C`$string = decode("utf8", $octets [, CHECK])\*(C'\fR.
376 The sequence of octets represented by \f(CW$octets\fR is decoded
377 from (loose, not strict) utf8 into a sequence of logical characters.
378 Because not all sequences of octets are valid not strict utf8,
379 it is quite possible for this function to fail.
380 For CHECK, see "Handling Malformed Data".
381 .PP
382 \&\fBCAVEAT\fR: the input \fR\f(CI$octets\fR\fI\fR might be modified in\-place depending on
383 what is set in CHECK. See "LEAVE_SRC" if you want your inputs to be
384 left unchanged.
385 .SS "Listing available encodings"
386 .IX Subsection "Listing available encodings"
387 .Vb 2
388 \& use Encode;
389 \& @list = Encode\->encodings();
390 .Ve
391 .PP
392 Returns a list of canonical names of available encodings that have already
393 been loaded. To get a list of all available encodings including those that
394 have not yet been loaded, say:
395 .PP
396 .Vb 1
397 \& @all_encodings = Encode\->encodings(":all");
398 .Ve
399 .PP
400 Or you can give the name of a specific module:
401 .PP
402 .Vb 1
403 \& @with_jp = Encode\->encodings("Encode::JP");
404 .Ve
405 .PP
406 When "\f(CW\*(C`::\*(C'\fR" is not in the name, "\f(CW\*(C`Encode::\*(C'\fR" is assumed.
407 .PP
408 .Vb 1
409 \& @ebcdic = Encode\->encodings("EBCDIC");
410 .Ve
411 .PP
412 To find out in detail which encodings are supported by this package,
413 see Encode::Supported.
414 .SS "Defining Aliases"
415 .IX Subsection "Defining Aliases"
416 To add a new alias to a given encoding, use:
417 .PP
418 .Vb 3
419 \& use Encode;
420 \& use Encode::Alias;
421 \& define_alias(NEWNAME => ENCODING);
422 .Ve
423 .PP
424 After that, \fINEWNAME\fR can be used as an alias for \fIENCODING\fR.
425 \&\fIENCODING\fR may be either the name of an encoding or an
426 \&\fIencoding object\fR.
427 .PP
428 Before you do that, first make sure the alias is nonexistent using
429 \&\f(CWresolve_alias()\fR, which returns the canonical name thereof.
430 For example:
431 .PP
432 .Vb 3
433 \& Encode::resolve_alias("latin1") eq "iso\-8859\-1" # true
434 \& Encode::resolve_alias("iso\-8859\-12") # false; nonexistent
435 \& Encode::resolve_alias($name) eq $name # true if $name is canonical
436 .Ve
437 .PP
438 \&\f(CWresolve_alias()\fR does not need \f(CW\*(C`use Encode::Alias\*(C'\fR; it can be
439 imported via \f(CW\*(C`use Encode qw(resolve_alias)\*(C'\fR.
440 .PP
441 See Encode::Alias for details.
442 .SS "Finding IANA Character Set Registry names"
443 .IX Subsection "Finding IANA Character Set Registry names"
444 The canonical name of a given encoding does not necessarily agree with
445 IANA Character Set Registry, commonly seen as \f(CW\*(C`Content\-Type:
446 text/plain; charset=\fR\f(CIWHATEVER\fR\f(CW\*(C'\fR. For most cases, the canonical name
447 works, but sometimes it does not, most notably with "utf\-8\-strict".
448 .PP
449 As of \f(CW\*(C`Encode\*(C'\fR version 2.21, a new method \f(CWmime_name()\fR is therefore added.
450 .PP
451 .Vb 4
452 \& use Encode;
453 \& my $enc = find_encoding("UTF\-8");
454 \& warn $enc\->name; # utf\-8\-strict
455 \& warn $enc\->mime_name; # UTF\-8
456 .Ve
457 .PP
458 See also: Encode::Encoding
459 .SH "Encoding via PerlIO"
460 .IX Header "Encoding via PerlIO"
461 If your perl supports \f(CW\*(C`PerlIO\*(C'\fR (which is the default), you can use a
462 \&\f(CW\*(C`PerlIO\*(C'\fR layer to decode and encode directly via a filehandle. The
463 following two examples are fully identical in functionality:
464 .PP
465 .Vb 10
466 \& ### Version 1 via PerlIO
467 \& open(INPUT, "< :encoding(shiftjis)", $infile)
468 \& || die "Can\*(Aqt open < $infile for reading: $!";
469 \& open(OUTPUT, "> :encoding(euc\-jp)", $outfile)
470 \& || die "Can\*(Aqt open > $output for writing: $!";
471 \& while (<INPUT>) { # auto decodes $_
472 \& print OUTPUT; # auto encodes $_
473 \& }
474 \& close(INPUT) || die "can\*(Aqt close $infile: $!";
475 \& close(OUTPUT) || die "can\*(Aqt close $outfile: $!";
476 \&
477 \& ### Version 2 via from_to()
478 \& open(INPUT, "< :raw", $infile)
479 \& || die "Can\*(Aqt open < $infile for reading: $!";
480 \& open(OUTPUT, "> :raw", $outfile)
481 \& || die "Can\*(Aqt open > $output for writing: $!";
482 \&
483 \& while (<INPUT>) {
484 \& from_to($_, "shiftjis", "euc\-jp", 1); # switch encoding
485 \& print OUTPUT; # emit raw (but properly encoded) data
486 \& }
487 \& close(INPUT) || die "can\*(Aqt close $infile: $!";
488 \& close(OUTPUT) || die "can\*(Aqt close $outfile: $!";
489 .Ve
490 .PP
491 In the first version above, you let the appropriate encoding layer
492 handle the conversion. In the second, you explicitly translate
493 from one encoding to the other.
494 .PP
495 Unfortunately, it may be that encodings are not \f(CW\*(C`PerlIO\*(C'\fR\-savvy. You can check
496 to see whether your encoding is supported by \f(CW\*(C`PerlIO\*(C'\fR by invoking the
497 \&\f(CW\*(C`perlio_ok\*(C'\fR method on it:
498 .PP
499 .Vb 2
500 \& Encode::perlio_ok("hz"); # false
501 \& find_encoding("euc\-cn")\->perlio_ok; # true wherever PerlIO is available
502 \&
503 \& use Encode qw(perlio_ok); # imported upon request
504 \& perlio_ok("euc\-jp")
505 .Ve
506 .PP
507 Fortunately, all encodings that come with \f(CW\*(C`Encode\*(C'\fR core are \f(CW\*(C`PerlIO\*(C'\fR\-savvy
508 except for \f(CW\*(C`hz\*(C'\fR and \f(CW\*(C`ISO\-2022\-kr\*(C'\fR. For the gory details, see
509 Encode::Encoding and Encode::PerlIO.
510 .SH "Handling Malformed Data"
511 .IX Header "Handling Malformed Data"
512 The optional \fICHECK\fR argument tells \f(CW\*(C`Encode\*(C'\fR what to do when
513 encountering malformed data. Without \fICHECK\fR, \f(CW\*(C`Encode::FB_DEFAULT\*(C'\fR
514 (== 0) is assumed.
515 .PP
516 As of version 2.12, \f(CW\*(C`Encode\*(C'\fR supports coderef values for \f(CW\*(C`CHECK\*(C'\fR;
517 see below.
518 .PP
519 \&\fBNOTE:\fR Not all encodings support this feature.
520 Some encodings ignore the \fICHECK\fR argument. For example,
521 Encode::Unicode ignores \fICHECK\fR and it always croaks on error.
522 .SS "List of \fICHECK\fP values"
523 .IX Subsection "List of CHECK values"
524 \fIFB_DEFAULT\fR
525 .IX Subsection "FB_DEFAULT"
526 .PP
527 .Vb 1
528 \& CHECK = Encode::FB_DEFAULT ( == 0)
529 .Ve
530 .PP
531 If \fICHECK\fR is 0, encoding and decoding replace any malformed character
532 with a \fIsubstitution character\fR. When you encode, \fISUBCHAR\fR is used.
533 When you decode, the Unicode REPLACEMENT CHARACTER, code point U+FFFD, is
534 used. If the data is supposed to be UTF\-8, an optional lexical warning of
535 warning category \f(CW"utf8"\fR is given.
536 .PP
537 \fIFB_CROAK\fR
538 .IX Subsection "FB_CROAK"
539 .PP
540 .Vb 1
541 \& CHECK = Encode::FB_CROAK ( == 1)
542 .Ve
543 .PP
544 If \fICHECK\fR is 1, methods immediately die with an error
545 message. Therefore, when \fICHECK\fR is 1, you should trap
546 exceptions with \f(CW\*(C`eval{}\*(C'\fR, unless you really want to let it \f(CW\*(C`die\*(C'\fR.
547 .PP
548 \fIFB_QUIET\fR
549 .IX Subsection "FB_QUIET"
550 .PP
551 .Vb 1
552 \& CHECK = Encode::FB_QUIET
553 .Ve
554 .PP
555 If \fICHECK\fR is set to \f(CW\*(C`Encode::FB_QUIET\*(C'\fR, encoding and decoding immediately
556 return the portion of the data that has been processed so far when an
557 error occurs. The data argument is overwritten with everything
558 after that point; that is, the unprocessed portion of the data. This is
559 handy when you have to call \f(CW\*(C`decode\*(C'\fR repeatedly in the case where your
560 source data may contain partial multi\-byte character sequences,
561 (that is, you are reading with a fixed\-width buffer). Here\*(Aqs some sample
562 code to do exactly that:
563 .PP
564 .Vb 5
565 \& my($buffer, $string) = ("", "");
566 \& while (read($fh, $buffer, 256, length($buffer))) {
567 \& $string .= decode($encoding, $buffer, Encode::FB_QUIET);
568 \& # $buffer now contains the unprocessed partial character
569 \& }
570 .Ve
571 .PP
572 \fIFB_WARN\fR
573 .IX Subsection "FB_WARN"
574 .PP
575 .Vb 1
576 \& CHECK = Encode::FB_WARN
577 .Ve
578 .PP
579 This is the same as \f(CW\*(C`FB_QUIET\*(C'\fR above, except that instead of being silent
580 on errors, it issues a warning. This is handy for when you are debugging.
581 .PP
582 \&\fBCAVEAT\fR: All warnings from Encode module are reported, independently of
583 pragma warnings settings. If you want to follow settings of
584 lexical warnings configured by pragma warnings then append
585 also check value \f(CW\*(C`ENCODE::ONLY_PRAGMA_WARNINGS\*(C'\fR. This value is available
586 since Encode version 2.99.
587 .PP
588 \fIFB_PERLQQ FB_HTMLCREF FB_XMLCREF\fR
589 .IX Subsection "FB_PERLQQ FB_HTMLCREF FB_XMLCREF"
590 .IP "perlqq mode (\fICHECK\fR = Encode::FB_PERLQQ)" 2
591 .IX Item "perlqq mode (CHECK = Encode::FB_PERLQQ)"
592 .PD 0
593 .IP "HTML charref mode (\fICHECK\fR = Encode::FB_HTMLCREF)" 2
594 .IX Item "HTML charref mode (CHECK = Encode::FB_HTMLCREF)"
595 .IP "XML charref mode (\fICHECK\fR = Encode::FB_XMLCREF)" 2
596 .IX Item "XML charref mode (CHECK = Encode::FB_XMLCREF)"
597 .PD
598 .PP
599 For encodings that are implemented by the \f(CW\*(C`Encode::XS\*(C'\fR module, \f(CW\*(C`CHECK\*(C'\fR \f(CW\*(C`==\*(C'\fR
600 \&\f(CW\*(C`Encode::FB_PERLQQ\*(C'\fR puts \f(CW\*(C`encode\*(C'\fR and \f(CW\*(C`decode\*(C'\fR into \f(CW\*(C`perlqq\*(C'\fR fallback mode.
601 .PP
602 When you decode, \f(CW\*(C`\ex\fR\f(CIHH\fR\f(CW\*(C'\fR is inserted for a malformed character, where
603 \&\fIHH\fR is the hex representation of the octet that could not be decoded to
604 utf8. When you encode, \f(CW\*(C`\ex{\fR\f(CIHHHH\fR\f(CW}\*(C'\fR will be inserted, where \fIHHHH\fR is
605 the Unicode code point (in any number of hex digits) of the character that
606 cannot be found in the character repertoire of the encoding.
607 .PP
608 The HTML/XML character reference modes are about the same. In place of
609 \&\f(CW\*(C`\ex{\fR\f(CIHHHH\fR\f(CW}\*(C'\fR, HTML uses \f(CW\*(C`&#\fR\f(CINNN\fR\f(CW;\*(C'\fR where \fINNN\fR is a decimal number, and
610 XML uses \f(CW\*(C`&#x\fR\f(CIHHHH\fR\f(CW;\*(C'\fR where \fIHHHH\fR is the hexadecimal number.
611 .PP
612 In \f(CW\*(C`Encode\*(C'\fR 2.10 or later, \f(CW\*(C`LEAVE_SRC\*(C'\fR is also implied.
613 .PP
614 \fIThe bitmask\fR
615 .IX Subsection "The bitmask"
616 .PP
617 These modes are all actually set via a bitmask. Here is how the \f(CW\*(C`FB_\fR\f(CIXXX\fR\f(CW\*(C'\fR
618 constants are laid out. You can import the \f(CW\*(C`FB_\fR\f(CIXXX\fR\f(CW\*(C'\fR constants via
619 \&\f(CW\*(C`use Encode qw(:fallbacks)\*(C'\fR, and you can import the generic bitmask
620 constants via \f(CW\*(C`use Encode qw(:fallback_all)\*(C'\fR.
621 .PP
622 .Vb 8
623 \& FB_DEFAULT FB_CROAK FB_QUIET FB_WARN FB_PERLQQ
624 \& DIE_ON_ERR 0x0001 X
625 \& WARN_ON_ERR 0x0002 X
626 \& RETURN_ON_ERR 0x0004 X X
627 \& LEAVE_SRC 0x0008 X
628 \& PERLQQ 0x0100 X
629 \& HTMLCREF 0x0200
630 \& XMLCREF 0x0400
631 .Ve
632 .PP
633 \fILEAVE_SRC\fR
634 .IX Subsection "LEAVE_SRC"
635 .PP
636 .Vb 1
637 \& Encode::LEAVE_SRC
638 .Ve
639 .PP
640 If the \f(CW\*(C`Encode::LEAVE_SRC\*(C'\fR bit is \fInot\fR set but \fICHECK\fR is set, then the
641 source string to \fBencode()\fR or \fBdecode()\fR will be overwritten in place.
642 If you\*(Aqre not interested in this, then bitwise\-OR it with the bitmask.
643 .SS "coderef for CHECK"
644 .IX Subsection "coderef for CHECK"
645 As of \f(CW\*(C`Encode\*(C'\fR 2.12, \f(CW\*(C`CHECK\*(C'\fR can also be a code reference which takes the
646 ordinal value of the unmapped character as an argument and returns
647 octets that represent the fallback character. For instance:
648 .PP
649 .Vb 1
650 \& $ascii = encode("ascii", $utf8, sub{ sprintf "<U+%04X>", shift });
651 .Ve
652 .PP
653 Acts like \f(CW\*(C`FB_PERLQQ\*(C'\fR but U+\fIXXXX\fR is used instead of \f(CW\*(C`\ex{\fR\f(CIXXXX\fR\f(CW}\*(C'\fR.
654 .PP
655 Fallback for \f(CW\*(C`decode\*(C'\fR must return decoded string (sequence of characters)
656 and takes a list of ordinal values as its arguments. So for
657 example if you wish to decode octets as UTF\-8, and use ISO\-8859\-15 as
658 a fallback for bytes that are not valid UTF\-8, you could write
659 .PP
660 .Vb 4
661 \& $str = decode \*(AqUTF\-8\*(Aq, $octets, sub {
662 \& my $tmp = join \*(Aq\*(Aq, map chr, @_;
663 \& return decode \*(AqISO\-8859\-15\*(Aq, $tmp;
664 \& };
665 .Ve
666 .SH "Defining Encodings"
667 .IX Header "Defining Encodings"
668 To define a new encoding, use:
669 .PP
670 .Vb 2
671 \& use Encode qw(define_encoding);
672 \& define_encoding($object, CANONICAL_NAME [, alias...]);
673 .Ve
674 .PP
675 \&\fICANONICAL_NAME\fR will be associated with \fR\f(CI$object\fR\fI\fR. The object
676 should provide the interface described in Encode::Encoding.
677 If more than two arguments are provided, additional
678 arguments are considered aliases for \fI\fR\f(CI$object\fR\fI\fR.
679 .PP
680 See Encode::Encoding for details.
681 .SH "The UTF8 flag"
682 .IX Header "The UTF8 flag"
683 Before the introduction of Unicode support in Perl, the \f(CW\*(C`eq\*(C'\fR operator
684 just compared the strings represented by two scalars. Beginning with
685 Perl 5.8, \f(CW\*(C`eq\*(C'\fR compares two strings with simultaneous consideration of
686 \&\fIthe UTF8 flag\fR. To explain why we made it so, I quote from page 402 of
687 \&\fIProgramming Perl, 3rd ed.\fR
688 .IP "Goal #1:" 2
689 .IX Item "Goal #1:"
690 Old byte\-oriented programs should not spontaneously break on the old
691 byte\-oriented data they used to work on.
692 .IP "Goal #2:" 2
693 .IX Item "Goal #2:"
694 Old byte\-oriented programs should magically start working on the new
695 character\-oriented data when appropriate.
696 .IP "Goal #3:" 2
697 .IX Item "Goal #3:"
698 Programs should run just as fast in the new character\-oriented mode
699 as in the old byte\-oriented mode.
700 .IP "Goal #4:" 2
701 .IX Item "Goal #4:"
702 Perl should remain one language, rather than forking into a
703 byte\-oriented Perl and a character\-oriented Perl.
704 .PP
705 When \fIProgramming Perl, 3rd ed.\fR was written, not even Perl 5.6.0 had been
706 born yet, many features documented in the book remained unimplemented for a
707 long time. Perl 5.8 corrected much of this, and the introduction of the
708 UTF8 flag is one of them. You can think of there being two fundamentally
709 different kinds of strings and string\-operations in Perl: one a
710 byte\-oriented mode for when the internal UTF8 flag is off, and the other a
711 character\-oriented mode for when the internal UTF8 flag is on.
712 .PP
713 This UTF8 flag is not visible in Perl scripts, exactly for the same reason
714 you cannot (or rather, you \fIdon\*(Aqt have to\fR) see whether a scalar contains
715 a string, an integer, or a floating\-point number. But you can still peek
716 and poke these if you will. See the next section.
717 .SS "Messing with Perl\*(Aqs Internals"
718 .IX Subsection "Messing with Perl's Internals"
719 The following API uses parts of Perl\*(Aqs internals in the current
720 implementation. As such, they are efficient but may change in a future
721 release.
722 .PP
723 \fIis_utf8\fR
724 .IX Subsection "is_utf8"
725 .PP
726 .Vb 1
727 \& is_utf8(STRING [, CHECK])
728 .Ve
729 .PP
730 [INTERNAL] Tests whether the UTF8 flag is turned on in the \fISTRING\fR.
731 If \fICHECK\fR is true, also checks whether \fISTRING\fR contains well\-formed
732 UTF\-8. Returns true if successful, false otherwise.
733 .PP
734 Typically only necessary for debugging and testing. Don\*(Aqt use this flag as
735 a marker to distinguish character and binary data, that should be decided
736 for each variable when you write your code.
737 .PP
738 \&\fBCAVEAT\fR: If \fISTRING\fR has UTF8 flag set, it does \fBNOT\fR mean that
739 \&\fISTRING\fR is UTF\-8 encoded and vice\-versa.
740 .PP
741 As of Perl 5.8.1, utf8 also has the \f(CW\*(C`utf8::is_utf8\*(C'\fR function.
742 .PP
743 \fI_utf8_on\fR
744 .IX Subsection "_utf8_on"
745 .PP
746 .Vb 1
747 \& _utf8_on(STRING)
748 .Ve
749 .PP
750 [INTERNAL] Turns the \fISTRING\fR\*(Aqs internal UTF8 flag \fBon\fR. The \fISTRING\fR
751 is \fInot\fR checked for containing only well\-formed UTF\-8. Do not use this
752 unless you \fIknow with absolute certainty\fR that the STRING holds only
753 well\-formed UTF\-8. Returns the previous state of the UTF8 flag (so please
754 don\*(Aqt treat the return value as indicating success or failure), or \f(CW\*(C`undef\*(C'\fR
755 if \fISTRING\fR is not a string.
756 .PP
757 \&\fBNOTE\fR: For security reasons, this function does not work on tainted values.
758 .PP
759 \fI_utf8_off\fR
760 .IX Subsection "_utf8_off"
761 .PP
762 .Vb 1
763 \& _utf8_off(STRING)
764 .Ve
765 .PP
766 [INTERNAL] Turns the \fISTRING\fR\*(Aqs internal UTF8 flag \fBoff\fR. Do not use
767 frivolously. Returns the previous state of the UTF8 flag, or \f(CW\*(C`undef\*(C'\fR if
768 \&\fISTRING\fR is not a string. Do not treat the return value as indicative of
769 success or failure, because that isn\*(Aqt what it means: it is only the
770 previous setting.
771 .PP
772 \&\fBNOTE\fR: For security reasons, this function does not work on tainted values.
773 .SH "UTF\-8 vs. utf8 vs. UTF8"
774 .IX Header "UTF-8 vs. utf8 vs. UTF8"
775 .Vb 3
776 \& ....We now view strings not as sequences of bytes, but as sequences
777 \& of numbers in the range 0 .. 2**32\-1 (or in the case of 64\-bit
778 \& computers, 0 .. 2**64\-1) \-\- Programming Perl, 3rd ed.
779 .Ve
780 .PP
781 That has historically been Perl\*(Aqs notion of UTF\-8, as that is how UTF\-8 was
782 first conceived by Ken Thompson when he invented it. However, thanks to
783 later revisions to the applicable standards, official UTF\-8 is now rather
784 stricter than that. For example, its range is much narrower (0 .. 0x10_FFFF
785 to cover only 21 bits instead of 32 or 64 bits) and some sequences
786 are not allowed, like those used in surrogate pairs, the 31 non\-character
787 code points 0xFDD0 .. 0xFDEF, the last two code points in \fIany\fR plane
788 (0x\fIXX\fR_FFFE and 0x\fIXX\fR_FFFF), all non\-shortest encodings, etc.
789 .PP
790 The former default in which Perl would always use a loose interpretation of
791 UTF\-8 has now been overruled:
792 .PP
793 .Vb 5
794 \& From: Larry Wall <[email protected]>
795 \& Date: December 04, 2004 11:51:58 JST
796 \& To: perl\[email protected]
797 \& Subject: Re: Make Encode.pm support the real UTF\-8
798 \& Message\-Id: <[email protected]>
799 \&
800 \& On Fri, Dec 03, 2004 at 10:12:12PM +0000, Tim Bunce wrote:
801 \& : I\*(Aqve no problem with \*(Aqutf8\*(Aq being perl\*(Aqs unrestricted uft8 encoding,
802 \& : but "UTF\-8" is the name of the standard and should give the
803 \& : corresponding behaviour.
804 \&
805 \& For what it\*(Aqs worth, that\*(Aqs how I\*(Aqve always kept them straight in my
806 \& head.
807 \&
808 \& Also for what it\*(Aqs worth, Perl 6 will mostly default to strict but
809 \& make it easy to switch back to lax.
810 \&
811 \& Larry
812 .Ve
813 .PP
814 Got that? As of Perl 5.8.7, \fB"UTF\-8"\fR means UTF\-8 in its current
815 sense, which is conservative and strict and security\-conscious, whereas
816 \&\fB"utf8"\fR means UTF\-8 in its former sense, which was liberal and loose and
817 lax. \f(CW\*(C`Encode\*(C'\fR version 2.10 or later thus groks this subtle but critically
818 important distinction between \f(CW"UTF\-8"\fR and \f(CW"utf8"\fR.
819 .PP
820 .Vb 2
821 \& encode("utf8", "\ex{FFFF_FFFF}", 1); # okay
822 \& encode("UTF\-8", "\ex{FFFF_FFFF}", 1); # croaks
823 .Ve
824 .PP
825 This distinction is also important for decoding. In the following,
826 \&\f(CW$s\fR stores character U+200000, which exceeds UTF\-8\*(Aqs allowed range.
827 \&\f(CW$s\fR thus stores an invalid Unicode code point:
828 .PP
829 .Vb 1
830 \& $s = decode("utf8", "\exf8\ex88\ex80\ex80\ex80");
831 .Ve
832 .PP
833 \&\f(CW"UTF\-8"\fR, by contrast, will either coerce the input to something valid:
834 .PP
835 .Vb 1
836 \& $s = decode("UTF\-8", "\exf8\ex88\ex80\ex80\ex80"); # U+FFFD
837 .Ve
838 .PP
839 \&.. or croak:
840 .PP
841 .Vb 1
842 \& decode("UTF\-8", "\exf8\ex88\ex80\ex80\ex80", FB_CROAK|LEAVE_SRC);
843 .Ve
844 .PP
845 In the \f(CW\*(C`Encode\*(C'\fR module, \f(CW"UTF\-8"\fR is actually a canonical name for
846 \&\f(CW"utf\-8\-strict"\fR. That hyphen between the \f(CW"UTF"\fR and the \f(CW"8"\fR is
847 critical; without it, \f(CW\*(C`Encode\*(C'\fR goes "liberal" and (perhaps overly\-)permissive:
848 .PP
849 .Vb 4
850 \& find_encoding("UTF\-8")\->name # is \*(Aqutf\-8\-strict\*(Aq
851 \& find_encoding("utf\-8")\->name # ditto. names are case insensitive
852 \& find_encoding("utf_8")\->name # ditto. "_" are treated as "\-"
853 \& find_encoding("UTF8")\->name # is \*(Aqutf8\*(Aq.
854 .Ve
855 .PP
856 Perl\*(Aqs internal UTF8 flag is called "UTF8", without a hyphen. It indicates
857 whether a string is internally encoded as "utf8", also without a hyphen.
858 .SH "SEE ALSO"
859 .IX Header "SEE ALSO"
860 Encode::Encoding,
861 Encode::Supported,
862 Encode::PerlIO,
863 encoding,
864 perlebcdic,
865 "open" in perlfunc,
866 perlunicode, perluniintro, perlunifaq, perlunitut
867 utf8,
868 the Perl Unicode Mailing List <http://lists.perl.org/list/perl\-unicode.html>
869 .SH MAINTAINER
870 .IX Header "MAINTAINER"
871 This project was originated by the late Nick Ing\-Simmons and later
872 maintained by Dan Kogai \fI<[email protected]>\fR. See AUTHORS
873 for a full list of people involved. For any questions, send mail to
874 \&\fI<perl\[email protected]>\fR so that we can all share.
875 .PP
876 While Dan Kogai retains the copyright as a maintainer, credit
877 should go to all those involved. See AUTHORS for a list of those
878 who submitted code to the project.
879 .SH COPYRIGHT
880 .IX Header "COPYRIGHT"
881 Copyright 2002\-2014 Dan Kogai \fI<[email protected]>\fR.
882 .PP
883 This library is free software; you can redistribute it and/or modify
884 it under the same terms as Perl itself.