| 1 |
.\" -*- mode: troff; coding: utf-8 -*- |
| 2 |
.\" Automatically generated by Pod::Man 5.0102 (Pod::Simple 3.45) |
| 3 |
.\" |
| 4 |
.\" Standard preamble: |
| 5 |
.\" ======================================================================== |
| 6 |
.de Sp \" Vertical space (when we can't use .PP) |
| 7 |
.if t .sp .5v |
| 8 |
.if n .sp |
| 9 |
.. |
| 10 |
.de Vb \" Begin verbatim text |
| 11 |
.ft CW |
| 12 |
.nf |
| 13 |
.ne \\$1 |
| 14 |
.. |
| 15 |
.de Ve \" End verbatim text |
| 16 |
.ft R |
| 17 |
.fi |
| 18 |
.. |
| 19 |
.\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>. |
| 20 |
.ie n \{\ |
| 21 |
. ds C` "" |
| 22 |
. ds C' "" |
| 23 |
'br\} |
| 24 |
.el\{\ |
| 25 |
. ds C` |
| 26 |
. ds C' |
| 27 |
'br\} |
| 28 |
.\" |
| 29 |
.\" Escape single quotes in literal strings from groff's Unicode transform. |
| 30 |
.ie \n(.g .ds Aq \(aq |
| 31 |
.el .ds Aq ' |
| 32 |
.\" |
| 33 |
.\" If the F register is >0, we'll generate index entries on stderr for |
| 34 |
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index |
| 35 |
.\" entries marked with X<> in POD. Of course, you'll have to process the |
| 36 |
.\" output yourself in some meaningful fashion. |
| 37 |
.\" |
| 38 |
.\" Avoid warning from groff about undefined register 'F'. |
| 39 |
.de IX |
| 40 |
.. |
| 41 |
.nr rF 0 |
| 42 |
.if \n(.g .if rF .nr rF 1 |
| 43 |
.if (\n(rF:(\n(.g==0)) \{\ |
| 44 |
. if \nF \{\ |
| 45 |
. de IX |
| 46 |
. tm Index:\\$1\t\\n%\t"\\$2" |
| 47 |
.. |
| 48 |
. if !\nF==2 \{\ |
| 49 |
. nr % 0 |
| 50 |
. nr F 2 |
| 51 |
. \} |
| 52 |
. \} |
| 53 |
.\} |
| 54 |
.rr rF |
| 55 |
.\" ======================================================================== |
| 56 |
.\" |
| 57 |
.IX Title "Encode::Unicode 3perl" |
| 58 |
.TH Encode::Unicode 3perl 2025-11-16 "perl v5.40.1" "Perl Programmers Reference Guide" |
| 59 |
.\" For nroff, turn off justification. Always turn off hyphenation; it makes |
| 60 |
.\" way too many mistakes in technical documents. |
| 61 |
.if n .ad l |
| 62 |
.nh |
| 63 |
.SH NAME |
| 64 |
Encode::Unicode \-\- Various Unicode Transformation Formats |
| 65 |
.SH SYNOPSIS |
| 66 |
.IX Header "SYNOPSIS" |
| 67 |
.Vb 3 |
| 68 |
\& use Encode qw/encode decode/; |
| 69 |
\& $ucs2 = encode("UCS\-2BE", $utf8); |
| 70 |
\& $utf8 = decode("UCS\-2BE", $ucs2); |
| 71 |
.Ve |
| 72 |
.SH ABSTRACT |
| 73 |
.IX Header "ABSTRACT" |
| 74 |
This module implements all Character Encoding Schemes of Unicode that |
| 75 |
are officially documented by Unicode Consortium (except, of course, |
| 76 |
for UTF\-8, which is a native format in perl). |
| 77 |
.IP "<http://www.unicode.org/glossary/> says:" 4 |
| 78 |
.IX Item "<http://www.unicode.org/glossary/> says:" |
| 79 |
\&\fICharacter Encoding Scheme\fR A character encoding form plus byte |
| 80 |
serialization. There are Seven character encoding schemes in Unicode: |
| 81 |
UTF\-8, UTF\-16, UTF\-16BE, UTF\-16LE, UTF\-32 (UCS\-4), UTF\-32BE (UCS\-4BE) and |
| 82 |
UTF\-32LE (UCS\-4LE), and UTF\-7. |
| 83 |
.Sp |
| 84 |
Since UTF\-7 is a 7\-bit (re)encoded version of UTF\-16BE, It is not part of |
| 85 |
Unicode's Character Encoding Scheme. It is separately implemented in |
| 86 |
Encode::Unicode::UTF7. For details see Encode::Unicode::UTF7. |
| 87 |
.IP "Quick Reference" 4 |
| 88 |
.IX Item "Quick Reference" |
| 89 |
.Vb 10 |
| 90 |
\& Decodes from ord(N) Encodes chr(N) to... |
| 91 |
\& octet/char BOM S.P d800\-dfff ord > 0xffff \ex{1abcd} == |
| 92 |
\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- |
| 93 |
\& UCS\-2BE 2 N N is bogus Not Available |
| 94 |
\& UCS\-2LE 2 N N bogus Not Available |
| 95 |
\& UTF\-16 2/4 Y Y is S.P S.P BE/LE |
| 96 |
\& UTF\-16BE 2/4 N Y S.P S.P 0xd82a,0xdfcd |
| 97 |
\& UTF\-16LE 2/4 N Y S.P S.P 0x2ad8,0xcddf |
| 98 |
\& UTF\-32 4 Y \- is bogus As is BE/LE |
| 99 |
\& UTF\-32BE 4 N \- bogus As is 0x0001abcd |
| 100 |
\& UTF\-32LE 4 N \- bogus As is 0xcdab0100 |
| 101 |
\& UTF\-8 1\-4 \- \- bogus >= 4 octets \exf0\ex9a\eaf\e8d |
| 102 |
\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- |
| 103 |
.Ve |
| 104 |
.SH "Size, Endianness, and BOM" |
| 105 |
.IX Header "Size, Endianness, and BOM" |
| 106 |
You can categorize these CES by 3 criteria: size of each character, |
| 107 |
endianness, and Byte Order Mark. |
| 108 |
.SS "by size" |
| 109 |
.IX Subsection "by size" |
| 110 |
UCS\-2 is a fixed-length encoding with each character taking 16 bits. |
| 111 |
It \fBdoes not\fR support \fIsurrogate pairs\fR. When a surrogate pair |
| 112 |
is encountered during \fBdecode()\fR, its place is filled with \ex{FFFD} |
| 113 |
if \fICHECK\fR is 0, or the routine croaks if \fICHECK\fR is 1. When a |
| 114 |
character whose ord value is larger than 0xFFFF is encountered, |
| 115 |
its place is filled with \ex{FFFD} if \fICHECK\fR is 0, or the routine |
| 116 |
croaks if \fICHECK\fR is 1. |
| 117 |
.PP |
| 118 |
UTF\-16 is almost the same as UCS\-2 but it supports \fIsurrogate pairs\fR. |
| 119 |
When it encounters a high surrogate (0xD800\-0xDBFF), it fetches the |
| 120 |
following low surrogate (0xDC00\-0xDFFF) and \f(CW\*(C`desurrogate\*(C'\fRs them to |
| 121 |
form a character. Bogus surrogates result in death. When \ex{10000} |
| 122 |
or above is encountered during \fBencode()\fR, it \f(CW\*(C`ensurrogate\*(C'\fRs them and |
| 123 |
pushes the surrogate pair to the output stream. |
| 124 |
.PP |
| 125 |
UTF\-32 (UCS\-4) is a fixed-length encoding with each character taking 32 bits. |
| 126 |
Since it is 32\-bit, there is no need for \fIsurrogate pairs\fR. |
| 127 |
.SS "by endianness" |
| 128 |
.IX Subsection "by endianness" |
| 129 |
The first (and now failed) goal of Unicode was to map all character |
| 130 |
repertoires into a fixed-length integer so that programmers are happy. |
| 131 |
Since each character is either a \fIshort\fR or \fIlong\fR in C, you have to |
| 132 |
pay attention to the endianness of each platform when you pass data |
| 133 |
to one another. |
| 134 |
.PP |
| 135 |
Anything marked as BE is Big Endian (or network byte order) and LE is |
| 136 |
Little Endian (aka VAX byte order). For anything not marked either |
| 137 |
BE or LE, a character called Byte Order Mark (BOM) indicating the |
| 138 |
endianness is prepended to the string. |
| 139 |
.PP |
| 140 |
CAVEAT: Though BOM in utf8 (\exEF\exBB\exBF) is valid, it is meaningless |
| 141 |
and as of this writing Encode suite just leave it as is (\ex{FeFF}). |
| 142 |
.IP "BOM as integer when fetched in network byte order" 4 |
| 143 |
.IX Item "BOM as integer when fetched in network byte order" |
| 144 |
.Vb 5 |
| 145 |
\& 16 32 bits/char |
| 146 |
\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- |
| 147 |
\& BE 0xFeFF 0x0000FeFF |
| 148 |
\& LE 0xFFFe 0xFFFe0000 |
| 149 |
\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- |
| 150 |
.Ve |
| 151 |
.PP |
| 152 |
This modules handles the BOM as follows. |
| 153 |
.IP \(bu 4 |
| 154 |
When BE or LE is explicitly stated as the name of encoding, BOM is |
| 155 |
simply treated as a normal character (ZERO WIDTH NO-BREAK SPACE). |
| 156 |
.IP \(bu 4 |
| 157 |
When BE or LE is omitted during \fBdecode()\fR, it checks if BOM is at the |
| 158 |
beginning of the string; if one is found, the endianness is set to |
| 159 |
what the BOM says. |
| 160 |
.IP \(bu 4 |
| 161 |
Default Byte Order |
| 162 |
.Sp |
| 163 |
When no BOM is found, Encode 2.76 and blow croaked. Since Encode |
| 164 |
2.77, it falls back to BE accordingly to RFC2781 and the Unicode |
| 165 |
Standard version 8.0 |
| 166 |
.IP \(bu 4 |
| 167 |
When BE or LE is omitted during \fBencode()\fR, it returns a BE-encoded |
| 168 |
string with BOM prepended. So when you want to encode a whole text |
| 169 |
file, make sure you \fBencode()\fR the whole text at once, not line by line |
| 170 |
or each line, not file, will have a BOM prepended. |
| 171 |
.IP \(bu 4 |
| 172 |
\&\f(CW\*(C`UCS\-2\*(C'\fR is an exception. Unlike others, this is an alias of UCS\-2BE. |
| 173 |
UCS\-2 is already registered by IANA and others that way. |
| 174 |
.SH "Surrogate Pairs" |
| 175 |
.IX Header "Surrogate Pairs" |
| 176 |
To say the least, surrogate pairs were the biggest mistake of the |
| 177 |
Unicode Consortium. But according to the late Douglas Adams in \fIThe |
| 178 |
Hitchhiker's Guide to the Galaxy\fR Trilogy, \f(CW\*(C`In the beginning the |
| 179 |
Universe was created. This has made a lot of people very angry and |
| 180 |
been widely regarded as a bad move\*(C'\fR. Their mistake was not of this |
| 181 |
magnitude so let's forgive them. |
| 182 |
.PP |
| 183 |
(I don't dare make any comparison with Unicode Consortium and the |
| 184 |
Vogons here ;) Or, comparing Encode to Babel Fish is completely |
| 185 |
appropriate \-\- if you can only stick this into your ear :) |
| 186 |
.PP |
| 187 |
Surrogate pairs were born when the Unicode Consortium finally |
| 188 |
admitted that 16 bits were not big enough to hold all the world's |
| 189 |
character repertoires. But they already made UCS\-2 16\-bit. What |
| 190 |
do we do? |
| 191 |
.PP |
| 192 |
Back then, the range 0xD800\-0xDFFF was not allocated. Let's split |
| 193 |
that range in half and use the first half to represent the \f(CW\*(C`upper |
| 194 |
half of a character\*(C'\fR and the second half to represent the \f(CW\*(C`lower |
| 195 |
half of a character\*(C'\fR. That way, you can represent 1024 * 1024 = |
| 196 |
1048576 more characters. Now we can store character ranges up to |
| 197 |
\&\ex{10ffff} even with 16\-bit encodings. This pair of half-character is |
| 198 |
now called a \fIsurrogate pair\fR and UTF\-16 is the name of the encoding |
| 199 |
that embraces them. |
| 200 |
.PP |
| 201 |
Here is a formula to ensurrogate a Unicode character \ex{10000} and |
| 202 |
above; |
| 203 |
.PP |
| 204 |
.Vb 2 |
| 205 |
\& $hi = ($uni \- 0x10000) / 0x400 + 0xD800; |
| 206 |
\& $lo = ($uni \- 0x10000) % 0x400 + 0xDC00; |
| 207 |
.Ve |
| 208 |
.PP |
| 209 |
And to desurrogate; |
| 210 |
.PP |
| 211 |
.Vb 1 |
| 212 |
\& $uni = 0x10000 + ($hi \- 0xD800) * 0x400 + ($lo \- 0xDC00); |
| 213 |
.Ve |
| 214 |
.PP |
| 215 |
Note this move has made \ex{D800}\-\ex{DFFF} into a forbidden zone but |
| 216 |
perl does not prohibit the use of characters within this range. To perl, |
| 217 |
every one of \ex{0000_0000} up to \ex{ffff_ffff} (*) is \fIa character\fR. |
| 218 |
.PP |
| 219 |
.Vb 2 |
| 220 |
\& (*) or \ex{ffff_ffff_ffff_ffff} if your perl is compiled with 64\-bit |
| 221 |
\& integer support! |
| 222 |
.Ve |
| 223 |
.SH "Error Checking" |
| 224 |
.IX Header "Error Checking" |
| 225 |
Unlike most encodings which accept various ways to handle errors, |
| 226 |
Unicode encodings simply croaks. |
| 227 |
.PP |
| 228 |
.Vb 6 |
| 229 |
\& % perl \-MEncode \-e\*(Aq$_ = "\exfe\exff\exd8\exd9\exda\exdb\e0\en"\*(Aq \e |
| 230 |
\& \-e\*(AqEncode::from_to($_, "utf16","shift_jis", 0); print\*(Aq |
| 231 |
\& UTF\-16:Malformed LO surrogate d8d9 at /path/to/Encode.pm line 184. |
| 232 |
\& % perl \-MEncode \-e\*(Aq$a = "BOM missing"\*(Aq \e |
| 233 |
\& \-e\*(Aq Encode::from_to($a, "utf16", "shift_jis", 0); print\*(Aq |
| 234 |
\& UTF\-16:Unrecognised BOM 424f at /path/to/Encode.pm line 184. |
| 235 |
.Ve |
| 236 |
.PP |
| 237 |
Unlike other encodings where mappings are not one-to-one against |
| 238 |
Unicode, UTFs are supposed to map 100% against one another. So Encode |
| 239 |
is more strict on UTFs. |
| 240 |
.PP |
| 241 |
Consider that "division by zero" of Encode :) |
| 242 |
.SH "SEE ALSO" |
| 243 |
.IX Header "SEE ALSO" |
| 244 |
Encode, Encode::Unicode::UTF7, <https://www.unicode.org/glossary/>, |
| 245 |
<https://www.unicode.org/faq/utf_bom.html>, |
| 246 |
.PP |
| 247 |
RFC 2781 <http://www.ietf.org/rfc/rfc2781.txt>, |
| 248 |
.PP |
| 249 |
The whole Unicode standard <https://www.unicode.org/standard/standard.html> |
| 250 |
.PP |
| 251 |
Ch. 6 pp. 275 of \f(CW\*(C`Programming Perl (3rd Edition)\*(C'\fR |
| 252 |
by Tom Christiansen, brian d foy & Larry Wall; |
| 253 |
O'Reilly & Associates; ISBN 978\-0\-596\-00492\-7 |