Debian Codegraph @

upstream/opensuse-tumbleweed/man1/perlunitut.1

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man v6.0.2 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\"
56 .\" Required to disable full justification in groff 1.23.0.
57 .if n .ds AD l
58 .\" ========================================================================
59 .\"
60 .IX Title "PERLUNITUT 1"
61 .TH PERLUNITUT 1 2026-03-09 "perl v5.42.1" "Perl Programmers Reference Guide"
62 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
63 .\" way too many mistakes in technical documents.
64 .if n .ad l
65 .nh
66 .SH NAME
67 perlunitut \- Perl Unicode Tutorial
68 .SH DESCRIPTION
69 .IX Header "DESCRIPTION"
70 The days of just flinging strings around are over. It\*(Aqs well established that
71 modern programs need to be capable of communicating funny accented letters, and
72 things like euro symbols. This means that programmers need new habits. It\*(Aqs
73 easy to program Unicode capable software, but it does require discipline to do
74 it right.
75 .PP
76 There\*(Aqs a lot to know about character sets, and text encodings. It\*(Aqs probably
77 best to spend a full day learning all this, but the basics can be learned in
78 minutes.
79 .PP
80 These are not the very basics, though. It is assumed that you already
81 know the difference between bytes and characters, and realise (and accept!)
82 that there are many different character sets and encodings, and that your
83 program has to be explicit about them. Recommended reading is "The Absolute
84 Minimum Every Software Developer Absolutely, Positively Must Know About Unicode
85 and Character Sets (No Excuses!)" by Joel Spolsky, at
86 <https://joelonsoftware.com/articles/Unicode.html>.
87 .PP
88 This tutorial speaks in rather absolute terms, and provides only a limited view
89 of the wealth of character string related features that Perl has to offer. For
90 most projects, this information will probably suffice.
91 .SS Definitions
92 .IX Subsection "Definitions"
93 It\*(Aqs important to set a few things straight first. This is the most important
94 part of this tutorial. This view may conflict with other information that you
95 may have found on the web, but that\*(Aqs mostly because many sources are wrong.
96 .PP
97 You may have to re\-read this entire section a few times...
98 .PP
99 \fIUnicode\fR
100 .IX Subsection "Unicode"
101 .PP
102 \&\fBUnicode\fR is a character set with room for lots of characters. The ordinal
103 value of a character is called a \fBcode point\fR. (But in practice, the
104 distinction between code point and character is blurred, so the terms often
105 are used interchangeably.)
106 .PP
107 There are many, many code points, but computers work with bytes, and a byte has
108 room for only 256 values. Unicode has many more characters than that,
109 so you need a method to make these accessible.
110 .PP
111 Unicode is encoded using several competing encodings, of which UTF\-8 is the
112 most used. In a Unicode encoding, multiple subsequent bytes can be used to
113 store a single code point, or simply: character.
114 .PP
115 \fIUTF\-8\fR
116 .IX Subsection "UTF-8"
117 .PP
118 \&\fBUTF\-8\fR is a Unicode encoding. Many people think that Unicode and UTF\-8 are
119 the same thing, but they\*(Aqre not. There are more Unicode encodings, but much of
120 the world has standardized on UTF\-8.
121 .PP
122 UTF\-8 treats the first 128 codepoints, 0..127, the same as ASCII. They take
123 only one byte per character. All other characters are encoded as two to
124 four bytes using a complex scheme. Fortunately, Perl handles this for
125 us, so we don\*(Aqt have to worry about this.
126 .PP
127 \fIText strings (character strings)\fR
128 .IX Subsection "Text strings (character strings)"
129 .PP
130 \&\fBText strings\fR, or \fBcharacter strings\fR are made of characters. Bytes are
131 irrelevant here, and so are encodings. Each character is just that: the
132 character.
133 .PP
134 On a text string, you would do things like:
135 .PP
136 .Vb 4
137 \& $text =~ s/foo/bar/;
138 \& if ($string =~ /^\ed+$/) { ... }
139 \& $text = ucfirst $text;
140 \& my $character_count = length $text;
141 .Ve
142 .PP
143 The value of a character (\f(CW\*(C`ord\*(C'\fR, \f(CW\*(C`chr\*(C'\fR) is the corresponding Unicode code
144 point.
145 .PP
146 \fIBinary strings (byte strings)\fR
147 .IX Subsection "Binary strings (byte strings)"
148 .PP
149 \&\fBBinary strings\fR, or \fBbyte strings\fR are made of bytes. Here, you don\*(Aqt have
150 characters, just bytes. All communication with the outside world (anything
151 outside of your current Perl process) is done in binary.
152 .PP
153 On a binary string, you would do things like:
154 .PP
155 .Vb 4
156 \& my (@length_content) = unpack "(V/a)*", $binary;
157 \& $binary =~ s/\ex00\ex0F/\exFF\exF0/; # for the brave :)
158 \& print {$fh} $binary;
159 \& my $byte_count = length $binary;
160 .Ve
161 .PP
162 \fIEncoding\fR
163 .IX Subsection "Encoding"
164 .PP
165 \&\fBEncoding\fR (as a verb) is the conversion from \fItext\fR to \fIbinary\fR. To encode,
166 you have to supply the target encoding, for example \f(CW\*(C`iso\-8859\-1\*(C'\fR or \f(CW\*(C`UTF\-8\*(C'\fR.
167 Some encodings, like the \f(CW\*(C`iso\-8859\*(C'\fR ("latin") range, do not support the full
168 Unicode standard; characters that can\*(Aqt be represented are lost in the
169 conversion.
170 .PP
171 \fIDecoding\fR
172 .IX Subsection "Decoding"
173 .PP
174 \&\fBDecoding\fR is the conversion from \fIbinary\fR to \fItext\fR. To decode, you have to
175 know what encoding was used during the encoding phase. And most of all, it must
176 be something decodable. It doesn\*(Aqt make much sense to decode a PNG image into a
177 text string.
178 .PP
179 \fIInternal format\fR
180 .IX Subsection "Internal format"
181 .PP
182 Perl has an \fBinternal format\fR, an encoding that it uses to encode text strings
183 so it can store them in memory. All text strings are in this internal format.
184 In fact, text strings are never in any other format!
185 .PP
186 You shouldn\*(Aqt worry about what this format is, because conversion is
187 automatically done when you decode or encode.
188 .SS "Your new toolkit"
189 .IX Subsection "Your new toolkit"
190 Add to your standard heading the following line:
191 .PP
192 .Vb 1
193 \& use Encode qw(encode decode);
194 .Ve
195 .PP
196 Or, if you\*(Aqre lazy, just:
197 .PP
198 .Vb 1
199 \& use Encode;
200 .Ve
201 .SS "I/O flow (the actual 5 minute tutorial)"
202 .IX Subsection "I/O flow (the actual 5 minute tutorial)"
203 The typical input/output flow of a program is:
204 .PP
205 .Vb 3
206 \& 1. Receive and decode
207 \& 2. Process
208 \& 3. Encode and output
209 .Ve
210 .PP
211 If your input is binary, and is supposed to remain binary, you shouldn\*(Aqt decode
212 it to a text string, of course. But in all other cases, you should decode it.
213 .PP
214 Decoding can\*(Aqt happen reliably if you don\*(Aqt know how the data was encoded. If
215 you get to choose, it\*(Aqs a good idea to standardize on UTF\-8.
216 .PP
217 .Vb 3
218 \& my $foo = decode(\*(AqUTF\-8\*(Aq, get \*(Aqhttp://example.com/\*(Aq);
219 \& my $bar = decode(\*(AqISO\-8859\-1\*(Aq, readline STDIN);
220 \& my $xyzzy = decode(\*(AqWindows\-1251\*(Aq, $cgi\->param(\*(Aqfoo\*(Aq));
221 .Ve
222 .PP
223 Processing happens as you knew before. The only difference is that you\*(Aqre now
224 using characters instead of bytes. That\*(Aqs very useful if you use things like
225 \&\f(CW\*(C`substr\*(C'\fR, or \f(CW\*(C`length\*(C'\fR.
226 .PP
227 It\*(Aqs important to realize that there are no bytes in a text string. Of course,
228 Perl has its internal encoding to store the string in memory, but ignore that.
229 If you have to do anything with the number of bytes, it\*(Aqs probably best to move
230 that part to step 3, just after you\*(Aqve encoded the string. Then you know
231 exactly how many bytes it will be in the destination string.
232 .PP
233 The syntax for encoding text strings to binary strings is as simple as decoding:
234 .PP
235 .Vb 1
236 \& $body = encode(\*(AqUTF\-8\*(Aq, $body);
237 .Ve
238 .PP
239 If you needed to know the length of the string in bytes, now\*(Aqs the perfect time
240 for that. Because \f(CW$body\fR is now a byte string, \f(CW\*(C`length\*(C'\fR will report the
241 number of bytes, instead of the number of characters. The number of
242 characters is no longer known, because characters only exist in text strings.
243 .PP
244 .Vb 1
245 \& my $byte_count = length $body;
246 .Ve
247 .PP
248 And if the protocol you\*(Aqre using supports a way of letting the recipient know
249 which character encoding you used, please help the receiving end by using that
250 feature! For example, E\-mail and HTTP support MIME headers, so you can use the
251 \&\f(CW\*(C`Content\-Type\*(C'\fR header. They can also have \f(CW\*(C`Content\-Length\*(C'\fR to indicate the
252 number of \fIbytes\fR, which is always a good idea to supply if the number is
253 known.
254 .PP
255 .Vb 2
256 \& "Content\-Type: text/plain; charset=UTF\-8",
257 \& "Content\-Length: $byte_count"
258 .Ve
259 .SH SUMMARY
260 .IX Header "SUMMARY"
261 Decode everything you receive, encode everything you send out. (If it\*(Aqs text
262 data.)
263 .SH "Q and A (or FAQ)"
264 .IX Header "Q and A (or FAQ)"
265 After reading this document, you ought to read perlunifaq too, then
266 perluniintro.
267 .SH ACKNOWLEDGEMENTS
268 .IX Header "ACKNOWLEDGEMENTS"
269 Thanks to Johan Vromans from Squirrel Consultancy. His UTF\-8 rants during the
270 Amsterdam Perl Mongers meetings got me interested and determined to find out
271 how to use character encodings in Perl in ways that don\*(Aqt break easily.
272 .PP
273 Thanks to Gerard Goossen from TTY. His presentation "UTF\-8 in the wild" (Dutch
274 Perl Workshop 2006) inspired me to publish my thoughts and write this tutorial.
275 .PP
276 Thanks to the people who asked about this kind of stuff in several Perl IRC
277 channels, and have constantly reminded me that a simpler explanation was
278 needed.
279 .PP
280 Thanks to the people who reviewed this document for me, before it went public.
281 They are: Benjamin Smith, Jan\-Pieter Cornet, Johan Vromans, Lukas Mai, Nathan
282 Gray.
283 .SH AUTHOR
284 .IX Header "AUTHOR"
285 Juerd Waalboer <#####@juerd.nl>
286 .SH "SEE ALSO"
287 .IX Header "SEE ALSO"
288 perlunifaq, perlunicode, perluniintro, Encode