Debian Codegraph @

upstream/debian-unstable/man1/perlunitut.1

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man 5.0102 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\" ========================================================================
56 .\"
57 .IX Title "PERLUNITUT 1"
58 .TH PERLUNITUT 1 2025-11-16 "perl v5.40.1" "Perl Programmers Reference Guide"
59 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
60 .\" way too many mistakes in technical documents.
61 .if n .ad l
62 .nh
63 .SH NAME
64 perlunitut \- Perl Unicode Tutorial
65 .SH DESCRIPTION
66 .IX Header "DESCRIPTION"
67 The days of just flinging strings around are over. It's well established that
68 modern programs need to be capable of communicating funny accented letters, and
69 things like euro symbols. This means that programmers need new habits. It's
70 easy to program Unicode capable software, but it does require discipline to do
71 it right.
72 .PP
73 There's a lot to know about character sets, and text encodings. It's probably
74 best to spend a full day learning all this, but the basics can be learned in
75 minutes.
76 .PP
77 These are not the very basics, though. It is assumed that you already
78 know the difference between bytes and characters, and realise (and accept!)
79 that there are many different character sets and encodings, and that your
80 program has to be explicit about them. Recommended reading is "The Absolute
81 Minimum Every Software Developer Absolutely, Positively Must Know About Unicode
82 and Character Sets (No Excuses!)" by Joel Spolsky, at
83 <https://joelonsoftware.com/articles/Unicode.html>.
84 .PP
85 This tutorial speaks in rather absolute terms, and provides only a limited view
86 of the wealth of character string related features that Perl has to offer. For
87 most projects, this information will probably suffice.
88 .SS Definitions
89 .IX Subsection "Definitions"
90 It's important to set a few things straight first. This is the most important
91 part of this tutorial. This view may conflict with other information that you
92 may have found on the web, but that's mostly because many sources are wrong.
93 .PP
94 You may have to re-read this entire section a few times...
95 .PP
96 \fIUnicode\fR
97 .IX Subsection "Unicode"
98 .PP
99 \&\fBUnicode\fR is a character set with room for lots of characters. The ordinal
100 value of a character is called a \fBcode point\fR. (But in practice, the
101 distinction between code point and character is blurred, so the terms often
102 are used interchangeably.)
103 .PP
104 There are many, many code points, but computers work with bytes, and a byte has
105 room for only 256 values. Unicode has many more characters than that,
106 so you need a method to make these accessible.
107 .PP
108 Unicode is encoded using several competing encodings, of which UTF\-8 is the
109 most used. In a Unicode encoding, multiple subsequent bytes can be used to
110 store a single code point, or simply: character.
111 .PP
112 \fIUTF\-8\fR
113 .IX Subsection "UTF-8"
114 .PP
115 \&\fBUTF\-8\fR is a Unicode encoding. Many people think that Unicode and UTF\-8 are
116 the same thing, but they're not. There are more Unicode encodings, but much of
117 the world has standardized on UTF\-8.
118 .PP
119 UTF\-8 treats the first 128 codepoints, 0..127, the same as ASCII. They take
120 only one byte per character. All other characters are encoded as two to
121 four bytes using a complex scheme. Fortunately, Perl handles this for
122 us, so we don't have to worry about this.
123 .PP
124 \fIText strings (character strings)\fR
125 .IX Subsection "Text strings (character strings)"
126 .PP
127 \&\fBText strings\fR, or \fBcharacter strings\fR are made of characters. Bytes are
128 irrelevant here, and so are encodings. Each character is just that: the
129 character.
130 .PP
131 On a text string, you would do things like:
132 .PP
133 .Vb 4
134 \& $text =~ s/foo/bar/;
135 \& if ($string =~ /^\ed+$/) { ... }
136 \& $text = ucfirst $text;
137 \& my $character_count = length $text;
138 .Ve
139 .PP
140 The value of a character (\f(CW\*(C`ord\*(C'\fR, \f(CW\*(C`chr\*(C'\fR) is the corresponding Unicode code
141 point.
142 .PP
143 \fIBinary strings (byte strings)\fR
144 .IX Subsection "Binary strings (byte strings)"
145 .PP
146 \&\fBBinary strings\fR, or \fBbyte strings\fR are made of bytes. Here, you don't have
147 characters, just bytes. All communication with the outside world (anything
148 outside of your current Perl process) is done in binary.
149 .PP
150 On a binary string, you would do things like:
151 .PP
152 .Vb 4
153 \& my (@length_content) = unpack "(V/a)*", $binary;
154 \& $binary =~ s/\ex00\ex0F/\exFF\exF0/; # for the brave :)
155 \& print {$fh} $binary;
156 \& my $byte_count = length $binary;
157 .Ve
158 .PP
159 \fIEncoding\fR
160 .IX Subsection "Encoding"
161 .PP
162 \&\fBEncoding\fR (as a verb) is the conversion from \fItext\fR to \fIbinary\fR. To encode,
163 you have to supply the target encoding, for example \f(CW\*(C`iso\-8859\-1\*(C'\fR or \f(CW\*(C`UTF\-8\*(C'\fR.
164 Some encodings, like the \f(CW\*(C`iso\-8859\*(C'\fR ("latin") range, do not support the full
165 Unicode standard; characters that can't be represented are lost in the
166 conversion.
167 .PP
168 \fIDecoding\fR
169 .IX Subsection "Decoding"
170 .PP
171 \&\fBDecoding\fR is the conversion from \fIbinary\fR to \fItext\fR. To decode, you have to
172 know what encoding was used during the encoding phase. And most of all, it must
173 be something decodable. It doesn't make much sense to decode a PNG image into a
174 text string.
175 .PP
176 \fIInternal format\fR
177 .IX Subsection "Internal format"
178 .PP
179 Perl has an \fBinternal format\fR, an encoding that it uses to encode text strings
180 so it can store them in memory. All text strings are in this internal format.
181 In fact, text strings are never in any other format!
182 .PP
183 You shouldn't worry about what this format is, because conversion is
184 automatically done when you decode or encode.
185 .SS "Your new toolkit"
186 .IX Subsection "Your new toolkit"
187 Add to your standard heading the following line:
188 .PP
189 .Vb 1
190 \& use Encode qw(encode decode);
191 .Ve
192 .PP
193 Or, if you're lazy, just:
194 .PP
195 .Vb 1
196 \& use Encode;
197 .Ve
198 .SS "I/O flow (the actual 5 minute tutorial)"
199 .IX Subsection "I/O flow (the actual 5 minute tutorial)"
200 The typical input/output flow of a program is:
201 .PP
202 .Vb 3
203 \& 1. Receive and decode
204 \& 2. Process
205 \& 3. Encode and output
206 .Ve
207 .PP
208 If your input is binary, and is supposed to remain binary, you shouldn't decode
209 it to a text string, of course. But in all other cases, you should decode it.
210 .PP
211 Decoding can't happen reliably if you don't know how the data was encoded. If
212 you get to choose, it's a good idea to standardize on UTF\-8.
213 .PP
214 .Vb 3
215 \& my $foo = decode(\*(AqUTF\-8\*(Aq, get \*(Aqhttp://example.com/\*(Aq);
216 \& my $bar = decode(\*(AqISO\-8859\-1\*(Aq, readline STDIN);
217 \& my $xyzzy = decode(\*(AqWindows\-1251\*(Aq, $cgi\->param(\*(Aqfoo\*(Aq));
218 .Ve
219 .PP
220 Processing happens as you knew before. The only difference is that you're now
221 using characters instead of bytes. That's very useful if you use things like
222 \&\f(CW\*(C`substr\*(C'\fR, or \f(CW\*(C`length\*(C'\fR.
223 .PP
224 It's important to realize that there are no bytes in a text string. Of course,
225 Perl has its internal encoding to store the string in memory, but ignore that.
226 If you have to do anything with the number of bytes, it's probably best to move
227 that part to step 3, just after you've encoded the string. Then you know
228 exactly how many bytes it will be in the destination string.
229 .PP
230 The syntax for encoding text strings to binary strings is as simple as decoding:
231 .PP
232 .Vb 1
233 \& $body = encode(\*(AqUTF\-8\*(Aq, $body);
234 .Ve
235 .PP
236 If you needed to know the length of the string in bytes, now's the perfect time
237 for that. Because \f(CW$body\fR is now a byte string, \f(CW\*(C`length\*(C'\fR will report the
238 number of bytes, instead of the number of characters. The number of
239 characters is no longer known, because characters only exist in text strings.
240 .PP
241 .Vb 1
242 \& my $byte_count = length $body;
243 .Ve
244 .PP
245 And if the protocol you're using supports a way of letting the recipient know
246 which character encoding you used, please help the receiving end by using that
247 feature! For example, E\-mail and HTTP support MIME headers, so you can use the
248 \&\f(CW\*(C`Content\-Type\*(C'\fR header. They can also have \f(CW\*(C`Content\-Length\*(C'\fR to indicate the
249 number of \fIbytes\fR, which is always a good idea to supply if the number is
250 known.
251 .PP
252 .Vb 2
253 \& "Content\-Type: text/plain; charset=UTF\-8",
254 \& "Content\-Length: $byte_count"
255 .Ve
256 .SH SUMMARY
257 .IX Header "SUMMARY"
258 Decode everything you receive, encode everything you send out. (If it's text
259 data.)
260 .SH "Q and A (or FAQ)"
261 .IX Header "Q and A (or FAQ)"
262 After reading this document, you ought to read perlunifaq too, then
263 perluniintro.
264 .SH ACKNOWLEDGEMENTS
265 .IX Header "ACKNOWLEDGEMENTS"
266 Thanks to Johan Vromans from Squirrel Consultancy. His UTF\-8 rants during the
267 Amsterdam Perl Mongers meetings got me interested and determined to find out
268 how to use character encodings in Perl in ways that don't break easily.
269 .PP
270 Thanks to Gerard Goossen from TTY. His presentation "UTF\-8 in the wild" (Dutch
271 Perl Workshop 2006) inspired me to publish my thoughts and write this tutorial.
272 .PP
273 Thanks to the people who asked about this kind of stuff in several Perl IRC
274 channels, and have constantly reminded me that a simpler explanation was
275 needed.
276 .PP
277 Thanks to the people who reviewed this document for me, before it went public.
278 They are: Benjamin Smith, Jan-Pieter Cornet, Johan Vromans, Lukas Mai, Nathan
279 Gray.
280 .SH AUTHOR
281 .IX Header "AUTHOR"
282 Juerd Waalboer <#####@juerd.nl>
283 .SH "SEE ALSO"
284 .IX Header "SEE ALSO"
285 perlunifaq, perlunicode, perluniintro, Encode