Debian Codegraph @

upstream/debian-unstable/man3/Encode::PerlIO.3perl

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man 5.0102 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\" ========================================================================
56 .\"
57 .IX Title "Encode::PerlIO 3perl"
58 .TH Encode::PerlIO 3perl 2025-11-16 "perl v5.40.1" "Perl Programmers Reference Guide"
59 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
60 .\" way too many mistakes in technical documents.
61 .if n .ad l
62 .nh
63 .SH NAME
64 Encode::PerlIO \-\- a detailed document on Encode and PerlIO
65 .SH Overview
66 .IX Header "Overview"
67 It is very common to want to do encoding transformations when
68 reading or writing files, network connections, pipes etc.
69 If Perl is configured to use the new 'perlio' IO system then
70 \&\f(CW\*(C`Encode\*(C'\fR provides a "layer" (see PerlIO) which can transform
71 data as it is read or written.
72 .PP
73 Here is how the blind poet would modernise the encoding:
74 .PP
75 .Vb 7
76 \& use Encode;
77 \& open(my $iliad,\*(Aq<:encoding(iso\-8859\-7)\*(Aq,\*(Aqiliad.greek\*(Aq);
78 \& open(my $utf8,\*(Aq>:utf8\*(Aq,\*(Aqiliad.utf8\*(Aq);
79 \& my @epic = <$iliad>;
80 \& print $utf8 @epic;
81 \& close($utf8);
82 \& close($illiad);
83 .Ve
84 .PP
85 In addition, the new IO system can also be configured to read/write
86 UTF\-8 encoded characters (as noted above, this is efficient):
87 .PP
88 .Vb 2
89 \& open(my $fh,\*(Aq>:utf8\*(Aq,\*(Aqanything\*(Aq);
90 \& print $fh "Any \ex{0021} string \eN{SMILEY FACE}\en";
91 .Ve
92 .PP
93 Either of the above forms of "layer" specifications can be made the default
94 for a lexical scope with the \f(CW\*(C`use open ...\*(C'\fR pragma. See open.
95 .PP
96 Once a handle is open, its layers can be altered using \f(CW\*(C`binmode\*(C'\fR.
97 .PP
98 Without any such configuration, or if Perl itself is built using the
99 system's own IO, then write operations assume that the file handle
100 accepts only \fIbytes\fR and will \f(CW\*(C`die\*(C'\fR if a character larger than 255 is
101 written to the handle. When reading, each octet from the handle becomes
102 a byte-in-a-character. Note that this default is the same behaviour
103 as bytes-only languages (including Perl before v5.6) would have,
104 and is sufficient to handle native 8\-bit encodings e.g. iso\-8859\-1,
105 EBCDIC etc. and any legacy mechanisms for handling other encodings
106 and binary data.
107 .PP
108 In other cases, it is the program's responsibility to transform
109 characters into bytes using the API above before doing writes, and to
110 transform the bytes read from a handle into characters before doing
111 "character operations" (e.g. \f(CW\*(C`lc\*(C'\fR, \f(CW\*(C`/\eW+/\*(C'\fR, ...).
112 .PP
113 You can also use PerlIO to convert larger amounts of data you don't
114 want to bring into memory. For example, to convert between ISO\-8859\-1
115 (Latin 1) and UTF\-8 (or UTF-EBCDIC in EBCDIC machines):
116 .PP
117 .Vb 3
118 \& open(F, "<:encoding(iso\-8859\-1)", "data.txt") or die $!;
119 \& open(G, ">:utf8", "data.utf") or die $!;
120 \& while (<F>) { print G }
121 \&
122 \& # Could also do "print G <F>" but that would pull
123 \& # the whole file into memory just to write it out again.
124 .Ve
125 .PP
126 More examples:
127 .PP
128 .Vb 3
129 \& open(my $f, "<:encoding(cp1252)")
130 \& open(my $g, ">:encoding(iso\-8859\-2)")
131 \& open(my $h, ">:encoding(latin9)") # iso\-8859\-15
132 .Ve
133 .PP
134 See also encoding for how to change the default encoding of the
135 data in your script.
136 .SH "How does it work?"
137 .IX Header "How does it work?"
138 Here is a crude diagram of how filehandle, PerlIO, and Encode
139 interact.
140 .PP
141 .Vb 3
142 \& filehandle <\-> PerlIO PerlIO <\-> scalar (read/printed)
143 \& \e /
144 \& Encode
145 .Ve
146 .PP
147 When PerlIO receives data from either direction, it fills a buffer
148 (currently with 1024 bytes) and passes the buffer to Encode.
149 Encode tries to convert the valid part and passes it back to PerlIO,
150 leaving invalid parts (usually a partial character) in the buffer.
151 PerlIO then appends more data to the buffer, calls Encode again,
152 and so on until the data stream ends.
153 .PP
154 To do so, PerlIO always calls (de|en)code methods with CHECK set to 1.
155 This ensures that the method stops at the right place when it
156 encounters partial character. The following is what happens when
157 PerlIO and Encode tries to encode (from utf8) more than 1024 bytes
158 and the buffer boundary happens to be in the middle of a character.
159 .PP
160 .Vb 5
161 \& A B C .... ~ \ex{3000} ....
162 \& 41 42 43 .... 7E e3 80 80 ....
163 \& <\- buffer \-\-\-\-\-\-\-\-\-\-\-\-\-\-\->
164 \& << encoded >>>>>>>>>>
165 \& <\- next buffer \-\-\-\-\-\-
166 .Ve
167 .PP
168 Encode converts from the beginning to \ex7E, leaving \exe3 in the buffer
169 because it is invalid (partial character).
170 .PP
171 Unfortunately, this scheme does not work well with escape-based
172 encodings such as ISO\-2022\-JP.
173 .SH "Line Buffering"
174 .IX Header "Line Buffering"
175 Now let's see what happens when you try to decode from ISO\-2022\-JP and
176 the buffer ends in the middle of a character.
177 .PP
178 .Vb 5
179 \& JIS208\-ESC \ex{5f3e}
180 \& A B C .... ~ \ee $ B |DAN | ....
181 \& 41 42 43 .... 7E 1b 24 41 43 46 ....
182 \& <\- buffer \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\->
183 \& << encoded >>>>>>>>>>>>>>>>>>>>>>>
184 .Ve
185 .PP
186 As you see, the next buffer begins with \ex43. But \ex43 is 'C' in
187 ASCII, which is wrong in this case because we are now in JISX 0208
188 area so it has to convert \ex43\ex46, not \ex43. Unlike utf8 and EUC,
189 in escape-based encodings you can't tell if a given octet is a whole
190 character or just part of it.
191 .PP
192 Fortunately PerlIO also supports line buffer if you tell PerlIO to use
193 one instead of fixed buffer. Since ISO\-2022\-JP is guaranteed to revert to ASCII at the end of the line, partial
194 character will never happen when line buffer is used.
195 .PP
196 To tell PerlIO to use line buffer, implement \->needs_lines method
197 for your encoding object. See Encode::Encoding for details.
198 .PP
199 Thanks to these efforts most encodings that come with Encode support
200 PerlIO but that still leaves following encodings.
201 .PP
202 .Vb 4
203 \& iso\-2022\-kr
204 \& MIME\-B
205 \& MIME\-Header
206 \& MIME\-Q
207 .Ve
208 .PP
209 Fortunately iso\-2022\-kr is hardly used (according to Jungshik) and
210 MIME\-* are very unlikely to be fed to PerlIO because they are for mail
211 headers. See Encode::MIME::Header for details.
212 .SS "How can I tell whether my encoding fully supports PerlIO ?"
213 .IX Subsection "How can I tell whether my encoding fully supports PerlIO ?"
214 As of this writing, any encoding whose class belongs to Encode::XS and
215 Encode::Unicode works. The Encode module has a \f(CW\*(C`perlio_ok\*(C'\fR method
216 which you can use before applying PerlIO encoding to the filehandle.
217 Here is an example:
218 .PP
219 .Vb 7
220 \& my $use_perlio = perlio_ok($enc);
221 \& my $layer = $use_perlio ? "<:raw" : "<:encoding($enc)";
222 \& open my $fh, $layer, $file or die "$file : $!";
223 \& while(<$fh>){
224 \& $_ = decode($enc, $_) unless $use_perlio;
225 \& # ....
226 \& }
227 .Ve
228 .SH "SEE ALSO"
229 .IX Header "SEE ALSO"
230 Encode::Encoding,
231 Encode::Supported,
232 Encode::PerlIO,
233 encoding,
234 perlebcdic,
235 "open" in perlfunc,
236 perlunicode,
237 utf8,
238 the Perl Unicode Mailing List <perl\[email protected]>