Debian Codegraph @

upstream/mageia-cauldron/man1/enc2xs.1

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man v6.0.2 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\"
56 .\" Required to disable full justification in groff 1.23.0.
57 .if n .ds AD l
58 .\" ========================================================================
59 .\"
60 .IX Title "ENC2XS 1"
61 .TH ENC2XS 1 2026-05-26 "perl v5.42.0" "Perl Programmers Reference Guide"
62 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
63 .\" way too many mistakes in technical documents.
64 .if n .ad l
65 .nh
66 .SH NAME
67 enc2xs \-\- Perl Encode Module Generator
68 .SH SYNOPSIS
69 .IX Header "SYNOPSIS"
70 .Vb 3
71 \& enc2xs \-[options]
72 \& enc2xs \-M ModName mapfiles...
73 \& enc2xs \-C
74 .Ve
75 .SH DESCRIPTION
76 .IX Header "DESCRIPTION"
77 \&\fIenc2xs\fR builds a Perl extension for use by Encode from either
78 Unicode Character Mapping files (.ucm) or Tcl Encoding Files (.enc).
79 Besides being used internally during the build process of the Encode
80 module, you can use \fIenc2xs\fR to add your own encoding to perl.
81 No knowledge of XS is necessary.
82 .SH "Quick Guide"
83 .IX Header "Quick Guide"
84 If you want to know as little about Perl as possible but need to
85 add a new encoding, just read this chapter and forget the rest.
86 .IP 0. 4
87 .IX Item "0."
88 Have a .ucm file ready. You can get it from somewhere or you can write
89 your own from scratch or you can grab one from the Encode distribution
90 and customize it. For the UCM format, see the next Chapter. In the
91 example below, I\*(Aqll call my theoretical encoding myascii, defined
92 in \fImy.ucm\fR. \f(CW\*(C`$\*(C'\fR is a shell prompt.
93 .Sp
94 .Vb 2
95 \& $ ls \-F
96 \& my.ucm
97 .Ve
98 .IP 1. 4
99 .IX Item "1."
100 Issue a command as follows;
101 .Sp
102 .Vb 5
103 \& $ enc2xs \-M My my.ucm
104 \& generating Makefile.PL
105 \& generating My.pm
106 \& generating README
107 \& generating Changes
108 .Ve
109 .Sp
110 Now take a look at your current directory. It should look like this.
111 .Sp
112 .Vb 2
113 \& $ ls \-F
114 \& Makefile.PL My.pm my.ucm t/
115 .Ve
116 .Sp
117 The following files were created.
118 .Sp
119 .Vb 3
120 \& Makefile.PL \- MakeMaker script
121 \& My.pm \- Encode submodule
122 \& t/My.t \- test file
123 .Ve
124 .RS 4
125 .IP 1.1. 4
126 .IX Item "1.1."
127 If you want *.ucm installed together with the modules, do as follows;
128 .Sp
129 .Vb 3
130 \& $ mkdir Encode
131 \& $ mv *.ucm Encode
132 \& $ enc2xs \-M My Encode/*ucm
133 .Ve
134 .RE
135 .RS 4
136 .RE
137 .IP 2. 4
138 .IX Item "2."
139 Edit the files generated. You don\*(Aqt have to if you have no time AND no
140 intention to give it to someone else. But it is a good idea to edit
141 the pod and to add more tests.
142 .IP 3. 4
143 .IX Item "3."
144 Now issue a command all Perl Mongers love:
145 .Sp
146 .Vb 2
147 \& $ perl Makefile.PL
148 \& Writing Makefile for Encode::My
149 .Ve
150 .IP 4. 4
151 .IX Item "4."
152 Now all you have to do is make.
153 .Sp
154 .Vb 12
155 \& $ make
156 \& cp My.pm blib/lib/Encode/My.pm
157 \& /usr/local/bin/perl /usr/local/bin/enc2xs \-Q \-O \e
158 \& \-o encode_t.c \-f encode_t.fnm
159 \& Reading myascii (myascii)
160 \& Writing compiled form
161 \& 128 bytes in string tables
162 \& 384 bytes (75%) saved spotting duplicates
163 \& 1 bytes (0.775%) saved using substrings
164 \& ....
165 \& chmod 644 blib/arch/auto/Encode/My/My.bs
166 \& $
167 .Ve
168 .Sp
169 The time it takes varies depending on how fast your machine is and
170 how large your encoding is. Unless you are working on something big
171 like euc\-tw, it won\*(Aqt take too long.
172 .IP 5. 4
173 .IX Item "5."
174 You can "make install" already but you should test first.
175 .Sp
176 .Vb 8
177 \& $ make test
178 \& PERL_DL_NONLAZY=1 /usr/local/bin/perl \-Iblib/arch \-Iblib/lib \e
179 \& \-e \*(Aquse Test::Harness qw(&runtests $verbose); \e
180 \& $verbose=0; runtests @ARGV;\*(Aq t/*.t
181 \& t/My....ok
182 \& All tests successful.
183 \& Files=1, Tests=2, 0 wallclock secs
184 \& ( 0.09 cusr + 0.01 csys = 0.09 CPU)
185 .Ve
186 .IP 6. 4
187 .IX Item "6."
188 If you are content with the test result, just "make install"
189 .IP 7. 4
190 .IX Item "7."
191 If you want to add your encoding to Encode\*(Aqs demand\-loading list
192 (so you don\*(Aqt have to "use Encode::YourEncoding"), run
193 .Sp
194 .Vb 1
195 \& enc2xs \-C
196 .Ve
197 .Sp
198 to update Encode::ConfigLocal, a module that controls local settings.
199 After that, "use Encode;" is enough to load your encodings on demand.
200 .SH "The Unicode Character Map"
201 .IX Header "The Unicode Character Map"
202 Encode uses the Unicode Character Map (UCM) format for source character
203 mappings. This format is used by IBM\*(Aqs ICU package and was adopted
204 by Nick Ing\-Simmons for use with the Encode module. Since UCM is
205 more flexible than Tcl\*(Aqs Encoding Map and far more user\-friendly,
206 this is the recommended format for Encode now.
207 .PP
208 A UCM file looks like this.
209 .PP
210 .Vb 10
211 \& #
212 \& # Comments
213 \& #
214 \& <code_set_name> "US\-ascii" # Required
215 \& <code_set_alias> "ascii" # Optional
216 \& <mb_cur_min> 1 # Required; usually 1
217 \& <mb_cur_max> 1 # Max. # of bytes/char
218 \& <subchar> \ex3F # Substitution char
219 \& #
220 \& CHARMAP
221 \& <U0000> \ex00 |0 # <control>
222 \& <U0001> \ex01 |0 # <control>
223 \& <U0002> \ex02 |0 # <control>
224 \& ....
225 \& <U007C> \ex7C |0 # VERTICAL LINE
226 \& <U007D> \ex7D |0 # RIGHT CURLY BRACKET
227 \& <U007E> \ex7E |0 # TILDE
228 \& <U007F> \ex7F |0 # <control>
229 \& END CHARMAP
230 .Ve
231 .IP \(bu 4
232 Anything that follows \f(CW\*(C`#\*(C'\fR is treated as a comment.
233 .IP \(bu 4
234 The header section continues until a line containing the word
235 CHARMAP. This section has a form of \fI<keyword> value\fR, one
236 pair per line. Strings used as values must be quoted. Barewords are
237 treated as numbers. \fI\exXX\fR represents a byte.
238 .Sp
239 Most of the keywords are self\-explanatory. \fIsubchar\fR means
240 substitution character, not subcharacter. When you decode a Unicode
241 sequence to this encoding but no matching character is found, the byte
242 sequence defined here will be used. For most cases, the value here is
243 \&\ex3F; in ASCII, this is a question mark.
244 .IP \(bu 4
245 CHARMAP starts the character map section. Each line has a form as
246 follows:
247 .Sp
248 .Vb 5
249 \& <UXXXX> \exXX.. |0 # comment
250 \& ^ ^ ^
251 \& | | +\- Fallback flag
252 \& | +\-\-\-\-\-\-\-\- Encoded byte sequence
253 \& +\-\-\-\-\-\-\-\-\-\-\-\-\-\- Unicode Character ID in hex
254 .Ve
255 .Sp
256 The format is roughly the same as a header section except for the
257 fallback flag: | followed by 0..3. The meaning of the possible
258 values is as follows:
259 .RS 4
260 .IP |0 4
261 .IX Item "|0"
262 Round trip safe. A character decoded to Unicode encodes back to the
263 same byte sequence. Most characters have this flag.
264 .IP |1 4
265 .IX Item "|1"
266 Fallback for unicode \-> encoding. When seen, enc2xs adds this
267 character for the encode map only.
268 .IP |2 4
269 .IX Item "|2"
270 Skip sub\-char mapping should there be no code point.
271 .IP |3 4
272 .IX Item "|3"
273 Fallback for encoding \-> unicode. When seen, enc2xs adds this
274 character for the decode map only.
275 .RE
276 .RS 4
277 .RE
278 .IP \(bu 4
279 And finally, END OF CHARMAP ends the section.
280 .PP
281 When you are manually creating a UCM file, you should copy ascii.ucm
282 or an existing encoding which is close to yours, rather than write
283 your own from scratch.
284 .PP
285 When you do so, make sure you leave at least \fBU0000\fR to \fBU0020\fR as
286 is, unless your environment is EBCDIC.
287 .PP
288 \&\fBCAVEAT\fR: not all features in UCM are implemented. For example,
289 icu:state is not used. Because of that, you need to write a perl
290 module if you want to support algorithmical encodings, notably
291 the ISO\-2022 series. Such modules include Encode::JP::2022_JP,
292 Encode::KR::2022_KR, and Encode::TW::HZ.
293 .SS "Coping with duplicate mappings"
294 .IX Subsection "Coping with duplicate mappings"
295 When you create a map, you SHOULD make your mappings round\-trip safe.
296 That is, \f(CWencode(\*(Aqyour\-encoding\*(Aq, decode(\*(Aqyour\-encoding\*(Aq, $data)) eq
297 $data\fR stands for all characters that are marked as \f(CW\*(C`|0\*(C'\fR. Here is
298 how to make sure:
299 .IP \(bu 4
300 Sort your map in Unicode order.
301 .IP \(bu 4
302 When you have a duplicate entry, mark either one with \*(Aq|1\*(Aq or \*(Aq|3\*(Aq.
303 .IP \(bu 4
304 And make sure the \*(Aq|1\*(Aq or \*(Aq|3\*(Aq entry FOLLOWS the \*(Aq|0\*(Aq entry.
305 .PP
306 Here is an example from big5\-eten.
307 .PP
308 .Vb 2
309 \& <U2550> \exF9\exF9 |0
310 \& <U2550> \exA2\exA4 |3
311 .Ve
312 .PP
313 Internally Encoding \-> Unicode and Unicode \-> Encoding Map looks like
314 this;
315 .PP
316 .Vb 4
317 \& E to U U to E
318 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
319 \& \exF9\exF9 => U2550 U2550 => \exF9\exF9
320 \& \exA2\exA4 => U2550
321 .Ve
322 .PP
323 So it is round\-trip safe for \exF9\exF9. But if the line above is upside
324 down, here is what happens.
325 .PP
326 .Vb 4
327 \& E to U U to E
328 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
329 \& \exA2\exA4 => U2550 U2550 => \exF9\exF9
330 \& (\exF9\exF9 => U2550 is now overwritten!)
331 .Ve
332 .PP
333 The Encode package comes with \fIucmlint\fR, a crude but sufficient
334 utility to check the integrity of a UCM file. Check under the
335 Encode/bin directory for this.
336 .PP
337 When in doubt, you can use \fIucmsort\fR, yet another utility under
338 Encode/bin directory.
339 .SH Bookmarks
340 .IX Header "Bookmarks"
341 .IP \(bu 4
342 ICU Home Page
343 <http://www.icu\-project.org/>
344 .IP \(bu 4
345 ICU Character Mapping Tables
346 <http://site.icu\-project.org/charts/charset>
347 .IP \(bu 4
348 ICU:Conversion Data
349 <http://www.icu\-project.org/userguide/conversion\-data.html>
350 .SH "SEE ALSO"
351 .IX Header "SEE ALSO"
352 Encode,
353 perlmod,
354 perlpod