Debian Codegraph @

upstream/debian-unstable/man1/enc2xs.1

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man 5.0102 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\" ========================================================================
56 .\"
57 .IX Title "ENC2XS 1"
58 .TH ENC2XS 1 2026-04-10 "perl v5.40.1" "Perl Programmers Reference Guide"
59 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
60 .\" way too many mistakes in technical documents.
61 .if n .ad l
62 .nh
63 .SH NAME
64 enc2xs \-\- Perl Encode Module Generator
65 .SH SYNOPSIS
66 .IX Header "SYNOPSIS"
67 .Vb 3
68 \& enc2xs \-[options]
69 \& enc2xs \-M ModName mapfiles...
70 \& enc2xs \-C
71 .Ve
72 .SH DESCRIPTION
73 .IX Header "DESCRIPTION"
74 \&\fIenc2xs\fR builds a Perl extension for use by Encode from either
75 Unicode Character Mapping files (.ucm) or Tcl Encoding Files (.enc).
76 Besides being used internally during the build process of the Encode
77 module, you can use \fIenc2xs\fR to add your own encoding to perl.
78 No knowledge of XS is necessary.
79 .SH "Quick Guide"
80 .IX Header "Quick Guide"
81 If you want to know as little about Perl as possible but need to
82 add a new encoding, just read this chapter and forget the rest.
83 .IP 0. 4
84 .IX Item "0."
85 Have a .ucm file ready. You can get it from somewhere or you can write
86 your own from scratch or you can grab one from the Encode distribution
87 and customize it. For the UCM format, see the next Chapter. In the
88 example below, I'll call my theoretical encoding myascii, defined
89 in \fImy.ucm\fR. \f(CW\*(C`$\*(C'\fR is a shell prompt.
90 .Sp
91 .Vb 2
92 \& $ ls \-F
93 \& my.ucm
94 .Ve
95 .IP 1. 4
96 .IX Item "1."
97 Issue a command as follows;
98 .Sp
99 .Vb 5
100 \& $ enc2xs \-M My my.ucm
101 \& generating Makefile.PL
102 \& generating My.pm
103 \& generating README
104 \& generating Changes
105 .Ve
106 .Sp
107 Now take a look at your current directory. It should look like this.
108 .Sp
109 .Vb 2
110 \& $ ls \-F
111 \& Makefile.PL My.pm my.ucm t/
112 .Ve
113 .Sp
114 The following files were created.
115 .Sp
116 .Vb 3
117 \& Makefile.PL \- MakeMaker script
118 \& My.pm \- Encode submodule
119 \& t/My.t \- test file
120 .Ve
121 .RS 4
122 .IP 1.1. 4
123 .IX Item "1.1."
124 If you want *.ucm installed together with the modules, do as follows;
125 .Sp
126 .Vb 3
127 \& $ mkdir Encode
128 \& $ mv *.ucm Encode
129 \& $ enc2xs \-M My Encode/*ucm
130 .Ve
131 .RE
132 .RS 4
133 .RE
134 .IP 2. 4
135 .IX Item "2."
136 Edit the files generated. You don't have to if you have no time AND no
137 intention to give it to someone else. But it is a good idea to edit
138 the pod and to add more tests.
139 .IP 3. 4
140 .IX Item "3."
141 Now issue a command all Perl Mongers love:
142 .Sp
143 .Vb 2
144 \& $ perl Makefile.PL
145 \& Writing Makefile for Encode::My
146 .Ve
147 .IP 4. 4
148 .IX Item "4."
149 Now all you have to do is make.
150 .Sp
151 .Vb 12
152 \& $ make
153 \& cp My.pm blib/lib/Encode/My.pm
154 \& /usr/local/bin/perl /usr/local/bin/enc2xs \-Q \-O \e
155 \& \-o encode_t.c \-f encode_t.fnm
156 \& Reading myascii (myascii)
157 \& Writing compiled form
158 \& 128 bytes in string tables
159 \& 384 bytes (75%) saved spotting duplicates
160 \& 1 bytes (0.775%) saved using substrings
161 \& ....
162 \& chmod 644 blib/arch/auto/Encode/My/My.bs
163 \& $
164 .Ve
165 .Sp
166 The time it takes varies depending on how fast your machine is and
167 how large your encoding is. Unless you are working on something big
168 like euc-tw, it won't take too long.
169 .IP 5. 4
170 .IX Item "5."
171 You can "make install" already but you should test first.
172 .Sp
173 .Vb 8
174 \& $ make test
175 \& PERL_DL_NONLAZY=1 /usr/local/bin/perl \-Iblib/arch \-Iblib/lib \e
176 \& \-e \*(Aquse Test::Harness qw(&runtests $verbose); \e
177 \& $verbose=0; runtests @ARGV;\*(Aq t/*.t
178 \& t/My....ok
179 \& All tests successful.
180 \& Files=1, Tests=2, 0 wallclock secs
181 \& ( 0.09 cusr + 0.01 csys = 0.09 CPU)
182 .Ve
183 .IP 6. 4
184 .IX Item "6."
185 If you are content with the test result, just "make install"
186 .IP 7. 4
187 .IX Item "7."
188 If you want to add your encoding to Encode's demand-loading list
189 (so you don't have to "use Encode::YourEncoding"), run
190 .Sp
191 .Vb 1
192 \& enc2xs \-C
193 .Ve
194 .Sp
195 to update Encode::ConfigLocal, a module that controls local settings.
196 After that, "use Encode;" is enough to load your encodings on demand.
197 .SH "The Unicode Character Map"
198 .IX Header "The Unicode Character Map"
199 Encode uses the Unicode Character Map (UCM) format for source character
200 mappings. This format is used by IBM's ICU package and was adopted
201 by Nick Ing-Simmons for use with the Encode module. Since UCM is
202 more flexible than Tcl's Encoding Map and far more user-friendly,
203 this is the recommended format for Encode now.
204 .PP
205 A UCM file looks like this.
206 .PP
207 .Vb 10
208 \& #
209 \& # Comments
210 \& #
211 \& <code_set_name> "US\-ascii" # Required
212 \& <code_set_alias> "ascii" # Optional
213 \& <mb_cur_min> 1 # Required; usually 1
214 \& <mb_cur_max> 1 # Max. # of bytes/char
215 \& <subchar> \ex3F # Substitution char
216 \& #
217 \& CHARMAP
218 \& <U0000> \ex00 |0 # <control>
219 \& <U0001> \ex01 |0 # <control>
220 \& <U0002> \ex02 |0 # <control>
221 \& ....
222 \& <U007C> \ex7C |0 # VERTICAL LINE
223 \& <U007D> \ex7D |0 # RIGHT CURLY BRACKET
224 \& <U007E> \ex7E |0 # TILDE
225 \& <U007F> \ex7F |0 # <control>
226 \& END CHARMAP
227 .Ve
228 .IP \(bu 4
229 Anything that follows \f(CW\*(C`#\*(C'\fR is treated as a comment.
230 .IP \(bu 4
231 The header section continues until a line containing the word
232 CHARMAP. This section has a form of \fI<keyword> value\fR, one
233 pair per line. Strings used as values must be quoted. Barewords are
234 treated as numbers. \fI\exXX\fR represents a byte.
235 .Sp
236 Most of the keywords are self-explanatory. \fIsubchar\fR means
237 substitution character, not subcharacter. When you decode a Unicode
238 sequence to this encoding but no matching character is found, the byte
239 sequence defined here will be used. For most cases, the value here is
240 \&\ex3F; in ASCII, this is a question mark.
241 .IP \(bu 4
242 CHARMAP starts the character map section. Each line has a form as
243 follows:
244 .Sp
245 .Vb 5
246 \& <UXXXX> \exXX.. |0 # comment
247 \& ^ ^ ^
248 \& | | +\- Fallback flag
249 \& | +\-\-\-\-\-\-\-\- Encoded byte sequence
250 \& +\-\-\-\-\-\-\-\-\-\-\-\-\-\- Unicode Character ID in hex
251 .Ve
252 .Sp
253 The format is roughly the same as a header section except for the
254 fallback flag: | followed by 0..3. The meaning of the possible
255 values is as follows:
256 .RS 4
257 .IP |0 4
258 .IX Item "|0"
259 Round trip safe. A character decoded to Unicode encodes back to the
260 same byte sequence. Most characters have this flag.
261 .IP |1 4
262 .IX Item "|1"
263 Fallback for unicode \-> encoding. When seen, enc2xs adds this
264 character for the encode map only.
265 .IP |2 4
266 .IX Item "|2"
267 Skip sub-char mapping should there be no code point.
268 .IP |3 4
269 .IX Item "|3"
270 Fallback for encoding \-> unicode. When seen, enc2xs adds this
271 character for the decode map only.
272 .RE
273 .RS 4
274 .RE
275 .IP \(bu 4
276 And finally, END OF CHARMAP ends the section.
277 .PP
278 When you are manually creating a UCM file, you should copy ascii.ucm
279 or an existing encoding which is close to yours, rather than write
280 your own from scratch.
281 .PP
282 When you do so, make sure you leave at least \fBU0000\fR to \fBU0020\fR as
283 is, unless your environment is EBCDIC.
284 .PP
285 \&\fBCAVEAT\fR: not all features in UCM are implemented. For example,
286 icu:state is not used. Because of that, you need to write a perl
287 module if you want to support algorithmical encodings, notably
288 the ISO\-2022 series. Such modules include Encode::JP::2022_JP,
289 Encode::KR::2022_KR, and Encode::TW::HZ.
290 .SS "Coping with duplicate mappings"
291 .IX Subsection "Coping with duplicate mappings"
292 When you create a map, you SHOULD make your mappings round-trip safe.
293 That is, \f(CWencode(\*(Aqyour\-encoding\*(Aq, decode(\*(Aqyour\-encoding\*(Aq, $data)) eq
294 $data\fR stands for all characters that are marked as \f(CW\*(C`|0\*(C'\fR. Here is
295 how to make sure:
296 .IP \(bu 4
297 Sort your map in Unicode order.
298 .IP \(bu 4
299 When you have a duplicate entry, mark either one with '|1' or '|3'.
300 .IP \(bu 4
301 And make sure the '|1' or '|3' entry FOLLOWS the '|0' entry.
302 .PP
303 Here is an example from big5\-eten.
304 .PP
305 .Vb 2
306 \& <U2550> \exF9\exF9 |0
307 \& <U2550> \exA2\exA4 |3
308 .Ve
309 .PP
310 Internally Encoding \-> Unicode and Unicode \-> Encoding Map looks like
311 this;
312 .PP
313 .Vb 4
314 \& E to U U to E
315 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
316 \& \exF9\exF9 => U2550 U2550 => \exF9\exF9
317 \& \exA2\exA4 => U2550
318 .Ve
319 .PP
320 So it is round-trip safe for \exF9\exF9. But if the line above is upside
321 down, here is what happens.
322 .PP
323 .Vb 4
324 \& E to U U to E
325 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
326 \& \exA2\exA4 => U2550 U2550 => \exF9\exF9
327 \& (\exF9\exF9 => U2550 is now overwritten!)
328 .Ve
329 .PP
330 The Encode package comes with \fIucmlint\fR, a crude but sufficient
331 utility to check the integrity of a UCM file. Check under the
332 Encode/bin directory for this.
333 .PP
334 When in doubt, you can use \fIucmsort\fR, yet another utility under
335 Encode/bin directory.
336 .SH Bookmarks
337 .IX Header "Bookmarks"
338 .IP \(bu 4
339 ICU Home Page
340 <http://www.icu\-project.org/>
341 .IP \(bu 4
342 ICU Character Mapping Tables
343 <http://site.icu\-project.org/charts/charset>
344 .IP \(bu 4
345 ICU:Conversion Data
346 <http://www.icu\-project.org/userguide/conversion\-data.html>
347 .SH "SEE ALSO"
348 .IX Header "SEE ALSO"
349 Encode,
350 perlmod,
351 perlpod