| 1 |
.\" -*- mode: troff; coding: utf-8 -*- |
| 2 |
.\" Automatically generated by Pod::Man 5.0102 (Pod::Simple 3.45) |
| 3 |
.\" |
| 4 |
.\" Standard preamble: |
| 5 |
.\" ======================================================================== |
| 6 |
.de Sp \" Vertical space (when we can't use .PP) |
| 7 |
.if t .sp .5v |
| 8 |
.if n .sp |
| 9 |
.. |
| 10 |
.de Vb \" Begin verbatim text |
| 11 |
.ft CW |
| 12 |
.nf |
| 13 |
.ne \\$1 |
| 14 |
.. |
| 15 |
.de Ve \" End verbatim text |
| 16 |
.ft R |
| 17 |
.fi |
| 18 |
.. |
| 19 |
.\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>. |
| 20 |
.ie n \{\ |
| 21 |
. ds C` "" |
| 22 |
. ds C' "" |
| 23 |
'br\} |
| 24 |
.el\{\ |
| 25 |
. ds C` |
| 26 |
. ds C' |
| 27 |
'br\} |
| 28 |
.\" |
| 29 |
.\" Escape single quotes in literal strings from groff's Unicode transform. |
| 30 |
.ie \n(.g .ds Aq \(aq |
| 31 |
.el .ds Aq ' |
| 32 |
.\" |
| 33 |
.\" If the F register is >0, we'll generate index entries on stderr for |
| 34 |
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index |
| 35 |
.\" entries marked with X<> in POD. Of course, you'll have to process the |
| 36 |
.\" output yourself in some meaningful fashion. |
| 37 |
.\" |
| 38 |
.\" Avoid warning from groff about undefined register 'F'. |
| 39 |
.de IX |
| 40 |
.. |
| 41 |
.nr rF 0 |
| 42 |
.if \n(.g .if rF .nr rF 1 |
| 43 |
.if (\n(rF:(\n(.g==0)) \{\ |
| 44 |
. if \nF \{\ |
| 45 |
. de IX |
| 46 |
. tm Index:\\$1\t\\n%\t"\\$2" |
| 47 |
.. |
| 48 |
. if !\nF==2 \{\ |
| 49 |
. nr % 0 |
| 50 |
. nr F 2 |
| 51 |
. \} |
| 52 |
. \} |
| 53 |
.\} |
| 54 |
.rr rF |
| 55 |
.\" ======================================================================== |
| 56 |
.\" |
| 57 |
.IX Title "ENC2XS 1" |
| 58 |
.TH ENC2XS 1 2026-04-10 "perl v5.40.1" "Perl Programmers Reference Guide" |
| 59 |
.\" For nroff, turn off justification. Always turn off hyphenation; it makes |
| 60 |
.\" way too many mistakes in technical documents. |
| 61 |
.if n .ad l |
| 62 |
.nh |
| 63 |
.SH NAME |
| 64 |
enc2xs \-\- Perl Encode Module Generator |
| 65 |
.SH SYNOPSIS |
| 66 |
.IX Header "SYNOPSIS" |
| 67 |
.Vb 3 |
| 68 |
\& enc2xs \-[options] |
| 69 |
\& enc2xs \-M ModName mapfiles... |
| 70 |
\& enc2xs \-C |
| 71 |
.Ve |
| 72 |
.SH DESCRIPTION |
| 73 |
.IX Header "DESCRIPTION" |
| 74 |
\&\fIenc2xs\fR builds a Perl extension for use by Encode from either |
| 75 |
Unicode Character Mapping files (.ucm) or Tcl Encoding Files (.enc). |
| 76 |
Besides being used internally during the build process of the Encode |
| 77 |
module, you can use \fIenc2xs\fR to add your own encoding to perl. |
| 78 |
No knowledge of XS is necessary. |
| 79 |
.SH "Quick Guide" |
| 80 |
.IX Header "Quick Guide" |
| 81 |
If you want to know as little about Perl as possible but need to |
| 82 |
add a new encoding, just read this chapter and forget the rest. |
| 83 |
.IP 0. 4 |
| 84 |
.IX Item "0." |
| 85 |
Have a .ucm file ready. You can get it from somewhere or you can write |
| 86 |
your own from scratch or you can grab one from the Encode distribution |
| 87 |
and customize it. For the UCM format, see the next Chapter. In the |
| 88 |
example below, I'll call my theoretical encoding myascii, defined |
| 89 |
in \fImy.ucm\fR. \f(CW\*(C`$\*(C'\fR is a shell prompt. |
| 90 |
.Sp |
| 91 |
.Vb 2 |
| 92 |
\& $ ls \-F |
| 93 |
\& my.ucm |
| 94 |
.Ve |
| 95 |
.IP 1. 4 |
| 96 |
.IX Item "1." |
| 97 |
Issue a command as follows; |
| 98 |
.Sp |
| 99 |
.Vb 5 |
| 100 |
\& $ enc2xs \-M My my.ucm |
| 101 |
\& generating Makefile.PL |
| 102 |
\& generating My.pm |
| 103 |
\& generating README |
| 104 |
\& generating Changes |
| 105 |
.Ve |
| 106 |
.Sp |
| 107 |
Now take a look at your current directory. It should look like this. |
| 108 |
.Sp |
| 109 |
.Vb 2 |
| 110 |
\& $ ls \-F |
| 111 |
\& Makefile.PL My.pm my.ucm t/ |
| 112 |
.Ve |
| 113 |
.Sp |
| 114 |
The following files were created. |
| 115 |
.Sp |
| 116 |
.Vb 3 |
| 117 |
\& Makefile.PL \- MakeMaker script |
| 118 |
\& My.pm \- Encode submodule |
| 119 |
\& t/My.t \- test file |
| 120 |
.Ve |
| 121 |
.RS 4 |
| 122 |
.IP 1.1. 4 |
| 123 |
.IX Item "1.1." |
| 124 |
If you want *.ucm installed together with the modules, do as follows; |
| 125 |
.Sp |
| 126 |
.Vb 3 |
| 127 |
\& $ mkdir Encode |
| 128 |
\& $ mv *.ucm Encode |
| 129 |
\& $ enc2xs \-M My Encode/*ucm |
| 130 |
.Ve |
| 131 |
.RE |
| 132 |
.RS 4 |
| 133 |
.RE |
| 134 |
.IP 2. 4 |
| 135 |
.IX Item "2." |
| 136 |
Edit the files generated. You don't have to if you have no time AND no |
| 137 |
intention to give it to someone else. But it is a good idea to edit |
| 138 |
the pod and to add more tests. |
| 139 |
.IP 3. 4 |
| 140 |
.IX Item "3." |
| 141 |
Now issue a command all Perl Mongers love: |
| 142 |
.Sp |
| 143 |
.Vb 2 |
| 144 |
\& $ perl Makefile.PL |
| 145 |
\& Writing Makefile for Encode::My |
| 146 |
.Ve |
| 147 |
.IP 4. 4 |
| 148 |
.IX Item "4." |
| 149 |
Now all you have to do is make. |
| 150 |
.Sp |
| 151 |
.Vb 12 |
| 152 |
\& $ make |
| 153 |
\& cp My.pm blib/lib/Encode/My.pm |
| 154 |
\& /usr/local/bin/perl /usr/local/bin/enc2xs \-Q \-O \e |
| 155 |
\& \-o encode_t.c \-f encode_t.fnm |
| 156 |
\& Reading myascii (myascii) |
| 157 |
\& Writing compiled form |
| 158 |
\& 128 bytes in string tables |
| 159 |
\& 384 bytes (75%) saved spotting duplicates |
| 160 |
\& 1 bytes (0.775%) saved using substrings |
| 161 |
\& .... |
| 162 |
\& chmod 644 blib/arch/auto/Encode/My/My.bs |
| 163 |
\& $ |
| 164 |
.Ve |
| 165 |
.Sp |
| 166 |
The time it takes varies depending on how fast your machine is and |
| 167 |
how large your encoding is. Unless you are working on something big |
| 168 |
like euc-tw, it won't take too long. |
| 169 |
.IP 5. 4 |
| 170 |
.IX Item "5." |
| 171 |
You can "make install" already but you should test first. |
| 172 |
.Sp |
| 173 |
.Vb 8 |
| 174 |
\& $ make test |
| 175 |
\& PERL_DL_NONLAZY=1 /usr/local/bin/perl \-Iblib/arch \-Iblib/lib \e |
| 176 |
\& \-e \*(Aquse Test::Harness qw(&runtests $verbose); \e |
| 177 |
\& $verbose=0; runtests @ARGV;\*(Aq t/*.t |
| 178 |
\& t/My....ok |
| 179 |
\& All tests successful. |
| 180 |
\& Files=1, Tests=2, 0 wallclock secs |
| 181 |
\& ( 0.09 cusr + 0.01 csys = 0.09 CPU) |
| 182 |
.Ve |
| 183 |
.IP 6. 4 |
| 184 |
.IX Item "6." |
| 185 |
If you are content with the test result, just "make install" |
| 186 |
.IP 7. 4 |
| 187 |
.IX Item "7." |
| 188 |
If you want to add your encoding to Encode's demand-loading list |
| 189 |
(so you don't have to "use Encode::YourEncoding"), run |
| 190 |
.Sp |
| 191 |
.Vb 1 |
| 192 |
\& enc2xs \-C |
| 193 |
.Ve |
| 194 |
.Sp |
| 195 |
to update Encode::ConfigLocal, a module that controls local settings. |
| 196 |
After that, "use Encode;" is enough to load your encodings on demand. |
| 197 |
.SH "The Unicode Character Map" |
| 198 |
.IX Header "The Unicode Character Map" |
| 199 |
Encode uses the Unicode Character Map (UCM) format for source character |
| 200 |
mappings. This format is used by IBM's ICU package and was adopted |
| 201 |
by Nick Ing-Simmons for use with the Encode module. Since UCM is |
| 202 |
more flexible than Tcl's Encoding Map and far more user-friendly, |
| 203 |
this is the recommended format for Encode now. |
| 204 |
.PP |
| 205 |
A UCM file looks like this. |
| 206 |
.PP |
| 207 |
.Vb 10 |
| 208 |
\& # |
| 209 |
\& # Comments |
| 210 |
\& # |
| 211 |
\& <code_set_name> "US\-ascii" # Required |
| 212 |
\& <code_set_alias> "ascii" # Optional |
| 213 |
\& <mb_cur_min> 1 # Required; usually 1 |
| 214 |
\& <mb_cur_max> 1 # Max. # of bytes/char |
| 215 |
\& <subchar> \ex3F # Substitution char |
| 216 |
\& # |
| 217 |
\& CHARMAP |
| 218 |
\& <U0000> \ex00 |0 # <control> |
| 219 |
\& <U0001> \ex01 |0 # <control> |
| 220 |
\& <U0002> \ex02 |0 # <control> |
| 221 |
\& .... |
| 222 |
\& <U007C> \ex7C |0 # VERTICAL LINE |
| 223 |
\& <U007D> \ex7D |0 # RIGHT CURLY BRACKET |
| 224 |
\& <U007E> \ex7E |0 # TILDE |
| 225 |
\& <U007F> \ex7F |0 # <control> |
| 226 |
\& END CHARMAP |
| 227 |
.Ve |
| 228 |
.IP \(bu 4 |
| 229 |
Anything that follows \f(CW\*(C`#\*(C'\fR is treated as a comment. |
| 230 |
.IP \(bu 4 |
| 231 |
The header section continues until a line containing the word |
| 232 |
CHARMAP. This section has a form of \fI<keyword> value\fR, one |
| 233 |
pair per line. Strings used as values must be quoted. Barewords are |
| 234 |
treated as numbers. \fI\exXX\fR represents a byte. |
| 235 |
.Sp |
| 236 |
Most of the keywords are self-explanatory. \fIsubchar\fR means |
| 237 |
substitution character, not subcharacter. When you decode a Unicode |
| 238 |
sequence to this encoding but no matching character is found, the byte |
| 239 |
sequence defined here will be used. For most cases, the value here is |
| 240 |
\&\ex3F; in ASCII, this is a question mark. |
| 241 |
.IP \(bu 4 |
| 242 |
CHARMAP starts the character map section. Each line has a form as |
| 243 |
follows: |
| 244 |
.Sp |
| 245 |
.Vb 5 |
| 246 |
\& <UXXXX> \exXX.. |0 # comment |
| 247 |
\& ^ ^ ^ |
| 248 |
\& | | +\- Fallback flag |
| 249 |
\& | +\-\-\-\-\-\-\-\- Encoded byte sequence |
| 250 |
\& +\-\-\-\-\-\-\-\-\-\-\-\-\-\- Unicode Character ID in hex |
| 251 |
.Ve |
| 252 |
.Sp |
| 253 |
The format is roughly the same as a header section except for the |
| 254 |
fallback flag: | followed by 0..3. The meaning of the possible |
| 255 |
values is as follows: |
| 256 |
.RS 4 |
| 257 |
.IP |0 4 |
| 258 |
.IX Item "|0" |
| 259 |
Round trip safe. A character decoded to Unicode encodes back to the |
| 260 |
same byte sequence. Most characters have this flag. |
| 261 |
.IP |1 4 |
| 262 |
.IX Item "|1" |
| 263 |
Fallback for unicode \-> encoding. When seen, enc2xs adds this |
| 264 |
character for the encode map only. |
| 265 |
.IP |2 4 |
| 266 |
.IX Item "|2" |
| 267 |
Skip sub-char mapping should there be no code point. |
| 268 |
.IP |3 4 |
| 269 |
.IX Item "|3" |
| 270 |
Fallback for encoding \-> unicode. When seen, enc2xs adds this |
| 271 |
character for the decode map only. |
| 272 |
.RE |
| 273 |
.RS 4 |
| 274 |
.RE |
| 275 |
.IP \(bu 4 |
| 276 |
And finally, END OF CHARMAP ends the section. |
| 277 |
.PP |
| 278 |
When you are manually creating a UCM file, you should copy ascii.ucm |
| 279 |
or an existing encoding which is close to yours, rather than write |
| 280 |
your own from scratch. |
| 281 |
.PP |
| 282 |
When you do so, make sure you leave at least \fBU0000\fR to \fBU0020\fR as |
| 283 |
is, unless your environment is EBCDIC. |
| 284 |
.PP |
| 285 |
\&\fBCAVEAT\fR: not all features in UCM are implemented. For example, |
| 286 |
icu:state is not used. Because of that, you need to write a perl |
| 287 |
module if you want to support algorithmical encodings, notably |
| 288 |
the ISO\-2022 series. Such modules include Encode::JP::2022_JP, |
| 289 |
Encode::KR::2022_KR, and Encode::TW::HZ. |
| 290 |
.SS "Coping with duplicate mappings" |
| 291 |
.IX Subsection "Coping with duplicate mappings" |
| 292 |
When you create a map, you SHOULD make your mappings round-trip safe. |
| 293 |
That is, \f(CWencode(\*(Aqyour\-encoding\*(Aq, decode(\*(Aqyour\-encoding\*(Aq, $data)) eq |
| 294 |
$data\fR stands for all characters that are marked as \f(CW\*(C`|0\*(C'\fR. Here is |
| 295 |
how to make sure: |
| 296 |
.IP \(bu 4 |
| 297 |
Sort your map in Unicode order. |
| 298 |
.IP \(bu 4 |
| 299 |
When you have a duplicate entry, mark either one with '|1' or '|3'. |
| 300 |
.IP \(bu 4 |
| 301 |
And make sure the '|1' or '|3' entry FOLLOWS the '|0' entry. |
| 302 |
.PP |
| 303 |
Here is an example from big5\-eten. |
| 304 |
.PP |
| 305 |
.Vb 2 |
| 306 |
\& <U2550> \exF9\exF9 |0 |
| 307 |
\& <U2550> \exA2\exA4 |3 |
| 308 |
.Ve |
| 309 |
.PP |
| 310 |
Internally Encoding \-> Unicode and Unicode \-> Encoding Map looks like |
| 311 |
this; |
| 312 |
.PP |
| 313 |
.Vb 4 |
| 314 |
\& E to U U to E |
| 315 |
\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- |
| 316 |
\& \exF9\exF9 => U2550 U2550 => \exF9\exF9 |
| 317 |
\& \exA2\exA4 => U2550 |
| 318 |
.Ve |
| 319 |
.PP |
| 320 |
So it is round-trip safe for \exF9\exF9. But if the line above is upside |
| 321 |
down, here is what happens. |
| 322 |
.PP |
| 323 |
.Vb 4 |
| 324 |
\& E to U U to E |
| 325 |
\& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\- |
| 326 |
\& \exA2\exA4 => U2550 U2550 => \exF9\exF9 |
| 327 |
\& (\exF9\exF9 => U2550 is now overwritten!) |
| 328 |
.Ve |
| 329 |
.PP |
| 330 |
The Encode package comes with \fIucmlint\fR, a crude but sufficient |
| 331 |
utility to check the integrity of a UCM file. Check under the |
| 332 |
Encode/bin directory for this. |
| 333 |
.PP |
| 334 |
When in doubt, you can use \fIucmsort\fR, yet another utility under |
| 335 |
Encode/bin directory. |
| 336 |
.SH Bookmarks |
| 337 |
.IX Header "Bookmarks" |
| 338 |
.IP \(bu 4 |
| 339 |
ICU Home Page |
| 340 |
<http://www.icu\-project.org/> |
| 341 |
.IP \(bu 4 |
| 342 |
ICU Character Mapping Tables |
| 343 |
<http://site.icu\-project.org/charts/charset> |
| 344 |
.IP \(bu 4 |
| 345 |
ICU:Conversion Data |
| 346 |
<http://www.icu\-project.org/userguide/conversion\-data.html> |
| 347 |
.SH "SEE ALSO" |
| 348 |
.IX Header "SEE ALSO" |
| 349 |
Encode, |
| 350 |
perlmod, |
| 351 |
perlpod |