| 1 |
.\" -*- mode: troff; coding: utf-8 -*- |
| 2 |
.\" Automatically generated by Pod::Man 5.0102 (Pod::Simple 3.45) |
| 3 |
.\" |
| 4 |
.\" Standard preamble: |
| 5 |
.\" ======================================================================== |
| 6 |
.de Sp \" Vertical space (when we can't use .PP) |
| 7 |
.if t .sp .5v |
| 8 |
.if n .sp |
| 9 |
.. |
| 10 |
.de Vb \" Begin verbatim text |
| 11 |
.ft CW |
| 12 |
.nf |
| 13 |
.ne \\$1 |
| 14 |
.. |
| 15 |
.de Ve \" End verbatim text |
| 16 |
.ft R |
| 17 |
.fi |
| 18 |
.. |
| 19 |
.\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>. |
| 20 |
.ie n \{\ |
| 21 |
. ds C` "" |
| 22 |
. ds C' "" |
| 23 |
'br\} |
| 24 |
.el\{\ |
| 25 |
. ds C` |
| 26 |
. ds C' |
| 27 |
'br\} |
| 28 |
.\" |
| 29 |
.\" Escape single quotes in literal strings from groff's Unicode transform. |
| 30 |
.ie \n(.g .ds Aq \(aq |
| 31 |
.el .ds Aq ' |
| 32 |
.\" |
| 33 |
.\" If the F register is >0, we'll generate index entries on stderr for |
| 34 |
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index |
| 35 |
.\" entries marked with X<> in POD. Of course, you'll have to process the |
| 36 |
.\" output yourself in some meaningful fashion. |
| 37 |
.\" |
| 38 |
.\" Avoid warning from groff about undefined register 'F'. |
| 39 |
.de IX |
| 40 |
.. |
| 41 |
.nr rF 0 |
| 42 |
.if \n(.g .if rF .nr rF 1 |
| 43 |
.if (\n(rF:(\n(.g==0)) \{\ |
| 44 |
. if \nF \{\ |
| 45 |
. de IX |
| 46 |
. tm Index:\\$1\t\\n%\t"\\$2" |
| 47 |
.. |
| 48 |
. if !\nF==2 \{\ |
| 49 |
. nr % 0 |
| 50 |
. nr F 2 |
| 51 |
. \} |
| 52 |
. \} |
| 53 |
.\} |
| 54 |
.rr rF |
| 55 |
.\" ======================================================================== |
| 56 |
.\" |
| 57 |
.IX Title "utf8 3perl" |
| 58 |
.TH utf8 3perl 2025-11-16 "perl v5.40.1" "Perl Programmers Reference Guide" |
| 59 |
.\" For nroff, turn off justification. Always turn off hyphenation; it makes |
| 60 |
.\" way too many mistakes in technical documents. |
| 61 |
.if n .ad l |
| 62 |
.nh |
| 63 |
.SH NAME |
| 64 |
utf8 \- Perl pragma to enable/disable UTF\-8 (or UTF\-EBCDIC) in source code |
| 65 |
.SH SYNOPSIS |
| 66 |
.IX Header "SYNOPSIS" |
| 67 |
.Vb 2 |
| 68 |
\& use utf8; |
| 69 |
\& no utf8; |
| 70 |
\& |
| 71 |
\& # Convert the internal representation of a Perl scalar to/from UTF\-8. |
| 72 |
\& |
| 73 |
\& $num_octets = utf8::upgrade($string); |
| 74 |
\& $success = utf8::downgrade($string[, $fail_ok]); |
| 75 |
\& |
| 76 |
\& # Change each character of a Perl scalar to/from a series of |
| 77 |
\& # characters that represent the UTF\-8 bytes of each original character. |
| 78 |
\& |
| 79 |
\& utf8::encode($string); # "\ex{100}" becomes "\exc4\ex80" |
| 80 |
\& utf8::decode($string); # "\exc4\ex80" becomes "\ex{100}" |
| 81 |
\& |
| 82 |
\& # Convert a code point from the platform native character set to |
| 83 |
\& # Unicode, and vice\-versa. |
| 84 |
\& $unicode = utf8::native_to_unicode(ord(\*(AqA\*(Aq)); # returns 65 on both |
| 85 |
\& # ASCII and EBCDIC |
| 86 |
\& # platforms |
| 87 |
\& $native = utf8::unicode_to_native(65); # returns 65 on ASCII |
| 88 |
\& # platforms; 193 on |
| 89 |
\& # EBCDIC |
| 90 |
\& |
| 91 |
\& $flag = utf8::is_utf8($string); # since Perl 5.8.1 |
| 92 |
\& $flag = utf8::valid($string); |
| 93 |
.Ve |
| 94 |
.SH DESCRIPTION |
| 95 |
.IX Header "DESCRIPTION" |
| 96 |
The \f(CW\*(C`use utf8\*(C'\fR pragma tells the Perl parser to allow UTF\-8 in the |
| 97 |
program text in the current lexical scope. The \f(CW\*(C`no utf8\*(C'\fR pragma tells Perl |
| 98 |
to switch back to treating the source text as literal bytes in the current |
| 99 |
lexical scope. (On EBCDIC platforms, technically it is allowing UTF-EBCDIC, |
| 100 |
and not UTF\-8, but this distinction is academic, so in this document the term |
| 101 |
UTF\-8 is used to mean both). |
| 102 |
.PP |
| 103 |
\&\fBDo not use this pragma for anything else than telling Perl that your |
| 104 |
script is written in UTF\-8.\fR The utility functions described below are |
| 105 |
directly usable without \f(CW\*(C`use utf8;\*(C'\fR. |
| 106 |
.PP |
| 107 |
Because it is not possible to reliably tell UTF\-8 from native 8 bit |
| 108 |
encodings, you need either a Byte Order Mark at the beginning of your |
| 109 |
source code, or \f(CW\*(C`use utf8;\*(C'\fR, to instruct perl. |
| 110 |
.PP |
| 111 |
When UTF\-8 becomes the standard source format, this pragma will |
| 112 |
effectively become a no-op. |
| 113 |
.PP |
| 114 |
See also the effects of the \f(CW\*(C`\-C\*(C'\fR switch and its cousin, the |
| 115 |
\&\f(CW\*(C`PERL_UNICODE\*(C'\fR environment variable, in perlrun. |
| 116 |
.PP |
| 117 |
Enabling the \f(CW\*(C`utf8\*(C'\fR pragma has the following effect: |
| 118 |
.IP \(bu 4 |
| 119 |
Bytes in the source text that are not in the ASCII character set will be |
| 120 |
treated as being part of a literal UTF\-8 sequence. This includes most |
| 121 |
literals such as identifier names, string constants, and constant |
| 122 |
regular expression patterns. |
| 123 |
.PP |
| 124 |
Note that if you have non-ASCII, non\-UTF\-8 bytes in your script (for example |
| 125 |
embedded Latin\-1 in your string literals), \f(CW\*(C`use utf8\*(C'\fR will be unhappy. If |
| 126 |
you want to have such bytes under \f(CW\*(C`use utf8\*(C'\fR, you can disable this pragma |
| 127 |
until the end the block (or file, if at top level) by \f(CW\*(C`no utf8;\*(C'\fR. |
| 128 |
.SS "Utility functions" |
| 129 |
.IX Subsection "Utility functions" |
| 130 |
The following functions are defined in the \f(CW\*(C`utf8::\*(C'\fR package by the |
| 131 |
Perl core. You do not need to say \f(CW\*(C`use utf8\*(C'\fR to use these and in fact |
| 132 |
you should not say that unless you really want to have UTF\-8 source code. |
| 133 |
.IP \(bu 4 |
| 134 |
\&\f(CW\*(C`$num_octets = utf8::upgrade($string)\*(C'\fR |
| 135 |
.Sp |
| 136 |
(Since Perl v5.8.0) |
| 137 |
Converts in-place the internal representation of the string from an octet |
| 138 |
sequence in the native encoding (Latin\-1 or EBCDIC) to UTF\-8. The |
| 139 |
logical character sequence itself is unchanged. If \fR\f(CI$string\fR\fI\fR is already |
| 140 |
upgraded, then this is a no-op. Returns the |
| 141 |
number of octets necessary to represent the string as UTF\-8. |
| 142 |
Since Perl v5.38, if \f(CW$string\fR is \f(CW\*(C`undef\*(C'\fR no action is taken; prior to that, |
| 143 |
it would be converted to be defined and zero-length. |
| 144 |
.Sp |
| 145 |
If your code needs to be compatible with versions of perl without |
| 146 |
\&\f(CW\*(C`use feature \*(Aqunicode_strings\*(Aq;\*(C'\fR, you can force Unicode semantics on |
| 147 |
a given string: |
| 148 |
.Sp |
| 149 |
.Vb 3 |
| 150 |
\& # force unicode semantics for $string without the |
| 151 |
\& # "unicode_strings" feature |
| 152 |
\& utf8::upgrade($string); |
| 153 |
.Ve |
| 154 |
.Sp |
| 155 |
For example: |
| 156 |
.Sp |
| 157 |
.Vb 7 |
| 158 |
\& # without explicit or implicit use feature \*(Aqunicode_strings\*(Aq |
| 159 |
\& my $x = "\exDF"; # LATIN SMALL LETTER SHARP S |
| 160 |
\& $x =~ /ss/i; # won\*(Aqt match |
| 161 |
\& my $y = uc($x); # won\*(Aqt convert |
| 162 |
\& utf8::upgrade($x); |
| 163 |
\& $x =~ /ss/i; # matches |
| 164 |
\& my $z = uc($x); # converts to "SS" |
| 165 |
.Ve |
| 166 |
.Sp |
| 167 |
\&\fBNote that this function does not handle arbitrary encodings\fR; |
| 168 |
use Encode instead. |
| 169 |
.IP \(bu 4 |
| 170 |
\&\f(CW\*(C`$success = utf8::downgrade($string[, $fail_ok])\*(C'\fR |
| 171 |
.Sp |
| 172 |
(Since Perl v5.8.0) |
| 173 |
Converts in-place the internal representation of the string from UTF\-8 to the |
| 174 |
equivalent octet sequence in the native encoding (Latin\-1 or EBCDIC). The |
| 175 |
logical character sequence itself is unchanged. If \fR\f(CI$string\fR\fI\fR is already |
| 176 |
stored as native 8 bit, then this is a no-op. Can be used to make sure that |
| 177 |
the UTF\-8 flag is off, e.g. when you want to make sure that the \fBsubstr()\fR or |
| 178 |
\&\fBlength()\fR function works with the usually faster byte algorithm. |
| 179 |
.Sp |
| 180 |
Fails if the original UTF\-8 sequence cannot be represented in the |
| 181 |
native 8 bit encoding. On failure dies or, if the value of \fR\f(CI$fail_ok\fR\fI\fR is |
| 182 |
true, returns false. |
| 183 |
.Sp |
| 184 |
Returns true on success. |
| 185 |
.Sp |
| 186 |
If your code expects an octet sequence this can be used to validate |
| 187 |
that you've received one: |
| 188 |
.Sp |
| 189 |
.Vb 2 |
| 190 |
\& # throw an exception if not representable as octets |
| 191 |
\& utf8::downgrade($string) |
| 192 |
\& |
| 193 |
\& # or do your own error handling |
| 194 |
\& utf8::downgrade($string, 1) or die "string must be octets"; |
| 195 |
.Ve |
| 196 |
.Sp |
| 197 |
\&\fBNote that this function does not handle arbitrary encodings\fR; |
| 198 |
use Encode instead. |
| 199 |
.IP \(bu 4 |
| 200 |
\&\f(CWutf8::encode($string)\fR |
| 201 |
.Sp |
| 202 |
(Since Perl v5.8.0) |
| 203 |
Converts in-place the character sequence to the corresponding octet |
| 204 |
sequence in Perl's extended UTF\-8. That is, every (possibly wide) character |
| 205 |
gets replaced with a sequence of one or more characters that represent the |
| 206 |
individual UTF\-8 bytes of the character. The UTF8 flag is turned off. |
| 207 |
Returns nothing. |
| 208 |
.Sp |
| 209 |
.Vb 4 |
| 210 |
\& my $x = "\ex{100}"; # $x contains one character, with ord 0x100 |
| 211 |
\& utf8::encode($x); # $x contains two characters, with ords (on |
| 212 |
\& # ASCII platforms) 0xc4 and 0x80. On EBCDIC |
| 213 |
\& # 1047, this would instead be 0x8C and 0x41. |
| 214 |
.Ve |
| 215 |
.Sp |
| 216 |
Similar to: |
| 217 |
.Sp |
| 218 |
.Vb 2 |
| 219 |
\& use Encode; |
| 220 |
\& $x = Encode::encode("utf8", $x); |
| 221 |
.Ve |
| 222 |
.Sp |
| 223 |
\&\fBNote that this function does not handle arbitrary encodings\fR; |
| 224 |
use Encode instead. |
| 225 |
.IP \(bu 4 |
| 226 |
\&\f(CW\*(C`$success = utf8::decode($string)\*(C'\fR |
| 227 |
.Sp |
| 228 |
(Since Perl v5.8.0) |
| 229 |
Attempts to convert in-place the octet sequence encoded in Perl's extended |
| 230 |
UTF\-8 to the corresponding character sequence. That is, it replaces each |
| 231 |
sequence of characters in the string whose ords represent a valid (extended) |
| 232 |
UTF\-8 byte sequence, with the corresponding single character. The UTF\-8 flag |
| 233 |
is turned on only if the source string contains multiple-byte UTF\-8 |
| 234 |
characters. If \fR\f(CI$string\fR\fI\fR is invalid as extended UTF\-8, returns false; |
| 235 |
otherwise returns true. |
| 236 |
.Sp |
| 237 |
.Vb 11 |
| 238 |
\& my $x = "\exc4\ex80"; # $x contains two characters, with ords |
| 239 |
\& # 0xc4 and 0x80 |
| 240 |
\& utf8::decode($x); # On ASCII platforms, $x contains one char, |
| 241 |
\& # with ord 0x100. Since these bytes aren\*(Aqt |
| 242 |
\& # legal UTF\-EBCDIC, on EBCDIC platforms, $x is |
| 243 |
\& # unchanged and the function returns FALSE. |
| 244 |
\& my $y = "\exc3\ex83\exc2\exab"; This has been encoded twice; this |
| 245 |
\& # example is only for ASCII platforms |
| 246 |
\& utf8::decode($y); # Converts $y to \exc3\exab, returns TRUE; |
| 247 |
\& utf8::decode($y); # Further converts to \exeb, returns TRUE; |
| 248 |
\& utf8::decode($y); # Returns FALSE, leaves $y unchanged |
| 249 |
.Ve |
| 250 |
.Sp |
| 251 |
\&\fBNote that this function does not handle arbitrary encodings\fR; |
| 252 |
use Encode instead. |
| 253 |
.IP \(bu 4 |
| 254 |
\&\f(CW\*(C`$unicode = utf8::native_to_unicode($code_point)\*(C'\fR |
| 255 |
.Sp |
| 256 |
(Since Perl v5.8.0) |
| 257 |
This takes an unsigned integer (which represents the ordinal number of a |
| 258 |
character (or a code point) on the platform the program is being run on) and |
| 259 |
returns its Unicode equivalent value. Since ASCII platforms natively use the |
| 260 |
Unicode code points, this function returns its input on them. On EBCDIC |
| 261 |
platforms it converts from EBCDIC to Unicode. |
| 262 |
.Sp |
| 263 |
A meaningless value will currently be returned if the input is not an unsigned |
| 264 |
integer. |
| 265 |
.Sp |
| 266 |
Since Perl v5.22.0, calls to this function are optimized out on ASCII |
| 267 |
platforms, so there is no performance hit in using it there. |
| 268 |
.IP \(bu 4 |
| 269 |
\&\f(CW\*(C`$native = utf8::unicode_to_native($code_point)\*(C'\fR |
| 270 |
.Sp |
| 271 |
(Since Perl v5.8.0) |
| 272 |
This is the inverse of \f(CWutf8::native_to_unicode()\fR, converting the other |
| 273 |
direction. Again, on ASCII platforms, this returns its input, but on EBCDIC |
| 274 |
platforms it will find the native platform code point, given any Unicode one. |
| 275 |
.Sp |
| 276 |
A meaningless value will currently be returned if the input is not an unsigned |
| 277 |
integer. |
| 278 |
.Sp |
| 279 |
Since Perl v5.22.0, calls to this function are optimized out on ASCII |
| 280 |
platforms, so there is no performance hit in using it there. |
| 281 |
.IP \(bu 4 |
| 282 |
\&\f(CW\*(C`$flag = utf8::is_utf8($string)\*(C'\fR |
| 283 |
.Sp |
| 284 |
(Since Perl 5.8.1) Test whether \fR\f(CI$string\fR\fI\fR is marked internally as encoded in |
| 285 |
UTF\-8. Functionally the same as \f(CWEncode::is_utf8($string)\fR. |
| 286 |
.Sp |
| 287 |
Typically only necessary for debugging and testing, if you need to |
| 288 |
dump the internals of an SV, Devel::Peek's \fBDump()\fR |
| 289 |
provides more detail in a compact form. |
| 290 |
.Sp |
| 291 |
If you still think you need this outside of debugging, testing or |
| 292 |
dealing with filenames, you should probably read perlunitut and |
| 293 |
"What is "the UTF8 flag"?" in perlunifaq. |
| 294 |
.Sp |
| 295 |
Don't use this flag as a marker to distinguish character and binary |
| 296 |
data: that should be decided for each variable when you write your |
| 297 |
code. |
| 298 |
.Sp |
| 299 |
To force unicode semantics in code portable to perl 5.8 and 5.10, call |
| 300 |
\&\f(CWutf8::upgrade($string)\fR unconditionally. |
| 301 |
.IP \(bu 4 |
| 302 |
\&\f(CW\*(C`$flag = utf8::valid($string)\*(C'\fR |
| 303 |
.Sp |
| 304 |
[INTERNAL] Test whether \fR\f(CI$string\fR\fI\fR is in a consistent state regarding |
| 305 |
UTF\-8. Will return true if it is well-formed Perl extended UTF\-8 and has the |
| 306 |
UTF\-8 flag |
| 307 |
on \fBor\fR if \fI\fR\f(CI$string\fR\fI\fR is held as bytes (both these states are 'consistent'). |
| 308 |
The main reason for this routine is to allow Perl's test suite to check |
| 309 |
that operations have left strings in a consistent state. |
| 310 |
.PP |
| 311 |
\&\f(CW\*(C`utf8::encode\*(C'\fR is like \f(CW\*(C`utf8::upgrade\*(C'\fR, but the UTF8 flag is |
| 312 |
cleared. See perlunicode, and the C API |
| 313 |
functions \f(CW\*(C`sv_utf8_upgrade\*(C'\fR, |
| 314 |
\&\f(CW\*(C`"sv_utf8_downgrade" in perlapi\*(C'\fR, \f(CW\*(C`"sv_utf8_encode" in perlapi\*(C'\fR, |
| 315 |
and \f(CW\*(C`"sv_utf8_decode" in perlapi\*(C'\fR, which are wrapped by the Perl functions |
| 316 |
\&\f(CW\*(C`utf8::upgrade\*(C'\fR, \f(CW\*(C`utf8::downgrade\*(C'\fR, \f(CW\*(C`utf8::encode\*(C'\fR and |
| 317 |
\&\f(CW\*(C`utf8::decode\*(C'\fR. Also, the functions \f(CW\*(C`utf8::is_utf8\*(C'\fR, \f(CW\*(C`utf8::valid\*(C'\fR, |
| 318 |
\&\f(CW\*(C`utf8::encode\*(C'\fR, \f(CW\*(C`utf8::decode\*(C'\fR, \f(CW\*(C`utf8::upgrade\*(C'\fR, and \f(CW\*(C`utf8::downgrade\*(C'\fR are |
| 319 |
actually internal, and thus always available, without a \f(CW\*(C`require utf8\*(C'\fR |
| 320 |
statement. |
| 321 |
.SH BUGS |
| 322 |
.IX Header "BUGS" |
| 323 |
Some filesystems may not support UTF\-8 file names, or they may be supported |
| 324 |
incompatibly with Perl. Therefore UTF\-8 names that are visible to the |
| 325 |
filesystem, such as module names may not work. |
| 326 |
.SH "SEE ALSO" |
| 327 |
.IX Header "SEE ALSO" |
| 328 |
perlunitut, perluniintro, perlrun, bytes, perlunicode |