| 1 |
NAME |
| 2 |
Test::utf8 - handy utf8 tests |
| 3 |
|
| 4 |
SYNOPSIS |
| 5 |
# check the string is good |
| 6 |
is_valid_string($string); # check the string is valid |
| 7 |
is_sane_utf8($string); # check not double encoded |
| 8 |
|
| 9 |
# check the string has certain attributes |
| 10 |
is_flagged_utf8($string1); # has utf8 flag set |
| 11 |
is_within_ascii($string2); # only has ascii chars in it |
| 12 |
isnt_within_ascii($string3); # has chars outside the ascii range |
| 13 |
is_within_latin_1($string4); # only has latin-1 chars in it |
| 14 |
isnt_within_ascii($string5); # has chars outside the latin-1 range |
| 15 |
|
| 16 |
DESCRIPTION |
| 17 |
This module is a collection of tests useful for dealing with utf8 |
| 18 |
strings in Perl. |
| 19 |
|
| 20 |
This module has two types of tests: The validity tests check if a string |
| 21 |
is valid and not corrupt, whereas the characteristics tests will check |
| 22 |
that string has a given set of characteristics. |
| 23 |
|
| 24 |
Validity Tests |
| 25 |
is_valid_string($string, $testname) |
| 26 |
Checks if the string is "valid", i.e. this passes and returns true |
| 27 |
unless the internal utf8 flag hasn't been set on scalar that isn't |
| 28 |
made up of a valid utf-8 byte sequence. |
| 29 |
|
| 30 |
This should *never* happen and, in theory, this test should always |
| 31 |
pass. Unless you (or a module you use) goes monkeying around inside |
| 32 |
a scalar using Encode's private functions or XS code you shouldn't |
| 33 |
ever end up in a situation where you've got a corrupt scalar. But if |
| 34 |
you do, and you do, then this function should help you detect the |
| 35 |
problem. |
| 36 |
|
| 37 |
To be clear, here's an example of the error case this can detect: |
| 38 |
|
| 39 |
my $mark = "Mark"; |
| 40 |
my $leon = "L\x{e9}on"; |
| 41 |
is_valid_string($mark); # passes, not utf-8 |
| 42 |
is_valid_string($leon); # passes, not utf-8 |
| 43 |
|
| 44 |
my $iloveny = "I \x{2665} NY"; |
| 45 |
is_valid_string($iloveny); # passes, proper utf-8 |
| 46 |
|
| 47 |
my $acme = "L\x{c3}\x{a9}on"; |
| 48 |
Encode::_utf8_on($acme); # (please don't do things like this) |
| 49 |
is_valid_string($acme); # passes, proper utf-8 byte sequence upgraded |
| 50 |
|
| 51 |
Encode::_utf8_on($leon); # (this is why you don't do things like this) |
| 52 |
is_valid_string($leon); # fails! the byte \x{e9} isn't valid utf-8 |
| 53 |
|
| 54 |
is_sane_utf8($string, $name) |
| 55 |
This test fails if the string contains something that looks like it |
| 56 |
might be dodgy utf8, i.e. containing something that looks like the |
| 57 |
multi-byte sequence for a latin-1 character but perl hasn't been |
| 58 |
instructed to treat as such. Strings that are not utf8 always |
| 59 |
automatically pass. |
| 60 |
|
| 61 |
Some examples may help: |
| 62 |
|
| 63 |
# This will pass as it's a normal latin-1 string |
| 64 |
is_sane_utf8("Hello L\x{e9}eon"); |
| 65 |
|
| 66 |
# this will fail because the \x{c3}\x{a9} looks like the |
| 67 |
# utf8 byte sequence for e-acute |
| 68 |
my $string = "Hello L\x{c3}\x{a9}on"; |
| 69 |
is_sane_utf8($string); |
| 70 |
|
| 71 |
# this will pass because the utf8 is correctly interpreted as utf8 |
| 72 |
Encode::_utf8_on($string) |
| 73 |
is_sane_utf8($string); |
| 74 |
|
| 75 |
Obviously this isn't a hundred percent reliable. The edge case where |
| 76 |
this will fail is where you have "\x{c2}" (which is "LATIN CAPITAL |
| 77 |
LETTER WITH CIRCUMFLEX") or "\x{c3}" (which is "LATIN CAPITAL LETTER |
| 78 |
WITH TILDE") followed by one of the latin-1 punctuation symbols. |
| 79 |
|
| 80 |
# a capital letter A with tilde surrounded by smart quotes |
| 81 |
# this will fail because it'll see the "\x{c2}\x{94}" and think |
| 82 |
# it's actually the utf8 sequence for the end smart quote |
| 83 |
is_sane_utf8("\x{93}\x{c2}\x{94}"); |
| 84 |
|
| 85 |
However, since this hardly comes up this test is reasonably reliable |
| 86 |
in most cases. Still, care should be applied in cases where dynamic |
| 87 |
data is placed next to latin-1 punctuation to avoid false negatives. |
| 88 |
|
| 89 |
There exists two situations to cause this test to fail; The string |
| 90 |
contains utf8 byte sequences and the string hasn't been flagged as |
| 91 |
utf8 (this normally means that you got it from an external source |
| 92 |
like a C library; When Perl needs to store a string internally as |
| 93 |
utf8 it does it's own encoding and flagging transparently) or a utf8 |
| 94 |
flagged string contains byte sequences that when translated to |
| 95 |
characters themselves look like a utf8 byte sequence. The test |
| 96 |
diagnostics tells you which is the case. |
| 97 |
|
| 98 |
String Characteristic Tests |
| 99 |
These routines allow you to check the range of characters in a string. |
| 100 |
Note that these routines are blind to the actual encoding perl |
| 101 |
internally uses to store the characters, they just check if the string |
| 102 |
contains only characters that can be represented in the named encoding: |
| 103 |
|
| 104 |
is_within_ascii |
| 105 |
Tests that a string only contains characters that are in the ASCII |
| 106 |
character set. |
| 107 |
|
| 108 |
is_within_latin_1 |
| 109 |
Tests that a string only contains characters that are in latin-1. |
| 110 |
|
| 111 |
Simply check if a scalar is or isn't flagged as utf8 by perl's |
| 112 |
internals: |
| 113 |
|
| 114 |
is_flagged_utf8($string, $name) |
| 115 |
Passes if the string is flagged by perl's internals as utf8, fails |
| 116 |
if it's not. |
| 117 |
|
| 118 |
isnt_flagged_utf8($string,$name) |
| 119 |
The opposite of "is_flagged_utf8", passes if and only if the string |
| 120 |
isn't flagged as utf8 by perl's internals. |
| 121 |
|
| 122 |
Note: you can refer to this function as "isn't_flagged_utf8" if you |
| 123 |
really want to. |
| 124 |
|
| 125 |
AUTHOR |
| 126 |
Written by Mark Fowler [email protected] |
| 127 |
|
| 128 |
COPYRIGHT |
| 129 |
Copyright Mark Fowler 2004,2012. All rights reserved. |
| 130 |
|
| 131 |
This program is free software; you can redistribute it and/or modify it |
| 132 |
under the same terms as Perl itself. |
| 133 |
|
| 134 |
BUGS |
| 135 |
None known. Please report any to me via the CPAN RT system. See |
| 136 |
http://rt.cpan.org/ for more details. |
| 137 |
|
| 138 |
SEE ALSO |
| 139 |
Test::DoubleEncodedEntities for testing for double encoded HTML |
| 140 |
entities. |
| 141 |
|