Debian Codegraph @

README

raw
1 NAME
2 Test::utf8 - handy utf8 tests
3
4 SYNOPSIS
5 # check the string is good
6 is_valid_string($string); # check the string is valid
7 is_sane_utf8($string); # check not double encoded
8
9 # check the string has certain attributes
10 is_flagged_utf8($string1); # has utf8 flag set
11 is_within_ascii($string2); # only has ascii chars in it
12 isnt_within_ascii($string3); # has chars outside the ascii range
13 is_within_latin_1($string4); # only has latin-1 chars in it
14 isnt_within_ascii($string5); # has chars outside the latin-1 range
15
16 DESCRIPTION
17 This module is a collection of tests useful for dealing with utf8
18 strings in Perl.
19
20 This module has two types of tests: The validity tests check if a string
21 is valid and not corrupt, whereas the characteristics tests will check
22 that string has a given set of characteristics.
23
24 Validity Tests
25 is_valid_string($string, $testname)
26 Checks if the string is "valid", i.e. this passes and returns true
27 unless the internal utf8 flag hasn't been set on scalar that isn't
28 made up of a valid utf-8 byte sequence.
29
30 This should *never* happen and, in theory, this test should always
31 pass. Unless you (or a module you use) goes monkeying around inside
32 a scalar using Encode's private functions or XS code you shouldn't
33 ever end up in a situation where you've got a corrupt scalar. But if
34 you do, and you do, then this function should help you detect the
35 problem.
36
37 To be clear, here's an example of the error case this can detect:
38
39 my $mark = "Mark";
40 my $leon = "L\x{e9}on";
41 is_valid_string($mark); # passes, not utf-8
42 is_valid_string($leon); # passes, not utf-8
43
44 my $iloveny = "I \x{2665} NY";
45 is_valid_string($iloveny); # passes, proper utf-8
46
47 my $acme = "L\x{c3}\x{a9}on";
48 Encode::_utf8_on($acme); # (please don't do things like this)
49 is_valid_string($acme); # passes, proper utf-8 byte sequence upgraded
50
51 Encode::_utf8_on($leon); # (this is why you don't do things like this)
52 is_valid_string($leon); # fails! the byte \x{e9} isn't valid utf-8
53
54 is_sane_utf8($string, $name)
55 This test fails if the string contains something that looks like it
56 might be dodgy utf8, i.e. containing something that looks like the
57 multi-byte sequence for a latin-1 character but perl hasn't been
58 instructed to treat as such. Strings that are not utf8 always
59 automatically pass.
60
61 Some examples may help:
62
63 # This will pass as it's a normal latin-1 string
64 is_sane_utf8("Hello L\x{e9}eon");
65
66 # this will fail because the \x{c3}\x{a9} looks like the
67 # utf8 byte sequence for e-acute
68 my $string = "Hello L\x{c3}\x{a9}on";
69 is_sane_utf8($string);
70
71 # this will pass because the utf8 is correctly interpreted as utf8
72 Encode::_utf8_on($string)
73 is_sane_utf8($string);
74
75 Obviously this isn't a hundred percent reliable. The edge case where
76 this will fail is where you have "\x{c2}" (which is "LATIN CAPITAL
77 LETTER WITH CIRCUMFLEX") or "\x{c3}" (which is "LATIN CAPITAL LETTER
78 WITH TILDE") followed by one of the latin-1 punctuation symbols.
79
80 # a capital letter A with tilde surrounded by smart quotes
81 # this will fail because it'll see the "\x{c2}\x{94}" and think
82 # it's actually the utf8 sequence for the end smart quote
83 is_sane_utf8("\x{93}\x{c2}\x{94}");
84
85 However, since this hardly comes up this test is reasonably reliable
86 in most cases. Still, care should be applied in cases where dynamic
87 data is placed next to latin-1 punctuation to avoid false negatives.
88
89 There exists two situations to cause this test to fail; The string
90 contains utf8 byte sequences and the string hasn't been flagged as
91 utf8 (this normally means that you got it from an external source
92 like a C library; When Perl needs to store a string internally as
93 utf8 it does it's own encoding and flagging transparently) or a utf8
94 flagged string contains byte sequences that when translated to
95 characters themselves look like a utf8 byte sequence. The test
96 diagnostics tells you which is the case.
97
98 String Characteristic Tests
99 These routines allow you to check the range of characters in a string.
100 Note that these routines are blind to the actual encoding perl
101 internally uses to store the characters, they just check if the string
102 contains only characters that can be represented in the named encoding:
103
104 is_within_ascii
105 Tests that a string only contains characters that are in the ASCII
106 character set.
107
108 is_within_latin_1
109 Tests that a string only contains characters that are in latin-1.
110
111 Simply check if a scalar is or isn't flagged as utf8 by perl's
112 internals:
113
114 is_flagged_utf8($string, $name)
115 Passes if the string is flagged by perl's internals as utf8, fails
116 if it's not.
117
118 isnt_flagged_utf8($string,$name)
119 The opposite of "is_flagged_utf8", passes if and only if the string
120 isn't flagged as utf8 by perl's internals.
121
122 Note: you can refer to this function as "isn't_flagged_utf8" if you
123 really want to.
124
125 AUTHOR
126 Written by Mark Fowler [email protected]
127
128 COPYRIGHT
129 Copyright Mark Fowler 2004,2012. All rights reserved.
130
131 This program is free software; you can redistribute it and/or modify it
132 under the same terms as Perl itself.
133
134 BUGS
135 None known. Please report any to me via the CPAN RT system. See
136 http://rt.cpan.org/ for more details.
137
138 SEE ALSO
139 Test::DoubleEncodedEntities for testing for double encoded HTML
140 entities.
141