Debian Codegraph @

README

raw
1 NAME
2
3 Test::Mojibake - check your source for encoding misbehavior.
4
5 VERSION
6
7 version 1.3
8
9 SYNOPSIS
10
11 # Test::Mojibake lets you check for inconsistencies in source/documentation encoding, and report its results in standard Test::Simple fashion.
12 no strict 'vars';
13
14 use Test::Mojibake;
15 file_encoding_ok($file, 'Valid encoding');
16 done_testing($num_tests);
17
18 DESCRIPTION
19
20 Many modern text editors automatically save files using UTF-8
21 codification, however, perl interpreter does not expects it by default.
22 Whereas this does not represent a big deal on (most) backend-oriented
23 programs, Web framework (Catalyst <http://www.catalystframework.org/>,
24 Mojolicious <http://mojolicio.us/>) based applications will suffer of
25 so-called Mojibake <http://en.wikipedia.org/wiki/Mojibake> (lit.
26 "unintelligible sequence of characters").
27
28 Even worse: if an editor saves BOM (Byte Order Mark, U+FEFF character
29 in Unicode) at the start of the script with executable bit set (on Unix
30 systems), it won't execute at all, due to shebang corruption.
31
32 Avoiding codification problems is quite simple:
33
34 * Always use utf8/use common::sense when saving source as UTF-8;
35
36 * Always specify =encoding UTF-8 when saving POD as UTF-8;
37
38 * Do neither of above when saving as ISO-8859-1;
39
40 * Never save BOM (not that it's wrong; just avoid it as you'll barely
41 notice it's presence when in trouble).
42
43 However, if you find yourself upgrading old code to use UTF-8 or trying
44 to standardize a big project with many developers each one using a
45 different platform/editor, reviewing all files manually can be quite
46 painful. Specially in cases when some files have multiple encodings
47 (note: it all started when I realized that Gedit & derivatives are
48 unable to open files with character conversion tables).
49
50 Enter the Test::Mojibake ;)
51
52 FUNCTIONS
53
54 file_encoding_ok( FILENAME[, TESTNAME ] )
55
56 Validates the codification of FILENAME.
57
58 When it fails, file_encoding_ok() will report the probable cause.
59
60 The optional second argument TESTNAME is the name of the test. If it is
61 omitted, file_encoding_ok() chooses a default test name "Mojibake test
62 for FILENAME".
63
64 all_files_encoding_ok( [@entries] )
65
66 Validates codification of all the files under @entries. It runs
67 all_files() on directories and assumes everything else to be a file to
68 be tested. It calls the plan() function for you (one test for each
69 file), so you can't have already called plan.
70
71 If @entries is empty or not passed, the function finds all
72 source/documentation files in files in the blib directory if it exists,
73 or the lib directory if not. A source/documentation file is one that
74 ends with .pod, .pl and .pm, or any file where the first line looks
75 like a shebang line.
76
77 all_files( [@dirs] )
78
79 Returns a list of all the Perl files in @dirs and in directories below.
80 If no directories are passed, it defaults to blib if blib exists, or
81 else lib if not. Skips any files in CVS, .svn, .git and similar
82 directories. See %Test::Mojibake::ignore_dirs for a list of them.
83
84 A Perl file is:
85
86 * Any file that ends in .PL, .pl, .pm, .pod, or .t;
87
88 * Any file that has a first line with a shebang and "perl" on it;
89
90 * Any file that ends in .bat and has a first line with "--*-Perl-*--"
91 on it.
92
93 The order of the files returned is machine-dependent. If you want them
94 sorted, you'll have to sort them yourself.
95
96 _detect_utf8( \$string )
97
98 Detects presence of UTF-8 encoded characters in a referenced octet
99 stream.
100
101 Return codes:
102
103 * 0 - 8-bit characters detected, does not validate as UTF-8;
104
105 * 1 - only 7-bit characters;
106
107 * 2 - 8-bit characters detected, validates as UTF-8.
108
109 Unicode::CheckUTF8 is highly recommended, however, it is optional and
110 this function will fallback to the Pure Perl implementation of the
111 following PHP code:
112 http://www.php.net/manual/en/function.utf8-encode.php#85293
113
114 SAMPLE TEST SCRIPT
115
116 Module authors can include the following in a t/mojibake.t file and
117 have Test::Mojibake automatically find and check all source files in a
118 module distribution:
119
120 #!perl -T
121 use strict;
122
123 BEGIN {
124 unless ($ENV{RELEASE_TESTING}) {
125 require Test::More;
126 Test::More::plan(skip_all => 'these tests are for release candidate testing');
127 }
128 }
129
130 use Test::More;
131
132 eval 'use Test::Mojibake';
133 plan skip_all => 'Test::Mojibake required for source encoding testing' if $@;
134
135 all_files_encoding_ok();
136
137 OPERATION
138
139 Test::Mojibake validates codification of both source (Perl code) and
140 documentation (POD). Both are assumed to be encoded in ISO-8859-1 (aka
141 latin1). Perl switches to UTF-8 through the statement:
142
143 use utf8;
144
145 or:
146
147 use utf8::all;
148
149 or even:
150
151 use common::sense;
152
153 Similarly, POD encoding can be changed via:
154
155 =encoding UTF-8
156
157 Correspondingly, no utf8/=encoding latin1 put Perl back into ISO-8859-1
158 mode.
159
160 Actually, Test::Mojibake only cares about UTF-8, as it is roughly safe
161 to be detected. So, when UTF-8 characters are detected without
162 preceding declaration, an error is reported. On the other way,
163 non-UTF-8 characters in UTF-8 mode are wrong, either.
164
165 If present, Unicode::CheckUTF8 module (XS wrapper) will be used to
166 validate UTF-8 strings, note that it is 30 times faster and a lot more
167 Unicode Consortium compliant than the built-in Pure Perl
168 implementation!
169
170 UTF-8 BOM (Byte Order Mark) is also detected as an error. While Perl is
171 OK handling BOM, your OS probably isn't. Check out:
172
173 ./bom.pl: line 1: $'\357\273\277#!/usr/bin/perl': command not found
174
175 Caveats
176
177 Whole-line source comments, like:
178
179 # this is a whole-line comment...
180 print "### hello world ###\n"; # ...and this os not
181
182 are not checked at all. This is mainly because many scripts/modules do
183 contain authors' names in headers, before the proper encoding
184 specification. So, if you happen to have some acutes/umlauts in your
185 name and your editor sign your code in the similar way, you probably
186 won't be happy with Test::Mojibake flooding you with (false) error
187 messages.
188
189 If you are wondering why only whole-line comments are stripped, check
190 the second line of the above example.
191
192 SEE ALSO
193
194 * scan_mojibake
195
196 * common::sense
197
198 * utf8::all
199
200 * Dist::Zilla::Plugin::MojibakeTests
201
202 * Test::Perl::Critic
203
204 * Test::Pod
205
206 * Test::Pod::Coverage
207
208 * Test::Kwalitee
209
210 ACKNOWLEDGEMENTS
211
212 This module is based on Test::Pod.
213
214 Thanks to Andy Lester, David Wheeler, Paul Miller and Peter Edwards for
215 contributions and to brian d foy for the original code.
216
217 AUTHOR
218
219 Stanislaw Pusep <[email protected]>
220
221 COPYRIGHT AND LICENSE
222
223 This software is copyright (c) 2017 by Stanislaw Pusep.
224
225 This is free software; you can redistribute it and/or modify it under
226 the same terms as the Perl 5 programming language system itself.
227
228 CONTRIBUTORS
229
230 * Dave Rolsky <[email protected]>
231
232 * Hunter McMillen <[email protected]>
233
234 * John SJ Anderson <[email protected]>
235
236 * Karen Etheridge <[email protected]>
237