| 1 |
NAME |
| 2 |
|
| 3 |
Test::Mojibake - check your source for encoding misbehavior. |
| 4 |
|
| 5 |
VERSION |
| 6 |
|
| 7 |
version 1.3 |
| 8 |
|
| 9 |
SYNOPSIS |
| 10 |
|
| 11 |
# Test::Mojibake lets you check for inconsistencies in source/documentation encoding, and report its results in standard Test::Simple fashion. |
| 12 |
no strict 'vars'; |
| 13 |
|
| 14 |
use Test::Mojibake; |
| 15 |
file_encoding_ok($file, 'Valid encoding'); |
| 16 |
done_testing($num_tests); |
| 17 |
|
| 18 |
DESCRIPTION |
| 19 |
|
| 20 |
Many modern text editors automatically save files using UTF-8 |
| 21 |
codification, however, perl interpreter does not expects it by default. |
| 22 |
Whereas this does not represent a big deal on (most) backend-oriented |
| 23 |
programs, Web framework (Catalyst <http://www.catalystframework.org/>, |
| 24 |
Mojolicious <http://mojolicio.us/>) based applications will suffer of |
| 25 |
so-called Mojibake <http://en.wikipedia.org/wiki/Mojibake> (lit. |
| 26 |
"unintelligible sequence of characters"). |
| 27 |
|
| 28 |
Even worse: if an editor saves BOM (Byte Order Mark, U+FEFF character |
| 29 |
in Unicode) at the start of the script with executable bit set (on Unix |
| 30 |
systems), it won't execute at all, due to shebang corruption. |
| 31 |
|
| 32 |
Avoiding codification problems is quite simple: |
| 33 |
|
| 34 |
* Always use utf8/use common::sense when saving source as UTF-8; |
| 35 |
|
| 36 |
* Always specify =encoding UTF-8 when saving POD as UTF-8; |
| 37 |
|
| 38 |
* Do neither of above when saving as ISO-8859-1; |
| 39 |
|
| 40 |
* Never save BOM (not that it's wrong; just avoid it as you'll barely |
| 41 |
notice it's presence when in trouble). |
| 42 |
|
| 43 |
However, if you find yourself upgrading old code to use UTF-8 or trying |
| 44 |
to standardize a big project with many developers each one using a |
| 45 |
different platform/editor, reviewing all files manually can be quite |
| 46 |
painful. Specially in cases when some files have multiple encodings |
| 47 |
(note: it all started when I realized that Gedit & derivatives are |
| 48 |
unable to open files with character conversion tables). |
| 49 |
|
| 50 |
Enter the Test::Mojibake ;) |
| 51 |
|
| 52 |
FUNCTIONS |
| 53 |
|
| 54 |
file_encoding_ok( FILENAME[, TESTNAME ] ) |
| 55 |
|
| 56 |
Validates the codification of FILENAME. |
| 57 |
|
| 58 |
When it fails, file_encoding_ok() will report the probable cause. |
| 59 |
|
| 60 |
The optional second argument TESTNAME is the name of the test. If it is |
| 61 |
omitted, file_encoding_ok() chooses a default test name "Mojibake test |
| 62 |
for FILENAME". |
| 63 |
|
| 64 |
all_files_encoding_ok( [@entries] ) |
| 65 |
|
| 66 |
Validates codification of all the files under @entries. It runs |
| 67 |
all_files() on directories and assumes everything else to be a file to |
| 68 |
be tested. It calls the plan() function for you (one test for each |
| 69 |
file), so you can't have already called plan. |
| 70 |
|
| 71 |
If @entries is empty or not passed, the function finds all |
| 72 |
source/documentation files in files in the blib directory if it exists, |
| 73 |
or the lib directory if not. A source/documentation file is one that |
| 74 |
ends with .pod, .pl and .pm, or any file where the first line looks |
| 75 |
like a shebang line. |
| 76 |
|
| 77 |
all_files( [@dirs] ) |
| 78 |
|
| 79 |
Returns a list of all the Perl files in @dirs and in directories below. |
| 80 |
If no directories are passed, it defaults to blib if blib exists, or |
| 81 |
else lib if not. Skips any files in CVS, .svn, .git and similar |
| 82 |
directories. See %Test::Mojibake::ignore_dirs for a list of them. |
| 83 |
|
| 84 |
A Perl file is: |
| 85 |
|
| 86 |
* Any file that ends in .PL, .pl, .pm, .pod, or .t; |
| 87 |
|
| 88 |
* Any file that has a first line with a shebang and "perl" on it; |
| 89 |
|
| 90 |
* Any file that ends in .bat and has a first line with "--*-Perl-*--" |
| 91 |
on it. |
| 92 |
|
| 93 |
The order of the files returned is machine-dependent. If you want them |
| 94 |
sorted, you'll have to sort them yourself. |
| 95 |
|
| 96 |
_detect_utf8( \$string ) |
| 97 |
|
| 98 |
Detects presence of UTF-8 encoded characters in a referenced octet |
| 99 |
stream. |
| 100 |
|
| 101 |
Return codes: |
| 102 |
|
| 103 |
* 0 - 8-bit characters detected, does not validate as UTF-8; |
| 104 |
|
| 105 |
* 1 - only 7-bit characters; |
| 106 |
|
| 107 |
* 2 - 8-bit characters detected, validates as UTF-8. |
| 108 |
|
| 109 |
Unicode::CheckUTF8 is highly recommended, however, it is optional and |
| 110 |
this function will fallback to the Pure Perl implementation of the |
| 111 |
following PHP code: |
| 112 |
http://www.php.net/manual/en/function.utf8-encode.php#85293 |
| 113 |
|
| 114 |
SAMPLE TEST SCRIPT |
| 115 |
|
| 116 |
Module authors can include the following in a t/mojibake.t file and |
| 117 |
have Test::Mojibake automatically find and check all source files in a |
| 118 |
module distribution: |
| 119 |
|
| 120 |
#!perl -T |
| 121 |
use strict; |
| 122 |
|
| 123 |
BEGIN { |
| 124 |
unless ($ENV{RELEASE_TESTING}) { |
| 125 |
require Test::More; |
| 126 |
Test::More::plan(skip_all => 'these tests are for release candidate testing'); |
| 127 |
} |
| 128 |
} |
| 129 |
|
| 130 |
use Test::More; |
| 131 |
|
| 132 |
eval 'use Test::Mojibake'; |
| 133 |
plan skip_all => 'Test::Mojibake required for source encoding testing' if $@; |
| 134 |
|
| 135 |
all_files_encoding_ok(); |
| 136 |
|
| 137 |
OPERATION |
| 138 |
|
| 139 |
Test::Mojibake validates codification of both source (Perl code) and |
| 140 |
documentation (POD). Both are assumed to be encoded in ISO-8859-1 (aka |
| 141 |
latin1). Perl switches to UTF-8 through the statement: |
| 142 |
|
| 143 |
use utf8; |
| 144 |
|
| 145 |
or: |
| 146 |
|
| 147 |
use utf8::all; |
| 148 |
|
| 149 |
or even: |
| 150 |
|
| 151 |
use common::sense; |
| 152 |
|
| 153 |
Similarly, POD encoding can be changed via: |
| 154 |
|
| 155 |
=encoding UTF-8 |
| 156 |
|
| 157 |
Correspondingly, no utf8/=encoding latin1 put Perl back into ISO-8859-1 |
| 158 |
mode. |
| 159 |
|
| 160 |
Actually, Test::Mojibake only cares about UTF-8, as it is roughly safe |
| 161 |
to be detected. So, when UTF-8 characters are detected without |
| 162 |
preceding declaration, an error is reported. On the other way, |
| 163 |
non-UTF-8 characters in UTF-8 mode are wrong, either. |
| 164 |
|
| 165 |
If present, Unicode::CheckUTF8 module (XS wrapper) will be used to |
| 166 |
validate UTF-8 strings, note that it is 30 times faster and a lot more |
| 167 |
Unicode Consortium compliant than the built-in Pure Perl |
| 168 |
implementation! |
| 169 |
|
| 170 |
UTF-8 BOM (Byte Order Mark) is also detected as an error. While Perl is |
| 171 |
OK handling BOM, your OS probably isn't. Check out: |
| 172 |
|
| 173 |
./bom.pl: line 1: $'\357\273\277#!/usr/bin/perl': command not found |
| 174 |
|
| 175 |
Caveats |
| 176 |
|
| 177 |
Whole-line source comments, like: |
| 178 |
|
| 179 |
# this is a whole-line comment... |
| 180 |
print "### hello world ###\n"; # ...and this os not |
| 181 |
|
| 182 |
are not checked at all. This is mainly because many scripts/modules do |
| 183 |
contain authors' names in headers, before the proper encoding |
| 184 |
specification. So, if you happen to have some acutes/umlauts in your |
| 185 |
name and your editor sign your code in the similar way, you probably |
| 186 |
won't be happy with Test::Mojibake flooding you with (false) error |
| 187 |
messages. |
| 188 |
|
| 189 |
If you are wondering why only whole-line comments are stripped, check |
| 190 |
the second line of the above example. |
| 191 |
|
| 192 |
SEE ALSO |
| 193 |
|
| 194 |
* scan_mojibake |
| 195 |
|
| 196 |
* common::sense |
| 197 |
|
| 198 |
* utf8::all |
| 199 |
|
| 200 |
* Dist::Zilla::Plugin::MojibakeTests |
| 201 |
|
| 202 |
* Test::Perl::Critic |
| 203 |
|
| 204 |
* Test::Pod |
| 205 |
|
| 206 |
* Test::Pod::Coverage |
| 207 |
|
| 208 |
* Test::Kwalitee |
| 209 |
|
| 210 |
ACKNOWLEDGEMENTS |
| 211 |
|
| 212 |
This module is based on Test::Pod. |
| 213 |
|
| 214 |
Thanks to Andy Lester, David Wheeler, Paul Miller and Peter Edwards for |
| 215 |
contributions and to brian d foy for the original code. |
| 216 |
|
| 217 |
AUTHOR |
| 218 |
|
| 219 |
Stanislaw Pusep <[email protected]> |
| 220 |
|
| 221 |
COPYRIGHT AND LICENSE |
| 222 |
|
| 223 |
This software is copyright (c) 2017 by Stanislaw Pusep. |
| 224 |
|
| 225 |
This is free software; you can redistribute it and/or modify it under |
| 226 |
the same terms as the Perl 5 programming language system itself. |
| 227 |
|
| 228 |
CONTRIBUTORS |
| 229 |
|
| 230 |
* Dave Rolsky <[email protected]> |
| 231 |
|
| 232 |
* Hunter McMillen <[email protected]> |
| 233 |
|
| 234 |
* John SJ Anderson <[email protected]> |
| 235 |
|
| 236 |
* Karen Etheridge <[email protected]> |
| 237 |
|