| 1 |
A Fast Method of Identifying Plain Text Files |
| 2 |
============================================= |
| 3 |
|
| 4 |
|
| 5 |
Introduction |
| 6 |
------------ |
| 7 |
|
| 8 |
Given a file coming from an unknown source, it is generally impossible |
| 9 |
to conclude automatically, and with 100% accuracy, whether that file is |
| 10 |
a plain text file, without performing a heavy-duty semantic analysis on |
| 11 |
the file contents. It is, however, possible to obtain a fairly high |
| 12 |
degree of accuracy, by employing various simple heuristics. |
| 13 |
|
| 14 |
Previous versions of the zip tools were using a crude detection scheme, |
| 15 |
originally used by PKWare in its PKZip programs: if more than 80% (4/5) |
| 16 |
of the bytes are within the range [7..127], the file is labeled as plain |
| 17 |
text, otherwise it is labeled as binary. A prominent limitation of this |
| 18 |
scheme is the restriction to Latin-based alphabets. Other alphabets, |
| 19 |
like Greek, Cyrillic or Asian, make extensive use of the bytes within |
| 20 |
the range [128..255], and texts using these alphabets are most often |
| 21 |
mis-identified by this scheme; in other words, the rate of false |
| 22 |
negatives is sometimes too high, which means that the recall is low. |
| 23 |
Another weakness of this scheme is a reduced precision, due to the false |
| 24 |
positives that may occur when binary files containing a large amount of |
| 25 |
textual characters are mis-identified as plain text. |
| 26 |
|
| 27 |
In this article we propose a new detection scheme, with a much increased |
| 28 |
accuracy and precision, and a near-100% recall. This scheme is designed |
| 29 |
to work on ASCII and ASCII-derived alphabets, and it handles single-byte |
| 30 |
alphabets (ISO-8859, OEM, KOI-8, etc.), and variable-sized alphabets |
| 31 |
(DBCS, UTF-8, etc.). However, it cannot handle fixed-sized, multi-byte |
| 32 |
alphabets (UCS-2, UCS-4), nor UTF-16. The principle used by this scheme |
| 33 |
can easily be adapted to non-ASCII alphabets like EBCDIC. |
| 34 |
|
| 35 |
|
| 36 |
The Algorithm |
| 37 |
------------- |
| 38 |
|
| 39 |
The algorithm works by dividing the set of bytes [0..255] into three |
| 40 |
categories: |
| 41 |
- The white list of textual bytecodes: |
| 42 |
9 (TAB), 10 (LF), 13 (CR), 20 (SPACE) to 255 |
| 43 |
- The gray list of tolerated bytecodes: |
| 44 |
7 (BEL), 8 (BS), 11 (VT), 12 (FF), 26 (SUB), 27 (ESC) |
| 45 |
- The black list of undesired, non-textual bytecodes: |
| 46 |
0 (NUL) to 6, 14 to 31. |
| 47 |
|
| 48 |
If a file contains at least one byte that belongs to the white list, and |
| 49 |
no byte that belongs to the black list, then the file is categorized as |
| 50 |
plain text. Otherwise, it is categorized as binary. |
| 51 |
|
| 52 |
|
| 53 |
Rationale |
| 54 |
--------- |
| 55 |
|
| 56 |
The idea behind this algorithm relies on two observations. |
| 57 |
|
| 58 |
The first observation is that, although the full range of 7-bit codes |
| 59 |
(0..127) is properly specified by the ASCII standard, most control |
| 60 |
characters in the range 0..31 are not used in practice. The only |
| 61 |
widely-used, almost universally-portable control codes are 9 (TAB), |
| 62 |
10 (LF), and 13 (CR). There are a few more control codes that are |
| 63 |
recognized on a reduced range of platforms and text viewers/editors: |
| 64 |
7 (BEL), 8 (BS), 11 (VT), 12 (FF), 26 (SUB), and 27 (ESC); but these |
| 65 |
codes are rarely (if ever) used alone, without being accompanied by |
| 66 |
some printable text. Even the newer, portable text formats, such as |
| 67 |
XML, avoid using control characters outside the list mentioned here. |
| 68 |
|
| 69 |
The second observation is that most of the binary files tend to contain |
| 70 |
control characters, especially 0 (NUL); even though the older text |
| 71 |
detection schemes observe the presence of non-ASCII codes from the range |
| 72 |
[128..255], the precision rarely has to suffer if this upper range is |
| 73 |
labeled as textual, because the files that are genuinely binary tend to |
| 74 |
contain both control characters, and codes from the upper range. On the |
| 75 |
other hand, the upper range needs to be labeled as textual, because it |
| 76 |
is being used by virtually all ASCII extensions. In particular, this |
| 77 |
range is being heavily used to encode non-Latin scripts. |
| 78 |
|
| 79 |
Given the two observations, the plain text detection algorithm becomes |
| 80 |
straightforward. There must be at least some printable material, or |
| 81 |
some portable whitespace such as TAB, CR or LF, otherwise the file is |
| 82 |
not labeled as plain text. (The boundary case, when the file is empty, |
| 83 |
automatically falls into this category.) However, there must be no |
| 84 |
non-portable control characters, otherwise it's very likely that the |
| 85 |
intended reader of that file is a machine, rather than a human. |
| 86 |
|
| 87 |
Since there is no counting involved, other than simply observing the |
| 88 |
presence or the absence of some byte values, the algorithm produces |
| 89 |
uniform results on any particular text file, no matter what alphabet |
| 90 |
encoding is being used for that text. (In contrast, if counting were |
| 91 |
involved, it could be possible to obtain different results on a text |
| 92 |
encoded, say, using ISO-8859-2 versus UTF-8.) There is the category |
| 93 |
of plain text files that are "polluted" with one or a few black-listed |
| 94 |
codes, either by mistake, or by peculiar design considerations. In such |
| 95 |
cases, a scheme that tolerates a small percentage of black-listed codes |
| 96 |
would provide an increased recall (i.e. more true positives). This, |
| 97 |
however, incurs a reduced precision, since false positives are also more |
| 98 |
likely to appear in binary files that contain large chunks of textual |
| 99 |
data. "Polluted" plain text may, in fact, be regarded as binary, on |
| 100 |
which text conversions should not be performed. Under this premise, it |
| 101 |
is safe to say that the detection method provides a near-100% recall. |
| 102 |
|
| 103 |
Experiments have been run on a large set of files of various categories, |
| 104 |
including plain old texts, system logs, source code, formatted office |
| 105 |
documents, compiled object code, etcetera. The results confirm the |
| 106 |
optimistic assumptions about the high accuracy, precision and recall |
| 107 |
offered by this algorithm. |
| 108 |
|
| 109 |
|
| 110 |
-- |
| 111 |
Cosmin Truta |
| 112 |
Last updated: 2005-Feb-27 |