| 1 |
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> |
| 2 |
<html> |
| 3 |
<head> |
| 4 |
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> |
| 5 |
<title>Bogotune FAQ</title> |
| 6 |
<style type="text/css"> |
| 7 |
h2 { |
| 8 |
margin-top: 1em; |
| 9 |
font-size: 125%; |
| 10 |
} |
| 11 |
h3 { |
| 12 |
margin-top: 1em; |
| 13 |
font-size: 110%; |
| 14 |
} |
| 15 |
p { |
| 16 |
margin-top : 0.5em; |
| 17 |
margin-bottom: 0.5em; |
| 18 |
} |
| 19 |
ul { |
| 20 |
margin-top: 1.5em; |
| 21 |
margin-bottom: 0.5em; |
| 22 |
} |
| 23 |
ul ul { |
| 24 |
margin-top: 0.25em; |
| 25 |
margin-bottom: 0; |
| 26 |
} |
| 27 |
li { |
| 28 |
margin-top: 0; |
| 29 |
margin-bottom: 1em; |
| 30 |
} |
| 31 |
li li { |
| 32 |
margin-bottom: 0.25em; |
| 33 |
} |
| 34 |
dt { |
| 35 |
margin-top: 0.5em; |
| 36 |
margin-bottom: 0; |
| 37 |
} |
| 38 |
hr { |
| 39 |
margin-top: 1em; |
| 40 |
margin-bottom: 1em; |
| 41 |
} |
| 42 |
</style> |
| 43 |
</head> |
| 44 |
|
| 45 |
<body> |
| 46 |
<h1>Bogotune FAQ</h1> |
| 47 |
|
| 48 |
<p>Official Versions: In |
| 49 |
<a href="http://bogofilter.sourceforge.net/-faq.html">bogotune-faq</a><br> |
| 50 |
Maintainer: David Relson <[email protected]></p> |
| 51 |
|
| 52 |
<p>This document is intended to answer frequently asked questions |
| 53 |
about bogotune.</p> |
| 54 |
|
| 55 |
<ul> |
| 56 |
<li><a href="#origin"> Where did bogotune come from?</a></li> |
| 57 |
<li><a href="#message-count"> What's the message count format?</a></li> |
| 58 |
<li><a href="#work"> How does bogotune work?</a></li> |
| 59 |
|
| 60 |
<li><a href="#ensure-count"> How does bogotune ensure the messages it |
| 61 |
works with are numerous enough, and well enough classified, to |
| 62 |
deliver useful recommendations?</a></li> |
| 63 |
|
| 64 |
<li><a href="#force"> Can I tell bogotune to do its work even |
| 65 |
though it doesn't like the data?</a></li> |
| 66 |
</ul> |
| 67 |
|
| 68 |
<hr> |
| 69 |
<h2 id="origin">Where did bogotune come from?</h2> |
| 70 |
|
| 71 |
<p>Greg Louis wrote the original Robinson geometric-mean and |
| 72 |
Robinson-Fisher algorithm code for bogofilter. To determine the |
| 73 |
optimal parameters for the Robinson-Fisher algorithm he wrote |
| 74 |
bogotune. The initial implementation was written in the R |
| 75 |
programming language. This was followed by the Perl |
| 76 |
implementation. Both of these implementations were slow because |
| 77 |
bogofilter had to be run for each message being scored. David |
| 78 |
Relson translated bogotune from Perl to C to provide more |
| 79 |
speed.</p> |
| 80 |
|
| 81 |
<hr> |
| 82 |
<h2 id="message-count">What's the message count format?</h2> |
| 83 |
|
| 84 |
<p>The parsing of a message by bogofilter takes some time. After |
| 85 |
parsing, finding the spam and non-spam counts for each token takes |
| 86 |
additional time. Having to repeate these steps every time |
| 87 |
bogotune needed a score was slow. It was realized that parsing |
| 88 |
and look-up could be done once with the results being saved in a |
| 89 |
special format. Initially this was called the bogolex format |
| 90 |
because the work was done by piping bogolexer output to bogoutil |
| 91 |
and formatting the result. Since each processed message begins |
| 92 |
with the .MSG_COUNT token the format became knowns as the message |
| 93 |
count format. The convention is to use a .mc extension for these |
| 94 |
files.</p> |
| 95 |
|
| 96 |
<hr> |
| 97 |
<h2 id="work">How does bogotune work?</h2> |
| 98 |
|
| 99 |
<p>First it reads all the files into memory, i.e. the wordlist and |
| 100 |
the ham messages and the spam messages. From the wordlist tokens, |
| 101 |
it computes an initial robx value which is used in the initial |
| 102 |
scan of the messages to ensure they're usable.</p> |
| 103 |
|
| 104 |
<p>Given the total number of messages in the test set, a target |
| 105 |
number of false positives is selected for use in determining spam |
| 106 |
cutoff values in the individual scans.</p> |
| 107 |
|
| 108 |
<p>Then comes the coarse scan. Using 225 combinations of values |
| 109 |
chosen to span the potentially useful ranges for robs, robx, and |
| 110 |
min_dev, all the ham messages are scored and the target value is |
| 111 |
used to find a spam_cutoff score. Then the spam messages are |
| 112 |
scored and the false negatives are counted. The scan finishes |
| 113 |
with a listing of the ten best sets of parameters and their scores |
| 114 |
(false negative and false positive counts and percent).</p> |
| 115 |
|
| 116 |
<p>From the results, the best non-outlying result is picked and |
| 117 |
these parameters become the starting point for the fine scan.</p> |
| 118 |
|
| 119 |
<p>The fine scan, as the name suggests, scans the region (range of |
| 120 |
values of robs, robx and min_dev) surrounding the optimum found in |
| 121 |
the coarse scan, with smaller intervals so as to determine the |
| 122 |
optimum values more precisely. </p> |
| 123 |
|
| 124 |
<hr> |
| 125 |
|
| 126 |
<h2 id="ensure-count">How does bogotune ensure the messages it works with |
| 127 |
are numerous enough, and well enough classified, to deliver useful |
| 128 |
recommendations?</h2> |
| 129 |
|
| 130 |
<p>It has certain minimum requirements that it checks for as it |
| 131 |
starts up. It will complain (and halt) if there are fewer than |
| 132 |
2,000 ham or 2,000 spam in the wordlist, or if there are fewer |
| 133 |
than 500 ham or 500 spam in the set of test messages. It will |
| 134 |
warn, but not halt, if there's too little scoring variation in the |
| 135 |
ham messages or the spam messages or if too many of the ham |
| 136 |
messages score as spam (or vice versa) on the initial pass. There |
| 137 |
are additional checks, but I'm sure you get the idea from these |
| 138 |
examples. For details, use the source :)</p> |
| 139 |
|
| 140 |
<hr> |
| 141 |
<h2 id="force">Can I tell bogotune to do its work even though it |
| 142 |
doesn't like the data?</h2> |
| 143 |
|
| 144 |
<p>No. At one time we had a -F option to force bogotune to run |
| 145 |
with unsuitable message data, but it was realized that this could |
| 146 |
be misleading and had little chance of being helpful. Bogotune |
| 147 |
will warn the operator if its conclusions are untrustworthy due to |
| 148 |
marginal input, and will not run if its input data are detectably |
| 149 |
inadequate.</p> |
| 150 |
|
| 151 |
<hr> |
| 152 |
</body> |
| 153 |
</html> |