Debian Codegraph @

doc/bogotune-faq.html

raw
1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
2 <html>
3 <head>
4 <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
5 <title>Bogotune FAQ</title>
6 <style type="text/css">
7 h2 {
8 margin-top: 1em;
9 font-size: 125%;
10 }
11 h3 {
12 margin-top: 1em;
13 font-size: 110%;
14 }
15 p {
16 margin-top : 0.5em;
17 margin-bottom: 0.5em;
18 }
19 ul {
20 margin-top: 1.5em;
21 margin-bottom: 0.5em;
22 }
23 ul ul {
24 margin-top: 0.25em;
25 margin-bottom: 0;
26 }
27 li {
28 margin-top: 0;
29 margin-bottom: 1em;
30 }
31 li li {
32 margin-bottom: 0.25em;
33 }
34 dt {
35 margin-top: 0.5em;
36 margin-bottom: 0;
37 }
38 hr {
39 margin-top: 1em;
40 margin-bottom: 1em;
41 }
42 </style>
43 </head>
44
45 <body>
46 <h1>Bogotune FAQ</h1>
47
48 <p>Official Versions: In
49 <a href="http://bogofilter.sourceforge.net/-faq.html">bogotune-faq</a><br>
50 Maintainer: David Relson &lt;[email protected]&gt;</p>
51
52 <p>This document is intended to answer frequently asked questions
53 about bogotune.</p>
54
55 <ul>
56 <li><a href="#origin"> Where did bogotune come from?</a></li>
57 <li><a href="#message-count"> What's the message count format?</a></li>
58 <li><a href="#work"> How does bogotune work?</a></li>
59
60 <li><a href="#ensure-count"> How does bogotune ensure the messages it
61 works with are numerous enough, and well enough classified, to
62 deliver useful recommendations?</a></li>
63
64 <li><a href="#force"> Can I tell bogotune to do its work even
65 though it doesn't like the data?</a></li>
66 </ul>
67
68 <hr>
69 <h2 id="origin">Where did bogotune come from?</h2>
70
71 <p>Greg Louis wrote the original Robinson geometric-mean and
72 Robinson-Fisher algorithm code for bogofilter. To determine the
73 optimal parameters for the Robinson-Fisher algorithm he wrote
74 bogotune. The initial implementation was written in the R
75 programming language. This was followed by the Perl
76 implementation. Both of these implementations were slow because
77 bogofilter had to be run for each message being scored. David
78 Relson translated bogotune from Perl to C to provide more
79 speed.</p>
80
81 <hr>
82 <h2 id="message-count">What's the message count format?</h2>
83
84 <p>The parsing of a message by bogofilter takes some time. After
85 parsing, finding the spam and non-spam counts for each token takes
86 additional time. Having to repeate these steps every time
87 bogotune needed a score was slow. It was realized that parsing
88 and look-up could be done once with the results being saved in a
89 special format. Initially this was called the bogolex format
90 because the work was done by piping bogolexer output to bogoutil
91 and formatting the result. Since each processed message begins
92 with the .MSG_COUNT token the format became knowns as the message
93 count format. The convention is to use a .mc extension for these
94 files.</p>
95
96 <hr>
97 <h2 id="work">How does bogotune work?</h2>
98
99 <p>First it reads all the files into memory, i.e. the wordlist and
100 the ham messages and the spam messages. From the wordlist tokens,
101 it computes an initial robx value which is used in the initial
102 scan of the messages to ensure they're usable.</p>
103
104 <p>Given the total number of messages in the test set, a target
105 number of false positives is selected for use in determining spam
106 cutoff values in the individual scans.</p>
107
108 <p>Then comes the coarse scan. Using 225 combinations of values
109 chosen to span the potentially useful ranges for robs, robx, and
110 min_dev, all the ham messages are scored and the target value is
111 used to find a spam_cutoff score. Then the spam messages are
112 scored and the false negatives are counted. The scan finishes
113 with a listing of the ten best sets of parameters and their scores
114 (false negative and false positive counts and percent).</p>
115
116 <p>From the results, the best non-outlying result is picked and
117 these parameters become the starting point for the fine scan.</p>
118
119 <p>The fine scan, as the name suggests, scans the region (range of
120 values of robs, robx and min_dev) surrounding the optimum found in
121 the coarse scan, with smaller intervals so as to determine the
122 optimum values more precisely. </p>
123
124 <hr>
125
126 <h2 id="ensure-count">How does bogotune ensure the messages it works with
127 are numerous enough, and well enough classified, to deliver useful
128 recommendations?</h2>
129
130 <p>It has certain minimum requirements that it checks for as it
131 starts up. It will complain (and halt) if there are fewer than
132 2,000 ham or 2,000 spam in the wordlist, or if there are fewer
133 than 500 ham or 500 spam in the set of test messages. It will
134 warn, but not halt, if there's too little scoring variation in the
135 ham messages or the spam messages or if too many of the ham
136 messages score as spam (or vice versa) on the initial pass. There
137 are additional checks, but I'm sure you get the idea from these
138 examples. For details, use the source :)</p>
139
140 <hr>
141 <h2 id="force">Can I tell bogotune to do its work even though it
142 doesn't like the data?</h2>
143
144 <p>No. At one time we had a -F option to force bogotune to run
145 with unsuitable message data, but it was realized that this could
146 be misleading and had little chance of being helpful. Bogotune
147 will warn the operator if its conclusions are untrustworthy due to
148 marginal input, and will not run if its input data are detectably
149 inadequate.</p>
150
151 <hr>
152 </body>
153 </html>