Debian Codegraph @

TODO

raw
1 TODO
2
3 0.
4 Explore use of Berkeley DB as faster alternative to MySQL.
5 It is likely that Berkeley DB will give a performance
6 increase over MySQL
7
8 1.
9 implement searching by substrings, by creating a lookup table
10 for _words with three letter combinations. this lookup table
11 will have two fields - word_id and substring (three letters)
12
13 for example, a search for 'bix::' would require a call to
14 $fts->contains('%bix::%') which would not utilize the index
15 in the _words table because of the first wildcard character '%'
16 This query, by the way, on the CPAN subsite should return
17 "DBIx::FullTextSearch".
18
19 the user would turn on this option at table create time by
20 setting 'use_substring' to true. the substring table can be
21 specified by setting 'substring_table', defaults to the name
22 of the index with the _substring suffix.
23
24 2.
25 implement the following Scoring Algorithm:
26
27 for a word in a doc
28
29 (occurances in doc) num docs
30 ------------------------------- * log ( ------------------ )
31 (max occur in doc of all words) num docs with word
32
33 to implement, note that you will have to store the maximum occurance
34 of a word for every doc in the *_docid table. Note that for the
35 default (and in some cases the table) frontend, you will have to
36 make sure that the *_docid table is created.
37
38 3.
39 implement Thesaurus (using Linuga::Wordnet???)
40
41 4.
42 extend the documentation to show how you can use your own
43 splitter and filter to strip down the HTML markup;
44
45 5.
46 extend the backends so that one DBIx::FullTextSearch index can store
47 more views of the documents -- for example for indexing
48 mailinglist archives, it'd be nice to index from,
49 subject and body of the messages separately yet in one
50 index, so that you could base the selection critaria on
51 what part of the documents (headers or body in this
52 case) contains the pattern; for HTML document, indexing
53 titles and body in this way seems natural requirement.
54
55 6.
56 make benchmarks showing the performance on various data sets and
57 various document types and lengths.