| 1 |
TODO |
| 2 |
|
| 3 |
0. |
| 4 |
Explore use of Berkeley DB as faster alternative to MySQL. |
| 5 |
It is likely that Berkeley DB will give a performance |
| 6 |
increase over MySQL |
| 7 |
|
| 8 |
1. |
| 9 |
implement searching by substrings, by creating a lookup table |
| 10 |
for _words with three letter combinations. this lookup table |
| 11 |
will have two fields - word_id and substring (three letters) |
| 12 |
|
| 13 |
for example, a search for 'bix::' would require a call to |
| 14 |
$fts->contains('%bix::%') which would not utilize the index |
| 15 |
in the _words table because of the first wildcard character '%' |
| 16 |
This query, by the way, on the CPAN subsite should return |
| 17 |
"DBIx::FullTextSearch". |
| 18 |
|
| 19 |
the user would turn on this option at table create time by |
| 20 |
setting 'use_substring' to true. the substring table can be |
| 21 |
specified by setting 'substring_table', defaults to the name |
| 22 |
of the index with the _substring suffix. |
| 23 |
|
| 24 |
2. |
| 25 |
implement the following Scoring Algorithm: |
| 26 |
|
| 27 |
for a word in a doc |
| 28 |
|
| 29 |
(occurances in doc) num docs |
| 30 |
------------------------------- * log ( ------------------ ) |
| 31 |
(max occur in doc of all words) num docs with word |
| 32 |
|
| 33 |
to implement, note that you will have to store the maximum occurance |
| 34 |
of a word for every doc in the *_docid table. Note that for the |
| 35 |
default (and in some cases the table) frontend, you will have to |
| 36 |
make sure that the *_docid table is created. |
| 37 |
|
| 38 |
3. |
| 39 |
implement Thesaurus (using Linuga::Wordnet???) |
| 40 |
|
| 41 |
4. |
| 42 |
extend the documentation to show how you can use your own |
| 43 |
splitter and filter to strip down the HTML markup; |
| 44 |
|
| 45 |
5. |
| 46 |
extend the backends so that one DBIx::FullTextSearch index can store |
| 47 |
more views of the documents -- for example for indexing |
| 48 |
mailinglist archives, it'd be nice to index from, |
| 49 |
subject and body of the messages separately yet in one |
| 50 |
index, so that you could base the selection critaria on |
| 51 |
what part of the documents (headers or body in this |
| 52 |
case) contains the pattern; for HTML document, indexing |
| 53 |
titles and body in this way seems natural requirement. |
| 54 |
|
| 55 |
6. |
| 56 |
make benchmarks showing the performance on various data sets and |
| 57 |
various document types and lengths. |