| 1 |
|
| 2 |
1. FTS3 Tokenizers |
| 3 |
|
| 4 |
When creating a new full-text table, FTS3 allows the user to select |
| 5 |
the text tokenizer implementation to be used when indexing text |
| 6 |
by specifying a "tokenize" clause as part of the CREATE VIRTUAL TABLE |
| 7 |
statement: |
| 8 |
|
| 9 |
CREATE VIRTUAL TABLE <table-name> USING fts3( |
| 10 |
<columns ...> [, tokenize <tokenizer-name> [<tokenizer-args>]] |
| 11 |
); |
| 12 |
|
| 13 |
The built-in tokenizers (valid values to pass as <tokenizer name>) are |
| 14 |
"simple", "porter" and "unicode". |
| 15 |
|
| 16 |
<tokenizer-args> should consist of zero or more white-space separated |
| 17 |
arguments to pass to the selected tokenizer implementation. The |
| 18 |
interpretation of the arguments, if any, depends on the individual |
| 19 |
tokenizer. |
| 20 |
|
| 21 |
2. Custom Tokenizers |
| 22 |
|
| 23 |
FTS3 allows users to provide custom tokenizer implementations. The |
| 24 |
interface used to create a new tokenizer is defined and described in |
| 25 |
the fts3_tokenizer.h source file. |
| 26 |
|
| 27 |
Registering a new FTS3 tokenizer is similar to registering a new |
| 28 |
virtual table module with SQLite. The user passes a pointer to a |
| 29 |
structure containing pointers to various callback functions that |
| 30 |
make up the implementation of the new tokenizer type. For tokenizers, |
| 31 |
the structure (defined in fts3_tokenizer.h) is called |
| 32 |
"sqlite3_tokenizer_module". |
| 33 |
|
| 34 |
FTS3 does not expose a C-function that users call to register new |
| 35 |
tokenizer types with a database handle. Instead, the pointer must |
| 36 |
be encoded as an SQL blob value and passed to FTS3 through the SQL |
| 37 |
engine by evaluating a special scalar function, "fts3_tokenizer()". |
| 38 |
The fts3_tokenizer() function may be called with one or two arguments, |
| 39 |
as follows: |
| 40 |
|
| 41 |
SELECT fts3_tokenizer(<tokenizer-name>); |
| 42 |
SELECT fts3_tokenizer(<tokenizer-name>, <sqlite3_tokenizer_module ptr>); |
| 43 |
|
| 44 |
Where <tokenizer-name> is a string identifying the tokenizer and |
| 45 |
<sqlite3_tokenizer_module ptr> is a pointer to an sqlite3_tokenizer_module |
| 46 |
structure encoded as an SQL blob. If the second argument is present, |
| 47 |
it is registered as tokenizer <tokenizer-name> and a copy of it |
| 48 |
returned. If only one argument is passed, a pointer to the tokenizer |
| 49 |
implementation currently registered as <tokenizer-name> is returned, |
| 50 |
encoded as a blob. Or, if no such tokenizer exists, an SQL exception |
| 51 |
(error) is raised. |
| 52 |
|
| 53 |
SECURITY: If the fts3 extension is used in an environment where potentially |
| 54 |
malicious users may execute arbitrary SQL (i.e. gears), they should be |
| 55 |
prevented from invoking the fts3_tokenizer() function. The |
| 56 |
fts3_tokenizer() function is disabled by default. It is only enabled |
| 57 |
by SQLITE_DBCONFIG_ENABLE_FTS3_TOKENIZER. Do not enable it in |
| 58 |
security sensitive environments. |
| 59 |
|
| 60 |
See "Sample code" below for an example of calling the fts3_tokenizer() |
| 61 |
function from C code. |
| 62 |
|
| 63 |
3. ICU Library Tokenizers |
| 64 |
|
| 65 |
If this extension is compiled with the SQLITE_ENABLE_ICU pre-processor |
| 66 |
symbol defined, then there exists a built-in tokenizer named "icu" |
| 67 |
implemented using the ICU library. The first argument passed to the |
| 68 |
xCreate() method (see fts3_tokenizer.h) of this tokenizer may be |
| 69 |
an ICU locale identifier. For example "tr_TR" for Turkish as used |
| 70 |
in Turkey, or "en_AU" for English as used in Australia. For example: |
| 71 |
|
| 72 |
"CREATE VIRTUAL TABLE thai_text USING fts3(text, tokenizer icu th_TH)" |
| 73 |
|
| 74 |
The ICU tokenizer implementation is very simple. It splits the input |
| 75 |
text according to the ICU rules for finding word boundaries and discards |
| 76 |
any tokens that consist entirely of white-space. This may be suitable |
| 77 |
for some applications in some locales, but not all. If more complex |
| 78 |
processing is required, for example to implement stemming or |
| 79 |
discard punctuation, this can be done by creating a tokenizer |
| 80 |
implementation that uses the ICU tokenizer as part of its implementation. |
| 81 |
|
| 82 |
When using the ICU tokenizer this way, it is safe to overwrite the |
| 83 |
contents of the strings returned by the xNext() method (see |
| 84 |
fts3_tokenizer.h). |
| 85 |
|
| 86 |
4. Sample code. |
| 87 |
|
| 88 |
The following two code samples illustrate the way C code should invoke |
| 89 |
the fts3_tokenizer() scalar function: |
| 90 |
|
| 91 |
int registerTokenizer( |
| 92 |
sqlite3 *db, |
| 93 |
char *zName, |
| 94 |
const sqlite3_tokenizer_module *p |
| 95 |
){ |
| 96 |
int rc; |
| 97 |
sqlite3_stmt *pStmt; |
| 98 |
const char zSql[] = "SELECT fts3_tokenizer(?, ?)"; |
| 99 |
|
| 100 |
rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0); |
| 101 |
if( rc!=SQLITE_OK ){ |
| 102 |
return rc; |
| 103 |
} |
| 104 |
|
| 105 |
sqlite3_bind_text(pStmt, 1, zName, -1, SQLITE_STATIC); |
| 106 |
sqlite3_bind_blob(pStmt, 2, &p, sizeof(p), SQLITE_STATIC); |
| 107 |
sqlite3_step(pStmt); |
| 108 |
|
| 109 |
return sqlite3_finalize(pStmt); |
| 110 |
} |
| 111 |
|
| 112 |
int queryTokenizer( |
| 113 |
sqlite3 *db, |
| 114 |
char *zName, |
| 115 |
const sqlite3_tokenizer_module **pp |
| 116 |
){ |
| 117 |
int rc; |
| 118 |
sqlite3_stmt *pStmt; |
| 119 |
const char zSql[] = "SELECT fts3_tokenizer(?)"; |
| 120 |
|
| 121 |
*pp = 0; |
| 122 |
rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, 0); |
| 123 |
if( rc!=SQLITE_OK ){ |
| 124 |
return rc; |
| 125 |
} |
| 126 |
|
| 127 |
sqlite3_bind_text(pStmt, 1, zName, -1, SQLITE_STATIC); |
| 128 |
if( SQLITE_ROW==sqlite3_step(pStmt) ){ |
| 129 |
if( sqlite3_column_type(pStmt, 0)==SQLITE_BLOB ){ |
| 130 |
memcpy(pp, sqlite3_column_blob(pStmt, 0), sizeof(*pp)); |
| 131 |
} |
| 132 |
} |
| 133 |
|
| 134 |
return sqlite3_finalize(pStmt); |
| 135 |
} |