| 1 |
src/backend/snowball/README |
| 2 |
|
| 3 |
Snowball-Based Stemming |
| 4 |
======================= |
| 5 |
|
| 6 |
This module uses the word stemming code developed by the Snowball project, |
| 7 |
http://snowballstem.org (formerly http://snowball.tartarus.org) |
| 8 |
which is released by them under a BSD-style license. |
| 9 |
|
| 10 |
The Snowball project does not often make formal releases; it's best |
| 11 |
to pull from their git repository |
| 12 |
|
| 13 |
git clone https://github.com/snowballstem/snowball.git |
| 14 |
|
| 15 |
and then building the derived files is as simple as |
| 16 |
|
| 17 |
cd snowball |
| 18 |
make |
| 19 |
|
| 20 |
At least on Linux, no platform-specific adjustment is needed. |
| 21 |
|
| 22 |
Postgres' files under src/backend/snowball/libstemmer/ and |
| 23 |
src/include/snowball/libstemmer/ are taken directly from the Snowball |
| 24 |
files, with only some minor adjustments of file inclusions. Note |
| 25 |
that most of these files are in fact derived files, not original source. |
| 26 |
The original sources are in the Snowball language, and are built using |
| 27 |
the Snowball-to-C compiler that is also part of the Snowball project. |
| 28 |
We choose to include the derived files in the PostgreSQL distribution |
| 29 |
because most installations will not have the Snowball compiler available. |
| 30 |
|
| 31 |
We are currently synced with the Snowball git commit |
| 32 |
d19326ac6c1b9a417fc872f7c2f845265a5e9ece |
| 33 |
of 2025-02-19. |
| 34 |
|
| 35 |
To update the PostgreSQL sources from a new Snowball version: |
| 36 |
|
| 37 |
0. If you didn't do it already, "make -C snowball". |
| 38 |
|
| 39 |
1. Copy the *.c files in snowball/src_c/ to src/backend/snowball/libstemmer |
| 40 |
with replacement of "../runtime/header.h" by "header.h", for example |
| 41 |
|
| 42 |
for f in .../snowball/src_c/*.c |
| 43 |
do |
| 44 |
sed 's|\.\./runtime/header\.h|header.h|' $f >libstemmer/`basename $f` |
| 45 |
done |
| 46 |
|
| 47 |
Do not copy stemmers that are listed in their libstemmer/modules.txt as |
| 48 |
nonstandard, such as "kraaij_pohlmann" or "lovins". |
| 49 |
|
| 50 |
2. Copy the *.c files in snowball/runtime/ to |
| 51 |
src/backend/snowball/libstemmer, and edit them to remove direct inclusions |
| 52 |
of system headers such as <stdio.h> --- they should only include "header.h". |
| 53 |
(This removal avoids portability problems on some platforms where <stdio.h> |
| 54 |
is sensitive to largefile compilation options.) |
| 55 |
|
| 56 |
3. Copy the *.h files in snowball/src_c/ and snowball/runtime/ |
| 57 |
to src/include/snowball/libstemmer. At this writing the header files |
| 58 |
do not require any changes. Again, omit the *.h files for nonstandard |
| 59 |
stemmers. |
| 60 |
|
| 61 |
4. Check whether any stemmer modules have been added or removed. If so, edit |
| 62 |
the OBJS list in Makefile, the dict_snowball_sources list in meson.build, |
| 63 |
the list of #include's and the stemmer_modules[] table in dict_snowball.c, |
| 64 |
and the sample \dFd output in the documentation in textsearch.sgml. |
| 65 |
You might also need to change the @languages array in snowball_create.pl |
| 66 |
and the tsearch_config_languages[] table in initdb.c. |
| 67 |
|
| 68 |
5. The various stopword files in stopwords/ must be downloaded |
| 69 |
individually from pages on the snowballstem.org website. |
| 70 |
Be careful that these files must be stored in UTF-8 encoding. |
| 71 |
Update the stop_files list in Makefile if any are added or removed |
| 72 |
(the meson tooling does not require adjustment for that, though). |