Debian Codegraph @

src/backend/snowball/README

raw
1 src/backend/snowball/README
2
3 Snowball-Based Stemming
4 =======================
5
6 This module uses the word stemming code developed by the Snowball project,
7 http://snowballstem.org (formerly http://snowball.tartarus.org)
8 which is released by them under a BSD-style license.
9
10 The Snowball project does not often make formal releases; it's best
11 to pull from their git repository
12
13 git clone https://github.com/snowballstem/snowball.git
14
15 and then building the derived files is as simple as
16
17 cd snowball
18 make
19
20 At least on Linux, no platform-specific adjustment is needed.
21
22 Postgres' files under src/backend/snowball/libstemmer/ and
23 src/include/snowball/libstemmer/ are taken directly from the Snowball
24 files, with only some minor adjustments of file inclusions. Note
25 that most of these files are in fact derived files, not original source.
26 The original sources are in the Snowball language, and are built using
27 the Snowball-to-C compiler that is also part of the Snowball project.
28 We choose to include the derived files in the PostgreSQL distribution
29 because most installations will not have the Snowball compiler available.
30
31 We are currently synced with the Snowball git commit
32 d19326ac6c1b9a417fc872f7c2f845265a5e9ece
33 of 2025-02-19.
34
35 To update the PostgreSQL sources from a new Snowball version:
36
37 0. If you didn't do it already, "make -C snowball".
38
39 1. Copy the *.c files in snowball/src_c/ to src/backend/snowball/libstemmer
40 with replacement of "../runtime/header.h" by "header.h", for example
41
42 for f in .../snowball/src_c/*.c
43 do
44 sed 's|\.\./runtime/header\.h|header.h|' $f >libstemmer/`basename $f`
45 done
46
47 Do not copy stemmers that are listed in their libstemmer/modules.txt as
48 nonstandard, such as "kraaij_pohlmann" or "lovins".
49
50 2. Copy the *.c files in snowball/runtime/ to
51 src/backend/snowball/libstemmer, and edit them to remove direct inclusions
52 of system headers such as <stdio.h> --- they should only include "header.h".
53 (This removal avoids portability problems on some platforms where <stdio.h>
54 is sensitive to largefile compilation options.)
55
56 3. Copy the *.h files in snowball/src_c/ and snowball/runtime/
57 to src/include/snowball/libstemmer. At this writing the header files
58 do not require any changes. Again, omit the *.h files for nonstandard
59 stemmers.
60
61 4. Check whether any stemmer modules have been added or removed. If so, edit
62 the OBJS list in Makefile, the dict_snowball_sources list in meson.build,
63 the list of #include's and the stemmer_modules[] table in dict_snowball.c,
64 and the sample \dFd output in the documentation in textsearch.sgml.
65 You might also need to change the @languages array in snowball_create.pl
66 and the tsearch_config_languages[] table in initdb.c.
67
68 5. The various stopword files in stopwords/ must be downloaded
69 individually from pages on the snowballstem.org website.
70 Be careful that these files must be stored in UTF-8 encoding.
71 Update the stop_files list in Makefile if any are added or removed
72 (the meson tooling does not require adjustment for that, though).