| 1 |
NAME |
| 2 |
|
| 3 |
Lingua::StopWords - Stop words for several languages. |
| 4 |
|
| 5 |
SYNOPSIS |
| 6 |
|
| 7 |
use Lingua::StopWords qw( getStopWords ); |
| 8 |
my $stopwords = getStopWords('en'); |
| 9 |
|
| 10 |
my @words = qw( i am the walrus goo goo g'joob ); |
| 11 |
|
| 12 |
# prints "walrus goo goo g'joob" |
| 13 |
print join ' ', grep { !$stopwords->{$_} } @words; |
| 14 |
|
| 15 |
DESCRIPTION |
| 16 |
|
| 17 |
In keyword search, it is common practice to suppress a collection of |
| 18 |
"stopwords": words such as "the", "and", "maybe", etc. which exist in |
| 19 |
in a large number of documents and do not tell you anything important |
| 20 |
about any document which contains them. This module provides such |
| 21 |
"stoplists" in several languages. |
| 22 |
|
| 23 |
Supported Languages |
| 24 |
|
| 25 |
|-----------------------------------------------------------| |
| 26 |
| Language | ISO code | default encoding | also available | |
| 27 |
|-----------------------------------------------------------| |
| 28 |
| Danish | da | ISO-8859-1 | UTF-8 | |
| 29 |
| Dutch | nl | ISO-8859-1 | UTF-8 | |
| 30 |
| English | en | ISO-8859-1 | UTF-8 | |
| 31 |
| Finnish | fi | ISO-8859-1 | UTF-8 | |
| 32 |
| French | fr | ISO-8859-1 | UTF-8 | |
| 33 |
| German | de | ISO-8859-1 | UTF-8 | |
| 34 |
| Hungarian | hu | ISO-8859-2 | UTF-8 | |
| 35 |
| Indonesian | id | ISO-8859-1 | UTF-8 | |
| 36 |
| Italian | it | ISO-8859-1 | UTF-8 | |
| 37 |
| Norwegian | no | ISO-8859-1 | UTF-8 | |
| 38 |
| Portuguese | pt | ISO-8859-1 | UTF-8 | |
| 39 |
| Romanian | ro | ISO-8859-2 | UTF-8 | |
| 40 |
| Spanish | es | ISO-8859-1 | UTF-8 | |
| 41 |
| Swedish | sv | ISO-8859-1 | UTF-8 | |
| 42 |
| Russian | ru | KOI8-R | UTF-8 | |
| 43 |
|-----------------------------------------------------------| |
| 44 |
|
| 45 |
FUNCTIONS |
| 46 |
|
| 47 |
getStopWords |
| 48 |
|
| 49 |
my $stoplist = getStopWords('en'); |
| 50 |
my $utf8_stoplist = getStopWords('en', 'UTF-8'); |
| 51 |
|
| 52 |
Retrieve a stoplist in the form of a hashref where the keys are all |
| 53 |
stopwords and the values are all 1. |
| 54 |
|
| 55 |
$stoplist = { |
| 56 |
and => 1, |
| 57 |
if => 1, |
| 58 |
# ... |
| 59 |
}; |
| 60 |
|
| 61 |
getStopWords() expects 1-2 arguments. The first, which is required, is |
| 62 |
an ISO code representing a supported language. If the ISO code cannot |
| 63 |
be found, getStopWords returns undef. |
| 64 |
|
| 65 |
The second argument should be 'UTF-8' if you want the stopwords encoded |
| 66 |
in UTF-8. The UTF-8 flag will be turned on, so make sure you understand |
| 67 |
all the implications of that. |
| 68 |
|
| 69 |
INSTALLATION |
| 70 |
|
| 71 |
To install this module type the following: |
| 72 |
|
| 73 |
perl Build.PL |
| 74 |
./Build |
| 75 |
./Build test |
| 76 |
./Build install |
| 77 |
|
| 78 |
SEE ALSO |
| 79 |
|
| 80 |
The stoplists supplied by this module were created as part of the |
| 81 |
Snowball project (see http://snowball.tartarus.org, |
| 82 |
Lingua::Stem::Snowball). |
| 83 |
|
| 84 |
Lingua::EN::StopWords provides a different stoplist for English. |
| 85 |
|
| 86 |
SOURCE REPOSITORY |
| 87 |
|
| 88 |
https://github.com/wollmers/Lingua-StopWords |
| 89 |
|
| 90 |
AUTHOR |
| 91 |
|
| 92 |
Maintained by Helmut Wollmersdorfer <[email protected]> and |
| 93 |
Marvin Humphrey <marvin at rectangular dot com>. Original author Fabien |
| 94 |
Potencier, <fabpot at cpan dot org>. |
| 95 |
|
| 96 |
COPYRIGHT |
| 97 |
|
| 98 |
Copyright 2021 Helmut Wollmersdorfer Copyright 2004-2008 Fabien |
| 99 |
Potencier, Marvin Humphrey |
| 100 |
|
| 101 |
LICENSE |
| 102 |
|
| 103 |
This library is free software; you can redistribute it and/or modify it |
| 104 |
under the same terms as Perl itself, either Perl version 5.8.3 or, at |
| 105 |
your option, any later version of Perl 5 you may have available. |
| 106 |
|