| 1 |
Open Text summarizer |
| 2 |
--------------------- |
| 3 |
Nadav Rotem 2003 |
| 4 |
[email protected] |
| 5 |
--------------------- |
| 6 |
|
| 7 |
The open text summarizer is an open source tool for summarizing texts. The program |
| 8 |
reads a text and decides which sentences are important and which are not. OTS will |
| 9 |
create a short summary or will highlight the main ideas in the text. |
| 10 |
|
| 11 |
OTS is both a library and a command line tool. Word processors such as AbiWord |
| 12 |
and KWord can link to the library and summarize documents while the command line tool |
| 13 |
lets you summarize text on the console. The program can either print the summarized |
| 14 |
text as text or HTML. If in HTML, the important sentences are highlighted. |
| 15 |
The program is multi lingual and works with UTF-8 encoding. |
| 16 |
|
| 17 |
Supported languages: |
| 18 |
(ca,bg,cs,cy,da,de,el,en,eo,es,et,eu,fi,fr,ga,gl,he,hu,ia,id,is,it,lv,mi,ms,mt,nl,nn, |
| 19 |
pl,pt,ro,ru,sv,tl,tr,uk,yi) |
| 20 |
|
| 21 |
Basque (seminal) |
| 22 |
Bulgarian (seminal) |
| 23 |
Catalan (seminal) |
| 24 |
Czech (seminal) |
| 25 |
Danish (seminal) |
| 26 |
Dutch |
| 27 |
English |
| 28 |
Esperanto (seminal) |
| 29 |
Estonian (seminal) |
| 30 |
Finnish (seminal) |
| 31 |
French (seminal) |
| 32 |
Galician (seminal) |
| 33 |
German (seminal) |
| 34 |
Greek (seminal) |
| 35 |
Hebrew |
| 36 |
Hungarian (seminal) |
| 37 |
Icelandic (seminal) |
| 38 |
Indonesian (seminal) |
| 39 |
Interlingua (seminal) |
| 40 |
Irish (Gaelic) (seminal) |
| 41 |
Italian (seminal) |
| 42 |
Latvian (seminal) |
| 43 |
Malaysian (seminal) |
| 44 |
Maltese (seminal) |
| 45 |
Maori (seminal) |
| 46 |
Norwegian Nynorsk (seminal) |
| 47 |
Polish (seminal) |
| 48 |
Portuguese (seminal) |
| 49 |
Romanian (seminal) |
| 50 |
Russian (seminal) |
| 51 |
Spanish (seminal) |
| 52 |
Swedish (seminal) |
| 53 |
Tagalog (Pilipino) (seminal) |
| 54 |
Turkish (seminal) |
| 55 |
Ukrainian (seminal) |
| 56 |
Welsh (seminal) |
| 57 |
Yiddish (seminal) |
| 58 |
|
| 59 |
Command line Usage: |
| 60 |
|
| 61 |
$ ots |
| 62 |
--ratio [0..100] :select the summary ratio |
| 63 |
--html :print summary in highlighted html form |
| 64 |
--keywords :print key words in the article |
| 65 |
--dic :use dictionary other then english, for example 'he' or 'es' |
| 66 |
--about :tells that the article talks about |
| 67 |
--file :input file |
| 68 |
--out :output file |
| 69 |
--help :print this help screen |
| 70 |
|
| 71 |
example1: |
| 72 |
ots articles/sacbee1.txt --ratio 20 --html |
| 73 |
|
| 74 |
This command will summarize the article from the Sacramento Bee and highlight the 20% of |
| 75 |
sentences most important to the content of the article. |
| 76 |
|
| 77 |
example2: |
| 78 |
ots --ratio 40 --file big_article.txt |
| 79 |
|
| 80 |
This command will summarize the article down to 40% and print it. |
| 81 |
|
| 82 |
|
| 83 |
How it works: |
| 84 |
|
| 85 |
The Idea behind OTS is that the important ideas in an article are described with |
| 86 |
many of the same words while redundant information uses less technical terms |
| 87 |
and is not related to the main subject of the article. Important lines are lines |
| 88 |
that are related to the subject of the article. The subject of the article is the |
| 89 |
list of ideas that are most discussed in the article. |
| 90 |
In practice , the article is scanned once and a list of all of the words and their |
| 91 |
occurrence in the text is stored in a list. After sorting this list by the |
| 92 |
occurrence of words in the text we will see a list similar to this: |
| 93 |
|
| 94 |
|
| 95 |
11 are |
| 96 |
17 is |
| 97 |
16 a |
| 98 |
14 Harry |
| 99 |
14 on |
| 100 |
13 Sally |
| 101 |
11 Love |
| 102 |
11 such |
| 103 |
4 an |
| 104 |
2 taxi |
| 105 |
1 university |
| 106 |
1 chicago |
| 107 |
1 meets |
| 108 |
... |
| 109 |
... |
| 110 |
|
| 111 |
Now we will remove all the words that are common in the English language using a |
| 112 |
dictionary file. Example of such words: "The","a","since","after","will". |
| 113 |
These words are words that don't teach us anything about the subject of the article. |
| 114 |
Knowing that the word "Police" is in a given text can tell us that the article might |
| 115 |
talk about the police , however knowing that the word "will" is in the text can't |
| 116 |
teach us anything about the subject of the text. |
| 117 |
|
| 118 |
After removing the redundant words the new list will look like this: |
| 119 |
|
| 120 |
14 Harry |
| 121 |
13 Sally |
| 122 |
11 Love |
| 123 |
2 taxi |
| 124 |
1 university |
| 125 |
1 chicago |
| 126 |
1 meets |
| 127 |
|
| 128 |
From this list we may assume that the text talks about "Harry , Sally , Love". |
| 129 |
So an important sentence in the text will be a sentence that talks about Harry, |
| 130 |
Sally and Love. A sentence that talks about the university of Chicago may be |
| 131 |
neglected because we find the word "chicago" only once in the text, so we may assume |
| 132 |
that "chicago" is not one of the main ideas in the text. Each sentence is given |
| 133 |
a grade based on the the keywords in it. A line that holds many important words will |
| 134 |
be given a high grade. To produce a 20% summary we print the top 20% sentences with |
| 135 |
the highest grade. |
| 136 |
|
| 137 |
Ots will not work on books because they cover too many topics. however, each |
| 138 |
page may be fed individually, this will produce a good result. Poems or other |
| 139 |
non-standard texts will not produce a good result because they are structured |
| 140 |
in a "funny" way. |
| 141 |
|
| 142 |
Every new language to be implemented needs to contain a list of VERY common words |
| 143 |
in the language. 250 words is more than enough. See en.dic for example. Writing such |
| 144 |
a dictionary is an iteratice process. It is suggested that the author of such a |
| 145 |
dictionary will try to scan a few articles using OTS and try it with the "--keywords" |
| 146 |
flag on. This will tell him if OTS thinks for some reason that "his" is the subject |
| 147 |
of the text. If this is the case then "his" needs to be added to the dictionary. |
| 148 |
|
| 149 |
|
| 150 |
Stemming: |
| 151 |
|
| 152 |
stemming is the ability to take a word such as "running" and trace it back to its |
| 153 |
original form to the word "run". We use this feature to group together all of the |
| 154 |
the derivatives of a certain words. Now what we group together these words |
| 155 |
ots will know what the subject of this article is the one word "run" and not |
| 156 |
("running","run", "runnable" ...). Now, when we encounter a sentence that talks |
| 157 |
about some "runner" we will know to that it is related to the topic of the article. |
| 158 |
|
| 159 |
In previous versions this has never been done because "running"!="run"; |
| 160 |
|
| 161 |
The stemming support is now integrated into OTS. To allow stemming I had to create a new |
| 162 |
file format that will hold the stemming rules. That format is XML based. libxml2 is being used. |
| 163 |
|
| 164 |
Example: |
| 165 |
|
| 166 |
[dam]==[dams] stem[dam] |
| 167 |
[asking]==[asked] stem[ask] |
| 168 |
[build]==[building] stem[build] |
| 169 |
[accidents]==[accident] stem[accident] |
| 170 |
[policing]==[police] stem[polic] |
| 171 |
[reported]==[report] stem[report] |
| 172 |
[years]==[year] stem[year] |
| 173 |
[increased]==[increase] stem[increas] |
| 174 |
[study]==[studies] stem[study] |
| 175 |
[reported]==[reports] stem[report] |
| 176 |
[gates]==[gate] stem[gat] |
| 177 |
[locked]==[lock] stem[lock] |
| 178 |
|
| 179 |
|