Debian Codegraph @

README

raw
1 Open Text summarizer
2 ---------------------
3 Nadav Rotem 2003
4 [email protected]
5 ---------------------
6
7 The open text summarizer is an open source tool for summarizing texts. The program
8 reads a text and decides which sentences are important and which are not. OTS will
9 create a short summary or will highlight the main ideas in the text.
10
11 OTS is both a library and a command line tool. Word processors such as AbiWord
12 and KWord can link to the library and summarize documents while the command line tool
13 lets you summarize text on the console. The program can either print the summarized
14 text as text or HTML. If in HTML, the important sentences are highlighted.
15 The program is multi lingual and works with UTF-8 encoding.
16
17 Supported languages:
18 (ca,bg,cs,cy,da,de,el,en,eo,es,et,eu,fi,fr,ga,gl,he,hu,ia,id,is,it,lv,mi,ms,mt,nl,nn,
19 pl,pt,ro,ru,sv,tl,tr,uk,yi)
20
21 Basque (seminal)
22 Bulgarian (seminal)
23 Catalan (seminal)
24 Czech (seminal)
25 Danish (seminal)
26 Dutch
27 English
28 Esperanto (seminal)
29 Estonian (seminal)
30 Finnish (seminal)
31 French (seminal)
32 Galician (seminal)
33 German (seminal)
34 Greek (seminal)
35 Hebrew
36 Hungarian (seminal)
37 Icelandic (seminal)
38 Indonesian (seminal)
39 Interlingua (seminal)
40 Irish (Gaelic) (seminal)
41 Italian (seminal)
42 Latvian (seminal)
43 Malaysian (seminal)
44 Maltese (seminal)
45 Maori (seminal)
46 Norwegian Nynorsk (seminal)
47 Polish (seminal)
48 Portuguese (seminal)
49 Romanian (seminal)
50 Russian (seminal)
51 Spanish (seminal)
52 Swedish (seminal)
53 Tagalog (Pilipino) (seminal)
54 Turkish (seminal)
55 Ukrainian (seminal)
56 Welsh (seminal)
57 Yiddish (seminal)
58
59 Command line Usage:
60
61 $ ots
62 --ratio [0..100] :select the summary ratio
63 --html :print summary in highlighted html form
64 --keywords :print key words in the article
65 --dic :use dictionary other then english, for example 'he' or 'es'
66 --about :tells that the article talks about
67 --file :input file
68 --out :output file
69 --help :print this help screen
70
71 example1:
72 ots articles/sacbee1.txt --ratio 20 --html
73
74 This command will summarize the article from the Sacramento Bee and highlight the 20% of
75 sentences most important to the content of the article.
76
77 example2:
78 ots --ratio 40 --file big_article.txt
79
80 This command will summarize the article down to 40% and print it.
81
82
83 How it works:
84
85 The Idea behind OTS is that the important ideas in an article are described with
86 many of the same words while redundant information uses less technical terms
87 and is not related to the main subject of the article. Important lines are lines
88 that are related to the subject of the article. The subject of the article is the
89 list of ideas that are most discussed in the article.
90 In practice , the article is scanned once and a list of all of the words and their
91 occurrence in the text is stored in a list. After sorting this list by the
92 occurrence of words in the text we will see a list similar to this:
93
94
95 11 are
96 17 is
97 16 a
98 14 Harry
99 14 on
100 13 Sally
101 11 Love
102 11 such
103 4 an
104 2 taxi
105 1 university
106 1 chicago
107 1 meets
108 ...
109 ...
110
111 Now we will remove all the words that are common in the English language using a
112 dictionary file. Example of such words: "The","a","since","after","will".
113 These words are words that don't teach us anything about the subject of the article.
114 Knowing that the word "Police" is in a given text can tell us that the article might
115 talk about the police , however knowing that the word "will" is in the text can't
116 teach us anything about the subject of the text.
117
118 After removing the redundant words the new list will look like this:
119
120 14 Harry
121 13 Sally
122 11 Love
123 2 taxi
124 1 university
125 1 chicago
126 1 meets
127
128 From this list we may assume that the text talks about "Harry , Sally , Love".
129 So an important sentence in the text will be a sentence that talks about Harry,
130 Sally and Love. A sentence that talks about the university of Chicago may be
131 neglected because we find the word "chicago" only once in the text, so we may assume
132 that "chicago" is not one of the main ideas in the text. Each sentence is given
133 a grade based on the the keywords in it. A line that holds many important words will
134 be given a high grade. To produce a 20% summary we print the top 20% sentences with
135 the highest grade.
136
137 Ots will not work on books because they cover too many topics. however, each
138 page may be fed individually, this will produce a good result. Poems or other
139 non-standard texts will not produce a good result because they are structured
140 in a "funny" way.
141
142 Every new language to be implemented needs to contain a list of VERY common words
143 in the language. 250 words is more than enough. See en.dic for example. Writing such
144 a dictionary is an iteratice process. It is suggested that the author of such a
145 dictionary will try to scan a few articles using OTS and try it with the "--keywords"
146 flag on. This will tell him if OTS thinks for some reason that "his" is the subject
147 of the text. If this is the case then "his" needs to be added to the dictionary.
148
149
150 Stemming:
151
152 stemming is the ability to take a word such as "running" and trace it back to its
153 original form to the word "run". We use this feature to group together all of the
154 the derivatives of a certain words. Now what we group together these words
155 ots will know what the subject of this article is the one word "run" and not
156 ("running","run", "runnable" ...). Now, when we encounter a sentence that talks
157 about some "runner" we will know to that it is related to the topic of the article.
158
159 In previous versions this has never been done because "running"!="run";
160
161 The stemming support is now integrated into OTS. To allow stemming I had to create a new
162 file format that will hold the stemming rules. That format is XML based. libxml2 is being used.
163
164 Example:
165
166 [dam]==[dams] stem[dam]
167 [asking]==[asked] stem[ask]
168 [build]==[building] stem[build]
169 [accidents]==[accident] stem[accident]
170 [policing]==[police] stem[polic]
171 [reported]==[report] stem[report]
172 [years]==[year] stem[year]
173 [increased]==[increase] stem[increas]
174 [study]==[studies] stem[study]
175 [reported]==[reports] stem[report]
176 [gates]==[gate] stem[gat]
177 [locked]==[lock] stem[lock]
178
179