Debian Codegraph @

docs/unsupervised-tutorials.md

raw
1 ---
2 id: unsupervised-tutorial
3 title: Word representations
4 ---
5 A popular idea in modern machine learning is to represent words by vectors. These vectors capture hidden information about a language, like word analogies or semantic. It is also used to improve performance of text classifiers.
6
7 In this tutorial, we show how to build these word vectors with the fastText tool. To download and install fastText, follow the first steps of [the tutorial on text classification](https://fasttext.cc/docs/en/supervised-tutorial.html).
8
9 ## Getting the data
10
11 In order to compute word vectors, you need a large text corpus. Depending on the corpus, the word vectors will capture different information. In this tutorial, we focus on Wikipedia's articles but other sources could be considered, like news or Webcrawl (more examples [here](http://statmt.org/)). To download a raw dump of Wikipedia, run the following command:
12
13 ```bash
14 wget https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2
15 ```
16
17 Downloading the Wikipedia corpus takes some time. Instead, lets restrict our study to the first 1 billion bytes of English Wikipedia. They can be found on Matt Mahoney's [website](http://mattmahoney.net/):
18
19 ```bash
20 $ mkdir data
21 $ wget -c http://mattmahoney.net/dc/enwik9.zip -P data
22 $ unzip data/enwik9.zip -d data
23 ```
24
25 A raw Wikipedia dump contains a lot of HTML / XML data. We pre-process it with the wikifil.pl script bundled with fastText (this script was originally developed by Matt Mahoney, and can be found on his [website](http://mattmahoney.net/)).
26
27 ```bash
28 $ perl wikifil.pl data/enwik9 > data/fil9
29 ```
30
31 We can check the file by running the following command:
32
33 ```bash
34 $ head -c 80 data/fil9
35 anarchism originated as a term of abuse first used against early working class
36 ```
37
38 The text is nicely pre-processed and can be used to learn our word vectors.
39
40 ## Training word vectors
41
42 <!--DOCUSAURUS_CODE_TABS-->
43 <!--Command line-->
44 Learning word vectors on this data can now be achieved with a single command:
45
46 ```bash
47 $ mkdir result
48 $ ./fasttext skipgram -input data/fil9 -output result/fil9
49 ```
50
51 To decompose this command line: ./fastext calls the binary fastText executable (see how to install fastText [here](https://fasttext.cc/docs/en/support.html)) with the 'skipgram' model (it can also be 'cbow'). We then specify the requires options '-input' for the location of the data and '-output' for the location where the word representations will be saved.
52
53 While fastText is running, the progress and estimated time to completion is shown on your screen. Once the program finishes, there should be two files in the result directory:
54
55 ```bash
56 $ ls -l result
57 -rw-r-r-- 1 bojanowski 1876110778 978480850 Dec 20 11:01 fil9.bin
58 -rw-r-r-- 1 bojanowski 1876110778 190004182 Dec 20 11:01 fil9.vec
59 ```
60
61 The `fil9.bin` file is a binary file that stores the whole fastText model and can be subsequently loaded. The `fil9.vec` file is a text file that contains the word vectors, one per line for each word in the vocabulary:
62
63 ```bash
64 $ head -n 4 result/fil9.vec
65 218316 100
66 the -0.10363 -0.063669 0.032436 -0.040798 0.53749 0.00097867 0.10083 0.24829 ...
67 of -0.0083724 0.0059414 -0.046618 -0.072735 0.83007 0.038895 -0.13634 0.60063 ...
68 one 0.32731 0.044409 -0.46484 0.14716 0.7431 0.24684 -0.11301 0.51721 0.73262 ...
69 ```
70
71 The first line is a header containing the number of words and the dimensionality of the vectors. The subsequent lines are the word vectors for all words in the vocabulary, sorted by decreasing frequency.
72 <!--Python-->
73 Learning word vectors on this data can now be achieved with a single command:
74 ```py
75 >>> import fasttext
76 >>> model = fasttext.train_unsupervised('data/fil9')
77 ```
78 While fastText is running, the progress and estimated time to completion is shown on your screen. Once the training finishes, `model` variable contains information on the trained model, and can be used for querying:
79
80 ```py
81 >>> model.words
82 [u'the', u'of', u'one', u'zero', u'and', u'in', u'two', u'a', u'nine', u'to', u'is', ...
83 ```
84 It returns all words in the vocabulary, sorted by decreasing frequency. We can get the word vector by:
85 ```py
86 >>> model.get_word_vector("the")
87 array([-0.03087516, 0.09221972, 0.17660329, 0.17308897, 0.12863874,
88 0.13912526, -0.09851588, 0.00739991, 0.37038437, -0.00845221,
89 ...
90 -0.21184735, -0.05048715, -0.34571868, 0.23765688, 0.23726143],
91 dtype=float32)
92 ```
93
94 We can save this model on disk as a binary file:
95 ```py
96 >>> model.save_model("result/fil9.bin")
97 ```
98 and reload it later instead of training again:
99 ```py
100 $ python
101 >>> import fasttext
102 >>> model = fasttext.load_model("result/fil9.bin")
103 ```
104
105 <!--END_DOCUSAURUS_CODE_TABS-->
106
107 ## Advanced readers: skipgram versus cbow
108
109 fastText provides two models for computing word representations: skipgram and cbow ('**c**ontinuous-**b**ag-**o**f-**w**ords').
110
111 The skipgram model learns to predict a target word thanks to a nearby word. On the other hand, the cbow model predicts the target word according to its context. The context is represented as a bag of the words contained in a fixed size window around the target word.
112
113 Let us illustrate this difference with an example: given the sentence *'Poets have been mysteriously silent on the subject of cheese'* and the target word '*silent*', a skipgram model tries to predict the target using a random close-by word, like '*subject' *or* '*mysteriously*'**. *The cbow model takes all the words in a surrounding window, like {*been, *mysteriously*, on, the*}, and uses the sum of their vectors to predict the target. The figure below summarizes this difference with another example.
114
115 ![cbow vs skipgram](https://fasttext.cc/img/cbo_vs_skipgram.png)
116 To train a cbow model with fastText, you run the following command:
117
118 <!--DOCUSAURUS_CODE_TABS-->
119 <!--Command line-->
120 ```bash
121 ./fasttext cbow -input data/fil9 -output result/fil9
122 ```
123 <!--Python-->
124 ```py
125 >>> import fasttext
126 >>> model = fasttext.train_unsupervised('data/fil9', "cbow")
127 ```
128 <!--END_DOCUSAURUS_CODE_TABS-->
129
130 In practice, we observe that skipgram models works better with subword information than cbow.
131
132 ## Advanced readers: playing with the parameters
133
134 So far, we run fastText with the default parameters, but depending on the data, these parameters may not be optimal. Let us give an introduction to some of the key parameters for word vectors.
135
136 The most important parameters of the model are its dimension and the range of size for the subwords. The dimension (*dim*) controls the size of the vectors, the larger they are the more information they can capture but requires more data to be learned. But, if they are too large, they are harder and slower to train. By default, we use 100 dimensions, but any value in the 100-300 range is as popular. The subwords are all the substrings contained in a word between the minimum size (*minn*) and the maximal size (*maxn*). By default, we take all the subword between 3 and 6 characters, but other range could be more appropriate to different languages:
137
138 <!--DOCUSAURUS_CODE_TABS-->
139 <!--Command line-->
140 ```bash
141 $ ./fasttext skipgram -input data/fil9 -output result/fil9 -minn 2 -maxn 5 -dim 300
142 ```
143 <!--Python-->
144 ```py
145 >>> import fasttext
146 >>> model = fasttext.train_unsupervised('data/fil9', minn=2, maxn=5, dim=300)
147 ```
148 <!--END_DOCUSAURUS_CODE_TABS-->
149
150 Depending on the quantity of data you have, you may want to change the parameters of the training. The *epoch* parameter controls how many times the model will loop over your data. By default, we loop over the dataset 5 times. If you dataset is extremely massive, you may want to loop over it less often. Another important parameter is the learning rate -*lr*. The higher the learning rate is, the faster the model converge to a solution but at the risk of overfitting to the dataset. The default value is 0.05 which is a good compromise. If you want to play with it we suggest to stay in the range of [0.01, 1]:
151
152 <!--DOCUSAURUS_CODE_TABS-->
153 <!--Command line-->
154 ```bash
155 $ ./fasttext skipgram -input data/fil9 -output result/fil9 -epoch 1 -lr 0.5
156 ```
157 <!--Python-->
158 ```py
159 >>> import fasttext
160 >>> model = fasttext.train_unsupervised('data/fil9', epoch=1, lr=0.5)
161 ```
162 <!--END_DOCUSAURUS_CODE_TABS-->
163
164 Finally , fastText is multi-threaded and uses 12 threads by default. If you have less CPU cores (say 4), you can easily set the number of threads using the *thread* flag:
165
166 <!--DOCUSAURUS_CODE_TABS-->
167 <!--Command line-->
168 ```bash
169 $ ./fasttext skipgram -input data/fil9 -output result/fil9 -thread 4
170 ```
171 <!--Python-->
172 ```py
173 >>> import fasttext
174 >>> model = fasttext.train_unsupervised('data/fil9', thread=4)
175 ```
176 <!--END_DOCUSAURUS_CODE_TABS-->
177
178
179 ## Printing word vectors
180
181 Searching and printing word vectors directly from the `fil9.vec` file is cumbersome. Fortunately, there is a `print-word-vectors` functionality in fastText.
182
183 For example, we can print the word vectors of words *asparagus,* *pidgey* and *yellow* with the following command:
184 <!--DOCUSAURUS_CODE_TABS-->
185 <!--Command line-->
186 ```bash
187 $ echo "asparagus pidgey yellow" | ./fasttext print-word-vectors result/fil9.bin
188 asparagus 0.46826 -0.20187 -0.29122 -0.17918 0.31289 -0.31679 0.17828 -0.04418 ...
189 pidgey -0.16065 -0.45867 0.10565 0.036952 -0.11482 0.030053 0.12115 0.39725 ...
190 yellow -0.39965 -0.41068 0.067086 -0.034611 0.15246 -0.12208 -0.040719 -0.30155 ...
191 ```
192 <!--Python-->
193 ```py
194 >>> [model.get_word_vector(x) for x in ["asparagus", "pidgey", "yellow"]]
195 [array([-0.25751096, -0.18716481, 0.06921121, 0.06455903, 0.29168844,
196 0.15426874, -0.33448914, -0.427215 , 0.7813013 , -0.10600132,
197 ...
198 0.37090245, 0.39266172, -0.4555302 , 0.27452755, 0.00467369],
199 dtype=float32),
200 array([-0.20613593, -0.25325796, -0.2422259 , -0.21067499, 0.32879013,
201 0.7269511 , 0.3782259 , 0.11274897, 0.246764 , -0.6423613 ,
202 ...
203 0.46302193, 0.2530962 , -0.35795924, 0.5755718 , 0.09843876],
204 dtype=float32),
205 array([-0.304823 , 0.2543754 , -0.2198013 , -0.25421786, 0.11219151,
206 0.38286993, -0.22636674, -0.54023844, 0.41095474, -0.3505803 ,
207 ...
208 0.54788435, 0.36740595, -0.5678512 , 0.07523401, -0.08701935],
209 dtype=float32)]
210
211 ```
212 <!--END_DOCUSAURUS_CODE_TABS-->
213
214 A nice feature is that you can also query for words that did not appear in your data! Indeed words are represented by the sum of its substrings. As long as the unknown word is made of known substrings, there is a representation of it!
215
216 As an example let's try with a misspelled word:
217
218 <!--DOCUSAURUS_CODE_TABS-->
219 <!--Command line-->
220 ```bash
221 $ echo "enviroment" | ./fasttext print-word-vectors result/fil9.bin
222 ```
223 <!--Python-->
224 ```py
225 >>> model.get_word_vector("enviroment")
226 ```
227 <!--END_DOCUSAURUS_CODE_TABS-->
228
229 You still get a word vector for it! But how good it is? Let's find out in the next sections!
230
231
232 ## Nearest neighbor queries
233
234 A simple way to check the quality of a word vector is to look at its nearest neighbors. This give an intuition of the type of semantic information the vectors are able to capture.
235
236 This can be achieved with the nearest neighbor (*nn*) functionality. For example, we can query the 10 nearest neighbors of a word by running the following command:
237
238 <!--DOCUSAURUS_CODE_TABS-->
239 <!--Command line-->
240 ```bash
241 $ ./fasttext nn result/fil9.bin
242 Pre-computing word vectors... done.
243 ```
244
245 Then we are prompted to type our query word, let us try *asparagus* :
246
247 ```bash
248 Query word? asparagus
249 beetroot 0.812384
250 tomato 0.806688
251 horseradish 0.805928
252 spinach 0.801483
253 licorice 0.791697
254 lingonberries 0.781507
255 asparagales 0.780756
256 lingonberry 0.778534
257 celery 0.774529
258 beets 0.773984
259 ```
260 <!--Python-->
261 ```py
262 >>> model.get_nearest_neighbors('asparagus')
263 [(0.812384, u'beetroot'), (0.806688, u'tomato'), (0.805928, u'horseradish'), (0.801483, u'spinach'), (0.791697, u'licorice'), (0.781507, u'lingonberries'), (0.780756, u'asparagales'), (0.778534, u'lingonberry'), (0.774529, u'celery'), (0.773984, u'beets')]
264 ```
265 <!--END_DOCUSAURUS_CODE_TABS-->
266
267 Nice! It seems that vegetable vectors are similar. Note that the nearest neighbor is the word *asparagus* itself, this means that this word appeared in the dataset. What about pokemons?
268
269 <!--DOCUSAURUS_CODE_TABS-->
270 <!--Command line-->
271 ```bash
272 Query word? pidgey
273 pidgeot 0.891801
274 pidgeotto 0.885109
275 pidge 0.884739
276 pidgeon 0.787351
277 pok 0.781068
278 pikachu 0.758688
279 charizard 0.749403
280 squirtle 0.742582
281 beedrill 0.741579
282 charmeleon 0.733625
283 ```
284 <!--Python-->
285 ```py
286 >>> model.get_nearest_neighbors('pidgey')
287 [(0.891801, u'pidgeot'), (0.885109, u'pidgeotto'), (0.884739, u'pidge'), (0.787351, u'pidgeon'), (0.781068, u'pok'), (0.758688, u'pikachu'), (0.749403, u'charizard'), (0.742582, u'squirtle'), (0.741579, u'beedrill'), (0.733625, u'charmeleon')]
288 ```
289 <!--END_DOCUSAURUS_CODE_TABS-->
290
291
292 Different evolution of the same Pokemon have close-by vectors! But what about our misspelled word, is its vector close to anything reasonable? Let s find out:
293
294 <!--DOCUSAURUS_CODE_TABS-->
295 <!--Command line-->
296 ```bash
297 Query word? enviroment
298 enviromental 0.907951
299 environ 0.87146
300 enviro 0.855381
301 environs 0.803349
302 environnement 0.772682
303 enviromission 0.761168
304 realclimate 0.716746
305 environment 0.702706
306 acclimatation 0.697196
307 ecotourism 0.697081
308 ```
309 <!--Python-->
310 ```py
311 >>> model.get_nearest_neighbors('enviroment')
312 [(0.907951, u'enviromental'), (0.87146, u'environ'), (0.855381, u'enviro'), (0.803349, u'environs'), (0.772682, u'environnement'), (0.761168, u'enviromission'), (0.716746, u'realclimate'), (0.702706, u'environment'), (0.697196, u'acclimatation'), (0.697081, u'ecotourism')]
313 ```
314 <!--END_DOCUSAURUS_CODE_TABS-->
315
316
317 Thanks to the information contained within the word, the vector of our misspelled word matches to reasonable words! It is not perfect but the main information has been captured.
318
319 ## Advanced reader: measure of similarity
320
321 In order to find nearest neighbors, we need to compute a similarity score between words. Our words are represented by continuous word vectors and we can thus apply simple similarities to them. In particular we use the cosine of the angles between two vectors. This similarity is computed for all words in the vocabulary, and the 10 most similar words are shown. Of course, if the word appears in the vocabulary, it will appear on top, with a similarity of 1.
322
323 ## Word analogies
324
325 In a similar spirit, one can play around with word analogies. For example, we can see if our model can guess what is to France, and what Berlin is to Germany.
326
327
328
329
330
331 This can be done with the *analogies* functionality. It takes a word triplet (like *Germany Berlin France*) and outputs the analogy:
332
333 <!--DOCUSAURUS_CODE_TABS-->
334 <!--Command line-->
335 ```bash
336 $ ./fasttext analogies result/fil9.bin
337 Pre-computing word vectors... done.
338 Query triplet (A - B + C)? berlin germany france
339 paris 0.896462
340 bourges 0.768954
341 louveciennes 0.765569
342 toulouse 0.761916
343 valenciennes 0.760251
344 montpellier 0.752747
345 strasbourg 0.744487
346 meudon 0.74143
347 bordeaux 0.740635
348 pigneaux 0.736122
349 ```
350 <!--Python-->
351 ```py
352 >>> model.get_analogies("berlin", "germany", "france")
353 [(0.896462, u'paris'), (0.768954, u'bourges'), (0.765569, u'louveciennes'), (0.761916, u'toulouse'), (0.760251, u'valenciennes'), (0.752747, u'montpellier'), (0.744487, u'strasbourg'), (0.74143, u'meudon'), (0.740635, u'bordeaux'), (0.736122, u'pigneaux')]
354 ```
355 <!--END_DOCUSAURUS_CODE_TABS-->
356
357 The answer provided by our model is *Paris*, which is correct. Let's have a look at a less obvious example:
358
359 <!--DOCUSAURUS_CODE_TABS-->
360 <!--Command line-->
361 ```bash
362 Query triplet (A - B + C)? psx sony nintendo
363 gamecube 0.803352
364 nintendogs 0.792646
365 playstation 0.77344
366 sega 0.772165
367 gameboy 0.767959
368 arcade 0.754774
369 playstationjapan 0.753473
370 gba 0.752909
371 dreamcast 0.74907
372 famicom 0.745298
373 ```
374 <!--Python-->
375 ```py
376 >>> model.get_analogies("psx", "sony", "nintendo")
377 [(0.803352, u'gamecube'), (0.792646, u'nintendogs'), (0.77344, u'playstation'), (0.772165, u'sega'), (0.767959, u'gameboy'), (0.754774, u'arcade'), (0.753473, u'playstationjapan'), (0.752909, u'gba'), (0.74907, u'dreamcast'), (0.745298, u'famicom')]
378 ```
379 <!--END_DOCUSAURUS_CODE_TABS-->
380
381
382 Our model considers that the *nintendo* analogy of a *psx* is the *gamecube*, which seems reasonable. Of course the quality of the analogies depend on the dataset used to train the model and one can only hope to cover fields only in the dataset.
383
384
385 ## Importance of character n-grams
386
387 Using subword-level information is particularly interesting to build vectors for unknown words. For example, the word *gearshift* does not exist on Wikipedia but we can still query its closest existing words:
388
389 <!--DOCUSAURUS_CODE_TABS-->
390 <!--Command line-->
391 ```bash
392 Query word? gearshift
393 gearing 0.790762
394 flywheels 0.779804
395 flywheel 0.777859
396 gears 0.776133
397 driveshafts 0.756345
398 driveshaft 0.755679
399 daisywheel 0.749998
400 wheelsets 0.748578
401 epicycles 0.744268
402 gearboxes 0.73986
403 ```
404 <!--Python-->
405 ```py
406 >>> model.get_nearest_neighbors('gearshift')
407 [(0.790762, u'gearing'), (0.779804, u'flywheels'), (0.777859, u'flywheel'), (0.776133, u'gears'), (0.756345, u'driveshafts'), (0.755679, u'driveshaft'), (0.749998, u'daisywheel'), (0.748578, u'wheelsets'), (0.744268, u'epicycles'), (0.73986, u'gearboxes')]
408 ```
409 <!--END_DOCUSAURUS_CODE_TABS-->
410
411
412
413 Most of the retrieved words share substantial substrings but a few are actually quite different, like *cogwheel*. You can try other words like *sunbathe* or *grandnieces*.
414
415 Now that we have seen the interest of subword information for unknown words, let's check how it compares to a model that does not use subword information. To train a model without subwords, just run the following command:
416
417 <!--DOCUSAURUS_CODE_TABS-->
418 <!--Command line-->
419 ```bash
420 $ ./fasttext skipgram -input data/fil9 -output result/fil9-none -maxn 0
421 ```
422 The results are saved in result/fil9-non.vec and result/fil9-non.bin.
423 <!--Python-->
424 ```py
425 >>> model_without_subwords = fasttext.train_unsupervised('data/fil9', maxn=0)
426 ```
427 <!--END_DOCUSAURUS_CODE_TABS-->
428
429
430 To illustrate the difference, let us take an uncommon word in Wikipedia, like *accomodation* which is a misspelling of *accommodation**.* Here is the nearest neighbors obtained without subwords:
431
432 <!--DOCUSAURUS_CODE_TABS-->
433 <!--Command line-->
434 ```bash
435 $ ./fasttext nn result/fil9-none.bin
436 Query word? accomodation
437 sunnhordland 0.775057
438 accomodations 0.769206
439 administrational 0.753011
440 laponian 0.752274
441 ammenities 0.750805
442 dachas 0.75026
443 vuosaari 0.74172
444 hostelling 0.739995
445 greenbelts 0.733975
446 asserbo 0.732465
447 ```
448 <!--Python-->
449 ```py
450 >>> model_without_subwords.get_nearest_neighbors('accomodation')
451 [(0.775057, u'sunnhordland'), (0.769206, u'accomodations'), (0.753011, u'administrational'), (0.752274, u'laponian'), (0.750805, u'ammenities'), (0.75026, u'dachas'), (0.74172, u'vuosaari'), (0.739995, u'hostelling'), (0.733975, u'greenbelts'), (0.732465, u'asserbo')]
452 ```
453 <!--END_DOCUSAURUS_CODE_TABS-->
454
455 The result does not make much sense, most of these words are unrelated. On the other hand, using subword information gives the following list of nearest neighbors:
456
457 <!--DOCUSAURUS_CODE_TABS-->
458 <!--Command line-->
459 ```bash
460 Query word? accomodation
461 accomodations 0.96342
462 accommodation 0.942124
463 accommodations 0.915427
464 accommodative 0.847751
465 accommodating 0.794353
466 accomodated 0.740381
467 amenities 0.729746
468 catering 0.725975
469 accomodate 0.703177
470 hospitality 0.701426
471 ```
472 <!--Python-->
473 ```py
474 >>> model.get_nearest_neighbors('accomodation')
475 [(0.96342, u'accomodations'), (0.942124, u'accommodation'), (0.915427, u'accommodations'), (0.847751, u'accommodative'), (0.794353, u'accommodating'), (0.740381, u'accomodated'), (0.729746, u'amenities'), (0.725975, u'catering'), (0.703177, u'accomodate'), (0.701426, u'hospitality')]
476 ```
477 <!--END_DOCUSAURUS_CODE_TABS-->
478
479 The nearest neighbors capture different variation around the word *accommodation*. We also get semantically related words such as *amenities* or *catering*.
480
481 ## Conclusion
482
483 In this tutorial, we show how to obtain word vectors from Wikipedia. This can be done for any language and we provide [pre-trained models](https://fasttext.cc/docs/en/pretrained-vectors.html) with the default setting for 294 of them.