Debian Codegraph @

docs/python-module.md

raw
1 ---
2 id: python-module
3 title: Python module
4 ---
5
6 In this document we present how to use fastText in python.
7
8 ## Table of contents
9
10 * [Requirements](#requirements)
11 * [Installation](#installation)
12 * [Usage overview](#usage-overview)
13 * [Word representation model](#word-representation-model)
14 * [Text classification model](#text-classification-model)
15 * [IMPORTANT: Preprocessing data / encoding conventions](#important-preprocessing-data-encoding-conventions)
16 * [More examples](#more-examples)
17 * [API](#api)
18 * [`train_unsupervised` parameters](#train_unsupervised-parameters)
19 * [`train_supervised` parameters](#train_supervised-parameters)
20 * [`model` object](#model-object)
21
22
23 # Requirements
24
25 [fastText](https://fasttext.cc/) builds on modern Mac OS and Linux distributions.
26 Since it uses C\++11 features, it requires a compiler with good C++11 support. You will need [Python](https://www.python.org/) (version 2.7 or ≥ 3.4), [NumPy](http://www.numpy.org/) & [SciPy](https://www.scipy.org/) and [pybind11](https://github.com/pybind/pybind11).
27
28
29 # Installation
30
31 To install the latest release, you can do :
32 ```bash
33 $ pip install fasttext
34 ```
35
36 or, to get the latest development version of fasttext, you can install from our github repository :
37 ```bash
38 $ git clone https://github.com/facebookresearch/fastText.git
39 $ cd fastText
40 $ sudo pip install .
41 $ # or :
42 $ sudo python setup.py install
43 ```
44
45 # Usage overview
46
47
48 ## Word representation model
49
50 In order to learn word vectors, as [described here](/docs/en/references.html#enriching-word-vectors-with-subword-information), we can use `fasttext.train_unsupervised` function like this:
51
52
53 ```py
54 import fasttext
55
56 # Skipgram model :
57 model = fasttext.train_unsupervised('data.txt', model='skipgram')
58
59 # or, cbow model :
60 model = fasttext.train_unsupervised('data.txt', model='cbow')
61
62 ```
63
64 where `data.txt` is a training file containing utf-8 encoded text.
65
66
67 The returned `model` object represents your learned model, and you can use it to retrieve information.
68
69 ```py
70 print(model.words) # list of words in dictionary
71 print(model['king']) # get the vector of the word 'king'
72 ```
73
74
75 ### Saving and loading a model object
76
77 You can save your trained model object by calling the function `save_model`.
78 ```py
79 model.save_model("model_filename.bin")
80 ```
81
82 and retrieve it later thanks to the function `load_model` :
83 ```py
84 model = fasttext.load_model("model_filename.bin")
85 ```
86
87 For more information about word representation usage of fasttext, you can refer to our [word representations tutorial](/docs/en/unsupervised-tutorial.html).
88
89
90 ## Text classification model
91
92 In order to train a text classifier using the method [described here](/docs/en/references.html#bag-of-tricks-for-efficient-text-classification), we can use `fasttext.train_supervised` function like this:
93
94
95 ```py
96 import fasttext
97
98 model = fasttext.train_supervised('data.train.txt')
99 ```
100
101 where `data.train.txt` is a text file containing a training sentence per line along with the labels. By default, we assume that labels are words that are prefixed by the string `__label__`
102
103 Once the model is trained, we can retrieve the list of words and labels:
104
105 ```py
106 print(model.words)
107 print(model.labels)
108 ```
109
110 To evaluate our model by computing the precision at 1 (P@1) and the recall on a test set, we use the `test` function:
111
112 ```py
113 def print_results(N, p, r):
114 print("N\t" + str(N))
115 print("P@{}\t{:.3f}".format(1, p))
116 print("R@{}\t{:.3f}".format(1, r))
117
118 print_results(*model.test('test.txt'))
119 ```
120
121 We can also predict labels for a specific text :
122
123 ```py
124 model.predict("Which baking dish is best to bake a banana bread ?")
125 ```
126
127 By default, `predict` returns only one label : the one with the highest probability. You can also predict more than one label by specifying the parameter `k`:
128 ```py
129 model.predict("Which baking dish is best to bake a banana bread ?", k=3)
130 ```
131
132 If you want to predict more than one sentence you can pass an array of strings :
133
134 ```py
135 model.predict(["Which baking dish is best to bake a banana bread ?", "Why not put knives in the dishwasher?"], k=3)
136 ```
137
138
139 Of course, you can also save and load a model to/from a file as [in the word representation usage](#saving-and-loading-a-model-object).
140
141 For more information about text classification usage of fasttext, you can refer to our [text classification tutorial](/docs/en/supervised-tutorial.html).
142
143
144
145
146 ### Compress model files with quantization
147
148 When you want to save a supervised model file, fastText can compress it in order to have a much smaller model file by sacrificing only a little bit performance.
149
150 ```py
151 # with the previously trained `model` object, call :
152 model.quantize(input='data.train.txt', retrain=True)
153
154 # then display results and save the new model :
155 print_results(*model.test(valid_data))
156 model.save_model("model_filename.ftz")
157 ```
158
159 `model_filename.ftz` will have a much smaller size than `model_filename.bin`.
160
161 For further reading on quantization, you can refer to [this paragraph from our blog post](/blog/2017/10/02/blog-post.html#model-compression).
162
163
164 ## IMPORTANT: Preprocessing data / encoding conventions
165
166 In general it is important to properly preprocess your data. In particular our example scripts in the [root folder](https://github.com/facebookresearch/fastText) do this.
167
168 fastText assumes UTF-8 encoded text. All text must be [unicode for Python2](https://docs.python.org/2/library/functions.html#unicode) and [str for Python3](https://docs.python.org/3.5/library/stdtypes.html#textseq). The passed text will be [encoded as UTF-8 by pybind11](https://pybind11.readthedocs.io/en/master/advanced/cast/strings.html?highlight=utf-8#strings-bytes-and-unicode-conversions) before passed to the fastText C++ library. This means it is important to use UTF-8 encoded text when building a model. On Unix-like systems you can convert text using [iconv](https://en.wikipedia.org/wiki/Iconv).
169
170 fastText will tokenize (split text into pieces) based on the following ASCII characters (bytes). In particular, it is not aware of UTF-8 whitespace. We advice the user to convert UTF-8 whitespace / word boundaries into one of the following symbols as appropiate.
171
172 * space
173 * tab
174 * vertical tab
175 * carriage return
176 * formfeed
177 * the null character
178
179 The newline character is used to delimit lines of text. In particular, the EOS token is appended to a line of text if a newline character is encountered. The only exception is if the number of tokens exceeds the MAX\_LINE\_SIZE constant as defined in the [Dictionary header](https://github.com/facebookresearch/fastText/blob/master/src/dictionary.h). This means if you have text that is not separate by newlines, such as the [fil9 dataset](http://mattmahoney.net/dc/textdata), it will be broken into chunks with MAX\_LINE\_SIZE of tokens and the EOS token is not appended.
180
181 The length of a token is the number of UTF-8 characters by considering the [leading two bits of a byte](https://en.wikipedia.org/wiki/UTF-8#Description) to identify [subsequent bytes of a multi-byte sequence](https://github.com/facebookresearch/fastText/blob/master/src/dictionary.cc). Knowing this is especially important when choosing the minimum and maximum length of subwords. Further, the EOS token (as specified in the [Dictionary header](https://github.com/facebookresearch/fastText/blob/master/src/dictionary.h)) is considered a character and will not be broken into subwords.
182
183 ## More examples
184
185 In order to have a better knowledge of fastText models, please consider the main [README](https://github.com/facebookresearch/fastText/blob/master/README.md) and in particular [the tutorials on our website](https://fasttext.cc/docs/en/supervised-tutorial.html).
186
187 You can find further python examples in [the doc folder](https://github.com/facebookresearch/fastText/tree/master/python/doc/examples).
188
189 As with any package you can get help on any Python function using the help function.
190
191 For example
192
193 ```
194 +>>> import fasttext
195 +>>> help(fasttext.FastText)
196
197 Help on module fasttext.FastText in fasttext:
198
199 NAME
200 fasttext.FastText
201
202 DESCRIPTION
203 # Copyright (c) 2017-present, Facebook, Inc.
204 # All rights reserved.
205 #
206 # This source code is licensed under the MIT license found in the
207 # LICENSE file in the root directory of this source tree.
208
209 FUNCTIONS
210 load_model(path)
211 Load a model given a filepath and return a model object.
212
213 tokenize(text)
214 Given a string of text, tokenize it and return a list of tokens
215 [...]
216 ```
217
218
219 # API
220
221
222 ## `train_unsupervised` parameters
223
224 ```python
225 input # training file path (required)
226 model # unsupervised fasttext model {cbow, skipgram} [skipgram]
227 lr # learning rate [0.05]
228 dim # size of word vectors [100]
229 ws # size of the context window [5]
230 epoch # number of epochs [5]
231 minCount # minimal number of word occurences [5]
232 minn # min length of char ngram [3]
233 maxn # max length of char ngram [6]
234 neg # number of negatives sampled [5]
235 wordNgrams # max length of word ngram [1]
236 loss # loss function {ns, hs, softmax, ova} [ns]
237 bucket # number of buckets [2000000]
238 thread # number of threads [number of cpus]
239 lrUpdateRate # change the rate of updates for the learning rate [100]
240 t # sampling threshold [0.0001]
241 verbose # verbose [2]
242 ```
243
244 ## `train_supervised` parameters
245
246 ```python
247 input # training file path (required)
248 lr # learning rate [0.1]
249 dim # size of word vectors [100]
250 ws # size of the context window [5]
251 epoch # number of epochs [5]
252 minCount # minimal number of word occurences [1]
253 minCountLabel # minimal number of label occurences [1]
254 minn # min length of char ngram [0]
255 maxn # max length of char ngram [0]
256 neg # number of negatives sampled [5]
257 wordNgrams # max length of word ngram [1]
258 loss # loss function {ns, hs, softmax, ova} [softmax]
259 bucket # number of buckets [2000000]
260 thread # number of threads [number of cpus]
261 lrUpdateRate # change the rate of updates for the learning rate [100]
262 t # sampling threshold [0.0001]
263 label # label prefix ['__label__']
264 verbose # verbose [2]
265 pretrainedVectors # pretrained word vectors (.vec file) for supervised learning []
266 ```
267
268 ## `model` object
269
270 `train_supervised`, `train_unsupervised` and `load_model` functions return an instance of `_FastText` class, that we generaly name `model` object.
271
272 This object exposes those training arguments as properties : `lr`, `dim`, `ws`, `epoch`, `minCount`, `minCountLabel`, `minn`, `maxn`, `neg`, `wordNgrams`, `loss`, `bucket`, `thread`, `lrUpdateRate`, `t`, `label`, `verbose`, `pretrainedVectors`. So `model.wordNgrams` will give you the max length of word ngram used for training this model.
273
274 In addition, the object exposes several functions :
275
276 ```python
277 get_dimension # Get the dimension (size) of a lookup vector (hidden layer).
278 # This is equivalent to `dim` property.
279 get_input_vector # Given an index, get the corresponding vector of the Input Matrix.
280 get_input_matrix # Get a copy of the full input matrix of a Model.
281 get_labels # Get the entire list of labels of the dictionary
282 # This is equivalent to `labels` property.
283 get_line # Split a line of text into words and labels.
284 get_output_matrix # Get a copy of the full output matrix of a Model.
285 get_sentence_vector # Given a string, get a single vector represenation. This function
286 # assumes to be given a single line of text. We split words on
287 # whitespace (space, newline, tab, vertical tab) and the control
288 # characters carriage return, formfeed and the null character.
289 get_subword_id # Given a subword, return the index (within input matrix) it hashes to.
290 get_subwords # Given a word, get the subwords and their indicies.
291 get_word_id # Given a word, get the word id within the dictionary.
292 get_word_vector # Get the vector representation of word.
293 get_words # Get the entire list of words of the dictionary
294 # This is equivalent to `words` property.
295 is_quantized # whether the model has been quantized
296 predict # Given a string, get a list of labels and a list of corresponding probabilities.
297 quantize # Quantize the model reducing the size of the model and it's memory footprint.
298 save_model # Save the model to the given path
299 test # Evaluate supervised model using file given by path
300 test_label # Return the precision and recall score for each label.
301 ```
302
303 The properties `words`, `labels` return the words and labels from the dictionary :
304 ```py
305 model.words # equivalent to model.get_words()
306 model.labels # equivalent to model.get_labels()
307 ```
308
309 The object overrides `__getitem__` and `__contains__` functions in order to return the representation of a word and to check if a word is in the vocabulary.
310
311 ```py
312 model['king'] # equivalent to model.get_word_vector('king')
313 'king' in model # equivalent to `'king' in model.get_words()`
314 ```