| 1 |
--- |
| 2 |
id: python-module |
| 3 |
title: Python module |
| 4 |
--- |
| 5 |
|
| 6 |
In this document we present how to use fastText in python. |
| 7 |
|
| 8 |
## Table of contents |
| 9 |
|
| 10 |
* [Requirements](#requirements) |
| 11 |
* [Installation](#installation) |
| 12 |
* [Usage overview](#usage-overview) |
| 13 |
* [Word representation model](#word-representation-model) |
| 14 |
* [Text classification model](#text-classification-model) |
| 15 |
* [IMPORTANT: Preprocessing data / encoding conventions](#important-preprocessing-data-encoding-conventions) |
| 16 |
* [More examples](#more-examples) |
| 17 |
* [API](#api) |
| 18 |
* [`train_unsupervised` parameters](#train_unsupervised-parameters) |
| 19 |
* [`train_supervised` parameters](#train_supervised-parameters) |
| 20 |
* [`model` object](#model-object) |
| 21 |
|
| 22 |
|
| 23 |
# Requirements |
| 24 |
|
| 25 |
[fastText](https://fasttext.cc/) builds on modern Mac OS and Linux distributions. |
| 26 |
Since it uses C\++11 features, it requires a compiler with good C++11 support. You will need [Python](https://www.python.org/) (version 2.7 or ≥ 3.4), [NumPy](http://www.numpy.org/) & [SciPy](https://www.scipy.org/) and [pybind11](https://github.com/pybind/pybind11). |
| 27 |
|
| 28 |
|
| 29 |
# Installation |
| 30 |
|
| 31 |
To install the latest release, you can do : |
| 32 |
```bash |
| 33 |
$ pip install fasttext |
| 34 |
``` |
| 35 |
|
| 36 |
or, to get the latest development version of fasttext, you can install from our github repository : |
| 37 |
```bash |
| 38 |
$ git clone https://github.com/facebookresearch/fastText.git |
| 39 |
$ cd fastText |
| 40 |
$ sudo pip install . |
| 41 |
$ # or : |
| 42 |
$ sudo python setup.py install |
| 43 |
``` |
| 44 |
|
| 45 |
# Usage overview |
| 46 |
|
| 47 |
|
| 48 |
## Word representation model |
| 49 |
|
| 50 |
In order to learn word vectors, as [described here](/docs/en/references.html#enriching-word-vectors-with-subword-information), we can use `fasttext.train_unsupervised` function like this: |
| 51 |
|
| 52 |
|
| 53 |
```py |
| 54 |
import fasttext |
| 55 |
|
| 56 |
# Skipgram model : |
| 57 |
model = fasttext.train_unsupervised('data.txt', model='skipgram') |
| 58 |
|
| 59 |
# or, cbow model : |
| 60 |
model = fasttext.train_unsupervised('data.txt', model='cbow') |
| 61 |
|
| 62 |
``` |
| 63 |
|
| 64 |
where `data.txt` is a training file containing utf-8 encoded text. |
| 65 |
|
| 66 |
|
| 67 |
The returned `model` object represents your learned model, and you can use it to retrieve information. |
| 68 |
|
| 69 |
```py |
| 70 |
print(model.words) # list of words in dictionary |
| 71 |
print(model['king']) # get the vector of the word 'king' |
| 72 |
``` |
| 73 |
|
| 74 |
|
| 75 |
### Saving and loading a model object |
| 76 |
|
| 77 |
You can save your trained model object by calling the function `save_model`. |
| 78 |
```py |
| 79 |
model.save_model("model_filename.bin") |
| 80 |
``` |
| 81 |
|
| 82 |
and retrieve it later thanks to the function `load_model` : |
| 83 |
```py |
| 84 |
model = fasttext.load_model("model_filename.bin") |
| 85 |
``` |
| 86 |
|
| 87 |
For more information about word representation usage of fasttext, you can refer to our [word representations tutorial](/docs/en/unsupervised-tutorial.html). |
| 88 |
|
| 89 |
|
| 90 |
## Text classification model |
| 91 |
|
| 92 |
In order to train a text classifier using the method [described here](/docs/en/references.html#bag-of-tricks-for-efficient-text-classification), we can use `fasttext.train_supervised` function like this: |
| 93 |
|
| 94 |
|
| 95 |
```py |
| 96 |
import fasttext |
| 97 |
|
| 98 |
model = fasttext.train_supervised('data.train.txt') |
| 99 |
``` |
| 100 |
|
| 101 |
where `data.train.txt` is a text file containing a training sentence per line along with the labels. By default, we assume that labels are words that are prefixed by the string `__label__` |
| 102 |
|
| 103 |
Once the model is trained, we can retrieve the list of words and labels: |
| 104 |
|
| 105 |
```py |
| 106 |
print(model.words) |
| 107 |
print(model.labels) |
| 108 |
``` |
| 109 |
|
| 110 |
To evaluate our model by computing the precision at 1 (P@1) and the recall on a test set, we use the `test` function: |
| 111 |
|
| 112 |
```py |
| 113 |
def print_results(N, p, r): |
| 114 |
print("N\t" + str(N)) |
| 115 |
print("P@{}\t{:.3f}".format(1, p)) |
| 116 |
print("R@{}\t{:.3f}".format(1, r)) |
| 117 |
|
| 118 |
print_results(*model.test('test.txt')) |
| 119 |
``` |
| 120 |
|
| 121 |
We can also predict labels for a specific text : |
| 122 |
|
| 123 |
```py |
| 124 |
model.predict("Which baking dish is best to bake a banana bread ?") |
| 125 |
``` |
| 126 |
|
| 127 |
By default, `predict` returns only one label : the one with the highest probability. You can also predict more than one label by specifying the parameter `k`: |
| 128 |
```py |
| 129 |
model.predict("Which baking dish is best to bake a banana bread ?", k=3) |
| 130 |
``` |
| 131 |
|
| 132 |
If you want to predict more than one sentence you can pass an array of strings : |
| 133 |
|
| 134 |
```py |
| 135 |
model.predict(["Which baking dish is best to bake a banana bread ?", "Why not put knives in the dishwasher?"], k=3) |
| 136 |
``` |
| 137 |
|
| 138 |
|
| 139 |
Of course, you can also save and load a model to/from a file as [in the word representation usage](#saving-and-loading-a-model-object). |
| 140 |
|
| 141 |
For more information about text classification usage of fasttext, you can refer to our [text classification tutorial](/docs/en/supervised-tutorial.html). |
| 142 |
|
| 143 |
|
| 144 |
|
| 145 |
|
| 146 |
### Compress model files with quantization |
| 147 |
|
| 148 |
When you want to save a supervised model file, fastText can compress it in order to have a much smaller model file by sacrificing only a little bit performance. |
| 149 |
|
| 150 |
```py |
| 151 |
# with the previously trained `model` object, call : |
| 152 |
model.quantize(input='data.train.txt', retrain=True) |
| 153 |
|
| 154 |
# then display results and save the new model : |
| 155 |
print_results(*model.test(valid_data)) |
| 156 |
model.save_model("model_filename.ftz") |
| 157 |
``` |
| 158 |
|
| 159 |
`model_filename.ftz` will have a much smaller size than `model_filename.bin`. |
| 160 |
|
| 161 |
For further reading on quantization, you can refer to [this paragraph from our blog post](/blog/2017/10/02/blog-post.html#model-compression). |
| 162 |
|
| 163 |
|
| 164 |
## IMPORTANT: Preprocessing data / encoding conventions |
| 165 |
|
| 166 |
In general it is important to properly preprocess your data. In particular our example scripts in the [root folder](https://github.com/facebookresearch/fastText) do this. |
| 167 |
|
| 168 |
fastText assumes UTF-8 encoded text. All text must be [unicode for Python2](https://docs.python.org/2/library/functions.html#unicode) and [str for Python3](https://docs.python.org/3.5/library/stdtypes.html#textseq). The passed text will be [encoded as UTF-8 by pybind11](https://pybind11.readthedocs.io/en/master/advanced/cast/strings.html?highlight=utf-8#strings-bytes-and-unicode-conversions) before passed to the fastText C++ library. This means it is important to use UTF-8 encoded text when building a model. On Unix-like systems you can convert text using [iconv](https://en.wikipedia.org/wiki/Iconv). |
| 169 |
|
| 170 |
fastText will tokenize (split text into pieces) based on the following ASCII characters (bytes). In particular, it is not aware of UTF-8 whitespace. We advice the user to convert UTF-8 whitespace / word boundaries into one of the following symbols as appropiate. |
| 171 |
|
| 172 |
* space |
| 173 |
* tab |
| 174 |
* vertical tab |
| 175 |
* carriage return |
| 176 |
* formfeed |
| 177 |
* the null character |
| 178 |
|
| 179 |
The newline character is used to delimit lines of text. In particular, the EOS token is appended to a line of text if a newline character is encountered. The only exception is if the number of tokens exceeds the MAX\_LINE\_SIZE constant as defined in the [Dictionary header](https://github.com/facebookresearch/fastText/blob/master/src/dictionary.h). This means if you have text that is not separate by newlines, such as the [fil9 dataset](http://mattmahoney.net/dc/textdata), it will be broken into chunks with MAX\_LINE\_SIZE of tokens and the EOS token is not appended. |
| 180 |
|
| 181 |
The length of a token is the number of UTF-8 characters by considering the [leading two bits of a byte](https://en.wikipedia.org/wiki/UTF-8#Description) to identify [subsequent bytes of a multi-byte sequence](https://github.com/facebookresearch/fastText/blob/master/src/dictionary.cc). Knowing this is especially important when choosing the minimum and maximum length of subwords. Further, the EOS token (as specified in the [Dictionary header](https://github.com/facebookresearch/fastText/blob/master/src/dictionary.h)) is considered a character and will not be broken into subwords. |
| 182 |
|
| 183 |
## More examples |
| 184 |
|
| 185 |
In order to have a better knowledge of fastText models, please consider the main [README](https://github.com/facebookresearch/fastText/blob/master/README.md) and in particular [the tutorials on our website](https://fasttext.cc/docs/en/supervised-tutorial.html). |
| 186 |
|
| 187 |
You can find further python examples in [the doc folder](https://github.com/facebookresearch/fastText/tree/master/python/doc/examples). |
| 188 |
|
| 189 |
As with any package you can get help on any Python function using the help function. |
| 190 |
|
| 191 |
For example |
| 192 |
|
| 193 |
``` |
| 194 |
+>>> import fasttext |
| 195 |
+>>> help(fasttext.FastText) |
| 196 |
|
| 197 |
Help on module fasttext.FastText in fasttext: |
| 198 |
|
| 199 |
NAME |
| 200 |
fasttext.FastText |
| 201 |
|
| 202 |
DESCRIPTION |
| 203 |
# Copyright (c) 2017-present, Facebook, Inc. |
| 204 |
# All rights reserved. |
| 205 |
# |
| 206 |
# This source code is licensed under the MIT license found in the |
| 207 |
# LICENSE file in the root directory of this source tree. |
| 208 |
|
| 209 |
FUNCTIONS |
| 210 |
load_model(path) |
| 211 |
Load a model given a filepath and return a model object. |
| 212 |
|
| 213 |
tokenize(text) |
| 214 |
Given a string of text, tokenize it and return a list of tokens |
| 215 |
[...] |
| 216 |
``` |
| 217 |
|
| 218 |
|
| 219 |
# API |
| 220 |
|
| 221 |
|
| 222 |
## `train_unsupervised` parameters |
| 223 |
|
| 224 |
```python |
| 225 |
input # training file path (required) |
| 226 |
model # unsupervised fasttext model {cbow, skipgram} [skipgram] |
| 227 |
lr # learning rate [0.05] |
| 228 |
dim # size of word vectors [100] |
| 229 |
ws # size of the context window [5] |
| 230 |
epoch # number of epochs [5] |
| 231 |
minCount # minimal number of word occurences [5] |
| 232 |
minn # min length of char ngram [3] |
| 233 |
maxn # max length of char ngram [6] |
| 234 |
neg # number of negatives sampled [5] |
| 235 |
wordNgrams # max length of word ngram [1] |
| 236 |
loss # loss function {ns, hs, softmax, ova} [ns] |
| 237 |
bucket # number of buckets [2000000] |
| 238 |
thread # number of threads [number of cpus] |
| 239 |
lrUpdateRate # change the rate of updates for the learning rate [100] |
| 240 |
t # sampling threshold [0.0001] |
| 241 |
verbose # verbose [2] |
| 242 |
``` |
| 243 |
|
| 244 |
## `train_supervised` parameters |
| 245 |
|
| 246 |
```python |
| 247 |
input # training file path (required) |
| 248 |
lr # learning rate [0.1] |
| 249 |
dim # size of word vectors [100] |
| 250 |
ws # size of the context window [5] |
| 251 |
epoch # number of epochs [5] |
| 252 |
minCount # minimal number of word occurences [1] |
| 253 |
minCountLabel # minimal number of label occurences [1] |
| 254 |
minn # min length of char ngram [0] |
| 255 |
maxn # max length of char ngram [0] |
| 256 |
neg # number of negatives sampled [5] |
| 257 |
wordNgrams # max length of word ngram [1] |
| 258 |
loss # loss function {ns, hs, softmax, ova} [softmax] |
| 259 |
bucket # number of buckets [2000000] |
| 260 |
thread # number of threads [number of cpus] |
| 261 |
lrUpdateRate # change the rate of updates for the learning rate [100] |
| 262 |
t # sampling threshold [0.0001] |
| 263 |
label # label prefix ['__label__'] |
| 264 |
verbose # verbose [2] |
| 265 |
pretrainedVectors # pretrained word vectors (.vec file) for supervised learning [] |
| 266 |
``` |
| 267 |
|
| 268 |
## `model` object |
| 269 |
|
| 270 |
`train_supervised`, `train_unsupervised` and `load_model` functions return an instance of `_FastText` class, that we generaly name `model` object. |
| 271 |
|
| 272 |
This object exposes those training arguments as properties : `lr`, `dim`, `ws`, `epoch`, `minCount`, `minCountLabel`, `minn`, `maxn`, `neg`, `wordNgrams`, `loss`, `bucket`, `thread`, `lrUpdateRate`, `t`, `label`, `verbose`, `pretrainedVectors`. So `model.wordNgrams` will give you the max length of word ngram used for training this model. |
| 273 |
|
| 274 |
In addition, the object exposes several functions : |
| 275 |
|
| 276 |
```python |
| 277 |
get_dimension # Get the dimension (size) of a lookup vector (hidden layer). |
| 278 |
# This is equivalent to `dim` property. |
| 279 |
get_input_vector # Given an index, get the corresponding vector of the Input Matrix. |
| 280 |
get_input_matrix # Get a copy of the full input matrix of a Model. |
| 281 |
get_labels # Get the entire list of labels of the dictionary |
| 282 |
# This is equivalent to `labels` property. |
| 283 |
get_line # Split a line of text into words and labels. |
| 284 |
get_output_matrix # Get a copy of the full output matrix of a Model. |
| 285 |
get_sentence_vector # Given a string, get a single vector represenation. This function |
| 286 |
# assumes to be given a single line of text. We split words on |
| 287 |
# whitespace (space, newline, tab, vertical tab) and the control |
| 288 |
# characters carriage return, formfeed and the null character. |
| 289 |
get_subword_id # Given a subword, return the index (within input matrix) it hashes to. |
| 290 |
get_subwords # Given a word, get the subwords and their indicies. |
| 291 |
get_word_id # Given a word, get the word id within the dictionary. |
| 292 |
get_word_vector # Get the vector representation of word. |
| 293 |
get_words # Get the entire list of words of the dictionary |
| 294 |
# This is equivalent to `words` property. |
| 295 |
is_quantized # whether the model has been quantized |
| 296 |
predict # Given a string, get a list of labels and a list of corresponding probabilities. |
| 297 |
quantize # Quantize the model reducing the size of the model and it's memory footprint. |
| 298 |
save_model # Save the model to the given path |
| 299 |
test # Evaluate supervised model using file given by path |
| 300 |
test_label # Return the precision and recall score for each label. |
| 301 |
``` |
| 302 |
|
| 303 |
The properties `words`, `labels` return the words and labels from the dictionary : |
| 304 |
```py |
| 305 |
model.words # equivalent to model.get_words() |
| 306 |
model.labels # equivalent to model.get_labels() |
| 307 |
``` |
| 308 |
|
| 309 |
The object overrides `__getitem__` and `__contains__` functions in order to return the representation of a word and to check if a word is in the vocabulary. |
| 310 |
|
| 311 |
```py |
| 312 |
model['king'] # equivalent to model.get_word_vector('king') |
| 313 |
'king' in model # equivalent to `'king' in model.get_words()` |
| 314 |
``` |