| 1 |
--- |
| 2 |
title: New release of python module |
| 3 |
author: Onur Çelebi |
| 4 |
authorURL: https://research.fb.com/people/celebi-onur/ |
| 5 |
authorFBID: 663146146 |
| 6 |
--- |
| 7 |
|
| 8 |
Today, we are happy to release a new version of the fastText python library. The main goal of this release is to merge two existing python modules: the official `fastText` module which was available on our github repository and the unofficial `fasttext` module which was available on pypi.org. We hope that this new version will address the confusion due to the previous existence of two similar, but different, python modules. |
| 9 |
|
| 10 |
The new version of our library is now available on [pypi.org](https://pypi.org/project/fasttext/) as well as on our github repository, and you can find [an overview of its API here](/docs/en/python-module.html). |
| 11 |
|
| 12 |
|
| 13 |
|
| 14 |
fastText vs fasttext: what happened? |
| 15 |
---------------------------------- |
| 16 |
There was an ongoing confusion among our user community about the existence of both `fastText` and `fasttext` modules. |
| 17 |
|
| 18 |
When fastText was first released in 2016, it was a command line only utility. Very soon, people wanted to use fastText's capabilities from python without having to call a binary for each action. In August 2016, [Bayu Aldi Yansyah](https://github.com/pyk), a developer outside of Facebook, published a python wrapper of fastText. His work was very helpful to a lot of people in our community and he published his unofficial python library on pypi with the pretty straighforward module name `fasttext` (note the lowercase `t`). |
| 19 |
|
| 20 |
Later, our team began to work on an official python binding of fastText, that was published under the same github repository as the C++ source code. However, the module name for this official library was `fastText` (note the uppercase `T`). |
| 21 |
|
| 22 |
Last year, Bayu Aldi Yansyah gave us admin access to the pypi project so that we could merge the two libraries. |
| 23 |
|
| 24 |
To sum up, we ended up with two libraries that had: |
| 25 |
|
| 26 |
- almost the same name |
| 27 |
- different APIs |
| 28 |
- different versions |
| 29 |
- different ways to install |
| 30 |
|
| 31 |
That was a very confusing situation for the community. |
| 32 |
|
| 33 |
What actions did we take? |
| 34 |
-------------------------- |
| 35 |
Today we are merging the two python libraries. We decided to keep the official API and top level functions such as `train_unsupervised` and `train_supervised` as well as returning numpy objects. We remove `cbow`, `skipgram` and `supervised` functions from the unofficial API. However, [we bring nice ideas](#wordvectormodel-and-supervisedmodel-objects) from the unofficial API to the official one. In particular, we liked the pythonic approach of `WordVectorModel`. This new python module is named `fasttext`, and is available on both [pypi](https://pypi.org/project/fasttext/) and our [github](https://github.com/facebookresearch/fastText) repository. |
| 36 |
|
| 37 |
From now, we will refer to the tool as "fastText", however the name of the python module is `fasttext`. |
| 38 |
|
| 39 |
|
| 40 |
|
| 41 |
What is the right way to do now? |
| 42 |
-------------------------------- |
| 43 |
|
| 44 |
Before, you would either use `fastText` (uppercase `T`): |
| 45 |
```python |
| 46 |
import fastText |
| 47 |
# and call: |
| 48 |
fastText.train_supervised |
| 49 |
fastText.train_unsupervised |
| 50 |
``` |
| 51 |
|
| 52 |
or use `fasttext` (lowercase `t`): |
| 53 |
```python |
| 54 |
import fasttext |
| 55 |
# and call: |
| 56 |
fasttext.cbow |
| 57 |
fasttext.skipgram |
| 58 |
fasttext.supervised |
| 59 |
``` |
| 60 |
|
| 61 |
|
| 62 |
Now, the right way to do is to |
| 63 |
`import fasttext` (lowercase `t`) |
| 64 |
and use |
| 65 |
```python |
| 66 |
import fasttext |
| 67 |
# and call: |
| 68 |
fasttext.train_supervised |
| 69 |
fasttext.train_unsupervised |
| 70 |
``` |
| 71 |
|
| 72 |
We are keeping the lowercase `fasttext` module name, while we keep the `fastText` API. |
| 73 |
|
| 74 |
This is because: |
| 75 |
|
| 76 |
- the standard way to name python modules is all lowercases |
| 77 |
- the API from `fastText` is exposing numpy arrays, which is widely used by the machine learning community. |
| 78 |
|
| 79 |
|
| 80 |
You can find a more comprehensive overview of our python API [here](/docs/en/python-module.html). |
| 81 |
|
| 82 |
Should I modify my existing code? |
| 83 |
--------------------------------- |
| 84 |
Depending on the version of the python module you were using, you might need to do some little modifications on your existing code. |
| 85 |
|
| 86 |
### 1) You were using the official `fastText` module: |
| 87 |
|
| 88 |
You don't have to do much. Just replace your `import fastText` lines by `import fasttext` and everything should work as usual. |
| 89 |
|
| 90 |
### 2) You were using the unofficial `fasttext` module: |
| 91 |
|
| 92 |
If you were using the functions `cbow`, `skipgram`, `supervised` and/or `WordVectorModel`, `SupervisedModel` objects, you were using the unofficial `fasttext` module. |
| 93 |
|
| 94 |
Updating your code should be pretty straightforward, but it still implies some little changes. |
| 95 |
|
| 96 |
#### `cbow` function: use `train_unsupervised` instead. |
| 97 |
For example, replace: |
| 98 |
|
| 99 |
``` |
| 100 |
fasttext.cbow("train.txt", "model_file", lr=0.05, dim=100, ws=5, epoch=5) |
| 101 |
``` |
| 102 |
with |
| 103 |
``` |
| 104 |
model = fasttext.train_unsupervised("train.txt", model='cbow', lr=0.05, dim=100, ws=5, epoch=5) |
| 105 |
model.save_model("model_file.bin") |
| 106 |
``` |
| 107 |
|
| 108 |
#### `skipgram` function: use `train_unsupervised` instead. |
| 109 |
For example, replace: |
| 110 |
|
| 111 |
``` |
| 112 |
fasttext.skipgram("train.txt", "model_file", lr=0.05, dim=100, ws=5, epoch=5) |
| 113 |
``` |
| 114 |
with |
| 115 |
``` |
| 116 |
model = fasttext.train_unsupervised("train.txt", model='skipgram', lr=0.05, dim=100, ws=5, epoch=5) |
| 117 |
model.save_model("model_file.bin") |
| 118 |
``` |
| 119 |
|
| 120 |
|
| 121 |
#### `supervised` function: use `train_supervised` instead |
| 122 |
For example, replace: |
| 123 |
``` |
| 124 |
fasttext.supervised("train.txt", "model_file", lr=0.1, dim=100, epoch=5, word_ngrams=2, loss='softmax') |
| 125 |
``` |
| 126 |
with |
| 127 |
``` |
| 128 |
model = fasttext.train_supervised("train.txt", lr=0.1, dim=100, epoch=5, , word_ngrams=2, loss='softmax') |
| 129 |
model.save_model("model_file.bin") |
| 130 |
``` |
| 131 |
|
| 132 |
#### Parameters |
| 133 |
|
| 134 |
- As you can see, you can use either `word_ngrams` or `wordNgrams` as parameter name. Because the parameter names from the unofficial API are mapped to the official ones: `min_count` to `minCount`, `word_ngrams` to `wordNgrams`, `lr_update_rate` to `lrUpdateRate`, `label_prefix` to `label` and `pretrained_vectors` to `pretrainedVectors`. |
| 135 |
- `silent` parameter is not supported. Use `verbose` parameter instead. |
| 136 |
- `encoding` parameter is not supported, every input should be encoded in `utf-8`. |
| 137 |
|
| 138 |
|
| 139 |
### `WordVectorModel` and `SupervisedModel` objects |
| 140 |
|
| 141 |
Instead of `WordVectorModel` and `SupervisedModel` objects, we return a model object that mimics some nice ideas from the unofficial API. |
| 142 |
|
| 143 |
```python |
| 144 |
model = fasttext.train_unsupervised("train.txt", model='skipgram') |
| 145 |
print(model.words) # list of words in dictionary |
| 146 |
print(model['king']) # get the vector of the word 'king' |
| 147 |
print('king' in model) # check if a word is in dictionary |
| 148 |
``` |
| 149 |
|
| 150 |
|
| 151 |
|
| 152 |
```python |
| 153 |
model = fasttext.train_supervised("train.txt") |
| 154 |
print(model.words) # list of words in dictionary |
| 155 |
print(model.labels) # list of labels |
| 156 |
``` |
| 157 |
|
| 158 |
The model object also contains the arguments of the training: |
| 159 |
|
| 160 |
```python |
| 161 |
print(model.epoch) |
| 162 |
print(model.loss) |
| 163 |
print(model.wordNgrams) |
| 164 |
``` |
| 165 |
|
| 166 |
Thank you! |
| 167 |
------------ |
| 168 |
We want to thank our incredible community. We truly appreciate your feedback, a big thank you to everyone reporting issues and contributing to the project. In particular we want to express how grateful we are to [Bayu Aldi Yansyah](https://github.com/pyk) who did a great job with his python library and for giving us the ownership of the pypi `fasttext` project. |