Debian Codegraph @

docs/autotune.md

raw
1 ---
2 id: autotune
3 title: Automatic hyperparameter optimization
4 ---
5
6 As we saw in [the tutorial](/docs/en/supervised-tutorial.html#more-epochs-and-larger-learning-rate), finding the best hyperparameters is crucial for building efficient models. However, searching the best hyperparameters manually is difficult. Parameters are dependent and the effect of each parameter vary from one dataset to another.
7
8 FastText's autotune feature allows you to find automatically the best hyperparameters for your dataset.
9
10 # How to use it
11
12 In order to activate hyperparameter optimization, we must provide a validation file with the `-autotune-validation` argument.
13
14 For example, using the same data as our [tutorial example](/docs/en/supervised-tutorial.html#our-first-classifier), the autotune can be used in the following way:
15
16 <!--DOCUSAURUS_CODE_TABS-->
17 <!--Command line-->
18 ```sh
19 >> ./fasttext supervised -input cooking.train -output model_cooking -autotune-validation cooking.valid
20 ```
21 <!--Python-->
22 ```py
23 >>> import fasttext
24 >>> model = fasttext.train_supervised(input='cooking.train', autotuneValidationFile='cooking.valid')
25 ```
26 <!--END_DOCUSAURUS_CODE_TABS-->
27
28
29 Then, fastText will search the hyperparameters that gives the best f1-score on `cooking.valid` file:
30 ```sh
31 Progress: 100.0% Trials: 27 Best score: 0.406763 ETA: 0h 0m 0s
32 ```
33
34 Now we can test the obtained model with:
35 <!--DOCUSAURUS_CODE_TABS-->
36 <!--Command line-->
37 ```sh
38 >> ./fasttext test model_cooking.bin cooking.valid
39 N 3000
40 P@1 0.666
41 R@1 0.288
42 ```
43 <!--Python-->
44 ```py
45 >>> model.test("cooking.valid")
46 (3000L, 0.666, 0.288)
47 ```
48 <!--END_DOCUSAURUS_CODE_TABS-->
49
50
51 By default, the search will take 5 minutes. You can set the timeout in seconds with the `-autotune-duration` argument. For example, if you want to set the limit to 10 minutes:
52
53 <!--DOCUSAURUS_CODE_TABS-->
54 <!--Command line-->
55 ```sh
56 >> ./fasttext supervised -input cooking.train -output model_cooking -autotune-validation cooking.valid -autotune-duration 600
57 ```
58 <!--Python-->
59 ```py
60 >>> import fasttext
61 >>> model = fasttext.train_supervised(input='cooking.train', autotuneValidationFile='cooking.valid', autotuneDuration=600)
62 ```
63 <!--END_DOCUSAURUS_CODE_TABS-->
64
65
66 While autotuning, fastText displays the best f1-score found so far. If we decide to stop the tuning before the time limit, we can send one `SIGINT` signal (via `CTLR-C` for example). FastText will then finish the current training, and retrain with the best parameters found so far.
67
68
69
70 # Constrain model size
71
72 As you may know, fastText can compress the model with [quantization](/docs/en/cheatsheet.html#quantization). However, this compression task comes with its own [hyperparameters](/docs/en/options.html) (`-cutoff`, `-retrain`, `-qnorm`, `-qout`, `-dsub`) that have a consequence on the accuracy and the size of the final model.
73
74 Fortunately, autotune can also find the hyperparameters for this compression task while targeting the desired model size. To this end, we can set the `-autotune-modelsize` argument:
75
76 <!--DOCUSAURUS_CODE_TABS-->
77 <!--Command line-->
78 ```sh
79 >> ./fasttext supervised -input cooking.train -output model_cooking -autotune-validation cooking.valid -autotune-modelsize 2M
80 ```
81 This will produce a `.ftz` file with the best accuracy having the desired size:
82 ```sh
83 >> ls -la model_cooking.ftz
84 -rw-r--r--. 1 celebio users 1990862 Aug 25 05:39 model_cooking.ftz
85 >> ./fasttext test model_cooking.ftz cooking.valid
86 N 3000
87 P@1 0.57
88 R@1 0.246
89 ```
90 <!--Python-->
91 ```py
92 >>> import fasttext
93 >>> model = fasttext.train_supervised(input='cooking.train', autotuneValidationFile='cooking.valid', autotuneModelSize="2M")
94 ```
95 If you save the model, you will obtain a model file with the desired size:
96 ```py
97 >>> model.save_model("model_cooking.ftz")
98 >>> import os
99 >>> os.stat("model_cooking.ftz").st_size
100 1990862
101 >>> model.test("cooking.valid")
102 (3000L, 0.57, 0.246)
103 ```
104 <!--END_DOCUSAURUS_CODE_TABS-->
105
106
107 # How to set the optimization metric?
108
109 <!--DOCUSAURUS_CODE_TABS-->
110 <!--Command line-->
111 <br />
112 By default, autotune will test the validation file you provide, exactly the same way as `./fasttext test model_cooking.bin cooking.valid` and try to optimize to get the highest [f1-score](https://en.wikipedia.org/wiki/F1_score).
113
114 But, if we want to optimize the score of a specific label, say `__label__baking`, we can set the `-autotune-metric` argument:
115
116 ```sh
117 >> ./fasttext supervised -input cooking.train -output model_cooking -autotune-validation cooking.valid -autotune-metric f1:__label__baking
118 ```
119
120 This is equivalent to manually optimize the f1-score we get when we test with `./fasttext test-label model_cooking.bin cooking.valid | grep __label__baking` in command line.
121
122 Sometimes, you may be interested in predicting more than one label. For example, if you were optimizing the hyperparameters manually to get the best score to predict two labels, you would test with `./fasttext test model_cooking.bin cooking.valid 2`. You can also tell autotune to optimize the parameters by testing two labels with the `-autotune-predictions` argument.
123 <!--Python-->
124 <br />
125 By default, autotune will test the validation file you provide, exactly the same way as `model.test("cooking.valid")` and try to optimize to get the highest [f1-score](https://en.wikipedia.org/wiki/F1_score).
126
127 But, if we want to optimize the score of a specific label, say `__label__baking`, we can set the `autotuneMetric` argument:
128
129 ```py
130 >>> import fasttext
131 >>> model = fasttext.train_supervised(input='cooking.train', autotuneValidationFile='cooking.valid', autotuneMetric="f1:__label__baking")
132 ```
133
134 This is equivalent to manually optimize the f1-score we get when we test with `model.test_label('cooking.valid')['__label__baking']`.
135
136 Sometimes, you may be interested in predicting more than one label. For example, if you were optimizing the hyperparameters manually to get the best score to predict two labels, you would test with `model.test("cooking.valid", k=2)`. You can also tell autotune to optimize the parameters by testing two labels with the `autotunePredictions` argument.
137 <!--END_DOCUSAURUS_CODE_TABS-->
138
139 You can also force autotune to optimize for the best precision for a given recall, or the best recall for a given precision, for all labels, or for a specific label:
140
141 For example, in order to get the best precision at recall = `30%`:
142 ```sh
143 >> ./fasttext supervised [...] -autotune-metric precisionAtRecall:30
144 ```
145 And to get the best precision at recall = `30%` for the label `__label__baking`:
146 ```sh
147 >> ./fasttext supervised [...] -autotune-metric precisionAtRecall:30:__label__baking
148 ```
149
150 Similarly, you can use `recallAtPrecision`:
151 ```sh
152 >> ./fasttext supervised [...] -autotune-metric recallAtPrecision:30
153 >> ./fasttext supervised [...] -autotune-metric recallAtPrecision:30:__label__baking
154 ```
155
156