| 1 |
# Lexd |
| 2 |
|
| 3 |
A lexicon compiler specialising in non-suffixational morphologies. |
| 4 |
|
| 5 |
This module compiles lexicons in a format loosely based on hfst-lexc and produces transducers in ATT format which are equivalent to those produced using the overgenerate-and-constrain approach with hfst-twolc (see [here](https://wiki.apertium.org/wiki/Morphotactic_constraints_with_twol) and [here](https://wiki.apertium.org/wiki/Replacement_for_flag_diacritics)). However, it is much faster (see below). |
| 6 |
|
| 7 |
See [Usage.md](Usage.md) for the rule file syntax. |
| 8 |
|
| 9 |
## Installation |
| 10 |
|
| 11 |
First, clone this repository. |
| 12 |
|
| 13 |
To build, do |
| 14 |
```bash |
| 15 |
./autogen.sh |
| 16 |
make |
| 17 |
make install |
| 18 |
``` |
| 19 |
|
| 20 |
If installing to a system-wide path, you may want to run `sudo make install` instead for the last step. |
| 21 |
|
| 22 |
To compile a lexicon file into a transducer, do |
| 23 |
```bash |
| 24 |
lexd lexicon_file att_file |
| 25 |
``` |
| 26 |
|
| 27 |
To get a speed comparison, do |
| 28 |
```bash |
| 29 |
make timing-test |
| 30 |
``` |
| 31 |
|
| 32 |
To run basic feature smoke-tests (fast), do |
| 33 |
```bash |
| 34 |
make check |
| 35 |
``` |
| 36 |
|
| 37 |
## Why is it faster? |
| 38 |
|
| 39 |
When dealing with prefixes, the overgenerate-and-constrain approach initially builds a transducer like this: |
| 40 |
|
| 41 |
 |
| 42 |
|
| 43 |
Then composes that with a twolc rule to turn it into somehting like this: |
| 44 |
|
| 45 |
 |
| 46 |
|
| 47 |
But compiling the rule needed to do that can take hundreds of times longer than compiling the lexicon. |
| 48 |
|
| 49 |
Lexd, meanwhile, makes multiple copies of the lexical portion and attaches one to each prefix, thus generating the second transducer directly in a similar amount of time to what is required to generate the first one. |
| 50 |
|
| 51 |
| Language | Wamesa | Hebrew | Navajo | Lingala | |
| 52 |
|---|---:|---:|---:|---:| |
| 53 |
| Stems | 262 | 127 | 19 | 1470 |
| 54 |
| Total forms | 12576 | 2540 | 473 | 1649496 |
| 55 |
| Path restrictions | 14 | 10 | 17 | 19 |
| 56 |
| **Lexc + Twolc** |
| 57 |
| Lexc compilation | 25ms | 15ms | 25ms | 230ms |
| 58 |
| Twolc compilation | 10245ms | 1360ms | 8460ms | 275525ms |
| 59 |
| Rule composition | 2050ms | 225ms | 1705ms | 45550ms |
| 60 |
| Minimization | 65ms | 5ms | 20ms | 155ms | |
| 61 |
| Total time | 12385ms | 1605ms | 10210ms | 321460ms | |
| 62 |
| **Lexd** |
| 63 |
| Lexd compilation | 210ms | 85ms | 10ms | 490ms | |
| 64 |
| Format conversion | 30ms | 5ms | 5ms | 55ms | |
| 65 |
| Total time | 240ms | 90ms | 15ms | 545ms | |
| 66 |
| **Speedup** | 52x | 18x | 681x | 590x | |