Debian Codegraph @

README

raw
1 # Lexd
2
3 A lexicon compiler specialising in non-suffixational morphologies.
4
5 This module compiles lexicons in a format loosely based on hfst-lexc and produces transducers in ATT format which are equivalent to those produced using the overgenerate-and-constrain approach with hfst-twolc (see [here](https://wiki.apertium.org/wiki/Morphotactic_constraints_with_twol) and [here](https://wiki.apertium.org/wiki/Replacement_for_flag_diacritics)). However, it is much faster (see below).
6
7 See [Usage.md](Usage.md) for the rule file syntax.
8
9 ## Installation
10
11 First, clone this repository.
12
13 To build, do
14 ```bash
15 ./autogen.sh
16 make
17 make install
18 ```
19
20 If installing to a system-wide path, you may want to run `sudo make install` instead for the last step.
21
22 To compile a lexicon file into a transducer, do
23 ```bash
24 lexd lexicon_file att_file
25 ```
26
27 To get a speed comparison, do
28 ```bash
29 make timing-test
30 ```
31
32 To run basic feature smoke-tests (fast), do
33 ```bash
34 make check
35 ```
36
37 ## Why is it faster?
38
39 When dealing with prefixes, the overgenerate-and-constrain approach initially builds a transducer like this:
40
41 ![transducer that overgenerates](https://github.com/apertium/lexd/raw/main/imgs/lexc.png)
42
43 Then composes that with a twolc rule to turn it into somehting like this:
44
45 ![correct transducer](https://github.com/apertium/lexd/raw/main/imgs/twoc.png)
46
47 But compiling the rule needed to do that can take hundreds of times longer than compiling the lexicon.
48
49 Lexd, meanwhile, makes multiple copies of the lexical portion and attaches one to each prefix, thus generating the second transducer directly in a similar amount of time to what is required to generate the first one.
50
51 | Language | Wamesa | Hebrew | Navajo | Lingala |
52 |---|---:|---:|---:|---:|
53 | Stems | 262 | 127 | 19 | 1470
54 | Total forms | 12576 | 2540 | 473 | 1649496
55 | Path restrictions | 14 | 10 | 17 | 19
56 | **Lexc + Twolc**
57 | Lexc compilation | 25ms | 15ms | 25ms | 230ms
58 | Twolc compilation | 10245ms | 1360ms | 8460ms | 275525ms
59 | Rule composition | 2050ms | 225ms | 1705ms | 45550ms
60 | Minimization | 65ms | 5ms | 20ms | 155ms |
61 | Total time | 12385ms | 1605ms | 10210ms | 321460ms |
62 | **Lexd**
63 | Lexd compilation | 210ms | 85ms | 10ms | 490ms |
64 | Format conversion | 30ms | 5ms | 5ms | 55ms |
65 | Total time | 240ms | 90ms | 15ms | 545ms |
66 | **Speedup** | 52x | 18x | 681x | 590x |