| 1 |
# Supporting new or Updating languages # |
| 2 |
|
| 3 |
We generate statistical language data using Wikipedia as natural |
| 4 |
language text resource. |
| 5 |
|
| 6 |
Right now, we have automated scripts only to generate statistical data |
| 7 |
for single-byte encodings. Multi-byte encodings usually requires more |
| 8 |
in-depth knowledge of its specification. |
| 9 |
|
| 10 |
## New single-byte encoding ## |
| 11 |
|
| 12 |
Uchardet uses language data, and therefore rather than supporting a |
| 13 |
charset, we in fact support a couple (language, charset). So for |
| 14 |
instance if uchardet supports (French, ISO-8859-15), it should be able |
| 15 |
to recognize French text encoded in ISO-8859-15, but may fail at |
| 16 |
detecting ISO-8859-15 for non-supported languages. |
| 17 |
|
| 18 |
This is why, though less flexible, it also makes uchardet much more |
| 19 |
accurate than other detection system, as well as making it an efficient |
| 20 |
language recognition system. |
| 21 |
Since many single-byte charsets actually share the same layout (or very |
| 22 |
similar ones), it is actually impossible to have an accurate single-byte |
| 23 |
encoding detector for random text. |
| 24 |
|
| 25 |
Therefore you need to describe the language and the codepoint layouts of |
| 26 |
every charset you want to add support for. |
| 27 |
|
| 28 |
I recommend having a look at langs/fr.py which is heavily commented as |
| 29 |
a base of a new language description, and charsets/windows-1252.py as a |
| 30 |
base for a new charset layout (note that charset layouts can be shared |
| 31 |
between languages. If yours is already there, you have nothing to do). |
| 32 |
The important name in the charset file are: |
| 33 |
|
| 34 |
- `name`: an iconv-compatible name. |
| 35 |
- `charmap`: fill it with CTR (control character), SYM (symbol), NUM |
| 36 |
(number), LET (letter), ILL (illegal codepoint). |
| 37 |
|
| 38 |
## Tools ## |
| 39 |
|
| 40 |
You must install Python 3 and the [`Wikipedia` Python |
| 41 |
tool](https://github.com/goldsmith/Wikipedia). |
| 42 |
|
| 43 |
## Run script ## |
| 44 |
|
| 45 |
Let's say you added (or modified) support for French (`fr`), run: |
| 46 |
|
| 47 |
> ./BuildLangModel.py fr --max-page=100 --max-depth=4 |
| 48 |
|
| 49 |
The options can be changed to any value. Bigger values mean the script |
| 50 |
will process more data, so more processing time now, but uchardet may |
| 51 |
possibly be more accurate in the end. |
| 52 |
|
| 53 |
## Updating core code ## |
| 54 |
|
| 55 |
If you were only updating data for a language model, you have nothing |
| 56 |
else to do. Just build `uchardet` again and test it. |
| 57 |
|
| 58 |
If you were creating new models though, you will have to add these in |
| 59 |
src/nsSBCSGroupProber.cpp and src/nsSBCharSetProber.h, and increase the |
| 60 |
value of `NUM_OF_SBCS_PROBERS` in src/nsSBCSGroupProber.h. |
| 61 |
Finally add the new file in src/CMakeLists.txt. |
| 62 |
|
| 63 |
I will be looking to make this step more straightforward in the future. |