Debian Codegraph @

script/README

raw
1 # Supporting new or Updating languages #
2
3 We generate statistical language data using Wikipedia as natural
4 language text resource.
5
6 Right now, we have automated scripts only to generate statistical data
7 for single-byte encodings. Multi-byte encodings usually requires more
8 in-depth knowledge of its specification.
9
10 ## New single-byte encoding ##
11
12 Uchardet uses language data, and therefore rather than supporting a
13 charset, we in fact support a couple (language, charset). So for
14 instance if uchardet supports (French, ISO-8859-15), it should be able
15 to recognize French text encoded in ISO-8859-15, but may fail at
16 detecting ISO-8859-15 for non-supported languages.
17
18 This is why, though less flexible, it also makes uchardet much more
19 accurate than other detection system, as well as making it an efficient
20 language recognition system.
21 Since many single-byte charsets actually share the same layout (or very
22 similar ones), it is actually impossible to have an accurate single-byte
23 encoding detector for random text.
24
25 Therefore you need to describe the language and the codepoint layouts of
26 every charset you want to add support for.
27
28 I recommend having a look at langs/fr.py which is heavily commented as
29 a base of a new language description, and charsets/windows-1252.py as a
30 base for a new charset layout (note that charset layouts can be shared
31 between languages. If yours is already there, you have nothing to do).
32 The important name in the charset file are:
33
34 - `name`: an iconv-compatible name.
35 - `charmap`: fill it with CTR (control character), SYM (symbol), NUM
36 (number), LET (letter), ILL (illegal codepoint).
37
38 ## Tools ##
39
40 You must install Python 3 and the [`Wikipedia` Python
41 tool](https://github.com/goldsmith/Wikipedia).
42
43 ## Run script ##
44
45 Let's say you added (or modified) support for French (`fr`), run:
46
47 > ./BuildLangModel.py fr --max-page=100 --max-depth=4
48
49 The options can be changed to any value. Bigger values mean the script
50 will process more data, so more processing time now, but uchardet may
51 possibly be more accurate in the end.
52
53 ## Updating core code ##
54
55 If you were only updating data for a language model, you have nothing
56 else to do. Just build `uchardet` again and test it.
57
58 If you were creating new models though, you will have to add these in
59 src/nsSBCSGroupProber.cpp and src/nsSBCharSetProber.h, and increase the
60 value of `NUM_OF_SBCS_PROBERS` in src/nsSBCSGroupProber.h.
61 Finally add the new file in src/CMakeLists.txt.
62
63 I will be looking to make this step more straightforward in the future.