Debian Codegraph @

doc/HOWTO-unicode.adoc

raw
1 = HOWTO: Updating character attributes to the latest Unicode standard
2
3 Gauche uses some of character attributes defined in Unicode
4 standard. They are extracted from files provided as Unicode
5 Character Database (UCD), but we have it in packed format for
6 the space efficiency.
7
8 We use two stage build process for it:
9
10
11 ----
12 [UCD data files]
13 |
14 |
15 make char-data
16 |
17 v
18 [src/unicode-data.scm]
19 |
20 |
21 make
22 |
23 v
24 [src/char_attr.c]
25 [src/gauche/priv/unicode_attr.h]
26 ----
27
28 The first step reads UCD data files and generates `src/unicode-data.scm`,
29 which only includes necessary information.
30 This step is only needed to be done when Unicode is updated; so we don't
31 require developers to do it. We check in `src/unicode-data.scm` to
32 the repo.
33
34 The second step is run when you build Gauche from the repository
35 source tree. When creating tarball, though, we pre-generate
36 the final `char_attr.c` and `unicode_attr.h` so that those who build
37 from the tarball won't need preinstalled `gosh`.
38
39 The rest of this document explains when you need to regenerate
40 `src/unicode-data.scm`.
41
42
43 == Preparing UCD data files
44
45 You need to obtain the following files from
46 link:http://www.unicode.org/Public/UCD/latest/ucd/[the Unicode site].
47
48 ----
49 UnicodeData.txt
50 SpecialCasing.txt
51 PropList.txt
52 EastAsianWidth.txt
53 auxiliary/GraphemeBreakProperty.txt
54 auxiliary/WordBreakProperty.txt
55 ----
56
57 Save those files in some directory (we call it `$UNICODEDIR`).
58 Note that you need to keep subdirectory (`auxiliary/`).
59
60 The list of files may grow in future versions of Gauche.
61
62 For your convenience, the script `src/gen-unicode.scm` can download
63 those files:
64
65 [source,console]
66 ----
67 $ gosh src/gen-unicode.scm --fetch $UNICODEDIR [$UNICODE_VERSION]
68 ----
69
70 This downloads those files into `$UNICODEDIR`. By default it downloads
71 the latest version; however, if the changes in newer versions of Unicode
72 trip the build process, you can give version number in the second argument
73 to download older versions of the data files.
74
75
76 == Review the changes
77
78 Property values may be added in the newer Unicode version,
79 so you want to check `src/gauche/char_attr.h` and the beginning of
80 `src/gen-unicode.scm` to see if we need to update the code. We take advantage
81 of property value distribution over codepoints for space-efficient
82 encoding (e.g. using flat tables on some ranges and binary trees
83 on others). You may need to tweak the code if new ranges of
84 characters are added.
85
86 Check changes in link:http://www.unicode.org/reports/tr29/[TR29], too.
87 The state-transition table is hardcoded in `ext/text/unicode.scm`
88 (they can't be extracted from UCD files). Any change in TR29
89 must be manually reflected to the code.
90
91
92 == Generate `unicode-data.scm`
93
94 There's a make rule for that.
95
96 [source,console]
97 ----
98 $ cd src
99 $ make UNICODEDATA=$UNICODEDIR char-data
100 ----
101
102 And it's done!
103
104
105 == Dealing with `unicode-data.scm`
106
107 The module `text.unicode.ucd` provides the utilities to read/write
108 `unicode-data.scm` and access information in it. Its format is described
109 in the source. However, it may be changed as Unicode evloves and/or
110 Gauche's Unicode support is enhanced, so do not count on the current
111 format.
112
113 In case if you need to hack the unicode data extraction step, here's a
114 brief description of `text.unicde.ucd`.
115
116 The file `unicode-data.scm` is a serialized format of `unichar-db` record.
117 You can read the file with `(ucd-load-db <iport>)` to get a `unichar-db`
118 instance, and write the serialized format to the current output port
119 with `(ucd-save-db <unichar-db>)`.
120
121 Once you get a `unichar-db` record, you can query
122
123 - `(ucd-get-entry <db> <codepoint>)` returns a `ucd-entry` instance,
124 that holds main character attributes such as character's general category,
125 case mapping, etc.
126 - `(ucd-get-break-property <db> <codepoint>)` returns a `ucd-break-property`
127 instance, that holds character's Grapheme_Break and Word_Break categories.
128 - `(ucd-get-east-asian-width <db> <codepoint>)` returns a symbol of character's
129 East Asian Width property.