| 1 |
= HOWTO: Updating character attributes to the latest Unicode standard |
| 2 |
|
| 3 |
Gauche uses some of character attributes defined in Unicode |
| 4 |
standard. They are extracted from files provided as Unicode |
| 5 |
Character Database (UCD), but we have it in packed format for |
| 6 |
the space efficiency. |
| 7 |
|
| 8 |
We use two stage build process for it: |
| 9 |
|
| 10 |
|
| 11 |
---- |
| 12 |
[UCD data files] |
| 13 |
| |
| 14 |
| |
| 15 |
make char-data |
| 16 |
| |
| 17 |
v |
| 18 |
[src/unicode-data.scm] |
| 19 |
| |
| 20 |
| |
| 21 |
make |
| 22 |
| |
| 23 |
v |
| 24 |
[src/char_attr.c] |
| 25 |
[src/gauche/priv/unicode_attr.h] |
| 26 |
---- |
| 27 |
|
| 28 |
The first step reads UCD data files and generates `src/unicode-data.scm`, |
| 29 |
which only includes necessary information. |
| 30 |
This step is only needed to be done when Unicode is updated; so we don't |
| 31 |
require developers to do it. We check in `src/unicode-data.scm` to |
| 32 |
the repo. |
| 33 |
|
| 34 |
The second step is run when you build Gauche from the repository |
| 35 |
source tree. When creating tarball, though, we pre-generate |
| 36 |
the final `char_attr.c` and `unicode_attr.h` so that those who build |
| 37 |
from the tarball won't need preinstalled `gosh`. |
| 38 |
|
| 39 |
The rest of this document explains when you need to regenerate |
| 40 |
`src/unicode-data.scm`. |
| 41 |
|
| 42 |
|
| 43 |
== Preparing UCD data files |
| 44 |
|
| 45 |
You need to obtain the following files from |
| 46 |
link:http://www.unicode.org/Public/UCD/latest/ucd/[the Unicode site]. |
| 47 |
|
| 48 |
---- |
| 49 |
UnicodeData.txt |
| 50 |
SpecialCasing.txt |
| 51 |
PropList.txt |
| 52 |
EastAsianWidth.txt |
| 53 |
auxiliary/GraphemeBreakProperty.txt |
| 54 |
auxiliary/WordBreakProperty.txt |
| 55 |
---- |
| 56 |
|
| 57 |
Save those files in some directory (we call it `$UNICODEDIR`). |
| 58 |
Note that you need to keep subdirectory (`auxiliary/`). |
| 59 |
|
| 60 |
The list of files may grow in future versions of Gauche. |
| 61 |
|
| 62 |
For your convenience, the script `src/gen-unicode.scm` can download |
| 63 |
those files: |
| 64 |
|
| 65 |
[source,console] |
| 66 |
---- |
| 67 |
$ gosh src/gen-unicode.scm --fetch $UNICODEDIR [$UNICODE_VERSION] |
| 68 |
---- |
| 69 |
|
| 70 |
This downloads those files into `$UNICODEDIR`. By default it downloads |
| 71 |
the latest version; however, if the changes in newer versions of Unicode |
| 72 |
trip the build process, you can give version number in the second argument |
| 73 |
to download older versions of the data files. |
| 74 |
|
| 75 |
|
| 76 |
== Review the changes |
| 77 |
|
| 78 |
Property values may be added in the newer Unicode version, |
| 79 |
so you want to check `src/gauche/char_attr.h` and the beginning of |
| 80 |
`src/gen-unicode.scm` to see if we need to update the code. We take advantage |
| 81 |
of property value distribution over codepoints for space-efficient |
| 82 |
encoding (e.g. using flat tables on some ranges and binary trees |
| 83 |
on others). You may need to tweak the code if new ranges of |
| 84 |
characters are added. |
| 85 |
|
| 86 |
Check changes in link:http://www.unicode.org/reports/tr29/[TR29], too. |
| 87 |
The state-transition table is hardcoded in `ext/text/unicode.scm` |
| 88 |
(they can't be extracted from UCD files). Any change in TR29 |
| 89 |
must be manually reflected to the code. |
| 90 |
|
| 91 |
|
| 92 |
== Generate `unicode-data.scm` |
| 93 |
|
| 94 |
There's a make rule for that. |
| 95 |
|
| 96 |
[source,console] |
| 97 |
---- |
| 98 |
$ cd src |
| 99 |
$ make UNICODEDATA=$UNICODEDIR char-data |
| 100 |
---- |
| 101 |
|
| 102 |
And it's done! |
| 103 |
|
| 104 |
|
| 105 |
== Dealing with `unicode-data.scm` |
| 106 |
|
| 107 |
The module `text.unicode.ucd` provides the utilities to read/write |
| 108 |
`unicode-data.scm` and access information in it. Its format is described |
| 109 |
in the source. However, it may be changed as Unicode evloves and/or |
| 110 |
Gauche's Unicode support is enhanced, so do not count on the current |
| 111 |
format. |
| 112 |
|
| 113 |
In case if you need to hack the unicode data extraction step, here's a |
| 114 |
brief description of `text.unicde.ucd`. |
| 115 |
|
| 116 |
The file `unicode-data.scm` is a serialized format of `unichar-db` record. |
| 117 |
You can read the file with `(ucd-load-db <iport>)` to get a `unichar-db` |
| 118 |
instance, and write the serialized format to the current output port |
| 119 |
with `(ucd-save-db <unichar-db>)`. |
| 120 |
|
| 121 |
Once you get a `unichar-db` record, you can query |
| 122 |
|
| 123 |
- `(ucd-get-entry <db> <codepoint>)` returns a `ucd-entry` instance, |
| 124 |
that holds main character attributes such as character's general category, |
| 125 |
case mapping, etc. |
| 126 |
- `(ucd-get-break-property <db> <codepoint>)` returns a `ucd-break-property` |
| 127 |
instance, that holds character's Grapheme_Break and Word_Break categories. |
| 128 |
- `(ucd-get-east-asian-width <db> <codepoint>)` returns a symbol of character's |
| 129 |
East Asian Width property. |