Debian Codegraph @

texmf-dist/doc/support/lua-uca/README.md

raw
1 \iffalse
2 # The `Lua-UCA` package
3 \fi
4
5 This package adds support for the [Unicode collation algorithm](https://unicode.org/reports/tr10/) for Lua 5.3 and later.
6 It is mainly intended for use with Lua\TeX and working \TeX\ distribution, but it can work also as a standalone
7 Lua module. You will need to install a required [Lua-uni-algos](https://github.com/latex3/lua-uni-algos) package by hand
8 in that case.
9
10
11 ## Usage
12
13 To sort a table using Czech collation rules:
14
15
16 kpse.set_program_name "luatex"
17 local ducet = require "lua-uca.lua-uca-ducet"
18 local collator = require "lua-uca.lua-uca-collator"
19 local languages = require "lua-uca.lua-uca-languages"
20
21 local collator_obj = collator.new(ducet)
22 -- load Czech rules
23 collator_obj = languages.cs(collator_obj)
24
25 local t = {"cihla", "chochol", "hudba", "jasan", "čáp"}
26
27 table.sort(t, function(a,b)
28 return collator_obj:compare_strings(a,b)
29 end)
30
31 for _, v in ipairs(t) do
32 print(v)
33 end
34
35 The output:
36
37 > cihla
38 > čáp
39 > hudba
40 > chochol
41 > jasan
42
43 More samples of the library usage can be found in the source repository of this package on [Github](https://github.com/michal-h21/lua-uca).
44 % See `HACKING.md` file in the repo for more information.
45
46 ## Use with Xindex processor
47
48 [Xindex](https://www.ctan.org/pkg/xindex) is flexible index processor written
49 in Lua by Herbert Voß. It has built-in `Lua-UCA` support starting with version
50 `0.23`. The support can be requested using the `-u` option:
51
52 xindex -u -l no -c norsk filename.idx
53
54 ## Use with LuaJIT
55
56 The default version of `lua-uca-ducet` fails with Luajit. You can use alternative version of this file, `lua-uca-ducet-jit`.
57
58 ## Change sorting rules
59
60 The simplest way to change the default sorting order is to use the
61 `tailor_string` method of the `collator_obj` object. It updates the collator object using
62 special syntax which is subset of the format used by the [Unicode locale data
63 markup
64 language](https://www.unicode.org/reports/tr35/tr35-collation.html#Orderings).
65
66 collator_obj:tailor_string "&a<b"
67
68 Full example with Czech rules:
69
70 kpse.set_program_name "luatex"
71 local ducet = require "lua-uca.lua-uca-ducet"
72 local collator = require "lua-uca.lua-uca-collator"
73 local languages = require "lua-uca.lua-uca-languages"
74
75 local collator_obj = collator.new(ducet)
76 local tailoring = function(s) collator_obj:tailor_string(s) end
77
78 tailoring "&c<č<<<Č"
79 tailoring "&h<ch<<<cH<<<Ch<<<CH"
80 tailoring "&R<ř<<<Ř"
81 tailoring "&s<š<<<Š"
82 tailoring "&z<ž<<<Ž"
83
84 Note that the sequence of letters `ch`, `Ch`, `cH` and `CH` will be sorted after `h`
85
86 It is also possible to expand a letter to multiple letters, like this example for DIN 2:
87
88
89 tailoring "&Ö=Oe"
90 tailoring "&ö=oe"
91
92 Some languages, like Norwegian, sort uppercase letters before lowercase. This
93 can be enabled using `collator_obj:uppercase_first()` function:
94
95 local tailoring = function(s) collator_obj:tailor_string(s) end
96 collator_obj:uppercase_first()
97 tailoring("&D<<đ<<<Đ<<ð<<<Ð")
98 tailoring("&th<<<þ")
99 tailoring("&TH<<<Þ")
100 tailoring("&Y<<ü<<<Ü<<ű<<<Ű")
101 tailoring("&ǀ<æ<<<Æ<<ä<<<Ä<ø<<<Ø<<ö<<<Ö<<ő<<<Ő<å<<<Å<<<aa<<<Aa<<<AA")
102 tailoring("&oe<<œ<<<Œ")
103
104
105 Some languages, for example Canadian French, sort accent backwards, like gêne < gëne < gêné.
106 In this case, you can set the `collator_obj.accents_backward` variable to `true`.
107
108
109 % More information on a new language support is in the `HACKING.md`
110 % document in the [`Lua-UCA` Github repo](https://github.com/michal-h21/lua-uca/blob/master/HACKING.md).
111
112 ### Script reordering
113
114 Many languages sort different scripts after the script this language uses. As
115 Latin based scripts are sorted first, it is necessary to reorder scripts in
116 such cases.
117
118 The `collator_obj:reorder` function takes table with scripts that need to be reordered.
119 For example Cyrillic can be sorted before Latin using:
120
121 collator_obj:reorder {"cyrillic"}
122
123 In German or Czech, numbers should be sorted after all other characters. This can be done using:
124
125 collator_obj:reorder {"others", "digits"}
126
127 The special keyword "others" means that the scripts that follows in the table
128 will be sorted at the very end.
129
130 ## Headers for index entries
131
132 In some languages, for example Czech, multiple letters may count as one
133 character. This is the case of the *ch* character.
134
135 Lua-UCA provides function `collator_obj:get_lowest_char()`. It returns table with UTF-8 codepoints
136 for correct first character for a given language that can be used for example as an index header.
137
138
139 local czech = collator.new(ducet)
140 languages.cs(czech)
141 -- first we need to convert string to codepoints
142 local codepoints = czech:string_to_codepoints("Chrobák")
143 local first_char = czech:get_lowest_char(codepoints)
144 -- it should print letters "ch"
145 print(utf8.char(table.unpack(first_char)))
146 -- you can also specify position of the character
147 local second_char = czech:get_lowest_char(codepoints, 2)
148 -- it should print letter "h", as it is second codepoint in the string
149 print(utf8.char(table.unpack(second_char)))
150
151
152 ## Unicode normalization
153
154 By default, no Unicode normalization is used internally. You can explicitly request normalization that use the
155 [Uninormalize package](https://ctan.org/pkg/uninormalize?lang=en). Note that it will significantly increase the
156 procesing time.
157
158 There are two normalization methods, NFC and NFD. They can be enabled using
159 `collation.use_nfc()` and `collation.use_nfd()` functions.
160
161
162 # What is missing
163
164 - Algorithm for setting implicit sort weights of characters that are not explicitly listed in DUCET.
165 - Special handling of CJK scripts.
166
167 \iffalse
168 # Copyright
169
170 Michal Hoftich, 2021–2024. See LICENSE file for more details.
171
172
173 \fi