| 1 |
\iffalse |
| 2 |
# The `Lua-UCA` package |
| 3 |
\fi |
| 4 |
|
| 5 |
This package adds support for the [Unicode collation algorithm](https://unicode.org/reports/tr10/) for Lua 5.3 and later. |
| 6 |
It is mainly intended for use with Lua\TeX and working \TeX\ distribution, but it can work also as a standalone |
| 7 |
Lua module. You will need to install a required [Lua-uni-algos](https://github.com/latex3/lua-uni-algos) package by hand |
| 8 |
in that case. |
| 9 |
|
| 10 |
|
| 11 |
## Usage |
| 12 |
|
| 13 |
To sort a table using Czech collation rules: |
| 14 |
|
| 15 |
|
| 16 |
kpse.set_program_name "luatex" |
| 17 |
local ducet = require "lua-uca.lua-uca-ducet" |
| 18 |
local collator = require "lua-uca.lua-uca-collator" |
| 19 |
local languages = require "lua-uca.lua-uca-languages" |
| 20 |
|
| 21 |
local collator_obj = collator.new(ducet) |
| 22 |
-- load Czech rules |
| 23 |
collator_obj = languages.cs(collator_obj) |
| 24 |
|
| 25 |
local t = {"cihla", "chochol", "hudba", "jasan", "čáp"} |
| 26 |
|
| 27 |
table.sort(t, function(a,b) |
| 28 |
return collator_obj:compare_strings(a,b) |
| 29 |
end) |
| 30 |
|
| 31 |
for _, v in ipairs(t) do |
| 32 |
print(v) |
| 33 |
end |
| 34 |
|
| 35 |
The output: |
| 36 |
|
| 37 |
> cihla |
| 38 |
> čáp |
| 39 |
> hudba |
| 40 |
> chochol |
| 41 |
> jasan |
| 42 |
|
| 43 |
More samples of the library usage can be found in the source repository of this package on [Github](https://github.com/michal-h21/lua-uca). |
| 44 |
% See `HACKING.md` file in the repo for more information. |
| 45 |
|
| 46 |
## Use with Xindex processor |
| 47 |
|
| 48 |
[Xindex](https://www.ctan.org/pkg/xindex) is flexible index processor written |
| 49 |
in Lua by Herbert Voß. It has built-in `Lua-UCA` support starting with version |
| 50 |
`0.23`. The support can be requested using the `-u` option: |
| 51 |
|
| 52 |
xindex -u -l no -c norsk filename.idx |
| 53 |
|
| 54 |
## Use with LuaJIT |
| 55 |
|
| 56 |
The default version of `lua-uca-ducet` fails with Luajit. You can use alternative version of this file, `lua-uca-ducet-jit`. |
| 57 |
|
| 58 |
## Change sorting rules |
| 59 |
|
| 60 |
The simplest way to change the default sorting order is to use the |
| 61 |
`tailor_string` method of the `collator_obj` object. It updates the collator object using |
| 62 |
special syntax which is subset of the format used by the [Unicode locale data |
| 63 |
markup |
| 64 |
language](https://www.unicode.org/reports/tr35/tr35-collation.html#Orderings). |
| 65 |
|
| 66 |
collator_obj:tailor_string "&a<b" |
| 67 |
|
| 68 |
Full example with Czech rules: |
| 69 |
|
| 70 |
kpse.set_program_name "luatex" |
| 71 |
local ducet = require "lua-uca.lua-uca-ducet" |
| 72 |
local collator = require "lua-uca.lua-uca-collator" |
| 73 |
local languages = require "lua-uca.lua-uca-languages" |
| 74 |
|
| 75 |
local collator_obj = collator.new(ducet) |
| 76 |
local tailoring = function(s) collator_obj:tailor_string(s) end |
| 77 |
|
| 78 |
tailoring "&c<č<<<Č" |
| 79 |
tailoring "&h<ch<<<cH<<<Ch<<<CH" |
| 80 |
tailoring "&R<ř<<<Ř" |
| 81 |
tailoring "&s<š<<<Š" |
| 82 |
tailoring "&z<ž<<<Ž" |
| 83 |
|
| 84 |
Note that the sequence of letters `ch`, `Ch`, `cH` and `CH` will be sorted after `h` |
| 85 |
|
| 86 |
It is also possible to expand a letter to multiple letters, like this example for DIN 2: |
| 87 |
|
| 88 |
|
| 89 |
tailoring "&Ö=Oe" |
| 90 |
tailoring "&ö=oe" |
| 91 |
|
| 92 |
Some languages, like Norwegian, sort uppercase letters before lowercase. This |
| 93 |
can be enabled using `collator_obj:uppercase_first()` function: |
| 94 |
|
| 95 |
local tailoring = function(s) collator_obj:tailor_string(s) end |
| 96 |
collator_obj:uppercase_first() |
| 97 |
tailoring("&D<<đ<<<Đ<<ð<<<Ð") |
| 98 |
tailoring("&th<<<þ") |
| 99 |
tailoring("&TH<<<Þ") |
| 100 |
tailoring("&Y<<ü<<<Ü<<ű<<<Ű") |
| 101 |
tailoring("&ǀ<æ<<<Æ<<ä<<<Ä<ø<<<Ø<<ö<<<Ö<<ő<<<Ő<å<<<Å<<<aa<<<Aa<<<AA") |
| 102 |
tailoring("&oe<<œ<<<Œ") |
| 103 |
|
| 104 |
|
| 105 |
Some languages, for example Canadian French, sort accent backwards, like gêne < gëne < gêné. |
| 106 |
In this case, you can set the `collator_obj.accents_backward` variable to `true`. |
| 107 |
|
| 108 |
|
| 109 |
% More information on a new language support is in the `HACKING.md` |
| 110 |
% document in the [`Lua-UCA` Github repo](https://github.com/michal-h21/lua-uca/blob/master/HACKING.md). |
| 111 |
|
| 112 |
### Script reordering |
| 113 |
|
| 114 |
Many languages sort different scripts after the script this language uses. As |
| 115 |
Latin based scripts are sorted first, it is necessary to reorder scripts in |
| 116 |
such cases. |
| 117 |
|
| 118 |
The `collator_obj:reorder` function takes table with scripts that need to be reordered. |
| 119 |
For example Cyrillic can be sorted before Latin using: |
| 120 |
|
| 121 |
collator_obj:reorder {"cyrillic"} |
| 122 |
|
| 123 |
In German or Czech, numbers should be sorted after all other characters. This can be done using: |
| 124 |
|
| 125 |
collator_obj:reorder {"others", "digits"} |
| 126 |
|
| 127 |
The special keyword "others" means that the scripts that follows in the table |
| 128 |
will be sorted at the very end. |
| 129 |
|
| 130 |
## Headers for index entries |
| 131 |
|
| 132 |
In some languages, for example Czech, multiple letters may count as one |
| 133 |
character. This is the case of the *ch* character. |
| 134 |
|
| 135 |
Lua-UCA provides function `collator_obj:get_lowest_char()`. It returns table with UTF-8 codepoints |
| 136 |
for correct first character for a given language that can be used for example as an index header. |
| 137 |
|
| 138 |
|
| 139 |
local czech = collator.new(ducet) |
| 140 |
languages.cs(czech) |
| 141 |
-- first we need to convert string to codepoints |
| 142 |
local codepoints = czech:string_to_codepoints("Chrobák") |
| 143 |
local first_char = czech:get_lowest_char(codepoints) |
| 144 |
-- it should print letters "ch" |
| 145 |
print(utf8.char(table.unpack(first_char))) |
| 146 |
-- you can also specify position of the character |
| 147 |
local second_char = czech:get_lowest_char(codepoints, 2) |
| 148 |
-- it should print letter "h", as it is second codepoint in the string |
| 149 |
print(utf8.char(table.unpack(second_char))) |
| 150 |
|
| 151 |
|
| 152 |
## Unicode normalization |
| 153 |
|
| 154 |
By default, no Unicode normalization is used internally. You can explicitly request normalization that use the |
| 155 |
[Uninormalize package](https://ctan.org/pkg/uninormalize?lang=en). Note that it will significantly increase the |
| 156 |
procesing time. |
| 157 |
|
| 158 |
There are two normalization methods, NFC and NFD. They can be enabled using |
| 159 |
`collation.use_nfc()` and `collation.use_nfd()` functions. |
| 160 |
|
| 161 |
|
| 162 |
# What is missing |
| 163 |
|
| 164 |
- Algorithm for setting implicit sort weights of characters that are not explicitly listed in DUCET. |
| 165 |
- Special handling of CJK scripts. |
| 166 |
|
| 167 |
\iffalse |
| 168 |
# Copyright |
| 169 |
|
| 170 |
Michal Hoftich, 2021–2024. See LICENSE file for more details. |
| 171 |
|
| 172 |
|
| 173 |
\fi |