Debian Codegraph @

texmf-dist/doc/luatex/lua-uni-algos/lua-uni-algos.tex

raw
1 \documentclass{article}
2 \usepackage{doc, shortvrb, metalogo, hyperref, fontspec}
3 % \setmainfont{Noto Serif}
4 % \setmonofont{FreeMono}
5 \title{Unicode algorithms for Lua\TeX\thanks{This document corresponds to \pkg{lua-uni-algos} v0.5.}}
6 \author{Marcel Krüger\thanks{E-Mail: \href{mailto:[email protected]}{\nolinkurl{[email protected]}}}}
7 \MakeShortVerb\|
8 \newcommand\pkg{\texttt}
9 \begin{document}
10 \maketitle
11 Dealing with general Unicode encoded data comes with many challenges because it has to respect individual concerns of many different scripts and languages. The Unicode consortium maintains multiple useful algorithms which can sometimes make this task much easier.
12
13 \pkg{lua-uni-algos} tries to make the most fundamental algorithms available for authors of Lua-based packages to aid in handling Unicode data.
14
15 Currently this package implements:
16 \begin{description}
17 \item[Unicode normalization] Normalize a given Lua string into any of the normalization forms NFC, NFD, NFKC, or NFKD as specified in the Unicode standard, section 2.12.
18 \item[Case folding] Fold Unicode codepoints into a form which eliminates all case distinctions. This can be used for case-independent matching of strings. Not to be confused with case mapping which maps all characters to lower/upper/titlecase: In contrast to case mapping, case folding is mostly locale independent but does not give results which should be shown to users.
19 \item[Grapheme cluster segmentation] Identify a grapheme cluster, a unit of text which is perceived as a single character by typical users, according to the rules in UAX \#29, section 3.
20 \item[Word boundary segmentation] Identify word boundaries according to the rules in UAX \#29, section 4.
21 \end{description}
22 \section{Normalization}
23 Unicode normalization is handled by the Lua module |lua-uni-normalize|.
24 You can either load it directly with
25 \begin{verbatim}
26 local normalize = require'lua-uni-normalize'
27 \end{verbatim}
28 or if you need access to all implemented algorithms you can use
29 \begin{verbatim}
30 local uni_algos = require'lua-uni-algos'
31 local normalize = uni_algos.normalize
32 \end{verbatim}
33
34 Then, four functions are available: |normalize.NFC|, |normalize.NFD|, |normalize.NFKC|, and |normalize.NFKD|.
35 If you do not know which of these you need, then you should probably |normalize.NFC|. All functions are used in the same way:
36 \begin{verbatim}
37 local str = "Äpfel…"
38 print("Original:", str)
39 print("NFC:", normalize.NFC(str))
40 print("NFD:", normalize.NFD(str))
41 print("NFKC:", normalize.NFKC(str))
42 print("NFKD:", normalize.NFKD(str))
43 \end{verbatim}
44 This results in
45 \begin{verbatim}
46 Original: Äpfel…
47 NFC: Äpfel…
48 NFD: Äpfel…
49 NFKC: Äpfel...
50 NFKD: Äpfel...
51 \end{verbatim}
52 (This example is shown in Latin Modern Mono which has the (for this purpose) very useful property of not handling combining character very well.
53 In a well-behaving font, the `...C` and `...D` lines should look the same.)
54
55 Additionally for direct normalization of Lua\TeX\ node lists is supported.
56 There are two functions |normalize.node.NFC| and |normalize.direct.NFC| taking upto four parameters: The first parameter is the head of the node list to be converted.
57 The second parameter is the font id of the affected character nodes. Only non-protected glyph nodes of the specified font will be normalized. Pass |nil| for the font
58 to normalize without respecting the font in the process. The third parameter is an optional table. If it is not |nil|, normalization is supressed if it might add glyph
59 which map to |false| (or |nil|) in this table. If the forth argument is |true|, normalization will never join two glyph nodes with different attributes.
60
61 For NFD and NFKD equivalent functions exists without the last parameter (since they never compose nodes, they never have to deal with composing nodes with different
62 attributes.
63
64 NFKC is not supported for node list normalization since the author is not convinced that there is any usecase for it. (Probably there isn't any usecase for node list
65 NFKD normalization either, but that was easy to implement while NFKC would need separate data tables.
66
67 \section{Case folding}
68 For case folding load the Lua module |lua-uni-case|.
69 You can either load it directly with
70 \begin{verbatim}
71 local uni_case = require'lua-uni-case'
72 \end{verbatim}
73 or if you need access to all implemented algorithms you can use
74 \begin{verbatim}
75 local uni_algos = require'lua-uni-algos'
76 local uni_case = uni_algos.case
77 \end{verbatim}
78
79 The main function is |uni_case.casefold(str, full, special)|. It accepts three parameters: A Lua string |str| to be case folded, a boolean |full| to specify if the number of codepoints is allowed to change in the progress (This should normally be set to |true|.) and a boolean |special| which enables special handling for Turkish languages (In most cases, this should be set to |false|.)
80 The function returns the case-folded string:
81 \begin{verbatim}
82 local str = "Straße…"
83 print("Original:", str)
84 print("Case folded (full=false):", uni_case.casefold(str, false, false))
85 print("Case folded (full=true):", uni_case.casefold(str, true, false))
86 \end{verbatim}
87 This results in
88
89 \noindent\begingroup
90 \ttfamily
91 \directlua{
92 local uni_case = require'lua-uni-case'
93 local str = "Straße…"
94 tex.sprint("Original:", str, '\\\\')
95 tex.sprint("Case folded (full=false):", uni_case.casefold(str, false, false), '\\\\')
96 tex.sprint("Case folded (full=true):", uni_case.casefold(str, true, false), '\\\\')
97 }\par
98 \endgroup
99
100 In most cases, you will want to normalize the string after casefolding.
101
102 For cases where you want to casefold something which is not given as a Lua string, you can use the function |uni_case.casefold_lookup(cp, full, special)|. Instead of a string, it accepts a codepoint as first parameter and returns a table of codepoints. A string can be casefolded by replacing every codepoints with the sequence of codepoints returned by |uni_case.casefold_lookup|. If |casefold_lookup| returns |false| or |nil|, the codepoint should not be changed.
103
104 \section{Grapheme clusters}
105 Grapheme cluster handling is handled by the Lua module |lua-uni-graphemes|.
106 You can either load it directly with
107 \begin{verbatim}
108 local graphemes = require'lua-uni-graphemes'
109 \end{verbatim}
110 or if you need access to all implemented algorithms you can use
111 \begin{verbatim}
112 local uni_algos = require'lua-uni-algos'
113 local graphemes = uni_algos.graphemes
114 \end{verbatim}
115
116 Sometimes we want to look at a single character of a string, but identifying what a character is is not that easy in Unicode. A simple example is the character from the previous section: ``Ä''
117 The NFD form is certainly a single character, but is encoded using two codepoints: U+0041 (LATIN CAPITAL LETTER A) and U+0308 (COMBINING DIAERESIS). Or the Tamil letter Ni which is encoded as U+0BA8 (TAMIL LETTER NA) followed by U+0BBF (TAMIL VOWEL SIGN I). But sometimes it can be useful to identify characters, e.g.\ for letterspacing or letterines.
118
119 There are two main interfaces for this: One iterator for iterating over grapheme clusters and one direct interface to the underlying state machine:
120
121 \begin{verbatim}
122 for final, first, grapheme in graphemes.graphemes'Äpfel' do
123 print(grapheme)
124 end
125 \end{verbatim}
126 % \begin{verbatim}
127 % for final, first, grapheme in graphemes.graphemes'Z͑ͫ̓ͪ̂ͫ̽͏̴̙̤̞͉͚̯̞̠͍A̴̵̜̰͔ͫ͗͢L̠ͨͧͩ͘G̴̻͈͍͔̹̑͗̎̅͛́Ǫ̵̹̻̝̳͂̌̌͘!͖̬̰̙̗̿̋ͥͥ̂ͣ̐́́͜͞' do
128 % print(grapheme)
129 % end
130 % \end{verbatim}
131
132 \noindent\begingroup
133 \ttfamily
134 \directlua{
135 local graphemes = require'./lua-uni-graphemes'
136 for final, first, grapheme in graphemes.graphemes'Äpfel' do
137 tex.sprint(grapheme, '\string\\\string\\')
138 end
139 }\par
140 \endgroup
141
142 The more powerful state machine interface |graphemes.read_codepoint| takes two parameters: A new codepoint and a state.
143 At the beginning, the state can be omitted.
144 For every codepoint in your input, call the function with the new codepoint and the last state. Then there are two return values: The first one is a boolean telling you if the current codepoint is the beginning of a new cluster, the second is a new state you have to pass with the next codepoint.
145
146 So e.g.\ to find cluster boundaries in the Unicode codepoint sequence U+0041 U+0308 U+0BA8 U+0BBF you could use
147
148 \begin{verbatim}
149 local graphemes = require'lua-uni-graphemes'
150 local new_cluster, state
151 new_cluster, state = graphemes.read_codepoint(0x0041, state)
152 print(new_cluster)
153 new_cluster, state = graphemes.read_codepoint(0x0308, state)
154 print(new_cluster)
155 new_cluster, state = graphemes.read_codepoint(0x0BA8, state)
156 print(new_cluster)
157 new_cluster, state = graphemes.read_codepoint(0x0BBF, state)
158 print(new_cluster)
159 \end{verbatim}
160
161 \noindent resulting in
162
163 \noindent\begingroup
164 \ttfamily
165 \directlua{
166 local graphemes = require'lua-uni-graphemes'
167 local new_cluster, state
168 new_cluster, state = graphemes.read_codepoint(0x0041, state)
169 tex.sprint(tostring(new_cluster), '\string\\\string\\')
170 new_cluster, state = graphemes.read_codepoint(0x0308, state)
171 tex.sprint(tostring(new_cluster), '\string\\\string\\')
172 new_cluster, state = graphemes.read_codepoint(0x0BA8, state)
173 tex.sprint(tostring(new_cluster), '\string\\\string\\')
174 new_cluster, state = graphemes.read_codepoint(0x0BBF, state)
175 tex.sprint(tostring(new_cluster), '\string\\\string\\')
176 }\par
177 \endgroup
178
179 \vskip-\baselineskip
180 \noindent meaning the first and third codepoint start a new cluster.
181
182 Do not try to interpret the |state|, it has no defined values and might change at any point.
183
184 \section{Word boundaries}
185 Word segmentation is handled by the Lua module |lua-uni-words|.
186 You can either load it directly with
187 \begin{verbatim}
188 local words = require'lua-uni-words'
189 \end{verbatim}
190 or if you need access to all implemented algorithms you can use
191 \begin{verbatim}
192 local uni_algos = require'lua-uni-algos'
193 local words = uni_algos.words
194 \end{verbatim}
195
196 This is used to identify word boundaries. Unicode describes these as the boundaries users would expect when searching in a text for whole words.
197
198 The UAX also suggests that for some use-cases useful words can be determined from these by taking any segment between word boundaries and filtering out all segments ``containing only whitespace, punctuation and similar characters''.
199
200 Currently only the string interface is recommended:
201
202 \begin{verbatim}
203 for final, first, word_segment in words.word_boundaries'This text will be split into words segments!' do
204 print('"' .. word_segment .. '"')
205 end
206 \end{verbatim}
207 % \begin{verbatim}
208 % for final, first, word_segment in words.word_boundaries'Z͑ͫ̓ͪ̂ͫ̽͏̴̙̤̞͉͚̯̞̠͍A̴̵̜̰͔ͫ͗͢L̠ͨͧͩ͘G̴̻͈͍͔̹̑͗̎̅͛́Ǫ̵̹̻̝̳͂̌̌͘!͖̬̰̙̗̿̋ͥͥ̂ͣ̐́́͜͞' do
209 % print(grapheme)
210 % end
211 % \end{verbatim}
212
213 \noindent\begingroup
214 \ttfamily
215 \directlua{
216 local words = require'./lua-uni-words'
217 for final, first, word_segment in words.word_boundaries'This text will be split into words segments!' do
218 tex.sprint('"' .. word_segment .. '"\string\\\string\\')
219 end
220 }\par
221 \endgroup
222
223 \end{document}