Debian Codegraph @

upstream/opensuse-tumbleweed/man3/Unicode::Collate.3pm

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man v6.0.2 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\"
56 .\" Required to disable full justification in groff 1.23.0.
57 .if n .ds AD l
58 .\" ========================================================================
59 .\"
60 .IX Title "Unicode::Collate 3pm"
61 .TH Unicode::Collate 3pm 2026-03-09 "perl v5.42.1" "Perl Programmers Reference Guide"
62 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
63 .\" way too many mistakes in technical documents.
64 .if n .ad l
65 .nh
66 .SH NAME
67 Unicode::Collate \- Unicode Collation Algorithm
68 .SH SYNOPSIS
69 .IX Header "SYNOPSIS"
70 .Vb 1
71 \& use Unicode::Collate;
72 \&
73 \& #construct
74 \& $Collator = Unicode::Collate\->new(%tailoring);
75 \&
76 \& #sort
77 \& @sorted = $Collator\->sort(@not_sorted);
78 \&
79 \& #compare
80 \& $result = $Collator\->cmp($a, $b); # returns 1, 0, or \-1.
81 .Ve
82 .PP
83 \&\fBNote:\fR Strings in \f(CW@not_sorted\fR, \f(CW$a\fR and \f(CW$b\fR are interpreted
84 according to Perl\*(Aqs Unicode support. See perlunicode,
85 perluniintro, perlunitut, perlunifaq, utf8.
86 Otherwise you can use \f(CW\*(C`preprocess\*(C'\fR or should decode them before.
87 .SH DESCRIPTION
88 .IX Header "DESCRIPTION"
89 This module is an implementation of Unicode Technical Standard #10
90 (a.k.a. UTS #10) \- Unicode Collation Algorithm (a.k.a. UCA).
91 .SS "Constructor and Tailoring"
92 .IX Subsection "Constructor and Tailoring"
93 The \f(CW\*(C`new\*(C'\fR method returns a collator object. If \fBnew()\fR is called
94 with no parameters, the collator should do the default collation.
95 .PP
96 .Vb 10
97 \& $Collator = Unicode::Collate\->new(
98 \& UCA_Version => $UCA_Version,
99 \& alternate => $alternate, # alias for \*(Aqvariable\*(Aq
100 \& backwards => $levelNumber, # or \e@levelNumbers
101 \& entry => $element,
102 \& hangul_terminator => $term_primary_weight,
103 \& highestFFFF => $bool,
104 \& identical => $bool,
105 \& ignoreName => qr/$ignoreName/,
106 \& ignoreChar => qr/$ignoreChar/,
107 \& ignore_level2 => $bool,
108 \& katakana_before_hiragana => $bool,
109 \& level => $collationLevel,
110 \& long_contraction => $bool,
111 \& minimalFFFE => $bool,
112 \& normalization => $normalization_form,
113 \& overrideCJK => \e&overrideCJK,
114 \& overrideHangul => \e&overrideHangul,
115 \& preprocess => \e&preprocess,
116 \& rearrange => \e@charList,
117 \& rewrite => \e&rewrite,
118 \& suppress => \e@charList,
119 \& table => $filename,
120 \& undefName => qr/$undefName/,
121 \& undefChar => qr/$undefChar/,
122 \& upper_before_lower => $bool,
123 \& variable => $variable,
124 \& );
125 .Ve
126 .IP UCA_Version 4
127 .IX Item "UCA_Version"
128 If the revision (previously "tracking version") number of UCA is given,
129 behavior of that revision is emulated on collating.
130 If omitted, the return value of \f(CWUCA_Version()\fR is used.
131 .Sp
132 The following revisions are supported. The default is 43.
133 .Sp
134 .Vb 10
135 \& UCA Unicode Standard DUCET (@version)
136 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
137 \& 8 3.1 3.0.1 (3.0.1d9)
138 \& 9 3.1 with Corrigendum 3 3.1.1
139 \& 11 4.0.0
140 \& 14 4.1.0
141 \& 16 5.0.0
142 \& 18 5.1.0
143 \& 20 5.2.0
144 \& 22 6.0.0
145 \& 24 6.1.0
146 \& 26 6.2.0
147 \& 28 6.3.0
148 \& 30 7.0.0
149 \& 32 8.0.0
150 \& 34 9.0.0
151 \& 36 10.0.0
152 \& 38 11.0.0
153 \& 40 12.0.0
154 \& 41 12.1.0
155 \& 43 13.0.0
156 .Ve
157 .Sp
158 * See below for \f(CW\*(C`long_contraction\*(C'\fR with \f(CW\*(C`UCA_Version\*(C'\fR 22 and 24.
159 .Sp
160 * Noncharacters (e.g. U+FFFF) are not ignored, and can be overridden
161 since \f(CW\*(C`UCA_Version\*(C'\fR 22.
162 .Sp
163 * Out\-of\-range codepoints (greater than U+10FFFF) are not ignored,
164 and can be overridden since \f(CW\*(C`UCA_Version\*(C'\fR 22.
165 .Sp
166 * Fully ignorable characters were ignored, and would not interrupt
167 contractions with \f(CW\*(C`UCA_Version\*(C'\fR 9 and 11.
168 .Sp
169 * Treatment of ignorables after variables and some behaviors
170 were changed at \f(CW\*(C`UCA_Version\*(C'\fR 9.
171 .Sp
172 * Characters regarded as CJK unified ideographs (cf. \f(CW\*(C`overrideCJK\*(C'\fR)
173 depend on \f(CW\*(C`UCA_Version\*(C'\fR.
174 .Sp
175 * Many hangul jamo are assigned at \f(CW\*(C`UCA_Version\*(C'\fR 20, that will affect
176 \&\f(CW\*(C`hangul_terminator\*(C'\fR.
177 .IP alternate 4
178 .IX Item "alternate"
179 \&\-\- see 3.2.2 Alternate Weighting, version 8 of UTS #10
180 .Sp
181 For backward compatibility, \f(CW\*(C`alternate\*(C'\fR (old name) can be used
182 as an alias for \f(CW\*(C`variable\*(C'\fR.
183 .IP backwards 4
184 .IX Item "backwards"
185 \&\-\- see 3.4 Backward Accents, UTS #10.
186 .Sp
187 .Vb 1
188 \& backwards => $levelNumber or \e@levelNumbers
189 .Ve
190 .Sp
191 Weights in reverse order; ex. level 2 (diacritic ordering) in French.
192 If omitted (or \f(CW$levelNumber\fR is \f(CW\*(C`undef\*(C'\fR or \f(CW\*(C`\e@levelNumbers\*(C'\fR is \f(CW\*(C`[]\*(C'\fR),
193 forwards at all the levels.
194 .IP entry 4
195 .IX Item "entry"
196 \&\-\- see 5 Tailoring; 9.1 Allkeys File Format, UTS #10.
197 .Sp
198 If the same character (or a sequence of characters) exists
199 in the collation element table through \f(CW\*(C`table\*(C'\fR,
200 mapping to collation elements is overridden.
201 If it does not exist, the mapping is defined additionally.
202 .Sp
203 .Vb 12
204 \& entry => <<\*(AqENTRY\*(Aq, # for DUCET v4.0.0 (allkeys\-4.0.0.txt)
205 \&0063 0068 ; [.0E6A.0020.0002.0063] # ch
206 \&0043 0068 ; [.0E6A.0020.0007.0043] # Ch
207 \&0043 0048 ; [.0E6A.0020.0008.0043] # CH
208 \&006C 006C ; [.0F4C.0020.0002.006C] # ll
209 \&004C 006C ; [.0F4C.0020.0007.004C] # Ll
210 \&004C 004C ; [.0F4C.0020.0008.004C] # LL
211 \&00F1 ; [.0F7B.0020.0002.00F1] # n\-tilde
212 \&006E 0303 ; [.0F7B.0020.0002.00F1] # n\-tilde
213 \&00D1 ; [.0F7B.0020.0008.00D1] # N\-tilde
214 \&004E 0303 ; [.0F7B.0020.0008.00D1] # N\-tilde
215 \&ENTRY
216 \&
217 \& entry => <<\*(AqENTRY\*(Aq, # for DUCET v4.0.0 (allkeys\-4.0.0.txt)
218 \&00E6 ; [.0E33.0020.0002.00E6][.0E8B.0020.0002.00E6] # ae ligature as <a><e>
219 \&00C6 ; [.0E33.0020.0008.00C6][.0E8B.0020.0008.00C6] # AE ligature as <A><E>
220 \&ENTRY
221 .Ve
222 .Sp
223 \&\fBNOTE:\fR The code point in the UCA file format (before \f(CW\*(Aq;\*(Aq\fR)
224 \&\fBmust\fR be a Unicode code point (defined as hexadecimal),
225 but not a native code point.
226 So \f(CW0063\fR must always denote \f(CW\*(C`U+0063\*(C'\fR,
227 but not a character of \f(CW"\ex63"\fR.
228 .Sp
229 Weighting may vary depending on collation element table.
230 So ensure the weights defined in \f(CW\*(C`entry\*(C'\fR will be consistent with
231 those in the collation element table loaded via \f(CW\*(C`table\*(C'\fR.
232 .Sp
233 In DUCET v4.0.0, primary weight of \f(CW\*(C`C\*(C'\fR is \f(CW0E60\fR
234 and that of \f(CW\*(C`D\*(C'\fR is \f(CW\*(C`0E6D\*(C'\fR. So setting primary weight of \f(CW\*(C`CH\*(C'\fR to \f(CW\*(C`0E6A\*(C'\fR
235 (as a value between \f(CW0E60\fR and \f(CW\*(C`0E6D\*(C'\fR)
236 makes ordering as \f(CW\*(C`C < CH < D\*(C'\fR.
237 Exactly speaking DUCET already has some characters between \f(CW\*(C`C\*(C'\fR and \f(CW\*(C`D\*(C'\fR:
238 \&\f(CW\*(C`small capital C\*(C'\fR (\f(CW\*(C`U+1D04\*(C'\fR) with primary weight \f(CW0E64\fR,
239 \&\f(CW\*(C`c\-hook/C\-hook\*(C'\fR (\f(CW\*(C`U+0188/U+0187\*(C'\fR) with \f(CW0E65\fR,
240 and \f(CW\*(C`c\-curl\*(C'\fR (\f(CW\*(C`U+0255\*(C'\fR) with \f(CW0E69\fR.
241 Then primary weight \f(CW\*(C`0E6A\*(C'\fR for \f(CW\*(C`CH\*(C'\fR makes \f(CW\*(C`CH\*(C'\fR
242 ordered between \f(CW\*(C`c\-curl\*(C'\fR and \f(CW\*(C`D\*(C'\fR.
243 .IP hangul_terminator 4
244 .IX Item "hangul_terminator"
245 \&\-\- see 7.1.4 Trailing Weights, UTS #10.
246 .Sp
247 If a true value is given (non\-zero but should be positive),
248 it will be added as a terminator primary weight to the end of
249 every standard Hangul syllable. Secondary and any higher weights
250 for terminator are set to zero.
251 If the value is false or \f(CW\*(C`hangul_terminator\*(C'\fR key does not exist,
252 insertion of terminator weights will not be performed.
253 .Sp
254 Boundaries of Hangul syllables are determined
255 according to conjoining Jamo behavior in \fIthe Unicode Standard\fR
256 and \fIHangulSyllableType.txt\fR.
257 .Sp
258 \&\fBImplementation Note:\fR
259 (1) For expansion mapping (Unicode character mapped
260 to a sequence of collation elements), a terminator will not be added
261 between collation elements, even if Hangul syllable boundary exists there.
262 Addition of terminator is restricted to the next position
263 to the last collation element.
264 .Sp
265 (2) Non\-conjoining Hangul letters
266 (Compatibility Jamo, halfwidth Jamo, and enclosed letters) are not
267 automatically terminated with a terminator primary weight.
268 These characters may need terminator included in a collation element
269 table beforehand.
270 .IP highestFFFF 4
271 .IX Item "highestFFFF"
272 \&\-\- see 2.4 Tailored noncharacter weights, UTS #35 (LDML) Part 5: Collation.
273 .Sp
274 If the parameter is made true, \f(CW\*(C`U+FFFF\*(C'\fR has a highest primary weight.
275 When a boolean of \f(CW\*(C`$coll\->ge($str, "abc")\*(C'\fR and
276 \&\f(CW\*(C`$coll\->le($str, "abc\ex{FFFF}")\*(C'\fR is true, it is expected that \f(CW$str\fR
277 begins with \f(CW"abc"\fR, or another primary equivalent.
278 \&\f(CW$str\fR may be \f(CW"abcd"\fR, \f(CW"abc012"\fR, but should not include \f(CW\*(C`U+FFFF\*(C'\fR
279 such as \f(CW"abc\ex{FFFF}xyz"\fR.
280 .Sp
281 \&\f(CW\*(C`$coll\->le($str, "abc\ex{FFFF}")\*(C'\fR works like \f(CW\*(C`$coll\->lt($str, "abd")\*(C'\fR
282 almost, but the latter has a problem that you should know which letter is
283 next to \f(CW\*(C`c\*(C'\fR. For a certain language where \f(CW\*(C`ch\*(C'\fR as the next letter,
284 \&\f(CW"abch"\fR is greater than \f(CW"abc\ex{FFFF}"\fR, but less than \f(CW"abd"\fR.
285 .Sp
286 Note:
287 This is equivalent to \f(CW\*(C`(entry => \*(AqFFFF ; [.FFFE.0020.0005.FFFF]\*(Aq)\*(C'\fR.
288 Any other character than \f(CW\*(C`U+FFFF\*(C'\fR can be tailored by \f(CW\*(C`entry\*(C'\fR.
289 .IP identical 4
290 .IX Item "identical"
291 \&\-\- see A.3 Deterministic Comparison, UTS #10.
292 .Sp
293 By default, strings whose weights are equal should be equal,
294 even though their code points are not equal.
295 Completely ignorable characters are ignored.
296 .Sp
297 If the parameter is made true, a final, tie\-breaking level is used.
298 If no difference of weights is found after the comparison through
299 all the level specified by \f(CW\*(C`level\*(C'\fR, the comparison with code points
300 will be performed.
301 For the tie\-breaking comparison, the sort key has code points
302 of the original string appended.
303 Completely ignorable characters are not ignored.
304 .Sp
305 If \f(CW\*(C`preprocess\*(C'\fR and/or \f(CW\*(C`normalization\*(C'\fR is applied, the code points
306 of the string after them (in NFD by default) are used.
307 .IP ignoreChar 4
308 .IX Item "ignoreChar"
309 .PD 0
310 .IP ignoreName 4
311 .IX Item "ignoreName"
312 .PD
313 \&\-\- see 3.6 Variable Weighting, UTS #10.
314 .Sp
315 Makes the entry in the table completely ignorable;
316 i.e. as if the weights were zero at all level.
317 .Sp
318 Through \f(CW\*(C`ignoreChar\*(C'\fR, any character matching \f(CW\*(C`qr/$ignoreChar/\*(C'\fR
319 will be ignored. Through \f(CW\*(C`ignoreName\*(C'\fR, any character whose name
320 (given in the \f(CW\*(C`table\*(C'\fR file as a comment) matches \f(CW\*(C`qr/$ignoreName/\*(C'\fR
321 will be ignored.
322 .Sp
323 E.g. when \*(Aqa\*(Aq and \*(Aqe\*(Aq are ignorable,
324 \&\*(Aqelement\*(Aq is equal to \*(Aqlament\*(Aq (or \*(Aqlmnt\*(Aq).
325 .IP ignore_level2 4
326 .IX Item "ignore_level2"
327 \&\-\- see 5.1 Parametric Tailoring, UTS #10.
328 .Sp
329 By default, case\-sensitive comparison (that is level 3 difference)
330 won\*(Aqt ignore accents (that is level 2 difference).
331 .Sp
332 If the parameter is made true, accents (and other primary ignorable
333 characters) are ignored, even though cases are taken into account.
334 .Sp
335 \&\fBNOTE\fR: \f(CW\*(C`level\*(C'\fR should be 3 or greater.
336 .IP katakana_before_hiragana 4
337 .IX Item "katakana_before_hiragana"
338 \&\-\- see 7.2 Tertiary Weight Table, UTS #10.
339 .Sp
340 By default, hiragana is before katakana.
341 If the parameter is made true, this is reversed.
342 .Sp
343 \&\fBNOTE\fR: This parameter simplemindedly assumes that any hiragana/katakana
344 distinctions must occur in level 3, and their weights at level 3 must be
345 same as those mentioned in 7.3.1, UTS #10.
346 If you define your collation elements which violate this requirement,
347 this parameter does not work validly.
348 .IP level 4
349 .IX Item "level"
350 \&\-\- see 4.3 Form Sort Key, UTS #10.
351 .Sp
352 Set the maximum level.
353 Any higher levels than the specified one are ignored.
354 .Sp
355 .Vb 4
356 \& Level 1: alphabetic ordering
357 \& Level 2: diacritic ordering
358 \& Level 3: case ordering
359 \& Level 4: tie\-breaking (e.g. in the case when variable is \*(Aqshifted\*(Aq)
360 \&
361 \& ex.level => 2,
362 .Ve
363 .Sp
364 If omitted, the maximum is the 4th.
365 .Sp
366 \&\fBNOTE:\fR The DUCET includes weights over 0xFFFF at the 4th level.
367 But this module only uses weights within 0xFFFF.
368 When \f(CW\*(C`variable\*(C'\fR is \*(Aqblanked\*(Aq or \*(Aqnon\-ignorable\*(Aq (other than \*(Aqshifted\*(Aq
369 and \*(Aqshift\-trimmed\*(Aq), the level 4 may be unreliable.
370 .Sp
371 See also \f(CW\*(C`identical\*(C'\fR.
372 .IP long_contraction 4
373 .IX Item "long_contraction"
374 \&\-\- see 3.8.2 Well\-Formedness of the DUCET, 4.2 Produce Array, UTS #10.
375 .Sp
376 If the parameter is made true, for a contraction with three or more
377 characters (here nicknamed "long contraction"), initial substrings
378 will be handled.
379 For example, a contraction ABC, where A is a starter, and B and C
380 are non\-starters (character with non\-zero combining character class),
381 will be detected even if there is not AB as a contraction.
382 .Sp
383 \&\fBDefault:\fR Usually false.
384 If \f(CW\*(C`UCA_Version\*(C'\fR is 22 or 24, and the value of \f(CW\*(C`long_contraction\*(C'\fR
385 is not specified in \f(CWnew()\fR, a true value is set implicitly.
386 This is a workaround to pass Conformance Tests for Unicode 6.0.0 and 6.1.0.
387 .Sp
388 \&\f(CWchange()\fR handles \f(CW\*(C`long_contraction\*(C'\fR explicitly only.
389 If \f(CW\*(C`long_contraction\*(C'\fR is not specified in \f(CWchange()\fR, even though
390 \&\f(CW\*(C`UCA_Version\*(C'\fR is changed, \f(CW\*(C`long_contraction\*(C'\fR will not be changed.
391 .Sp
392 \&\fBLimitation:\fR Scanning non\-starters is one\-way (no back tracking).
393 If AB is found but not ABC is not found, other long contraction where
394 the first character is A and the second is not B may not be found.
395 .Sp
396 Under \f(CW\*(C`(normalization => undef)\*(C'\fR, detection step of discontiguous
397 contractions will be skipped.
398 .Sp
399 \&\fBNote:\fR The following contractions in DUCET are not considered
400 in steps S2.1.1 to S2.1.3, where they are discontiguous.
401 .Sp
402 .Vb 2
403 \& 0FB2 0F71 0F80 (TIBETAN VOWEL SIGN VOCALIC RR)
404 \& 0FB3 0F71 0F80 (TIBETAN VOWEL SIGN VOCALIC LL)
405 .Ve
406 .Sp
407 For example \f(CW\*(C`TIBETAN VOWEL SIGN VOCALIC RR\*(C'\fR with \f(CW\*(C`COMBINING TILDE OVERLAY\*(C'\fR
408 (\f(CW\*(C`U+0344\*(C'\fR) is \f(CW\*(C`0FB2 0344 0F71 0F80\*(C'\fR in NFD.
409 In this case \f(CW\*(C`0FB2 0F80\*(C'\fR (\f(CW\*(C`TIBETAN VOWEL SIGN VOCALIC R\*(C'\fR) is detected,
410 instead of \f(CW\*(C`0FB2 0F71 0F80\*(C'\fR.
411 Inserted \f(CW0344\fR makes \f(CW\*(C`0FB2 0F71 0F80\*(C'\fR discontiguous and lack of
412 contraction \f(CW\*(C`0FB2 0F71\*(C'\fR prohibits \f(CW\*(C`0FB2 0F71 0F80\*(C'\fR from being detected.
413 .IP minimalFFFE 4
414 .IX Item "minimalFFFE"
415 \&\-\- see 1.1.1 U+FFFE, UTS #35 (LDML) Part 5: Collation.
416 .Sp
417 If the parameter is made true, \f(CW\*(C`U+FFFE\*(C'\fR has a minimal primary weight.
418 The comparison between \f(CW"$a1\ex{FFFE}$a2"\fR and \f(CW"$b1\ex{FFFE}$b2"\fR
419 first compares \f(CW$a1\fR and \f(CW$b1\fR at level 1, and
420 then \f(CW$a2\fR and \f(CW$b2\fR at level 1, as followed.
421 .Sp
422 .Vb 12
423 \& "ab\ex{FFFE}a"
424 \& "Ab\ex{FFFE}a"
425 \& "ab\ex{FFFE}c"
426 \& "Ab\ex{FFFE}c"
427 \& "ab\ex{FFFE}xyz"
428 \& "abc\ex{FFFE}def"
429 \& "abc\ex{FFFE}xYz"
430 \& "aBc\ex{FFFE}xyz"
431 \& "abcX\ex{FFFE}def"
432 \& "abcx\ex{FFFE}xyz"
433 \& "b\ex{FFFE}aaa"
434 \& "bbb\ex{FFFE}a"
435 .Ve
436 .Sp
437 Note:
438 This is equivalent to \f(CW\*(C`(entry => \*(AqFFFE ; [.0001.0020.0005.FFFE]\*(Aq)\*(C'\fR.
439 Any other character than \f(CW\*(C`U+FFFE\*(C'\fR can be tailored by \f(CW\*(C`entry\*(C'\fR.
440 .IP normalization 4
441 .IX Item "normalization"
442 \&\-\- see 4.1 Normalize, UTS #10.
443 .Sp
444 If specified, strings are normalized before preparation of sort keys
445 (the normalization is executed after preprocess).
446 .Sp
447 A form name \f(CWUnicode::Normalize::normalize()\fR accepts will be applied
448 as \f(CW$normalization_form\fR.
449 Acceptable names include \f(CW\*(AqNFD\*(Aq\fR, \f(CW\*(AqNFC\*(Aq\fR, \f(CW\*(AqNFKD\*(Aq\fR, and \f(CW\*(AqNFKC\*(Aq\fR.
450 See \f(CWUnicode::Normalize::normalize()\fR for detail.
451 If omitted, \f(CW\*(AqNFD\*(Aq\fR is used.
452 .Sp
453 \&\f(CW\*(C`normalization\*(C'\fR is performed after \f(CW\*(C`preprocess\*(C'\fR (if defined).
454 .Sp
455 Furthermore, special values, \f(CW\*(C`undef\*(C'\fR and \f(CW"prenormalized"\fR, can be used,
456 though they are not concerned with \f(CWUnicode::Normalize::normalize()\fR.
457 .Sp
458 If \f(CW\*(C`undef\*(C'\fR (not a string \f(CW"undef"\fR) is passed explicitly
459 as the value for this key,
460 any normalization is not carried out (this may make tailoring easier
461 if any normalization is not desired). Under \f(CW\*(C`(normalization => undef)\*(C'\fR,
462 only contiguous contractions are resolved;
463 e.g. even if \f(CW\*(C`A\-ring\*(C'\fR (and \f(CW\*(C`A\-ring\-cedilla\*(C'\fR) is ordered after \f(CW\*(C`Z\*(C'\fR,
464 \&\f(CW\*(C`A\-cedilla\-ring\*(C'\fR would be primary equal to \f(CW\*(C`A\*(C'\fR.
465 In this point,
466 \&\f(CW\*(C`(normalization => undef, preprocess => sub { NFD(shift) })\*(C'\fR
467 \&\fBis not\fR equivalent to \f(CW\*(C`(normalization => \*(AqNFD\*(Aq)\*(C'\fR.
468 .Sp
469 In the case of \f(CW\*(C`(normalization => "prenormalized")\*(C'\fR,
470 any normalization is not performed, but
471 discontiguous contractions with combining characters are performed.
472 Therefore
473 \&\f(CW\*(C`(normalization => \*(Aqprenormalized\*(Aq, preprocess => sub { NFD(shift) })\*(C'\fR
474 \&\fBis\fR equivalent to \f(CW\*(C`(normalization => \*(AqNFD\*(Aq)\*(C'\fR.
475 If source strings are finely prenormalized,
476 \&\f(CW\*(C`(normalization => \*(Aqprenormalized\*(Aq)\*(C'\fR may save time for normalization.
477 .Sp
478 Except \f(CW\*(C`(normalization => undef)\*(C'\fR,
479 \&\fBUnicode::Normalize\fR is required (see also \fBCAVEAT\fR).
480 .IP overrideCJK 4
481 .IX Item "overrideCJK"
482 \&\-\- see 7.1 Derived Collation Elements, UTS #10.
483 .Sp
484 By default, CJK unified ideographs are ordered in Unicode codepoint
485 order, but those in the CJK Unified Ideographs block are less than
486 those in the CJK Unified Ideographs Extension A etc.
487 .Sp
488 .Vb 10
489 \& In the CJK Unified Ideographs block:
490 \& U+4E00..U+9FA5 if UCA_Version is 8, 9 or 11.
491 \& U+4E00..U+9FBB if UCA_Version is 14 or 16.
492 \& U+4E00..U+9FC3 if UCA_Version is 18.
493 \& U+4E00..U+9FCB if UCA_Version is 20 or 22.
494 \& U+4E00..U+9FCC if UCA_Version is 24 to 30.
495 \& U+4E00..U+9FD5 if UCA_Version is 32 or 34.
496 \& U+4E00..U+9FEA if UCA_Version is 36.
497 \& U+4E00..U+9FEF if UCA_Version is 38, 40 or 41.
498 \& U+4E00..U+9FFC if UCA_Version is 43.
499 \&
500 \& In the CJK Unified Ideographs Extension blocks:
501 \& Ext.A (U+3400..U+4DB5) if UCA_Version is 8 to 41.
502 \& Ext.A (U+3400..U+4DBF) if UCA_Version is 43.
503 \& Ext.B (U+20000..U+2A6D6) if UCA_Version is 8 to 41.
504 \& Ext.B (U+20000..U+2A6DD) if UCA_Version is 43.
505 \& Ext.C (U+2A700..U+2B734) if UCA_Version is 20 or later.
506 \& Ext.D (U+2B740..U+2B81D) if UCA_Version is 22 or later.
507 \& Ext.E (U+2B820..U+2CEA1) if UCA_Version is 32 or later.
508 \& Ext.F (U+2CEB0..U+2EBE0) if UCA_Version is 36 or later.
509 \& Ext.G (U+30000..U+3134A) if UCA_Version is 43.
510 .Ve
511 .Sp
512 Through \f(CW\*(C`overrideCJK\*(C'\fR, ordering of CJK unified ideographs (including
513 extensions) can be overridden.
514 .Sp
515 ex. CJK unified ideographs in the JIS code point order.
516 .Sp
517 .Vb 7
518 \& overrideCJK => sub {
519 \& my $u = shift; # get a Unicode codepoint
520 \& my $b = pack(\*(Aqn\*(Aq, $u); # to UTF\-16BE
521 \& my $s = your_unicode_to_sjis_converter($b); # convert
522 \& my $n = unpack(\*(Aqn\*(Aq, $s); # convert sjis to short
523 \& [ $n, 0x20, 0x2, $u ]; # return the collation element
524 \& },
525 .Ve
526 .Sp
527 The return value may be an arrayref of 1st to 4th weights as shown
528 above. The return value may be an integer as the primary weight
529 as shown below. If \f(CW\*(C`undef\*(C'\fR is returned, the default derived
530 collation element will be used.
531 .Sp
532 .Vb 7
533 \& overrideCJK => sub {
534 \& my $u = shift; # get a Unicode codepoint
535 \& my $b = pack(\*(Aqn\*(Aq, $u); # to UTF\-16BE
536 \& my $s = your_unicode_to_sjis_converter($b); # convert
537 \& my $n = unpack(\*(Aqn\*(Aq, $s); # convert sjis to short
538 \& return $n; # return the primary weight
539 \& },
540 .Ve
541 .Sp
542 The return value may be a list containing zero or more of
543 an arrayref, an integer, or \f(CW\*(C`undef\*(C'\fR.
544 .Sp
545 ex. ignores all CJK unified ideographs.
546 .Sp
547 .Vb 1
548 \& overrideCJK => sub {()}, # CODEREF returning empty list
549 \&
550 \& # where \->eq("Pe\ex{4E00}rl", "Perl") is true
551 \& # as U+4E00 is a CJK unified ideograph and to be ignorable.
552 .Ve
553 .Sp
554 If a false value (including \f(CW\*(C`undef\*(C'\fR) is passed, \f(CW\*(C`overrideCJK\*(C'\fR
555 has no effect.
556 \&\f(CW\*(C`$Collator\->change(overrideCJK => 0)\*(C'\fR resets the old one.
557 .Sp
558 But assignment of weight for CJK unified ideographs
559 in \f(CW\*(C`table\*(C'\fR or \f(CW\*(C`entry\*(C'\fR is still valid.
560 If \f(CW\*(C`undef\*(C'\fR is passed explicitly as the value for this key,
561 weights for CJK unified ideographs are treated as undefined.
562 However when \f(CW\*(C`UCA_Version\*(C'\fR > 8, \f(CW\*(C`(overrideCJK => undef)\*(C'\fR
563 has no special meaning.
564 .Sp
565 \&\fBNote:\fR In addition to them, 12 CJK compatibility ideographs (\f(CW\*(C`U+FA0E\*(C'\fR,
566 \&\f(CW\*(C`U+FA0F\*(C'\fR, \f(CW\*(C`U+FA11\*(C'\fR, \f(CW\*(C`U+FA13\*(C'\fR, \f(CW\*(C`U+FA14\*(C'\fR, \f(CW\*(C`U+FA1F\*(C'\fR, \f(CW\*(C`U+FA21\*(C'\fR, \f(CW\*(C`U+FA23\*(C'\fR,
567 \&\f(CW\*(C`U+FA24\*(C'\fR, \f(CW\*(C`U+FA27\*(C'\fR, \f(CW\*(C`U+FA28\*(C'\fR, \f(CW\*(C`U+FA29\*(C'\fR) are also treated as CJK unified
568 ideographs. But they can\*(Aqt be overridden via \f(CW\*(C`overrideCJK\*(C'\fR when you use
569 DUCET, as the table includes weights for them. \f(CW\*(C`table\*(C'\fR or \f(CW\*(C`entry\*(C'\fR has
570 priority over \f(CW\*(C`overrideCJK\*(C'\fR.
571 .IP overrideHangul 4
572 .IX Item "overrideHangul"
573 \&\-\- see 7.1 Derived Collation Elements, UTS #10.
574 .Sp
575 By default, Hangul syllables are decomposed into Hangul Jamo,
576 even if \f(CW\*(C`(normalization => undef)\*(C'\fR.
577 But the mapping of Hangul syllables may be overridden.
578 .Sp
579 This parameter works like \f(CW\*(C`overrideCJK\*(C'\fR, so see there for examples.
580 .Sp
581 If you want to override the mapping of Hangul syllables,
582 NFD and NFKD are not appropriate, since NFD and NFKD will decompose
583 Hangul syllables before overriding. FCD may decompose Hangul syllables
584 as the case may be.
585 .Sp
586 If a false value (but not \f(CW\*(C`undef\*(C'\fR) is passed, \f(CW\*(C`overrideHangul\*(C'\fR
587 has no effect.
588 \&\f(CW\*(C`$Collator\->change(overrideHangul => 0)\*(C'\fR resets the old one.
589 .Sp
590 If \f(CW\*(C`undef\*(C'\fR is passed explicitly as the value for this key,
591 weight for Hangul syllables is treated as undefined
592 without decomposition into Hangul Jamo.
593 But definition of weight for Hangul syllables
594 in \f(CW\*(C`table\*(C'\fR or \f(CW\*(C`entry\*(C'\fR is still valid.
595 .IP overrideOut 4
596 .IX Item "overrideOut"
597 \&\-\- see 7.1.1 Handling Ill\-Formed Code Unit Sequences, UTS #10.
598 .Sp
599 Perl seems to allow out\-of\-range values (greater than 0x10FFFF).
600 By default, out\-of\-range values are replaced with \f(CW\*(C`U+FFFD\*(C'\fR
601 (REPLACEMENT CHARACTER) when \f(CW\*(C`UCA_Version\*(C'\fR >= 22,
602 or ignored when \f(CW\*(C`UCA_Version\*(C'\fR <= 20.
603 .Sp
604 When \f(CW\*(C`UCA_Version\*(C'\fR >= 22, the weights of out\-of\-range values
605 can be overridden. Though \f(CW\*(C`table\*(C'\fR or \f(CW\*(C`entry\*(C'\fR are available for them,
606 out\-of\-range values are too many.
607 .Sp
608 \&\f(CW\*(C`overrideOut\*(C'\fR can perform it algorithmically.
609 This parameter works like \f(CW\*(C`overrideCJK\*(C'\fR, so see there for examples.
610 .Sp
611 ex. ignores all out\-of\-range values.
612 .Sp
613 .Vb 1
614 \& overrideOut => sub {()}, # CODEREF returning empty list
615 .Ve
616 .Sp
617 If a false value (including \f(CW\*(C`undef\*(C'\fR) is passed, \f(CW\*(C`overrideOut\*(C'\fR
618 has no effect.
619 \&\f(CW\*(C`$Collator\->change(overrideOut => 0)\*(C'\fR resets the old one.
620 .Sp
621 \&\fBNOTE ABOUT U+FFFD:\fR
622 .Sp
623 UCA recommends that out\-of\-range values should not be ignored for security
624 reasons. Say, \f(CW"pe\ex{110000}rl"\fR should not be equal to \f(CW"perl"\fR.
625 However, \f(CW\*(C`U+FFFD\*(C'\fR is wrongly mapped to a variable collation element
626 in DUCET for Unicode 6.0.0 to 6.2.0, that means out\-of\-range values will be
627 ignored when \f(CW\*(C`variable\*(C'\fR isn\*(Aqt \f(CW\*(C`Non\-ignorable\*(C'\fR.
628 .Sp
629 The mapping of \f(CW\*(C`U+FFFD\*(C'\fR is corrected in Unicode 6.3.0.
630 see <http://www.unicode.org/reports/tr10/tr10\-28.html#Trailing_Weights>
631 (7.1.4 Trailing Weights). Such a correction is reproduced by this.
632 .Sp
633 .Vb 1
634 \& overrideOut => sub { 0xFFFD }, # CODEREF returning a very large integer
635 .Ve
636 .Sp
637 This workaround is unnecessary since Unicode 6.3.0.
638 .IP preprocess 4
639 .IX Item "preprocess"
640 \&\-\- see 5.4 Preprocessing, UTS #10.
641 .Sp
642 If specified, the coderef is used to preprocess each string
643 before the formation of sort keys.
644 .Sp
645 ex. dropping English articles, such as "a" or "the".
646 Then, "the pen" is before "a pencil".
647 .Sp
648 .Vb 5
649 \& preprocess => sub {
650 \& my $str = shift;
651 \& $str =~ s/\eb(?:an?|the)\es+//gi;
652 \& return $str;
653 \& },
654 .Ve
655 .Sp
656 \&\f(CW\*(C`preprocess\*(C'\fR is performed before \f(CW\*(C`normalization\*(C'\fR (if defined).
657 .Sp
658 ex. decoding strings in a legacy encoding such as shift\-jis:
659 .Sp
660 .Vb 4
661 \& $sjis_collator = Unicode::Collate\->new(
662 \& preprocess => \e&your_shiftjis_to_unicode_decoder,
663 \& );
664 \& @result = $sjis_collator\->sort(@shiftjis_strings);
665 .Ve
666 .Sp
667 \&\fBNote:\fR Strings returned from the coderef will be interpreted
668 according to Perl\*(Aqs Unicode support. See perlunicode,
669 perluniintro, perlunitut, perlunifaq, utf8.
670 .IP rearrange 4
671 .IX Item "rearrange"
672 \&\-\- see 3.5 Rearrangement, UTS #10.
673 .Sp
674 Characters that are not coded in logical order and to be rearranged.
675 If \f(CW\*(C`UCA_Version\*(C'\fR is equal to or less than 11, default is:
676 .Sp
677 .Vb 1
678 \& rearrange => [ 0x0E40..0x0E44, 0x0EC0..0x0EC4 ],
679 .Ve
680 .Sp
681 If you want to disallow any rearrangement, pass \f(CW\*(C`undef\*(C'\fR or \f(CW\*(C`[]\*(C'\fR
682 (a reference to empty list) as the value for this key.
683 .Sp
684 If \f(CW\*(C`UCA_Version\*(C'\fR is equal to or greater than 14, default is \f(CW\*(C`[]\*(C'\fR
685 (i.e. no rearrangement).
686 .Sp
687 \&\fBAccording to the version 9 of UCA, this parameter shall not be used;
688 but it is not warned at present.\fR
689 .IP rewrite 4
690 .IX Item "rewrite"
691 If specified, the coderef is used to rewrite lines in \f(CW\*(C`table\*(C'\fR or \f(CW\*(C`entry\*(C'\fR.
692 The coderef will get each line, and then should return a rewritten line
693 according to the UCA file format.
694 If the coderef returns an empty line, the line will be skipped.
695 .Sp
696 e.g. any primary ignorable characters into tertiary ignorable:
697 .Sp
698 .Vb 5
699 \& rewrite => sub {
700 \& my $line = shift;
701 \& $line =~ s/\e[\e.0000\e..{4}\e..{4}\e./[.0000.0000.0000./g;
702 \& return $line;
703 \& },
704 .Ve
705 .Sp
706 This example shows rewriting weights. \f(CW\*(C`rewrite\*(C'\fR is allowed to
707 affect code points, weights, and the name.
708 .Sp
709 \&\fBNOTE\fR: \f(CW\*(C`table\*(C'\fR is available to use another table file;
710 preparing a modified table once would be more efficient than
711 rewriting lines on reading an unmodified table every time.
712 .IP suppress 4
713 .IX Item "suppress"
714 \&\-\- see 3.12 Special\-Purpose Commands, UTS #35 (LDML) Part 5: Collation.
715 .Sp
716 Contractions beginning with the specified characters are suppressed,
717 even if those contractions are defined in \f(CW\*(C`table\*(C'\fR.
718 .Sp
719 An example for Russian and some languages using the Cyrillic script:
720 .Sp
721 .Vb 1
722 \& suppress => [0x0400..0x0417, 0x041A..0x0437, 0x043A..0x045F],
723 .Ve
724 .Sp
725 where 0x0400 stands for \f(CW\*(C`U+0400\*(C'\fR, CYRILLIC CAPITAL LETTER IE WITH GRAVE.
726 .Sp
727 \&\fBNOTE\fR: Contractions via \f(CW\*(C`entry\*(C'\fR will not be suppressed.
728 .IP table 4
729 .IX Item "table"
730 \&\-\- see 3.8 Default Unicode Collation Element Table, UTS #10.
731 .Sp
732 You can use another collation element table if desired.
733 .Sp
734 The table file should locate in the \fIUnicode/Collate\fR directory
735 on \f(CW@INC\fR. Say, if the filename is \fIFoo.txt\fR,
736 the table file is searched as \fIUnicode/Collate/Foo.txt\fR in \f(CW@INC\fR.
737 .Sp
738 By default, \fIallkeys.txt\fR (as the filename of DUCET) is used.
739 If you will prepare your own table file, any name other than \fIallkeys.txt\fR
740 may be better to avoid namespace conflict.
741 .Sp
742 \&\fBNOTE\fR: When XSUB is used, the DUCET is compiled on building this
743 module, and it may save time at the run time.
744 Explicit saying \f(CW\*(C`(table => \*(Aqallkeys.txt\*(Aq)\*(C'\fR, or using another table,
745 or using \f(CW\*(C`ignoreChar\*(C'\fR, \f(CW\*(C`ignoreName\*(C'\fR, \f(CW\*(C`undefChar\*(C'\fR, \f(CW\*(C`undefName\*(C'\fR or
746 \&\f(CW\*(C`rewrite\*(C'\fR will prevent this module from using the compiled DUCET.
747 .Sp
748 If \f(CW\*(C`undef\*(C'\fR is passed explicitly as the value for this key,
749 no file is read (but you can define collation elements via \f(CW\*(C`entry\*(C'\fR).
750 .Sp
751 A typical way to define a collation element table
752 without any file of table:
753 .Sp
754 .Vb 11
755 \& $onlyABC = Unicode::Collate\->new(
756 \& table => undef,
757 \& entry => << \*(AqENTRIES\*(Aq,
758 \&0061 ; [.0101.0020.0002.0061] # LATIN SMALL LETTER A
759 \&0041 ; [.0101.0020.0008.0041] # LATIN CAPITAL LETTER A
760 \&0062 ; [.0102.0020.0002.0062] # LATIN SMALL LETTER B
761 \&0042 ; [.0102.0020.0008.0042] # LATIN CAPITAL LETTER B
762 \&0063 ; [.0103.0020.0002.0063] # LATIN SMALL LETTER C
763 \&0043 ; [.0103.0020.0008.0043] # LATIN CAPITAL LETTER C
764 \&ENTRIES
765 \& );
766 .Ve
767 .Sp
768 If \f(CW\*(C`ignoreName\*(C'\fR or \f(CW\*(C`undefName\*(C'\fR is used, character names should be
769 specified as a comment (following \f(CW\*(C`#\*(C'\fR) on each line.
770 .IP undefChar 4
771 .IX Item "undefChar"
772 .PD 0
773 .IP undefName 4
774 .IX Item "undefName"
775 .PD
776 \&\-\- see 6.3.3 Reducing the Repertoire, UTS #10.
777 .Sp
778 Undefines the collation element as if it were unassigned in the \f(CW\*(C`table\*(C'\fR.
779 This reduces the size of the table.
780 If an unassigned character appears in the string to be collated,
781 the sort key is made from its codepoint
782 as a single\-character collation element,
783 as it is greater than any other assigned collation elements
784 (in the codepoint order among the unassigned characters).
785 But, it\*(Aqd be better to ignore characters
786 unfamiliar to you and maybe never used.
787 .Sp
788 Through \f(CW\*(C`undefChar\*(C'\fR, any character matching \f(CW\*(C`qr/$undefChar/\*(C'\fR
789 will be undefined. Through \f(CW\*(C`undefName\*(C'\fR, any character whose name
790 (given in the \f(CW\*(C`table\*(C'\fR file as a comment) matches \f(CW\*(C`qr/$undefName/\*(C'\fR
791 will be undefined.
792 .Sp
793 ex. Collation weights for beyond\-BMP characters are not stored in object:
794 .Sp
795 .Vb 1
796 \& undefChar => qr/[^\e0\-\ex{fffd}]/,
797 .Ve
798 .IP upper_before_lower 4
799 .IX Item "upper_before_lower"
800 \&\-\- see 6.6 Case Comparisons, UTS #10.
801 .Sp
802 By default, lowercase is before uppercase.
803 If the parameter is made true, this is reversed.
804 .Sp
805 \&\fBNOTE\fR: This parameter simplemindedly assumes that any lowercase/uppercase
806 distinctions must occur in level 3, and their weights at level 3 must be
807 same as those mentioned in 7.3.1, UTS #10.
808 If you define your collation elements which differs from this requirement,
809 this parameter doesn\*(Aqt work validly.
810 .IP variable 4
811 .IX Item "variable"
812 \&\-\- see 3.6 Variable Weighting, UTS #10.
813 .Sp
814 This key allows for variable weighting of variable collation elements,
815 which are marked with an ASTERISK in the table
816 (NOTE: Many punctuation marks and symbols are variable in \fIallkeys.txt\fR).
817 .Sp
818 .Vb 1
819 \& variable => \*(Aqblanked\*(Aq, \*(Aqnon\-ignorable\*(Aq, \*(Aqshifted\*(Aq, or \*(Aqshift\-trimmed\*(Aq.
820 .Ve
821 .Sp
822 These names are case\-insensitive.
823 By default (if specification is omitted), \*(Aqshifted\*(Aq is adopted.
824 .Sp
825 .Vb 2
826 \& \*(AqBlanked\*(Aq Variable elements are made ignorable at levels 1 through 3;
827 \& considered at the 4th level.
828 \&
829 \& \*(AqNon\-Ignorable\*(Aq Variable elements are not reset to ignorable.
830 \&
831 \& \*(AqShifted\*(Aq Variable elements are made ignorable at levels 1 through 3
832 \& their level 4 weight is replaced by the old level 1 weight.
833 \& Level 4 weight for Non\-Variable elements is 0xFFFF.
834 \&
835 \& \*(AqShift\-Trimmed\*(Aq Same as \*(Aqshifted\*(Aq, but all FFFF\*(Aqs at the 4th level
836 \& are trimmed.
837 .Ve
838 .SS "Methods for Collation"
839 .IX Subsection "Methods for Collation"
840 .ie n .IP """@sorted = $Collator\->sort(@not_sorted)""" 4
841 .el .IP "\f(CW@sorted = $Collator\->sort(@not_sorted)\fR" 4
842 .IX Item "@sorted = $Collator->sort(@not_sorted)"
843 Sorts a list of strings.
844 .ie n .IP """$result = $Collator\->cmp($a, $b)""" 4
845 .el .IP "\f(CW$result = $Collator\->cmp($a, $b)\fR" 4
846 .IX Item "$result = $Collator->cmp($a, $b)"
847 Returns 1 (when \f(CW$a\fR is greater than \f(CW$b\fR)
848 or 0 (when \f(CW$a\fR is equal to \f(CW$b\fR)
849 or \-1 (when \f(CW$a\fR is less than \f(CW$b\fR).
850 .ie n .IP """$result = $Collator\->eq($a, $b)""" 4
851 .el .IP "\f(CW$result = $Collator\->eq($a, $b)\fR" 4
852 .IX Item "$result = $Collator->eq($a, $b)"
853 .PD 0
854 .ie n .IP """$result = $Collator\->ne($a, $b)""" 4
855 .el .IP "\f(CW$result = $Collator\->ne($a, $b)\fR" 4
856 .IX Item "$result = $Collator->ne($a, $b)"
857 .ie n .IP """$result = $Collator\->lt($a, $b)""" 4
858 .el .IP "\f(CW$result = $Collator\->lt($a, $b)\fR" 4
859 .IX Item "$result = $Collator->lt($a, $b)"
860 .ie n .IP """$result = $Collator\->le($a, $b)""" 4
861 .el .IP "\f(CW$result = $Collator\->le($a, $b)\fR" 4
862 .IX Item "$result = $Collator->le($a, $b)"
863 .ie n .IP """$result = $Collator\->gt($a, $b)""" 4
864 .el .IP "\f(CW$result = $Collator\->gt($a, $b)\fR" 4
865 .IX Item "$result = $Collator->gt($a, $b)"
866 .ie n .IP """$result = $Collator\->ge($a, $b)""" 4
867 .el .IP "\f(CW$result = $Collator\->ge($a, $b)\fR" 4
868 .IX Item "$result = $Collator->ge($a, $b)"
869 .PD
870 They works like the same name operators as theirs.
871 .Sp
872 .Vb 6
873 \& eq : whether $a is equal to $b.
874 \& ne : whether $a is not equal to $b.
875 \& lt : whether $a is less than $b.
876 \& le : whether $a is less than $b or equal to $b.
877 \& gt : whether $a is greater than $b.
878 \& ge : whether $a is greater than $b or equal to $b.
879 .Ve
880 .ie n .IP """$sortKey = $Collator\->getSortKey($string)""" 4
881 .el .IP "\f(CW$sortKey = $Collator\->getSortKey($string)\fR" 4
882 .IX Item "$sortKey = $Collator->getSortKey($string)"
883 \&\-\- see 4.3 Form Sort Key, UTS #10.
884 .Sp
885 Returns a sort key.
886 .Sp
887 You compare the sort keys using a binary comparison
888 and get the result of the comparison of the strings using UCA.
889 .Sp
890 .Vb 1
891 \& $Collator\->getSortKey($a) cmp $Collator\->getSortKey($b)
892 \&
893 \& is equivalent to
894 \&
895 \& $Collator\->cmp($a, $b)
896 .Ve
897 .ie n .IP """$sortKeyForm = $Collator\->viewSortKey($string)""" 4
898 .el .IP "\f(CW$sortKeyForm = $Collator\->viewSortKey($string)\fR" 4
899 .IX Item "$sortKeyForm = $Collator->viewSortKey($string)"
900 Converts a sorting key into its representation form.
901 If \f(CW\*(C`UCA_Version\*(C'\fR is 8, the output is slightly different.
902 .Sp
903 .Vb 3
904 \& use Unicode::Collate;
905 \& my $c = Unicode::Collate\->new();
906 \& print $c\->viewSortKey("Perl"),"\en";
907 \&
908 \& # output:
909 \& # [0B67 0A65 0B7F 0B03 | 0020 0020 0020 0020 | 0008 0002 0002 0002 | FFFF FFFF FFFF FFFF]
910 \& # Level 1 Level 2 Level 3 Level 4
911 .Ve
912 .SS "Methods for Searching"
913 .IX Subsection "Methods for Searching"
914 The \f(CW\*(C`match\*(C'\fR, \f(CW\*(C`gmatch\*(C'\fR, \f(CW\*(C`subst\*(C'\fR, \f(CW\*(C`gsubst\*(C'\fR methods work
915 like \f(CW\*(C`m//\*(C'\fR, \f(CW\*(C`m//g\*(C'\fR, \f(CW\*(C`s///\*(C'\fR, \f(CW\*(C`s///g\*(C'\fR, respectively,
916 but they are not aware of any pattern, but only a literal substring.
917 .PP
918 \&\fBDISCLAIMER:\fR If \f(CW\*(C`preprocess\*(C'\fR or \f(CW\*(C`normalization\*(C'\fR parameter is true
919 for \f(CW$Collator\fR, calling these methods (\f(CW\*(C`index\*(C'\fR, \f(CW\*(C`match\*(C'\fR, \f(CW\*(C`gmatch\*(C'\fR,
920 \&\f(CW\*(C`subst\*(C'\fR, \f(CW\*(C`gsubst\*(C'\fR) is croaked, as the position and the length might
921 differ from those on the specified string.
922 .PP
923 \&\f(CW\*(C`rearrange\*(C'\fR and \f(CW\*(C`hangul_terminator\*(C'\fR parameters are neglected.
924 \&\f(CW\*(C`katakana_before_hiragana\*(C'\fR and \f(CW\*(C`upper_before_lower\*(C'\fR don\*(Aqt affect
925 matching and searching, as it doesn\*(Aqt matter whether greater or less.
926 .ie n .IP """$position = $Collator\->index($string, $substring[, $position])""" 4
927 .el .IP "\f(CW$position = $Collator\->index($string, $substring[, $position])\fR" 4
928 .IX Item "$position = $Collator->index($string, $substring[, $position])"
929 .PD 0
930 .ie n .IP """($position, $length) = $Collator\->index($string, $substring[, $position])""" 4
931 .el .IP "\f(CW($position, $length) = $Collator\->index($string, $substring[, $position])\fR" 4
932 .IX Item "($position, $length) = $Collator->index($string, $substring[, $position])"
933 .PD
934 If \f(CW$substring\fR matches a part of \f(CW$string\fR, returns
935 the position of the first occurrence of the matching part in scalar context;
936 in list context, returns a two\-element list of
937 the position and the length of the matching part.
938 .Sp
939 If \f(CW$substring\fR does not match any part of \f(CW$string\fR,
940 returns \f(CW\-1\fR in scalar context and
941 an empty list in list context.
942 .Sp
943 e.g. when the content of \f(CW$str\fR is \f(CW\*(C`"Ich mu\*(C'\fRß\f(CW\*(C` studieren Perl."\*(C'\fR,
944 you say the following where \f(CW$sub\fR is \f(CW\*(C`"M\*(C'\fRü\f(CW\*(C`SS"\*(C'\fR,
945 .Sp
946 .Vb 6
947 \& my $Collator = Unicode::Collate\->new( normalization => undef, level => 1 );
948 \& # (normalization => undef) is REQUIRED.
949 \& my $match;
950 \& if (my($pos,$len) = $Collator\->index($str, $sub)) {
951 \& $match = substr($str, $pos, $len);
952 \& }
953 .Ve
954 .Sp
955 and get \f(CW\*(C`"mu\*(C'\fRß\f(CW\*(C`"\*(C'\fR in \f(CW$match\fR, since \f(CW\*(C`"mu\*(C'\fRß\f(CW\*(C`"\*(C'\fR
956 is primary equal to \f(CW\*(C`"M\*(C'\fRü\f(CW\*(C`SS"\*(C'\fR.
957 .ie n .IP """$match_ref = $Collator\->match($string, $substring)""" 4
958 .el .IP "\f(CW$match_ref = $Collator\->match($string, $substring)\fR" 4
959 .IX Item "$match_ref = $Collator->match($string, $substring)"
960 .PD 0
961 .ie n .IP """($match) = $Collator\->match($string, $substring)""" 4
962 .el .IP "\f(CW($match) = $Collator\->match($string, $substring)\fR" 4
963 .IX Item "($match) = $Collator->match($string, $substring)"
964 .PD
965 If \f(CW$substring\fR matches a part of \f(CW$string\fR, in scalar context, returns
966 \&\fBa reference to\fR the first occurrence of the matching part
967 (\f(CW$match_ref\fR is always true if matches,
968 since every reference is \fBtrue\fR);
969 in list context, returns the first occurrence of the matching part.
970 .Sp
971 If \f(CW$substring\fR does not match any part of \f(CW$string\fR,
972 returns \f(CW\*(C`undef\*(C'\fR in scalar context and
973 an empty list in list context.
974 .Sp
975 e.g.
976 .Sp
977 .Vb 5
978 \& if ($match_ref = $Collator\->match($str, $sub)) { # scalar context
979 \& print "matches [$$match_ref].\en";
980 \& } else {
981 \& print "doesn\*(Aqt match.\en";
982 \& }
983 \&
984 \& or
985 \&
986 \& if (($match) = $Collator\->match($str, $sub)) { # list context
987 \& print "matches [$match].\en";
988 \& } else {
989 \& print "doesn\*(Aqt match.\en";
990 \& }
991 .Ve
992 .ie n .IP """@match = $Collator\->gmatch($string, $substring)""" 4
993 .el .IP "\f(CW@match = $Collator\->gmatch($string, $substring)\fR" 4
994 .IX Item "@match = $Collator->gmatch($string, $substring)"
995 If \f(CW$substring\fR matches a part of \f(CW$string\fR, returns
996 all the matching parts (or matching count in scalar context).
997 .Sp
998 If \f(CW$substring\fR does not match any part of \f(CW$string\fR,
999 returns an empty list.
1000 .ie n .IP """$count = $Collator\->subst($string, $substring, $replacement)""" 4
1001 .el .IP "\f(CW$count = $Collator\->subst($string, $substring, $replacement)\fR" 4
1002 .IX Item "$count = $Collator->subst($string, $substring, $replacement)"
1003 If \f(CW$substring\fR matches a part of \f(CW$string\fR,
1004 the first occurrence of the matching part is replaced by \f(CW$replacement\fR
1005 (\f(CW$string\fR is modified) and \f(CW$count\fR (always equals to \f(CW1\fR) is returned.
1006 .Sp
1007 \&\f(CW$replacement\fR can be a \f(CW\*(C`CODEREF\*(C'\fR,
1008 taking the matching part as an argument,
1009 and returning a string to replace the matching part
1010 (a bit similar to \f(CW\*(C`s/(..)/$coderef\->($1)/e\*(C'\fR).
1011 .ie n .IP """$count = $Collator\->gsubst($string, $substring, $replacement)""" 4
1012 .el .IP "\f(CW$count = $Collator\->gsubst($string, $substring, $replacement)\fR" 4
1013 .IX Item "$count = $Collator->gsubst($string, $substring, $replacement)"
1014 If \f(CW$substring\fR matches a part of \f(CW$string\fR,
1015 all the occurrences of the matching part are replaced by \f(CW$replacement\fR
1016 (\f(CW$string\fR is modified) and \f(CW$count\fR is returned.
1017 .Sp
1018 \&\f(CW$replacement\fR can be a \f(CW\*(C`CODEREF\*(C'\fR,
1019 taking the matching part as an argument,
1020 and returning a string to replace the matching part
1021 (a bit similar to \f(CW\*(C`s/(..)/$coderef\->($1)/eg\*(C'\fR).
1022 .Sp
1023 e.g.
1024 .Sp
1025 .Vb 4
1026 \& my $Collator = Unicode::Collate\->new( normalization => undef, level => 1 );
1027 \& # (normalization => undef) is REQUIRED.
1028 \& my $str = "Camel donkey zebra came\ex{301}l CAMEL horse cam\e0e\e0l...";
1029 \& $Collator\->gsubst($str, "camel", sub { "<b>$_[0]</b>" });
1030 \&
1031 \& # now $str is "<b>Camel</b> donkey zebra <b>came\ex{301}l</b> <b>CAMEL</b> horse <b>cam\e0e\e0l</b>...";
1032 \& # i.e., all the camels are made bold\-faced.
1033 \&
1034 \& Examples: levels and ignore_level2 \- what does camel match?
1035 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
1036 \& level ignore_level2 | camel Camel came\ex{301}l c\-a\-m\-e\-l cam\e0e\e0l
1037 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-|\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
1038 \& 1 false | yes yes yes yes yes
1039 \& 2 false | yes yes no yes yes
1040 \& 3 false | yes no no yes yes
1041 \& 4 false | yes no no no yes
1042 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-|\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
1043 \& 1 true | yes yes yes yes yes
1044 \& 2 true | yes yes yes yes yes
1045 \& 3 true | yes no yes yes yes
1046 \& 4 true | yes no yes no yes
1047 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
1048 \& note: if variable => non\-ignorable, camel doesn\*(Aqt match c\-a\-m\-e\-l
1049 \& at any level.
1050 .Ve
1051 .SS "Other Methods"
1052 .IX Subsection "Other Methods"
1053 .ie n .IP """%old_tailoring = $Collator\->change(%new_tailoring)""" 4
1054 .el .IP "\f(CW%old_tailoring = $Collator\->change(%new_tailoring)\fR" 4
1055 .IX Item "%old_tailoring = $Collator->change(%new_tailoring)"
1056 .PD 0
1057 .ie n .IP """$modified_collator = $Collator\->change(%new_tailoring)""" 4
1058 .el .IP "\f(CW$modified_collator = $Collator\->change(%new_tailoring)\fR" 4
1059 .IX Item "$modified_collator = $Collator->change(%new_tailoring)"
1060 .PD
1061 Changes the value of specified keys and returns the changed part.
1062 .Sp
1063 .Vb 1
1064 \& $Collator = Unicode::Collate\->new(level => 4);
1065 \&
1066 \& $Collator\->eq("perl", "PERL"); # false
1067 \&
1068 \& %old = $Collator\->change(level => 2); # returns (level => 4).
1069 \&
1070 \& $Collator\->eq("perl", "PERL"); # true
1071 \&
1072 \& $Collator\->change(%old); # returns (level => 2).
1073 \&
1074 \& $Collator\->eq("perl", "PERL"); # false
1075 .Ve
1076 .Sp
1077 Not all \f(CW\*(C`(key,value)\*(C'\fRs are allowed to be changed.
1078 See also \f(CW@Unicode::Collate::ChangeOK\fR and \f(CW@Unicode::Collate::ChangeNG\fR.
1079 .Sp
1080 In the scalar context, returns the modified collator
1081 (but it is \fBnot\fR a clone from the original).
1082 .Sp
1083 .Vb 1
1084 \& $Collator\->change(level => 2)\->eq("perl", "PERL"); # true
1085 \&
1086 \& $Collator\->eq("perl", "PERL"); # true; now max level is 2nd.
1087 \&
1088 \& $Collator\->change(level => 4)\->eq("perl", "PERL"); # false
1089 .Ve
1090 .ie n .IP """$version = $Collator\->version()""" 4
1091 .el .IP "\f(CW$version = $Collator\->version()\fR" 4
1092 .IX Item "$version = $Collator->version()"
1093 Returns the version number (a string) of the Unicode Standard
1094 which the \f(CW\*(C`table\*(C'\fR file used by the collator object is based on.
1095 If the table does not include a version line (starting with \f(CW@version\fR),
1096 returns \f(CW"unknown"\fR.
1097 .ie n .IP UCA_Version() 4
1098 .el .IP \f(CWUCA_Version()\fR 4
1099 .IX Item "UCA_Version()"
1100 Returns the revision number of UTS #10 this module consults,
1101 that should correspond with the DUCET incorporated.
1102 .ie n .IP Base_Unicode_Version() 4
1103 .el .IP \f(CWBase_Unicode_Version()\fR 4
1104 .IX Item "Base_Unicode_Version()"
1105 Returns the version number of UTS #10 this module consults,
1106 that should correspond with the DUCET incorporated.
1107 .SH EXPORT
1108 .IX Header "EXPORT"
1109 No method will be exported.
1110 .SH INSTALL
1111 .IX Header "INSTALL"
1112 Though this module can be used without any \f(CW\*(C`table\*(C'\fR file,
1113 to use this module easily, it is recommended to install a table file
1114 in the UCA format, by copying it under the directory
1115 <a place in \f(CW@INC\fR>/Unicode/Collate.
1116 .PP
1117 The most preferable one is "The Default Unicode Collation Element Table"
1118 (aka DUCET), available from the Unicode Consortium\*(Aqs website:
1119 .PP
1120 .Vb 1
1121 \& http://www.unicode.org/Public/UCA/
1122 \&
1123 \& http://www.unicode.org/Public/UCA/latest/allkeys.txt
1124 \& (latest version)
1125 .Ve
1126 .PP
1127 If DUCET is not installed, it is recommended to copy the file
1128 from http://www.unicode.org/Public/UCA/latest/allkeys.txt
1129 to <a place in \f(CW@INC\fR>/Unicode/Collate/allkeys.txt
1130 manually.
1131 .SH CAVEATS
1132 .IX Header "CAVEATS"
1133 .IP Normalization 4
1134 .IX Item "Normalization"
1135 Use of the \f(CW\*(C`normalization\*(C'\fR parameter requires the \fBUnicode::Normalize\fR
1136 module (see Unicode::Normalize).
1137 .Sp
1138 If you need not it (say, in the case when you need not
1139 handle any combining characters),
1140 assign \f(CW\*(C`(normalization => undef)\*(C'\fR explicitly.
1141 .Sp
1142 \&\-\- see 6.5 Avoiding Normalization, UTS #10.
1143 .IP "Conformance Test" 4
1144 .IX Item "Conformance Test"
1145 The Conformance Test for the UCA is available
1146 under <http://www.unicode.org/Public/UCA/>.
1147 .Sp
1148 For \fICollationTest_SHIFTED.txt\fR,
1149 a collator via \f(CW\*(C`Unicode::Collate\->new( )\*(C'\fR should be used;
1150 for \fICollationTest_NON_IGNORABLE.txt\fR, a collator via
1151 \&\f(CW\*(C`Unicode::Collate\->new(variable => "non\-ignorable", level => 3)\*(C'\fR.
1152 .Sp
1153 If \f(CW\*(C`UCA_Version\*(C'\fR is 26 or later, the \f(CW\*(C`identical\*(C'\fR level is preferred;
1154 \&\f(CW\*(C`Unicode::Collate\->new(identical => 1)\*(C'\fR and
1155 \&\f(CW\*(C`Unicode::Collate\->new(identical => 1,\*(C'\fR
1156 \&\f(CW\*(C`variable => "non\-ignorable", level => 3)\*(C'\fR should be used.
1157 .Sp
1158 \&\fBUnicode::Normalize is required to try The Conformance Test.\fR
1159 .Sp
1160 \&\fBEBCDIC\-SUPPORT IS EXPERIMENTAL.\fR
1161 .SH "AUTHOR, COPYRIGHT AND LICENSE"
1162 .IX Header "AUTHOR, COPYRIGHT AND LICENSE"
1163 The Unicode::Collate module for perl was written by SADAHIRO Tomoyuki,
1164 <[email protected]>. This module is Copyright(C) 2001\-2021,
1165 SADAHIRO Tomoyuki. Japan. All rights reserved.
1166 .PP
1167 This module is free software; you can redistribute it and/or
1168 modify it under the same terms as Perl itself.
1169 .PP
1170 The file Unicode/Collate/allkeys.txt was copied verbatim
1171 from <http://www.unicode.org/Public/UCA/13.0.0/allkeys.txt>.
1172 For this file, Copyright (c) 2020 Unicode, Inc.; distributed
1173 under the Terms of Use in <http://www.unicode.org/terms_of_use.html>
1174 .SH "SEE ALSO"
1175 .IX Header "SEE ALSO"
1176 .IP "Unicode Collation Algorithm \- UTS #10" 4
1177 .IX Item "Unicode Collation Algorithm - UTS #10"
1178 <http://www.unicode.org/reports/tr10/>
1179 .IP "The Default Unicode Collation Element Table (DUCET)" 4
1180 .IX Item "The Default Unicode Collation Element Table (DUCET)"
1181 <http://www.unicode.org/Public/UCA/latest/allkeys.txt>
1182 .IP "The conformance test for the UCA" 4
1183 .IX Item "The conformance test for the UCA"
1184 <http://www.unicode.org/Public/UCA/latest/CollationTest.html>
1185 .Sp
1186 <http://www.unicode.org/Public/UCA/latest/CollationTest.zip>
1187 .IP "Hangul Syllable Type" 4
1188 .IX Item "Hangul Syllable Type"
1189 <http://www.unicode.org/Public/UNIDATA/HangulSyllableType.txt>
1190 .IP "Unicode Normalization Forms \- UAX #15" 4
1191 .IX Item "Unicode Normalization Forms - UAX #15"
1192 <http://www.unicode.org/reports/tr15/>
1193 .IP "Unicode Locale Data Markup Language (LDML) \- UTS #35" 4
1194 .IX Item "Unicode Locale Data Markup Language (LDML) - UTS #35"
1195 <http://www.unicode.org/reports/tr35/>