Debian Codegraph @

upstream/debian-unstable/man3/Unicode::Collate.3perl

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man 5.0102 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\" ========================================================================
56 .\"
57 .IX Title "Unicode::Collate 3perl"
58 .TH Unicode::Collate 3perl 2025-11-16 "perl v5.40.1" "Perl Programmers Reference Guide"
59 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
60 .\" way too many mistakes in technical documents.
61 .if n .ad l
62 .nh
63 .SH NAME
64 Unicode::Collate \- Unicode Collation Algorithm
65 .SH SYNOPSIS
66 .IX Header "SYNOPSIS"
67 .Vb 1
68 \& use Unicode::Collate;
69 \&
70 \& #construct
71 \& $Collator = Unicode::Collate\->new(%tailoring);
72 \&
73 \& #sort
74 \& @sorted = $Collator\->sort(@not_sorted);
75 \&
76 \& #compare
77 \& $result = $Collator\->cmp($a, $b); # returns 1, 0, or \-1.
78 .Ve
79 .PP
80 \&\fBNote:\fR Strings in \f(CW@not_sorted\fR, \f(CW$a\fR and \f(CW$b\fR are interpreted
81 according to Perl's Unicode support. See perlunicode,
82 perluniintro, perlunitut, perlunifaq, utf8.
83 Otherwise you can use \f(CW\*(C`preprocess\*(C'\fR or should decode them before.
84 .SH DESCRIPTION
85 .IX Header "DESCRIPTION"
86 This module is an implementation of Unicode Technical Standard #10
87 (a.k.a. UTS #10) \- Unicode Collation Algorithm (a.k.a. UCA).
88 .SS "Constructor and Tailoring"
89 .IX Subsection "Constructor and Tailoring"
90 The \f(CW\*(C`new\*(C'\fR method returns a collator object. If \fBnew()\fR is called
91 with no parameters, the collator should do the default collation.
92 .PP
93 .Vb 10
94 \& $Collator = Unicode::Collate\->new(
95 \& UCA_Version => $UCA_Version,
96 \& alternate => $alternate, # alias for \*(Aqvariable\*(Aq
97 \& backwards => $levelNumber, # or \e@levelNumbers
98 \& entry => $element,
99 \& hangul_terminator => $term_primary_weight,
100 \& highestFFFF => $bool,
101 \& identical => $bool,
102 \& ignoreName => qr/$ignoreName/,
103 \& ignoreChar => qr/$ignoreChar/,
104 \& ignore_level2 => $bool,
105 \& katakana_before_hiragana => $bool,
106 \& level => $collationLevel,
107 \& long_contraction => $bool,
108 \& minimalFFFE => $bool,
109 \& normalization => $normalization_form,
110 \& overrideCJK => \e&overrideCJK,
111 \& overrideHangul => \e&overrideHangul,
112 \& preprocess => \e&preprocess,
113 \& rearrange => \e@charList,
114 \& rewrite => \e&rewrite,
115 \& suppress => \e@charList,
116 \& table => $filename,
117 \& undefName => qr/$undefName/,
118 \& undefChar => qr/$undefChar/,
119 \& upper_before_lower => $bool,
120 \& variable => $variable,
121 \& );
122 .Ve
123 .IP UCA_Version 4
124 .IX Item "UCA_Version"
125 If the revision (previously "tracking version") number of UCA is given,
126 behavior of that revision is emulated on collating.
127 If omitted, the return value of \f(CWUCA_Version()\fR is used.
128 .Sp
129 The following revisions are supported. The default is 43.
130 .Sp
131 .Vb 10
132 \& UCA Unicode Standard DUCET (@version)
133 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
134 \& 8 3.1 3.0.1 (3.0.1d9)
135 \& 9 3.1 with Corrigendum 3 3.1.1
136 \& 11 4.0.0
137 \& 14 4.1.0
138 \& 16 5.0.0
139 \& 18 5.1.0
140 \& 20 5.2.0
141 \& 22 6.0.0
142 \& 24 6.1.0
143 \& 26 6.2.0
144 \& 28 6.3.0
145 \& 30 7.0.0
146 \& 32 8.0.0
147 \& 34 9.0.0
148 \& 36 10.0.0
149 \& 38 11.0.0
150 \& 40 12.0.0
151 \& 41 12.1.0
152 \& 43 13.0.0
153 .Ve
154 .Sp
155 * See below for \f(CW\*(C`long_contraction\*(C'\fR with \f(CW\*(C`UCA_Version\*(C'\fR 22 and 24.
156 .Sp
157 * Noncharacters (e.g. U+FFFF) are not ignored, and can be overridden
158 since \f(CW\*(C`UCA_Version\*(C'\fR 22.
159 .Sp
160 * Out-of-range codepoints (greater than U+10FFFF) are not ignored,
161 and can be overridden since \f(CW\*(C`UCA_Version\*(C'\fR 22.
162 .Sp
163 * Fully ignorable characters were ignored, and would not interrupt
164 contractions with \f(CW\*(C`UCA_Version\*(C'\fR 9 and 11.
165 .Sp
166 * Treatment of ignorables after variables and some behaviors
167 were changed at \f(CW\*(C`UCA_Version\*(C'\fR 9.
168 .Sp
169 * Characters regarded as CJK unified ideographs (cf. \f(CW\*(C`overrideCJK\*(C'\fR)
170 depend on \f(CW\*(C`UCA_Version\*(C'\fR.
171 .Sp
172 * Many hangul jamo are assigned at \f(CW\*(C`UCA_Version\*(C'\fR 20, that will affect
173 \&\f(CW\*(C`hangul_terminator\*(C'\fR.
174 .IP alternate 4
175 .IX Item "alternate"
176 \&\-\- see 3.2.2 Alternate Weighting, version 8 of UTS #10
177 .Sp
178 For backward compatibility, \f(CW\*(C`alternate\*(C'\fR (old name) can be used
179 as an alias for \f(CW\*(C`variable\*(C'\fR.
180 .IP backwards 4
181 .IX Item "backwards"
182 \&\-\- see 3.4 Backward Accents, UTS #10.
183 .Sp
184 .Vb 1
185 \& backwards => $levelNumber or \e@levelNumbers
186 .Ve
187 .Sp
188 Weights in reverse order; ex. level 2 (diacritic ordering) in French.
189 If omitted (or \f(CW$levelNumber\fR is \f(CW\*(C`undef\*(C'\fR or \f(CW\*(C`\e@levelNumbers\*(C'\fR is \f(CW\*(C`[]\*(C'\fR),
190 forwards at all the levels.
191 .IP entry 4
192 .IX Item "entry"
193 \&\-\- see 5 Tailoring; 9.1 Allkeys File Format, UTS #10.
194 .Sp
195 If the same character (or a sequence of characters) exists
196 in the collation element table through \f(CW\*(C`table\*(C'\fR,
197 mapping to collation elements is overridden.
198 If it does not exist, the mapping is defined additionally.
199 .Sp
200 .Vb 12
201 \& entry => <<\*(AqENTRY\*(Aq, # for DUCET v4.0.0 (allkeys\-4.0.0.txt)
202 \&0063 0068 ; [.0E6A.0020.0002.0063] # ch
203 \&0043 0068 ; [.0E6A.0020.0007.0043] # Ch
204 \&0043 0048 ; [.0E6A.0020.0008.0043] # CH
205 \&006C 006C ; [.0F4C.0020.0002.006C] # ll
206 \&004C 006C ; [.0F4C.0020.0007.004C] # Ll
207 \&004C 004C ; [.0F4C.0020.0008.004C] # LL
208 \&00F1 ; [.0F7B.0020.0002.00F1] # n\-tilde
209 \&006E 0303 ; [.0F7B.0020.0002.00F1] # n\-tilde
210 \&00D1 ; [.0F7B.0020.0008.00D1] # N\-tilde
211 \&004E 0303 ; [.0F7B.0020.0008.00D1] # N\-tilde
212 \&ENTRY
213 \&
214 \& entry => <<\*(AqENTRY\*(Aq, # for DUCET v4.0.0 (allkeys\-4.0.0.txt)
215 \&00E6 ; [.0E33.0020.0002.00E6][.0E8B.0020.0002.00E6] # ae ligature as <a><e>
216 \&00C6 ; [.0E33.0020.0008.00C6][.0E8B.0020.0008.00C6] # AE ligature as <A><E>
217 \&ENTRY
218 .Ve
219 .Sp
220 \&\fBNOTE:\fR The code point in the UCA file format (before \f(CW\*(Aq;\*(Aq\fR)
221 \&\fBmust\fR be a Unicode code point (defined as hexadecimal),
222 but not a native code point.
223 So \f(CW0063\fR must always denote \f(CW\*(C`U+0063\*(C'\fR,
224 but not a character of \f(CW"\ex63"\fR.
225 .Sp
226 Weighting may vary depending on collation element table.
227 So ensure the weights defined in \f(CW\*(C`entry\*(C'\fR will be consistent with
228 those in the collation element table loaded via \f(CW\*(C`table\*(C'\fR.
229 .Sp
230 In DUCET v4.0.0, primary weight of \f(CW\*(C`C\*(C'\fR is \f(CW0E60\fR
231 and that of \f(CW\*(C`D\*(C'\fR is \f(CW\*(C`0E6D\*(C'\fR. So setting primary weight of \f(CW\*(C`CH\*(C'\fR to \f(CW\*(C`0E6A\*(C'\fR
232 (as a value between \f(CW0E60\fR and \f(CW\*(C`0E6D\*(C'\fR)
233 makes ordering as \f(CW\*(C`C < CH < D\*(C'\fR.
234 Exactly speaking DUCET already has some characters between \f(CW\*(C`C\*(C'\fR and \f(CW\*(C`D\*(C'\fR:
235 \&\f(CW\*(C`small capital C\*(C'\fR (\f(CW\*(C`U+1D04\*(C'\fR) with primary weight \f(CW0E64\fR,
236 \&\f(CW\*(C`c\-hook/C\-hook\*(C'\fR (\f(CW\*(C`U+0188/U+0187\*(C'\fR) with \f(CW0E65\fR,
237 and \f(CW\*(C`c\-curl\*(C'\fR (\f(CW\*(C`U+0255\*(C'\fR) with \f(CW0E69\fR.
238 Then primary weight \f(CW\*(C`0E6A\*(C'\fR for \f(CW\*(C`CH\*(C'\fR makes \f(CW\*(C`CH\*(C'\fR
239 ordered between \f(CW\*(C`c\-curl\*(C'\fR and \f(CW\*(C`D\*(C'\fR.
240 .IP hangul_terminator 4
241 .IX Item "hangul_terminator"
242 \&\-\- see 7.1.4 Trailing Weights, UTS #10.
243 .Sp
244 If a true value is given (non-zero but should be positive),
245 it will be added as a terminator primary weight to the end of
246 every standard Hangul syllable. Secondary and any higher weights
247 for terminator are set to zero.
248 If the value is false or \f(CW\*(C`hangul_terminator\*(C'\fR key does not exist,
249 insertion of terminator weights will not be performed.
250 .Sp
251 Boundaries of Hangul syllables are determined
252 according to conjoining Jamo behavior in \fIthe Unicode Standard\fR
253 and \fIHangulSyllableType.txt\fR.
254 .Sp
255 \&\fBImplementation Note:\fR
256 (1) For expansion mapping (Unicode character mapped
257 to a sequence of collation elements), a terminator will not be added
258 between collation elements, even if Hangul syllable boundary exists there.
259 Addition of terminator is restricted to the next position
260 to the last collation element.
261 .Sp
262 (2) Non-conjoining Hangul letters
263 (Compatibility Jamo, halfwidth Jamo, and enclosed letters) are not
264 automatically terminated with a terminator primary weight.
265 These characters may need terminator included in a collation element
266 table beforehand.
267 .IP highestFFFF 4
268 .IX Item "highestFFFF"
269 \&\-\- see 2.4 Tailored noncharacter weights, UTS #35 (LDML) Part 5: Collation.
270 .Sp
271 If the parameter is made true, \f(CW\*(C`U+FFFF\*(C'\fR has a highest primary weight.
272 When a boolean of \f(CW\*(C`$coll\->ge($str, "abc")\*(C'\fR and
273 \&\f(CW\*(C`$coll\->le($str, "abc\ex{FFFF}")\*(C'\fR is true, it is expected that \f(CW$str\fR
274 begins with \f(CW"abc"\fR, or another primary equivalent.
275 \&\f(CW$str\fR may be \f(CW"abcd"\fR, \f(CW"abc012"\fR, but should not include \f(CW\*(C`U+FFFF\*(C'\fR
276 such as \f(CW"abc\ex{FFFF}xyz"\fR.
277 .Sp
278 \&\f(CW\*(C`$coll\->le($str, "abc\ex{FFFF}")\*(C'\fR works like \f(CW\*(C`$coll\->lt($str, "abd")\*(C'\fR
279 almost, but the latter has a problem that you should know which letter is
280 next to \f(CW\*(C`c\*(C'\fR. For a certain language where \f(CW\*(C`ch\*(C'\fR as the next letter,
281 \&\f(CW"abch"\fR is greater than \f(CW"abc\ex{FFFF}"\fR, but less than \f(CW"abd"\fR.
282 .Sp
283 Note:
284 This is equivalent to \f(CW\*(C`(entry => \*(AqFFFF ; [.FFFE.0020.0005.FFFF]\*(Aq)\*(C'\fR.
285 Any other character than \f(CW\*(C`U+FFFF\*(C'\fR can be tailored by \f(CW\*(C`entry\*(C'\fR.
286 .IP identical 4
287 .IX Item "identical"
288 \&\-\- see A.3 Deterministic Comparison, UTS #10.
289 .Sp
290 By default, strings whose weights are equal should be equal,
291 even though their code points are not equal.
292 Completely ignorable characters are ignored.
293 .Sp
294 If the parameter is made true, a final, tie-breaking level is used.
295 If no difference of weights is found after the comparison through
296 all the level specified by \f(CW\*(C`level\*(C'\fR, the comparison with code points
297 will be performed.
298 For the tie-breaking comparison, the sort key has code points
299 of the original string appended.
300 Completely ignorable characters are not ignored.
301 .Sp
302 If \f(CW\*(C`preprocess\*(C'\fR and/or \f(CW\*(C`normalization\*(C'\fR is applied, the code points
303 of the string after them (in NFD by default) are used.
304 .IP ignoreChar 4
305 .IX Item "ignoreChar"
306 .PD 0
307 .IP ignoreName 4
308 .IX Item "ignoreName"
309 .PD
310 \&\-\- see 3.6 Variable Weighting, UTS #10.
311 .Sp
312 Makes the entry in the table completely ignorable;
313 i.e. as if the weights were zero at all level.
314 .Sp
315 Through \f(CW\*(C`ignoreChar\*(C'\fR, any character matching \f(CW\*(C`qr/$ignoreChar/\*(C'\fR
316 will be ignored. Through \f(CW\*(C`ignoreName\*(C'\fR, any character whose name
317 (given in the \f(CW\*(C`table\*(C'\fR file as a comment) matches \f(CW\*(C`qr/$ignoreName/\*(C'\fR
318 will be ignored.
319 .Sp
320 E.g. when 'a' and 'e' are ignorable,
321 \&'element' is equal to 'lament' (or 'lmnt').
322 .IP ignore_level2 4
323 .IX Item "ignore_level2"
324 \&\-\- see 5.1 Parametric Tailoring, UTS #10.
325 .Sp
326 By default, case-sensitive comparison (that is level 3 difference)
327 won't ignore accents (that is level 2 difference).
328 .Sp
329 If the parameter is made true, accents (and other primary ignorable
330 characters) are ignored, even though cases are taken into account.
331 .Sp
332 \&\fBNOTE\fR: \f(CW\*(C`level\*(C'\fR should be 3 or greater.
333 .IP katakana_before_hiragana 4
334 .IX Item "katakana_before_hiragana"
335 \&\-\- see 7.2 Tertiary Weight Table, UTS #10.
336 .Sp
337 By default, hiragana is before katakana.
338 If the parameter is made true, this is reversed.
339 .Sp
340 \&\fBNOTE\fR: This parameter simplemindedly assumes that any hiragana/katakana
341 distinctions must occur in level 3, and their weights at level 3 must be
342 same as those mentioned in 7.3.1, UTS #10.
343 If you define your collation elements which violate this requirement,
344 this parameter does not work validly.
345 .IP level 4
346 .IX Item "level"
347 \&\-\- see 4.3 Form Sort Key, UTS #10.
348 .Sp
349 Set the maximum level.
350 Any higher levels than the specified one are ignored.
351 .Sp
352 .Vb 4
353 \& Level 1: alphabetic ordering
354 \& Level 2: diacritic ordering
355 \& Level 3: case ordering
356 \& Level 4: tie\-breaking (e.g. in the case when variable is \*(Aqshifted\*(Aq)
357 \&
358 \& ex.level => 2,
359 .Ve
360 .Sp
361 If omitted, the maximum is the 4th.
362 .Sp
363 \&\fBNOTE:\fR The DUCET includes weights over 0xFFFF at the 4th level.
364 But this module only uses weights within 0xFFFF.
365 When \f(CW\*(C`variable\*(C'\fR is 'blanked' or 'non\-ignorable' (other than 'shifted'
366 and 'shift\-trimmed'), the level 4 may be unreliable.
367 .Sp
368 See also \f(CW\*(C`identical\*(C'\fR.
369 .IP long_contraction 4
370 .IX Item "long_contraction"
371 \&\-\- see 3.8.2 Well-Formedness of the DUCET, 4.2 Produce Array, UTS #10.
372 .Sp
373 If the parameter is made true, for a contraction with three or more
374 characters (here nicknamed "long contraction"), initial substrings
375 will be handled.
376 For example, a contraction ABC, where A is a starter, and B and C
377 are non-starters (character with non-zero combining character class),
378 will be detected even if there is not AB as a contraction.
379 .Sp
380 \&\fBDefault:\fR Usually false.
381 If \f(CW\*(C`UCA_Version\*(C'\fR is 22 or 24, and the value of \f(CW\*(C`long_contraction\*(C'\fR
382 is not specified in \f(CWnew()\fR, a true value is set implicitly.
383 This is a workaround to pass Conformance Tests for Unicode 6.0.0 and 6.1.0.
384 .Sp
385 \&\f(CWchange()\fR handles \f(CW\*(C`long_contraction\*(C'\fR explicitly only.
386 If \f(CW\*(C`long_contraction\*(C'\fR is not specified in \f(CWchange()\fR, even though
387 \&\f(CW\*(C`UCA_Version\*(C'\fR is changed, \f(CW\*(C`long_contraction\*(C'\fR will not be changed.
388 .Sp
389 \&\fBLimitation:\fR Scanning non-starters is one-way (no back tracking).
390 If AB is found but not ABC is not found, other long contraction where
391 the first character is A and the second is not B may not be found.
392 .Sp
393 Under \f(CW\*(C`(normalization => undef)\*(C'\fR, detection step of discontiguous
394 contractions will be skipped.
395 .Sp
396 \&\fBNote:\fR The following contractions in DUCET are not considered
397 in steps S2.1.1 to S2.1.3, where they are discontiguous.
398 .Sp
399 .Vb 2
400 \& 0FB2 0F71 0F80 (TIBETAN VOWEL SIGN VOCALIC RR)
401 \& 0FB3 0F71 0F80 (TIBETAN VOWEL SIGN VOCALIC LL)
402 .Ve
403 .Sp
404 For example \f(CW\*(C`TIBETAN VOWEL SIGN VOCALIC RR\*(C'\fR with \f(CW\*(C`COMBINING TILDE OVERLAY\*(C'\fR
405 (\f(CW\*(C`U+0344\*(C'\fR) is \f(CW\*(C`0FB2 0344 0F71 0F80\*(C'\fR in NFD.
406 In this case \f(CW\*(C`0FB2 0F80\*(C'\fR (\f(CW\*(C`TIBETAN VOWEL SIGN VOCALIC R\*(C'\fR) is detected,
407 instead of \f(CW\*(C`0FB2 0F71 0F80\*(C'\fR.
408 Inserted \f(CW0344\fR makes \f(CW\*(C`0FB2 0F71 0F80\*(C'\fR discontiguous and lack of
409 contraction \f(CW\*(C`0FB2 0F71\*(C'\fR prohibits \f(CW\*(C`0FB2 0F71 0F80\*(C'\fR from being detected.
410 .IP minimalFFFE 4
411 .IX Item "minimalFFFE"
412 \&\-\- see 1.1.1 U+FFFE, UTS #35 (LDML) Part 5: Collation.
413 .Sp
414 If the parameter is made true, \f(CW\*(C`U+FFFE\*(C'\fR has a minimal primary weight.
415 The comparison between \f(CW"$a1\ex{FFFE}$a2"\fR and \f(CW"$b1\ex{FFFE}$b2"\fR
416 first compares \f(CW$a1\fR and \f(CW$b1\fR at level 1, and
417 then \f(CW$a2\fR and \f(CW$b2\fR at level 1, as followed.
418 .Sp
419 .Vb 12
420 \& "ab\ex{FFFE}a"
421 \& "Ab\ex{FFFE}a"
422 \& "ab\ex{FFFE}c"
423 \& "Ab\ex{FFFE}c"
424 \& "ab\ex{FFFE}xyz"
425 \& "abc\ex{FFFE}def"
426 \& "abc\ex{FFFE}xYz"
427 \& "aBc\ex{FFFE}xyz"
428 \& "abcX\ex{FFFE}def"
429 \& "abcx\ex{FFFE}xyz"
430 \& "b\ex{FFFE}aaa"
431 \& "bbb\ex{FFFE}a"
432 .Ve
433 .Sp
434 Note:
435 This is equivalent to \f(CW\*(C`(entry => \*(AqFFFE ; [.0001.0020.0005.FFFE]\*(Aq)\*(C'\fR.
436 Any other character than \f(CW\*(C`U+FFFE\*(C'\fR can be tailored by \f(CW\*(C`entry\*(C'\fR.
437 .IP normalization 4
438 .IX Item "normalization"
439 \&\-\- see 4.1 Normalize, UTS #10.
440 .Sp
441 If specified, strings are normalized before preparation of sort keys
442 (the normalization is executed after preprocess).
443 .Sp
444 A form name \f(CWUnicode::Normalize::normalize()\fR accepts will be applied
445 as \f(CW$normalization_form\fR.
446 Acceptable names include \f(CW\*(AqNFD\*(Aq\fR, \f(CW\*(AqNFC\*(Aq\fR, \f(CW\*(AqNFKD\*(Aq\fR, and \f(CW\*(AqNFKC\*(Aq\fR.
447 See \f(CWUnicode::Normalize::normalize()\fR for detail.
448 If omitted, \f(CW\*(AqNFD\*(Aq\fR is used.
449 .Sp
450 \&\f(CW\*(C`normalization\*(C'\fR is performed after \f(CW\*(C`preprocess\*(C'\fR (if defined).
451 .Sp
452 Furthermore, special values, \f(CW\*(C`undef\*(C'\fR and \f(CW"prenormalized"\fR, can be used,
453 though they are not concerned with \f(CWUnicode::Normalize::normalize()\fR.
454 .Sp
455 If \f(CW\*(C`undef\*(C'\fR (not a string \f(CW"undef"\fR) is passed explicitly
456 as the value for this key,
457 any normalization is not carried out (this may make tailoring easier
458 if any normalization is not desired). Under \f(CW\*(C`(normalization => undef)\*(C'\fR,
459 only contiguous contractions are resolved;
460 e.g. even if \f(CW\*(C`A\-ring\*(C'\fR (and \f(CW\*(C`A\-ring\-cedilla\*(C'\fR) is ordered after \f(CW\*(C`Z\*(C'\fR,
461 \&\f(CW\*(C`A\-cedilla\-ring\*(C'\fR would be primary equal to \f(CW\*(C`A\*(C'\fR.
462 In this point,
463 \&\f(CW\*(C`(normalization => undef, preprocess => sub { NFD(shift) })\*(C'\fR
464 \&\fBis not\fR equivalent to \f(CW\*(C`(normalization => \*(AqNFD\*(Aq)\*(C'\fR.
465 .Sp
466 In the case of \f(CW\*(C`(normalization => "prenormalized")\*(C'\fR,
467 any normalization is not performed, but
468 discontiguous contractions with combining characters are performed.
469 Therefore
470 \&\f(CW\*(C`(normalization => \*(Aqprenormalized\*(Aq, preprocess => sub { NFD(shift) })\*(C'\fR
471 \&\fBis\fR equivalent to \f(CW\*(C`(normalization => \*(AqNFD\*(Aq)\*(C'\fR.
472 If source strings are finely prenormalized,
473 \&\f(CW\*(C`(normalization => \*(Aqprenormalized\*(Aq)\*(C'\fR may save time for normalization.
474 .Sp
475 Except \f(CW\*(C`(normalization => undef)\*(C'\fR,
476 \&\fBUnicode::Normalize\fR is required (see also \fBCAVEAT\fR).
477 .IP overrideCJK 4
478 .IX Item "overrideCJK"
479 \&\-\- see 7.1 Derived Collation Elements, UTS #10.
480 .Sp
481 By default, CJK unified ideographs are ordered in Unicode codepoint
482 order, but those in the CJK Unified Ideographs block are less than
483 those in the CJK Unified Ideographs Extension A etc.
484 .Sp
485 .Vb 10
486 \& In the CJK Unified Ideographs block:
487 \& U+4E00..U+9FA5 if UCA_Version is 8, 9 or 11.
488 \& U+4E00..U+9FBB if UCA_Version is 14 or 16.
489 \& U+4E00..U+9FC3 if UCA_Version is 18.
490 \& U+4E00..U+9FCB if UCA_Version is 20 or 22.
491 \& U+4E00..U+9FCC if UCA_Version is 24 to 30.
492 \& U+4E00..U+9FD5 if UCA_Version is 32 or 34.
493 \& U+4E00..U+9FEA if UCA_Version is 36.
494 \& U+4E00..U+9FEF if UCA_Version is 38, 40 or 41.
495 \& U+4E00..U+9FFC if UCA_Version is 43.
496 \&
497 \& In the CJK Unified Ideographs Extension blocks:
498 \& Ext.A (U+3400..U+4DB5) if UCA_Version is 8 to 41.
499 \& Ext.A (U+3400..U+4DBF) if UCA_Version is 43.
500 \& Ext.B (U+20000..U+2A6D6) if UCA_Version is 8 to 41.
501 \& Ext.B (U+20000..U+2A6DD) if UCA_Version is 43.
502 \& Ext.C (U+2A700..U+2B734) if UCA_Version is 20 or later.
503 \& Ext.D (U+2B740..U+2B81D) if UCA_Version is 22 or later.
504 \& Ext.E (U+2B820..U+2CEA1) if UCA_Version is 32 or later.
505 \& Ext.F (U+2CEB0..U+2EBE0) if UCA_Version is 36 or later.
506 \& Ext.G (U+30000..U+3134A) if UCA_Version is 43.
507 .Ve
508 .Sp
509 Through \f(CW\*(C`overrideCJK\*(C'\fR, ordering of CJK unified ideographs (including
510 extensions) can be overridden.
511 .Sp
512 ex. CJK unified ideographs in the JIS code point order.
513 .Sp
514 .Vb 7
515 \& overrideCJK => sub {
516 \& my $u = shift; # get a Unicode codepoint
517 \& my $b = pack(\*(Aqn\*(Aq, $u); # to UTF\-16BE
518 \& my $s = your_unicode_to_sjis_converter($b); # convert
519 \& my $n = unpack(\*(Aqn\*(Aq, $s); # convert sjis to short
520 \& [ $n, 0x20, 0x2, $u ]; # return the collation element
521 \& },
522 .Ve
523 .Sp
524 The return value may be an arrayref of 1st to 4th weights as shown
525 above. The return value may be an integer as the primary weight
526 as shown below. If \f(CW\*(C`undef\*(C'\fR is returned, the default derived
527 collation element will be used.
528 .Sp
529 .Vb 7
530 \& overrideCJK => sub {
531 \& my $u = shift; # get a Unicode codepoint
532 \& my $b = pack(\*(Aqn\*(Aq, $u); # to UTF\-16BE
533 \& my $s = your_unicode_to_sjis_converter($b); # convert
534 \& my $n = unpack(\*(Aqn\*(Aq, $s); # convert sjis to short
535 \& return $n; # return the primary weight
536 \& },
537 .Ve
538 .Sp
539 The return value may be a list containing zero or more of
540 an arrayref, an integer, or \f(CW\*(C`undef\*(C'\fR.
541 .Sp
542 ex. ignores all CJK unified ideographs.
543 .Sp
544 .Vb 1
545 \& overrideCJK => sub {()}, # CODEREF returning empty list
546 \&
547 \& # where \->eq("Pe\ex{4E00}rl", "Perl") is true
548 \& # as U+4E00 is a CJK unified ideograph and to be ignorable.
549 .Ve
550 .Sp
551 If a false value (including \f(CW\*(C`undef\*(C'\fR) is passed, \f(CW\*(C`overrideCJK\*(C'\fR
552 has no effect.
553 \&\f(CW\*(C`$Collator\->change(overrideCJK => 0)\*(C'\fR resets the old one.
554 .Sp
555 But assignment of weight for CJK unified ideographs
556 in \f(CW\*(C`table\*(C'\fR or \f(CW\*(C`entry\*(C'\fR is still valid.
557 If \f(CW\*(C`undef\*(C'\fR is passed explicitly as the value for this key,
558 weights for CJK unified ideographs are treated as undefined.
559 However when \f(CW\*(C`UCA_Version\*(C'\fR > 8, \f(CW\*(C`(overrideCJK => undef)\*(C'\fR
560 has no special meaning.
561 .Sp
562 \&\fBNote:\fR In addition to them, 12 CJK compatibility ideographs (\f(CW\*(C`U+FA0E\*(C'\fR,
563 \&\f(CW\*(C`U+FA0F\*(C'\fR, \f(CW\*(C`U+FA11\*(C'\fR, \f(CW\*(C`U+FA13\*(C'\fR, \f(CW\*(C`U+FA14\*(C'\fR, \f(CW\*(C`U+FA1F\*(C'\fR, \f(CW\*(C`U+FA21\*(C'\fR, \f(CW\*(C`U+FA23\*(C'\fR,
564 \&\f(CW\*(C`U+FA24\*(C'\fR, \f(CW\*(C`U+FA27\*(C'\fR, \f(CW\*(C`U+FA28\*(C'\fR, \f(CW\*(C`U+FA29\*(C'\fR) are also treated as CJK unified
565 ideographs. But they can't be overridden via \f(CW\*(C`overrideCJK\*(C'\fR when you use
566 DUCET, as the table includes weights for them. \f(CW\*(C`table\*(C'\fR or \f(CW\*(C`entry\*(C'\fR has
567 priority over \f(CW\*(C`overrideCJK\*(C'\fR.
568 .IP overrideHangul 4
569 .IX Item "overrideHangul"
570 \&\-\- see 7.1 Derived Collation Elements, UTS #10.
571 .Sp
572 By default, Hangul syllables are decomposed into Hangul Jamo,
573 even if \f(CW\*(C`(normalization => undef)\*(C'\fR.
574 But the mapping of Hangul syllables may be overridden.
575 .Sp
576 This parameter works like \f(CW\*(C`overrideCJK\*(C'\fR, so see there for examples.
577 .Sp
578 If you want to override the mapping of Hangul syllables,
579 NFD and NFKD are not appropriate, since NFD and NFKD will decompose
580 Hangul syllables before overriding. FCD may decompose Hangul syllables
581 as the case may be.
582 .Sp
583 If a false value (but not \f(CW\*(C`undef\*(C'\fR) is passed, \f(CW\*(C`overrideHangul\*(C'\fR
584 has no effect.
585 \&\f(CW\*(C`$Collator\->change(overrideHangul => 0)\*(C'\fR resets the old one.
586 .Sp
587 If \f(CW\*(C`undef\*(C'\fR is passed explicitly as the value for this key,
588 weight for Hangul syllables is treated as undefined
589 without decomposition into Hangul Jamo.
590 But definition of weight for Hangul syllables
591 in \f(CW\*(C`table\*(C'\fR or \f(CW\*(C`entry\*(C'\fR is still valid.
592 .IP overrideOut 4
593 .IX Item "overrideOut"
594 \&\-\- see 7.1.1 Handling Ill-Formed Code Unit Sequences, UTS #10.
595 .Sp
596 Perl seems to allow out-of-range values (greater than 0x10FFFF).
597 By default, out-of-range values are replaced with \f(CW\*(C`U+FFFD\*(C'\fR
598 (REPLACEMENT CHARACTER) when \f(CW\*(C`UCA_Version\*(C'\fR >= 22,
599 or ignored when \f(CW\*(C`UCA_Version\*(C'\fR <= 20.
600 .Sp
601 When \f(CW\*(C`UCA_Version\*(C'\fR >= 22, the weights of out-of-range values
602 can be overridden. Though \f(CW\*(C`table\*(C'\fR or \f(CW\*(C`entry\*(C'\fR are available for them,
603 out-of-range values are too many.
604 .Sp
605 \&\f(CW\*(C`overrideOut\*(C'\fR can perform it algorithmically.
606 This parameter works like \f(CW\*(C`overrideCJK\*(C'\fR, so see there for examples.
607 .Sp
608 ex. ignores all out-of-range values.
609 .Sp
610 .Vb 1
611 \& overrideOut => sub {()}, # CODEREF returning empty list
612 .Ve
613 .Sp
614 If a false value (including \f(CW\*(C`undef\*(C'\fR) is passed, \f(CW\*(C`overrideOut\*(C'\fR
615 has no effect.
616 \&\f(CW\*(C`$Collator\->change(overrideOut => 0)\*(C'\fR resets the old one.
617 .Sp
618 \&\fBNOTE ABOUT U+FFFD:\fR
619 .Sp
620 UCA recommends that out-of-range values should not be ignored for security
621 reasons. Say, \f(CW"pe\ex{110000}rl"\fR should not be equal to \f(CW"perl"\fR.
622 However, \f(CW\*(C`U+FFFD\*(C'\fR is wrongly mapped to a variable collation element
623 in DUCET for Unicode 6.0.0 to 6.2.0, that means out-of-range values will be
624 ignored when \f(CW\*(C`variable\*(C'\fR isn't \f(CW\*(C`Non\-ignorable\*(C'\fR.
625 .Sp
626 The mapping of \f(CW\*(C`U+FFFD\*(C'\fR is corrected in Unicode 6.3.0.
627 see <http://www.unicode.org/reports/tr10/tr10\-28.html#Trailing_Weights>
628 (7.1.4 Trailing Weights). Such a correction is reproduced by this.
629 .Sp
630 .Vb 1
631 \& overrideOut => sub { 0xFFFD }, # CODEREF returning a very large integer
632 .Ve
633 .Sp
634 This workaround is unnecessary since Unicode 6.3.0.
635 .IP preprocess 4
636 .IX Item "preprocess"
637 \&\-\- see 5.4 Preprocessing, UTS #10.
638 .Sp
639 If specified, the coderef is used to preprocess each string
640 before the formation of sort keys.
641 .Sp
642 ex. dropping English articles, such as "a" or "the".
643 Then, "the pen" is before "a pencil".
644 .Sp
645 .Vb 5
646 \& preprocess => sub {
647 \& my $str = shift;
648 \& $str =~ s/\eb(?:an?|the)\es+//gi;
649 \& return $str;
650 \& },
651 .Ve
652 .Sp
653 \&\f(CW\*(C`preprocess\*(C'\fR is performed before \f(CW\*(C`normalization\*(C'\fR (if defined).
654 .Sp
655 ex. decoding strings in a legacy encoding such as shift-jis:
656 .Sp
657 .Vb 4
658 \& $sjis_collator = Unicode::Collate\->new(
659 \& preprocess => \e&your_shiftjis_to_unicode_decoder,
660 \& );
661 \& @result = $sjis_collator\->sort(@shiftjis_strings);
662 .Ve
663 .Sp
664 \&\fBNote:\fR Strings returned from the coderef will be interpreted
665 according to Perl's Unicode support. See perlunicode,
666 perluniintro, perlunitut, perlunifaq, utf8.
667 .IP rearrange 4
668 .IX Item "rearrange"
669 \&\-\- see 3.5 Rearrangement, UTS #10.
670 .Sp
671 Characters that are not coded in logical order and to be rearranged.
672 If \f(CW\*(C`UCA_Version\*(C'\fR is equal to or less than 11, default is:
673 .Sp
674 .Vb 1
675 \& rearrange => [ 0x0E40..0x0E44, 0x0EC0..0x0EC4 ],
676 .Ve
677 .Sp
678 If you want to disallow any rearrangement, pass \f(CW\*(C`undef\*(C'\fR or \f(CW\*(C`[]\*(C'\fR
679 (a reference to empty list) as the value for this key.
680 .Sp
681 If \f(CW\*(C`UCA_Version\*(C'\fR is equal to or greater than 14, default is \f(CW\*(C`[]\*(C'\fR
682 (i.e. no rearrangement).
683 .Sp
684 \&\fBAccording to the version 9 of UCA, this parameter shall not be used;
685 but it is not warned at present.\fR
686 .IP rewrite 4
687 .IX Item "rewrite"
688 If specified, the coderef is used to rewrite lines in \f(CW\*(C`table\*(C'\fR or \f(CW\*(C`entry\*(C'\fR.
689 The coderef will get each line, and then should return a rewritten line
690 according to the UCA file format.
691 If the coderef returns an empty line, the line will be skipped.
692 .Sp
693 e.g. any primary ignorable characters into tertiary ignorable:
694 .Sp
695 .Vb 5
696 \& rewrite => sub {
697 \& my $line = shift;
698 \& $line =~ s/\e[\e.0000\e..{4}\e..{4}\e./[.0000.0000.0000./g;
699 \& return $line;
700 \& },
701 .Ve
702 .Sp
703 This example shows rewriting weights. \f(CW\*(C`rewrite\*(C'\fR is allowed to
704 affect code points, weights, and the name.
705 .Sp
706 \&\fBNOTE\fR: \f(CW\*(C`table\*(C'\fR is available to use another table file;
707 preparing a modified table once would be more efficient than
708 rewriting lines on reading an unmodified table every time.
709 .IP suppress 4
710 .IX Item "suppress"
711 \&\-\- see 3.12 Special-Purpose Commands, UTS #35 (LDML) Part 5: Collation.
712 .Sp
713 Contractions beginning with the specified characters are suppressed,
714 even if those contractions are defined in \f(CW\*(C`table\*(C'\fR.
715 .Sp
716 An example for Russian and some languages using the Cyrillic script:
717 .Sp
718 .Vb 1
719 \& suppress => [0x0400..0x0417, 0x041A..0x0437, 0x043A..0x045F],
720 .Ve
721 .Sp
722 where 0x0400 stands for \f(CW\*(C`U+0400\*(C'\fR, CYRILLIC CAPITAL LETTER IE WITH GRAVE.
723 .Sp
724 \&\fBNOTE\fR: Contractions via \f(CW\*(C`entry\*(C'\fR will not be suppressed.
725 .IP table 4
726 .IX Item "table"
727 \&\-\- see 3.8 Default Unicode Collation Element Table, UTS #10.
728 .Sp
729 You can use another collation element table if desired.
730 .Sp
731 The table file should locate in the \fIUnicode/Collate\fR directory
732 on \f(CW@INC\fR. Say, if the filename is \fIFoo.txt\fR,
733 the table file is searched as \fIUnicode/Collate/Foo.txt\fR in \f(CW@INC\fR.
734 .Sp
735 By default, \fIallkeys.txt\fR (as the filename of DUCET) is used.
736 If you will prepare your own table file, any name other than \fIallkeys.txt\fR
737 may be better to avoid namespace conflict.
738 .Sp
739 \&\fBNOTE\fR: When XSUB is used, the DUCET is compiled on building this
740 module, and it may save time at the run time.
741 Explicit saying \f(CW\*(C`(table => \*(Aqallkeys.txt\*(Aq)\*(C'\fR, or using another table,
742 or using \f(CW\*(C`ignoreChar\*(C'\fR, \f(CW\*(C`ignoreName\*(C'\fR, \f(CW\*(C`undefChar\*(C'\fR, \f(CW\*(C`undefName\*(C'\fR or
743 \&\f(CW\*(C`rewrite\*(C'\fR will prevent this module from using the compiled DUCET.
744 .Sp
745 If \f(CW\*(C`undef\*(C'\fR is passed explicitly as the value for this key,
746 no file is read (but you can define collation elements via \f(CW\*(C`entry\*(C'\fR).
747 .Sp
748 A typical way to define a collation element table
749 without any file of table:
750 .Sp
751 .Vb 11
752 \& $onlyABC = Unicode::Collate\->new(
753 \& table => undef,
754 \& entry => << \*(AqENTRIES\*(Aq,
755 \&0061 ; [.0101.0020.0002.0061] # LATIN SMALL LETTER A
756 \&0041 ; [.0101.0020.0008.0041] # LATIN CAPITAL LETTER A
757 \&0062 ; [.0102.0020.0002.0062] # LATIN SMALL LETTER B
758 \&0042 ; [.0102.0020.0008.0042] # LATIN CAPITAL LETTER B
759 \&0063 ; [.0103.0020.0002.0063] # LATIN SMALL LETTER C
760 \&0043 ; [.0103.0020.0008.0043] # LATIN CAPITAL LETTER C
761 \&ENTRIES
762 \& );
763 .Ve
764 .Sp
765 If \f(CW\*(C`ignoreName\*(C'\fR or \f(CW\*(C`undefName\*(C'\fR is used, character names should be
766 specified as a comment (following \f(CW\*(C`#\*(C'\fR) on each line.
767 .IP undefChar 4
768 .IX Item "undefChar"
769 .PD 0
770 .IP undefName 4
771 .IX Item "undefName"
772 .PD
773 \&\-\- see 6.3.3 Reducing the Repertoire, UTS #10.
774 .Sp
775 Undefines the collation element as if it were unassigned in the \f(CW\*(C`table\*(C'\fR.
776 This reduces the size of the table.
777 If an unassigned character appears in the string to be collated,
778 the sort key is made from its codepoint
779 as a single-character collation element,
780 as it is greater than any other assigned collation elements
781 (in the codepoint order among the unassigned characters).
782 But, it'd be better to ignore characters
783 unfamiliar to you and maybe never used.
784 .Sp
785 Through \f(CW\*(C`undefChar\*(C'\fR, any character matching \f(CW\*(C`qr/$undefChar/\*(C'\fR
786 will be undefined. Through \f(CW\*(C`undefName\*(C'\fR, any character whose name
787 (given in the \f(CW\*(C`table\*(C'\fR file as a comment) matches \f(CW\*(C`qr/$undefName/\*(C'\fR
788 will be undefined.
789 .Sp
790 ex. Collation weights for beyond-BMP characters are not stored in object:
791 .Sp
792 .Vb 1
793 \& undefChar => qr/[^\e0\-\ex{fffd}]/,
794 .Ve
795 .IP upper_before_lower 4
796 .IX Item "upper_before_lower"
797 \&\-\- see 6.6 Case Comparisons, UTS #10.
798 .Sp
799 By default, lowercase is before uppercase.
800 If the parameter is made true, this is reversed.
801 .Sp
802 \&\fBNOTE\fR: This parameter simplemindedly assumes that any lowercase/uppercase
803 distinctions must occur in level 3, and their weights at level 3 must be
804 same as those mentioned in 7.3.1, UTS #10.
805 If you define your collation elements which differs from this requirement,
806 this parameter doesn't work validly.
807 .IP variable 4
808 .IX Item "variable"
809 \&\-\- see 3.6 Variable Weighting, UTS #10.
810 .Sp
811 This key allows for variable weighting of variable collation elements,
812 which are marked with an ASTERISK in the table
813 (NOTE: Many punctuation marks and symbols are variable in \fIallkeys.txt\fR).
814 .Sp
815 .Vb 1
816 \& variable => \*(Aqblanked\*(Aq, \*(Aqnon\-ignorable\*(Aq, \*(Aqshifted\*(Aq, or \*(Aqshift\-trimmed\*(Aq.
817 .Ve
818 .Sp
819 These names are case-insensitive.
820 By default (if specification is omitted), 'shifted' is adopted.
821 .Sp
822 .Vb 2
823 \& \*(AqBlanked\*(Aq Variable elements are made ignorable at levels 1 through 3;
824 \& considered at the 4th level.
825 \&
826 \& \*(AqNon\-Ignorable\*(Aq Variable elements are not reset to ignorable.
827 \&
828 \& \*(AqShifted\*(Aq Variable elements are made ignorable at levels 1 through 3
829 \& their level 4 weight is replaced by the old level 1 weight.
830 \& Level 4 weight for Non\-Variable elements is 0xFFFF.
831 \&
832 \& \*(AqShift\-Trimmed\*(Aq Same as \*(Aqshifted\*(Aq, but all FFFF\*(Aqs at the 4th level
833 \& are trimmed.
834 .Ve
835 .SS "Methods for Collation"
836 .IX Subsection "Methods for Collation"
837 .ie n .IP """@sorted = $Collator\->sort(@not_sorted)""" 4
838 .el .IP "\f(CW@sorted = $Collator\->sort(@not_sorted)\fR" 4
839 .IX Item "@sorted = $Collator->sort(@not_sorted)"
840 Sorts a list of strings.
841 .ie n .IP """$result = $Collator\->cmp($a, $b)""" 4
842 .el .IP "\f(CW$result = $Collator\->cmp($a, $b)\fR" 4
843 .IX Item "$result = $Collator->cmp($a, $b)"
844 Returns 1 (when \f(CW$a\fR is greater than \f(CW$b\fR)
845 or 0 (when \f(CW$a\fR is equal to \f(CW$b\fR)
846 or \-1 (when \f(CW$a\fR is less than \f(CW$b\fR).
847 .ie n .IP """$result = $Collator\->eq($a, $b)""" 4
848 .el .IP "\f(CW$result = $Collator\->eq($a, $b)\fR" 4
849 .IX Item "$result = $Collator->eq($a, $b)"
850 .PD 0
851 .ie n .IP """$result = $Collator\->ne($a, $b)""" 4
852 .el .IP "\f(CW$result = $Collator\->ne($a, $b)\fR" 4
853 .IX Item "$result = $Collator->ne($a, $b)"
854 .ie n .IP """$result = $Collator\->lt($a, $b)""" 4
855 .el .IP "\f(CW$result = $Collator\->lt($a, $b)\fR" 4
856 .IX Item "$result = $Collator->lt($a, $b)"
857 .ie n .IP """$result = $Collator\->le($a, $b)""" 4
858 .el .IP "\f(CW$result = $Collator\->le($a, $b)\fR" 4
859 .IX Item "$result = $Collator->le($a, $b)"
860 .ie n .IP """$result = $Collator\->gt($a, $b)""" 4
861 .el .IP "\f(CW$result = $Collator\->gt($a, $b)\fR" 4
862 .IX Item "$result = $Collator->gt($a, $b)"
863 .ie n .IP """$result = $Collator\->ge($a, $b)""" 4
864 .el .IP "\f(CW$result = $Collator\->ge($a, $b)\fR" 4
865 .IX Item "$result = $Collator->ge($a, $b)"
866 .PD
867 They works like the same name operators as theirs.
868 .Sp
869 .Vb 6
870 \& eq : whether $a is equal to $b.
871 \& ne : whether $a is not equal to $b.
872 \& lt : whether $a is less than $b.
873 \& le : whether $a is less than $b or equal to $b.
874 \& gt : whether $a is greater than $b.
875 \& ge : whether $a is greater than $b or equal to $b.
876 .Ve
877 .ie n .IP """$sortKey = $Collator\->getSortKey($string)""" 4
878 .el .IP "\f(CW$sortKey = $Collator\->getSortKey($string)\fR" 4
879 .IX Item "$sortKey = $Collator->getSortKey($string)"
880 \&\-\- see 4.3 Form Sort Key, UTS #10.
881 .Sp
882 Returns a sort key.
883 .Sp
884 You compare the sort keys using a binary comparison
885 and get the result of the comparison of the strings using UCA.
886 .Sp
887 .Vb 1
888 \& $Collator\->getSortKey($a) cmp $Collator\->getSortKey($b)
889 \&
890 \& is equivalent to
891 \&
892 \& $Collator\->cmp($a, $b)
893 .Ve
894 .ie n .IP """$sortKeyForm = $Collator\->viewSortKey($string)""" 4
895 .el .IP "\f(CW$sortKeyForm = $Collator\->viewSortKey($string)\fR" 4
896 .IX Item "$sortKeyForm = $Collator->viewSortKey($string)"
897 Converts a sorting key into its representation form.
898 If \f(CW\*(C`UCA_Version\*(C'\fR is 8, the output is slightly different.
899 .Sp
900 .Vb 3
901 \& use Unicode::Collate;
902 \& my $c = Unicode::Collate\->new();
903 \& print $c\->viewSortKey("Perl"),"\en";
904 \&
905 \& # output:
906 \& # [0B67 0A65 0B7F 0B03 | 0020 0020 0020 0020 | 0008 0002 0002 0002 | FFFF FFFF FFFF FFFF]
907 \& # Level 1 Level 2 Level 3 Level 4
908 .Ve
909 .SS "Methods for Searching"
910 .IX Subsection "Methods for Searching"
911 The \f(CW\*(C`match\*(C'\fR, \f(CW\*(C`gmatch\*(C'\fR, \f(CW\*(C`subst\*(C'\fR, \f(CW\*(C`gsubst\*(C'\fR methods work
912 like \f(CW\*(C`m//\*(C'\fR, \f(CW\*(C`m//g\*(C'\fR, \f(CW\*(C`s///\*(C'\fR, \f(CW\*(C`s///g\*(C'\fR, respectively,
913 but they are not aware of any pattern, but only a literal substring.
914 .PP
915 \&\fBDISCLAIMER:\fR If \f(CW\*(C`preprocess\*(C'\fR or \f(CW\*(C`normalization\*(C'\fR parameter is true
916 for \f(CW$Collator\fR, calling these methods (\f(CW\*(C`index\*(C'\fR, \f(CW\*(C`match\*(C'\fR, \f(CW\*(C`gmatch\*(C'\fR,
917 \&\f(CW\*(C`subst\*(C'\fR, \f(CW\*(C`gsubst\*(C'\fR) is croaked, as the position and the length might
918 differ from those on the specified string.
919 .PP
920 \&\f(CW\*(C`rearrange\*(C'\fR and \f(CW\*(C`hangul_terminator\*(C'\fR parameters are neglected.
921 \&\f(CW\*(C`katakana_before_hiragana\*(C'\fR and \f(CW\*(C`upper_before_lower\*(C'\fR don't affect
922 matching and searching, as it doesn't matter whether greater or less.
923 .ie n .IP """$position = $Collator\->index($string, $substring[, $position])""" 4
924 .el .IP "\f(CW$position = $Collator\->index($string, $substring[, $position])\fR" 4
925 .IX Item "$position = $Collator->index($string, $substring[, $position])"
926 .PD 0
927 .ie n .IP """($position, $length) = $Collator\->index($string, $substring[, $position])""" 4
928 .el .IP "\f(CW($position, $length) = $Collator\->index($string, $substring[, $position])\fR" 4
929 .IX Item "($position, $length) = $Collator->index($string, $substring[, $position])"
930 .PD
931 If \f(CW$substring\fR matches a part of \f(CW$string\fR, returns
932 the position of the first occurrence of the matching part in scalar context;
933 in list context, returns a two-element list of
934 the position and the length of the matching part.
935 .Sp
936 If \f(CW$substring\fR does not match any part of \f(CW$string\fR,
937 returns \f(CW\-1\fR in scalar context and
938 an empty list in list context.
939 .Sp
940 e.g. when the content of \f(CW$str\fR is \f(CW\*(C`"Ich mu\*(C'\fRß\f(CW\*(C` studieren Perl."\*(C'\fR,
941 you say the following where \f(CW$sub\fR is \f(CW\*(C`"M\*(C'\fRü\f(CW\*(C`SS"\*(C'\fR,
942 .Sp
943 .Vb 6
944 \& my $Collator = Unicode::Collate\->new( normalization => undef, level => 1 );
945 \& # (normalization => undef) is REQUIRED.
946 \& my $match;
947 \& if (my($pos,$len) = $Collator\->index($str, $sub)) {
948 \& $match = substr($str, $pos, $len);
949 \& }
950 .Ve
951 .Sp
952 and get \f(CW\*(C`"mu\*(C'\fRß\f(CW\*(C`"\*(C'\fR in \f(CW$match\fR, since \f(CW\*(C`"mu\*(C'\fRß\f(CW\*(C`"\*(C'\fR
953 is primary equal to \f(CW\*(C`"M\*(C'\fRü\f(CW\*(C`SS"\*(C'\fR.
954 .ie n .IP """$match_ref = $Collator\->match($string, $substring)""" 4
955 .el .IP "\f(CW$match_ref = $Collator\->match($string, $substring)\fR" 4
956 .IX Item "$match_ref = $Collator->match($string, $substring)"
957 .PD 0
958 .ie n .IP """($match) = $Collator\->match($string, $substring)""" 4
959 .el .IP "\f(CW($match) = $Collator\->match($string, $substring)\fR" 4
960 .IX Item "($match) = $Collator->match($string, $substring)"
961 .PD
962 If \f(CW$substring\fR matches a part of \f(CW$string\fR, in scalar context, returns
963 \&\fBa reference to\fR the first occurrence of the matching part
964 (\f(CW$match_ref\fR is always true if matches,
965 since every reference is \fBtrue\fR);
966 in list context, returns the first occurrence of the matching part.
967 .Sp
968 If \f(CW$substring\fR does not match any part of \f(CW$string\fR,
969 returns \f(CW\*(C`undef\*(C'\fR in scalar context and
970 an empty list in list context.
971 .Sp
972 e.g.
973 .Sp
974 .Vb 5
975 \& if ($match_ref = $Collator\->match($str, $sub)) { # scalar context
976 \& print "matches [$$match_ref].\en";
977 \& } else {
978 \& print "doesn\*(Aqt match.\en";
979 \& }
980 \&
981 \& or
982 \&
983 \& if (($match) = $Collator\->match($str, $sub)) { # list context
984 \& print "matches [$match].\en";
985 \& } else {
986 \& print "doesn\*(Aqt match.\en";
987 \& }
988 .Ve
989 .ie n .IP """@match = $Collator\->gmatch($string, $substring)""" 4
990 .el .IP "\f(CW@match = $Collator\->gmatch($string, $substring)\fR" 4
991 .IX Item "@match = $Collator->gmatch($string, $substring)"
992 If \f(CW$substring\fR matches a part of \f(CW$string\fR, returns
993 all the matching parts (or matching count in scalar context).
994 .Sp
995 If \f(CW$substring\fR does not match any part of \f(CW$string\fR,
996 returns an empty list.
997 .ie n .IP """$count = $Collator\->subst($string, $substring, $replacement)""" 4
998 .el .IP "\f(CW$count = $Collator\->subst($string, $substring, $replacement)\fR" 4
999 .IX Item "$count = $Collator->subst($string, $substring, $replacement)"
1000 If \f(CW$substring\fR matches a part of \f(CW$string\fR,
1001 the first occurrence of the matching part is replaced by \f(CW$replacement\fR
1002 (\f(CW$string\fR is modified) and \f(CW$count\fR (always equals to \f(CW1\fR) is returned.
1003 .Sp
1004 \&\f(CW$replacement\fR can be a \f(CW\*(C`CODEREF\*(C'\fR,
1005 taking the matching part as an argument,
1006 and returning a string to replace the matching part
1007 (a bit similar to \f(CW\*(C`s/(..)/$coderef\->($1)/e\*(C'\fR).
1008 .ie n .IP """$count = $Collator\->gsubst($string, $substring, $replacement)""" 4
1009 .el .IP "\f(CW$count = $Collator\->gsubst($string, $substring, $replacement)\fR" 4
1010 .IX Item "$count = $Collator->gsubst($string, $substring, $replacement)"
1011 If \f(CW$substring\fR matches a part of \f(CW$string\fR,
1012 all the occurrences of the matching part are replaced by \f(CW$replacement\fR
1013 (\f(CW$string\fR is modified) and \f(CW$count\fR is returned.
1014 .Sp
1015 \&\f(CW$replacement\fR can be a \f(CW\*(C`CODEREF\*(C'\fR,
1016 taking the matching part as an argument,
1017 and returning a string to replace the matching part
1018 (a bit similar to \f(CW\*(C`s/(..)/$coderef\->($1)/eg\*(C'\fR).
1019 .Sp
1020 e.g.
1021 .Sp
1022 .Vb 4
1023 \& my $Collator = Unicode::Collate\->new( normalization => undef, level => 1 );
1024 \& # (normalization => undef) is REQUIRED.
1025 \& my $str = "Camel donkey zebra came\ex{301}l CAMEL horse cam\e0e\e0l...";
1026 \& $Collator\->gsubst($str, "camel", sub { "<b>$_[0]</b>" });
1027 \&
1028 \& # now $str is "<b>Camel</b> donkey zebra <b>came\ex{301}l</b> <b>CAMEL</b> horse <b>cam\e0e\e0l</b>...";
1029 \& # i.e., all the camels are made bold\-faced.
1030 \&
1031 \& Examples: levels and ignore_level2 \- what does camel match?
1032 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
1033 \& level ignore_level2 | camel Camel came\ex{301}l c\-a\-m\-e\-l cam\e0e\e0l
1034 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-|\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
1035 \& 1 false | yes yes yes yes yes
1036 \& 2 false | yes yes no yes yes
1037 \& 3 false | yes no no yes yes
1038 \& 4 false | yes no no no yes
1039 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-|\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
1040 \& 1 true | yes yes yes yes yes
1041 \& 2 true | yes yes yes yes yes
1042 \& 3 true | yes no yes yes yes
1043 \& 4 true | yes no yes no yes
1044 \& \-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
1045 \& note: if variable => non\-ignorable, camel doesn\*(Aqt match c\-a\-m\-e\-l
1046 \& at any level.
1047 .Ve
1048 .SS "Other Methods"
1049 .IX Subsection "Other Methods"
1050 .ie n .IP """%old_tailoring = $Collator\->change(%new_tailoring)""" 4
1051 .el .IP "\f(CW%old_tailoring = $Collator\->change(%new_tailoring)\fR" 4
1052 .IX Item "%old_tailoring = $Collator->change(%new_tailoring)"
1053 .PD 0
1054 .ie n .IP """$modified_collator = $Collator\->change(%new_tailoring)""" 4
1055 .el .IP "\f(CW$modified_collator = $Collator\->change(%new_tailoring)\fR" 4
1056 .IX Item "$modified_collator = $Collator->change(%new_tailoring)"
1057 .PD
1058 Changes the value of specified keys and returns the changed part.
1059 .Sp
1060 .Vb 1
1061 \& $Collator = Unicode::Collate\->new(level => 4);
1062 \&
1063 \& $Collator\->eq("perl", "PERL"); # false
1064 \&
1065 \& %old = $Collator\->change(level => 2); # returns (level => 4).
1066 \&
1067 \& $Collator\->eq("perl", "PERL"); # true
1068 \&
1069 \& $Collator\->change(%old); # returns (level => 2).
1070 \&
1071 \& $Collator\->eq("perl", "PERL"); # false
1072 .Ve
1073 .Sp
1074 Not all \f(CW\*(C`(key,value)\*(C'\fRs are allowed to be changed.
1075 See also \f(CW@Unicode::Collate::ChangeOK\fR and \f(CW@Unicode::Collate::ChangeNG\fR.
1076 .Sp
1077 In the scalar context, returns the modified collator
1078 (but it is \fBnot\fR a clone from the original).
1079 .Sp
1080 .Vb 1
1081 \& $Collator\->change(level => 2)\->eq("perl", "PERL"); # true
1082 \&
1083 \& $Collator\->eq("perl", "PERL"); # true; now max level is 2nd.
1084 \&
1085 \& $Collator\->change(level => 4)\->eq("perl", "PERL"); # false
1086 .Ve
1087 .ie n .IP """$version = $Collator\->version()""" 4
1088 .el .IP "\f(CW$version = $Collator\->version()\fR" 4
1089 .IX Item "$version = $Collator->version()"
1090 Returns the version number (a string) of the Unicode Standard
1091 which the \f(CW\*(C`table\*(C'\fR file used by the collator object is based on.
1092 If the table does not include a version line (starting with \f(CW@version\fR),
1093 returns \f(CW"unknown"\fR.
1094 .ie n .IP UCA_Version() 4
1095 .el .IP \f(CWUCA_Version()\fR 4
1096 .IX Item "UCA_Version()"
1097 Returns the revision number of UTS #10 this module consults,
1098 that should correspond with the DUCET incorporated.
1099 .ie n .IP Base_Unicode_Version() 4
1100 .el .IP \f(CWBase_Unicode_Version()\fR 4
1101 .IX Item "Base_Unicode_Version()"
1102 Returns the version number of UTS #10 this module consults,
1103 that should correspond with the DUCET incorporated.
1104 .SH EXPORT
1105 .IX Header "EXPORT"
1106 No method will be exported.
1107 .SH INSTALL
1108 .IX Header "INSTALL"
1109 Though this module can be used without any \f(CW\*(C`table\*(C'\fR file,
1110 to use this module easily, it is recommended to install a table file
1111 in the UCA format, by copying it under the directory
1112 <a place in \f(CW@INC\fR>/Unicode/Collate.
1113 .PP
1114 The most preferable one is "The Default Unicode Collation Element Table"
1115 (aka DUCET), available from the Unicode Consortium's website:
1116 .PP
1117 .Vb 1
1118 \& http://www.unicode.org/Public/UCA/
1119 \&
1120 \& http://www.unicode.org/Public/UCA/latest/allkeys.txt
1121 \& (latest version)
1122 .Ve
1123 .PP
1124 If DUCET is not installed, it is recommended to copy the file
1125 from http://www.unicode.org/Public/UCA/latest/allkeys.txt
1126 to <a place in \f(CW@INC\fR>/Unicode/Collate/allkeys.txt
1127 manually.
1128 .SH CAVEATS
1129 .IX Header "CAVEATS"
1130 .IP Normalization 4
1131 .IX Item "Normalization"
1132 Use of the \f(CW\*(C`normalization\*(C'\fR parameter requires the \fBUnicode::Normalize\fR
1133 module (see Unicode::Normalize).
1134 .Sp
1135 If you need not it (say, in the case when you need not
1136 handle any combining characters),
1137 assign \f(CW\*(C`(normalization => undef)\*(C'\fR explicitly.
1138 .Sp
1139 \&\-\- see 6.5 Avoiding Normalization, UTS #10.
1140 .IP "Conformance Test" 4
1141 .IX Item "Conformance Test"
1142 The Conformance Test for the UCA is available
1143 under <http://www.unicode.org/Public/UCA/>.
1144 .Sp
1145 For \fICollationTest_SHIFTED.txt\fR,
1146 a collator via \f(CW\*(C`Unicode::Collate\->new( )\*(C'\fR should be used;
1147 for \fICollationTest_NON_IGNORABLE.txt\fR, a collator via
1148 \&\f(CW\*(C`Unicode::Collate\->new(variable => "non\-ignorable", level => 3)\*(C'\fR.
1149 .Sp
1150 If \f(CW\*(C`UCA_Version\*(C'\fR is 26 or later, the \f(CW\*(C`identical\*(C'\fR level is preferred;
1151 \&\f(CW\*(C`Unicode::Collate\->new(identical => 1)\*(C'\fR and
1152 \&\f(CW\*(C`Unicode::Collate\->new(identical => 1,\*(C'\fR
1153 \&\f(CW\*(C`variable => "non\-ignorable", level => 3)\*(C'\fR should be used.
1154 .Sp
1155 \&\fBUnicode::Normalize is required to try The Conformance Test.\fR
1156 .Sp
1157 \&\fBEBCDIC-SUPPORT IS EXPERIMENTAL.\fR
1158 .SH "AUTHOR, COPYRIGHT AND LICENSE"
1159 .IX Header "AUTHOR, COPYRIGHT AND LICENSE"
1160 The Unicode::Collate module for perl was written by SADAHIRO Tomoyuki,
1161 <[email protected]>. This module is Copyright(C) 2001\-2021,
1162 SADAHIRO Tomoyuki. Japan. All rights reserved.
1163 .PP
1164 This module is free software; you can redistribute it and/or
1165 modify it under the same terms as Perl itself.
1166 .PP
1167 The file Unicode/Collate/allkeys.txt was copied verbatim
1168 from <http://www.unicode.org/Public/UCA/13.0.0/allkeys.txt>.
1169 For this file, Copyright (c) 2020 Unicode, Inc.; distributed
1170 under the Terms of Use in <http://www.unicode.org/terms_of_use.html>
1171 .SH "SEE ALSO"
1172 .IX Header "SEE ALSO"
1173 .IP "Unicode Collation Algorithm \- UTS #10" 4
1174 .IX Item "Unicode Collation Algorithm - UTS #10"
1175 <http://www.unicode.org/reports/tr10/>
1176 .IP "The Default Unicode Collation Element Table (DUCET)" 4
1177 .IX Item "The Default Unicode Collation Element Table (DUCET)"
1178 <http://www.unicode.org/Public/UCA/latest/allkeys.txt>
1179 .IP "The conformance test for the UCA" 4
1180 .IX Item "The conformance test for the UCA"
1181 <http://www.unicode.org/Public/UCA/latest/CollationTest.html>
1182 .Sp
1183 <http://www.unicode.org/Public/UCA/latest/CollationTest.zip>
1184 .IP "Hangul Syllable Type" 4
1185 .IX Item "Hangul Syllable Type"
1186 <http://www.unicode.org/Public/UNIDATA/HangulSyllableType.txt>
1187 .IP "Unicode Normalization Forms \- UAX #15" 4
1188 .IX Item "Unicode Normalization Forms - UAX #15"
1189 <http://www.unicode.org/reports/tr15/>
1190 .IP "Unicode Locale Data Markup Language (LDML) \- UTS #35" 4
1191 .IX Item "Unicode Locale Data Markup Language (LDML) - UTS #35"
1192 <http://www.unicode.org/reports/tr35/>