Debian Codegraph @

.cargo/vendor/regex/UNICODE.md

raw
1 # Unicode conformance
2
3 This document describes the regex crate's conformance to Unicode's
4 [UTS#18](https://unicode.org/reports/tr18/)
5 report, which lays out 3 levels of support: Basic, Extended and Tailored.
6
7 Full support for Level 1 ("Basic Unicode Support") is provided with two
8 exceptions:
9
10 1. Line boundaries are not Unicode aware. Namely, only the `\n`
11 (`END OF LINE`) character is recognized as a line boundary by default.
12 One can opt into `\r\n|\r|\n` being a line boundary via CRLF mode.
13 2. The compatibility properties specified by
14 [RL1.2a](https://unicode.org/reports/tr18/#RL1.2a)
15 are ASCII-only definitions.
16
17 Little to no support is provided for either Level 2 or Level 3. For the most
18 part, this is because the features are either complex/hard to implement, or at
19 the very least, very difficult to implement without sacrificing performance.
20 For example, tackling canonical equivalence such that matching worked as one
21 would expect regardless of normalization form would be a significant
22 undertaking. This is at least partially a result of the fact that this regex
23 engine is based on finite automata, which admits less flexibility normally
24 associated with backtracking implementations.
25
26
27 ## RL1.1 Hex Notation
28
29 [UTS#18 RL1.1](https://unicode.org/reports/tr18/#Hex_notation)
30
31 Hex Notation refers to the ability to specify a Unicode code point in a regular
32 expression via its hexadecimal code point representation. This is useful in
33 environments that have poor Unicode font rendering or if you need to express a
34 code point that is not normally displayable. All forms of hexadecimal notation
35 are supported
36
37 \x7F hex character code (exactly two digits)
38 \x{10FFFF} any hex character code corresponding to a Unicode code point
39 \u007F hex character code (exactly four digits)
40 \u{7F} any hex character code corresponding to a Unicode code point
41 \U0000007F hex character code (exactly eight digits)
42 \U{7F} any hex character code corresponding to a Unicode code point
43
44 Briefly, the `\x{...}`, `\u{...}` and `\U{...}` are all exactly equivalent ways
45 of expressing hexadecimal code points. Any number of digits can be written
46 within the brackets. In contrast, `\xNN`, `\uNNNN`, `\UNNNNNNNN` are all
47 fixed-width variants of the same idea.
48
49 Note that when Unicode mode is disabled, any non-ASCII Unicode codepoint is
50 banned. Additionally, the `\xNN` syntax represents arbitrary bytes when Unicode
51 mode is disabled. That is, the regex `\xFF` matches the Unicode codepoint
52 U+00FF (encoded as `\xC3\xBF` in UTF-8) while the regex `(?-u)\xFF` matches
53 the literal byte `\xFF`.
54
55
56 ## RL1.2 Properties
57
58 [UTS#18 RL1.2](https://unicode.org/reports/tr18/#Categories)
59
60 Full support for Unicode property syntax is provided. Unicode properties
61 provide a convenient way to construct character classes of groups of code
62 points specified by Unicode. The regex crate does not provide exhaustive
63 support, but covers a useful subset. In particular:
64
65 * [General categories](https://unicode.org/reports/tr18/#General_Category_Property)
66 * [Scripts and Script Extensions](https://unicode.org/reports/tr18/#Script_Property)
67 * [Age](https://unicode.org/reports/tr18/#Age)
68 * A smattering of boolean properties, including all of those specified by
69 [RL1.2](https://unicode.org/reports/tr18/#RL1.2) explicitly.
70
71 In all cases, property name and value abbreviations are supported, and all
72 names/values are matched loosely without regard for case, whitespace or
73 underscores. Property name aliases can be found in Unicode's
74 [`PropertyAliases.txt`](https://www.unicode.org/Public/UCD/latest/ucd/PropertyAliases.txt)
75 file, while property value aliases can be found in Unicode's
76 [`PropertyValueAliases.txt`](https://www.unicode.org/Public/UCD/latest/ucd/PropertyValueAliases.txt)
77 file.
78
79 The syntax supported is also consistent with the UTS#18 recommendation:
80
81 * `\p{Greek}` selects the `Greek` script. Equivalent expressions follow:
82 `\p{sc:Greek}`, `\p{Script:Greek}`, `\p{Sc=Greek}`, `\p{script=Greek}`,
83 `\P{sc!=Greek}`. Similarly for `General_Category` (or `gc` for short) and
84 `Script_Extensions` (or `scx` for short).
85 * `\p{age:3.2}` selects all code points in Unicode 3.2.
86 * `\p{Alphabetic}` selects the "alphabetic" property and can be abbreviated
87 via `\p{alpha}` (for example).
88 * Single letter variants for properties with single letter abbreviations.
89 For example, `\p{Letter}` can be equivalently written as `\pL`.
90
91 The following is a list of all properties supported by the regex crate (starred
92 properties correspond to properties required by RL1.2):
93
94 * `General_Category` \* (including `Any`, `ASCII` and `Assigned`)
95 * `Script` \*
96 * `Script_Extensions` \*
97 * `Age`
98 * `ASCII_Hex_Digit`
99 * `Alphabetic` \*
100 * `Bidi_Control`
101 * `Case_Ignorable`
102 * `Cased`
103 * `Changes_When_Casefolded`
104 * `Changes_When_Casemapped`
105 * `Changes_When_Lowercased`
106 * `Changes_When_Titlecased`
107 * `Changes_When_Uppercased`
108 * `Dash`
109 * `Default_Ignorable_Code_Point` \*
110 * `Deprecated`
111 * `Diacritic`
112 * `Emoji`
113 * `Emoji_Presentation`
114 * `Emoji_Modifier`
115 * `Emoji_Modifier_Base`
116 * `Emoji_Component`
117 * `Extended_Pictographic`
118 * `Extender`
119 * `Grapheme_Base`
120 * `Grapheme_Cluster_Break`
121 * `Grapheme_Extend`
122 * `Hex_Digit`
123 * `IDS_Binary_Operator`
124 * `IDS_Trinary_Operator`
125 * `ID_Continue`
126 * `ID_Start`
127 * `Join_Control`
128 * `Logical_Order_Exception`
129 * `Lowercase` \*
130 * `Math`
131 * `Noncharacter_Code_Point` \*
132 * `Pattern_Syntax`
133 * `Pattern_White_Space`
134 * `Prepended_Concatenation_Mark`
135 * `Quotation_Mark`
136 * `Radical`
137 * `Regional_Indicator`
138 * `Sentence_Break`
139 * `Sentence_Terminal`
140 * `Soft_Dotted`
141 * `Terminal_Punctuation`
142 * `Unified_Ideograph`
143 * `Uppercase` \*
144 * `Variation_Selector`
145 * `White_Space` \*
146 * `Word_Break`
147 * `XID_Continue`
148 * `XID_Start`
149
150
151 ## RL1.2a Compatibility Properties
152
153 [UTS#18 RL1.2a](https://unicode.org/reports/tr18/#RL1.2a)
154
155 The regex crate only provides ASCII definitions of the
156 [compatibility properties documented in UTS#18 Annex C](https://unicode.org/reports/tr18/#Compatibility_Properties)
157 (sans the `\X` class, for matching grapheme clusters, which isn't provided
158 at all). This is because it seems to be consistent with most other regular
159 expression engines, and in particular, because these are often referred to as
160 "ASCII" or "POSIX" character classes.
161
162 Note that the `\w`, `\s` and `\d` character classes **are** Unicode aware.
163 Their traditional ASCII definition can be used by disabling Unicode. That is,
164 `[[:word:]]` and `(?-u)\w` are equivalent.
165
166
167 ## RL1.3 Subtraction and Intersection
168
169 [UTS#18 RL1.3](https://unicode.org/reports/tr18/#Subtraction_and_Intersection)
170
171 The regex crate provides full support for nested character classes, along with
172 union, intersection (`&&`), difference (`--`) and symmetric difference (`~~`)
173 operations on arbitrary character classes.
174
175 For example, to match all non-ASCII letters, you could use either
176 `[\p{Letter}--\p{Ascii}]` (difference) or `[\p{Letter}&&[^\p{Ascii}]]`
177 (intersecting the negation).
178
179
180 ## RL1.4 Simple Word Boundaries
181
182 [UTS#18 RL1.4](https://unicode.org/reports/tr18/#Simple_Word_Boundaries)
183
184 The regex crate provides basic Unicode aware word boundary assertions. A word
185 boundary assertion can be written as `\b`, or `\B` as its negation. A word
186 boundary negation corresponds to a zero-width match, where its adjacent
187 characters correspond to word and non-word, or non-word and word characters.
188
189 Conformance in this case chooses to define word character in the same way that
190 the `\w` character class is defined: a code point that is a member of one of
191 the following classes:
192
193 * `\p{Alphabetic}`
194 * `\p{Join_Control}`
195 * `\p{gc:Mark}`
196 * `\p{gc:Decimal_Number}`
197 * `\p{gc:Connector_Punctuation}`
198
199 In particular, this differs slightly from the
200 [prescription given in RL1.4](https://unicode.org/reports/tr18/#Simple_Word_Boundaries)
201 but is permissible according to
202 [UTS#18 Annex C](https://unicode.org/reports/tr18/#Compatibility_Properties).
203 Namely, it is convenient and simpler to have `\w` and `\b` be in sync with
204 one another.
205
206 Finally, Unicode word boundaries can be disabled, which will cause ASCII word
207 boundaries to be used instead. That is, `\b` is a Unicode word boundary while
208 `(?-u)\b` is an ASCII-only word boundary. This can occasionally be beneficial
209 if performance is important, since the implementation of Unicode word
210 boundaries is currently sub-optimal on non-ASCII text.
211
212
213 ## RL1.5 Simple Loose Matches
214
215 [UTS#18 RL1.5](https://unicode.org/reports/tr18/#Simple_Loose_Matches)
216
217 The regex crate provides full support for case insensitive matching in
218 accordance with RL1.5. That is, it uses the "simple" case folding mapping. The
219 "simple" mapping was chosen because of a key convenient property: every
220 "simple" mapping is a mapping from exactly one code point to exactly one other
221 code point. This makes case insensitive matching of character classes, for
222 example, straight-forward to implement.
223
224 When case insensitive mode is enabled (e.g., `(?i)[a]` is equivalent to `a|A`),
225 then all characters classes are case folded as well.
226
227
228 ## RL1.6 Line Boundaries
229
230 [UTS#18 RL1.6](https://unicode.org/reports/tr18/#Line_Boundaries)
231
232 The regex crate only provides support for recognizing the `\n` (`END OF LINE`)
233 character as a line boundary by default. One can also opt into treating
234 `\r\n|\r|\n` as a line boundary via CRLF mode. This choice was made mostly for
235 implementation convenience, and to avoid performance cliffs that Unicode word
236 boundaries are subject to.
237
238
239 ## RL1.7 Code Points
240
241 [UTS#18 RL1.7](https://unicode.org/reports/tr18/#Supplementary_Characters)
242
243 The regex crate provides full support for Unicode code point matching. Namely,
244 the fundamental atom of any match is always a single code point.
245
246 Given Rust's strong ties to UTF-8, the following guarantees are also provided:
247
248 * All matches are reported on valid UTF-8 code unit boundaries. That is, any
249 match range returned by the public regex API is guaranteed to successfully
250 slice the string that was searched.
251 * By consequence of the above, it is impossible to match surrogode code points.
252 No support for UTF-16 is provided, so this is never necessary.
253
254 Note that when Unicode mode is disabled, the fundamental atom of matching is
255 no longer a code point but a single byte. When Unicode mode is disabled, many
256 Unicode features are disabled as well. For example, `(?-u)\pL` is not a valid
257 regex but `\pL(?-u)\xFF` (matches any Unicode `Letter` followed by the literal
258 byte `\xFF`) is, for example.