| 1 |
# Unicode conformance |
| 2 |
|
| 3 |
This document describes the regex crate's conformance to Unicode's |
| 4 |
[UTS#18](https://unicode.org/reports/tr18/) |
| 5 |
report, which lays out 3 levels of support: Basic, Extended and Tailored. |
| 6 |
|
| 7 |
Full support for Level 1 ("Basic Unicode Support") is provided with two |
| 8 |
exceptions: |
| 9 |
|
| 10 |
1. Line boundaries are not Unicode aware. Namely, only the `\n` |
| 11 |
(`END OF LINE`) character is recognized as a line boundary by default. |
| 12 |
One can opt into `\r\n|\r|\n` being a line boundary via CRLF mode. |
| 13 |
2. The compatibility properties specified by |
| 14 |
[RL1.2a](https://unicode.org/reports/tr18/#RL1.2a) |
| 15 |
are ASCII-only definitions. |
| 16 |
|
| 17 |
Little to no support is provided for either Level 2 or Level 3. For the most |
| 18 |
part, this is because the features are either complex/hard to implement, or at |
| 19 |
the very least, very difficult to implement without sacrificing performance. |
| 20 |
For example, tackling canonical equivalence such that matching worked as one |
| 21 |
would expect regardless of normalization form would be a significant |
| 22 |
undertaking. This is at least partially a result of the fact that this regex |
| 23 |
engine is based on finite automata, which admits less flexibility normally |
| 24 |
associated with backtracking implementations. |
| 25 |
|
| 26 |
|
| 27 |
## RL1.1 Hex Notation |
| 28 |
|
| 29 |
[UTS#18 RL1.1](https://unicode.org/reports/tr18/#Hex_notation) |
| 30 |
|
| 31 |
Hex Notation refers to the ability to specify a Unicode code point in a regular |
| 32 |
expression via its hexadecimal code point representation. This is useful in |
| 33 |
environments that have poor Unicode font rendering or if you need to express a |
| 34 |
code point that is not normally displayable. All forms of hexadecimal notation |
| 35 |
are supported |
| 36 |
|
| 37 |
\x7F hex character code (exactly two digits) |
| 38 |
\x{10FFFF} any hex character code corresponding to a Unicode code point |
| 39 |
\u007F hex character code (exactly four digits) |
| 40 |
\u{7F} any hex character code corresponding to a Unicode code point |
| 41 |
\U0000007F hex character code (exactly eight digits) |
| 42 |
\U{7F} any hex character code corresponding to a Unicode code point |
| 43 |
|
| 44 |
Briefly, the `\x{...}`, `\u{...}` and `\U{...}` are all exactly equivalent ways |
| 45 |
of expressing hexadecimal code points. Any number of digits can be written |
| 46 |
within the brackets. In contrast, `\xNN`, `\uNNNN`, `\UNNNNNNNN` are all |
| 47 |
fixed-width variants of the same idea. |
| 48 |
|
| 49 |
Note that when Unicode mode is disabled, any non-ASCII Unicode codepoint is |
| 50 |
banned. Additionally, the `\xNN` syntax represents arbitrary bytes when Unicode |
| 51 |
mode is disabled. That is, the regex `\xFF` matches the Unicode codepoint |
| 52 |
U+00FF (encoded as `\xC3\xBF` in UTF-8) while the regex `(?-u)\xFF` matches |
| 53 |
the literal byte `\xFF`. |
| 54 |
|
| 55 |
|
| 56 |
## RL1.2 Properties |
| 57 |
|
| 58 |
[UTS#18 RL1.2](https://unicode.org/reports/tr18/#Categories) |
| 59 |
|
| 60 |
Full support for Unicode property syntax is provided. Unicode properties |
| 61 |
provide a convenient way to construct character classes of groups of code |
| 62 |
points specified by Unicode. The regex crate does not provide exhaustive |
| 63 |
support, but covers a useful subset. In particular: |
| 64 |
|
| 65 |
* [General categories](https://unicode.org/reports/tr18/#General_Category_Property) |
| 66 |
* [Scripts and Script Extensions](https://unicode.org/reports/tr18/#Script_Property) |
| 67 |
* [Age](https://unicode.org/reports/tr18/#Age) |
| 68 |
* A smattering of boolean properties, including all of those specified by |
| 69 |
[RL1.2](https://unicode.org/reports/tr18/#RL1.2) explicitly. |
| 70 |
|
| 71 |
In all cases, property name and value abbreviations are supported, and all |
| 72 |
names/values are matched loosely without regard for case, whitespace or |
| 73 |
underscores. Property name aliases can be found in Unicode's |
| 74 |
[`PropertyAliases.txt`](https://www.unicode.org/Public/UCD/latest/ucd/PropertyAliases.txt) |
| 75 |
file, while property value aliases can be found in Unicode's |
| 76 |
[`PropertyValueAliases.txt`](https://www.unicode.org/Public/UCD/latest/ucd/PropertyValueAliases.txt) |
| 77 |
file. |
| 78 |
|
| 79 |
The syntax supported is also consistent with the UTS#18 recommendation: |
| 80 |
|
| 81 |
* `\p{Greek}` selects the `Greek` script. Equivalent expressions follow: |
| 82 |
`\p{sc:Greek}`, `\p{Script:Greek}`, `\p{Sc=Greek}`, `\p{script=Greek}`, |
| 83 |
`\P{sc!=Greek}`. Similarly for `General_Category` (or `gc` for short) and |
| 84 |
`Script_Extensions` (or `scx` for short). |
| 85 |
* `\p{age:3.2}` selects all code points in Unicode 3.2. |
| 86 |
* `\p{Alphabetic}` selects the "alphabetic" property and can be abbreviated |
| 87 |
via `\p{alpha}` (for example). |
| 88 |
* Single letter variants for properties with single letter abbreviations. |
| 89 |
For example, `\p{Letter}` can be equivalently written as `\pL`. |
| 90 |
|
| 91 |
The following is a list of all properties supported by the regex crate (starred |
| 92 |
properties correspond to properties required by RL1.2): |
| 93 |
|
| 94 |
* `General_Category` \* (including `Any`, `ASCII` and `Assigned`) |
| 95 |
* `Script` \* |
| 96 |
* `Script_Extensions` \* |
| 97 |
* `Age` |
| 98 |
* `ASCII_Hex_Digit` |
| 99 |
* `Alphabetic` \* |
| 100 |
* `Bidi_Control` |
| 101 |
* `Case_Ignorable` |
| 102 |
* `Cased` |
| 103 |
* `Changes_When_Casefolded` |
| 104 |
* `Changes_When_Casemapped` |
| 105 |
* `Changes_When_Lowercased` |
| 106 |
* `Changes_When_Titlecased` |
| 107 |
* `Changes_When_Uppercased` |
| 108 |
* `Dash` |
| 109 |
* `Default_Ignorable_Code_Point` \* |
| 110 |
* `Deprecated` |
| 111 |
* `Diacritic` |
| 112 |
* `Emoji` |
| 113 |
* `Emoji_Presentation` |
| 114 |
* `Emoji_Modifier` |
| 115 |
* `Emoji_Modifier_Base` |
| 116 |
* `Emoji_Component` |
| 117 |
* `Extended_Pictographic` |
| 118 |
* `Extender` |
| 119 |
* `Grapheme_Base` |
| 120 |
* `Grapheme_Cluster_Break` |
| 121 |
* `Grapheme_Extend` |
| 122 |
* `Hex_Digit` |
| 123 |
* `IDS_Binary_Operator` |
| 124 |
* `IDS_Trinary_Operator` |
| 125 |
* `ID_Continue` |
| 126 |
* `ID_Start` |
| 127 |
* `Join_Control` |
| 128 |
* `Logical_Order_Exception` |
| 129 |
* `Lowercase` \* |
| 130 |
* `Math` |
| 131 |
* `Noncharacter_Code_Point` \* |
| 132 |
* `Pattern_Syntax` |
| 133 |
* `Pattern_White_Space` |
| 134 |
* `Prepended_Concatenation_Mark` |
| 135 |
* `Quotation_Mark` |
| 136 |
* `Radical` |
| 137 |
* `Regional_Indicator` |
| 138 |
* `Sentence_Break` |
| 139 |
* `Sentence_Terminal` |
| 140 |
* `Soft_Dotted` |
| 141 |
* `Terminal_Punctuation` |
| 142 |
* `Unified_Ideograph` |
| 143 |
* `Uppercase` \* |
| 144 |
* `Variation_Selector` |
| 145 |
* `White_Space` \* |
| 146 |
* `Word_Break` |
| 147 |
* `XID_Continue` |
| 148 |
* `XID_Start` |
| 149 |
|
| 150 |
|
| 151 |
## RL1.2a Compatibility Properties |
| 152 |
|
| 153 |
[UTS#18 RL1.2a](https://unicode.org/reports/tr18/#RL1.2a) |
| 154 |
|
| 155 |
The regex crate only provides ASCII definitions of the |
| 156 |
[compatibility properties documented in UTS#18 Annex C](https://unicode.org/reports/tr18/#Compatibility_Properties) |
| 157 |
(sans the `\X` class, for matching grapheme clusters, which isn't provided |
| 158 |
at all). This is because it seems to be consistent with most other regular |
| 159 |
expression engines, and in particular, because these are often referred to as |
| 160 |
"ASCII" or "POSIX" character classes. |
| 161 |
|
| 162 |
Note that the `\w`, `\s` and `\d` character classes **are** Unicode aware. |
| 163 |
Their traditional ASCII definition can be used by disabling Unicode. That is, |
| 164 |
`[[:word:]]` and `(?-u)\w` are equivalent. |
| 165 |
|
| 166 |
|
| 167 |
## RL1.3 Subtraction and Intersection |
| 168 |
|
| 169 |
[UTS#18 RL1.3](https://unicode.org/reports/tr18/#Subtraction_and_Intersection) |
| 170 |
|
| 171 |
The regex crate provides full support for nested character classes, along with |
| 172 |
union, intersection (`&&`), difference (`--`) and symmetric difference (`~~`) |
| 173 |
operations on arbitrary character classes. |
| 174 |
|
| 175 |
For example, to match all non-ASCII letters, you could use either |
| 176 |
`[\p{Letter}--\p{Ascii}]` (difference) or `[\p{Letter}&&[^\p{Ascii}]]` |
| 177 |
(intersecting the negation). |
| 178 |
|
| 179 |
|
| 180 |
## RL1.4 Simple Word Boundaries |
| 181 |
|
| 182 |
[UTS#18 RL1.4](https://unicode.org/reports/tr18/#Simple_Word_Boundaries) |
| 183 |
|
| 184 |
The regex crate provides basic Unicode aware word boundary assertions. A word |
| 185 |
boundary assertion can be written as `\b`, or `\B` as its negation. A word |
| 186 |
boundary negation corresponds to a zero-width match, where its adjacent |
| 187 |
characters correspond to word and non-word, or non-word and word characters. |
| 188 |
|
| 189 |
Conformance in this case chooses to define word character in the same way that |
| 190 |
the `\w` character class is defined: a code point that is a member of one of |
| 191 |
the following classes: |
| 192 |
|
| 193 |
* `\p{Alphabetic}` |
| 194 |
* `\p{Join_Control}` |
| 195 |
* `\p{gc:Mark}` |
| 196 |
* `\p{gc:Decimal_Number}` |
| 197 |
* `\p{gc:Connector_Punctuation}` |
| 198 |
|
| 199 |
In particular, this differs slightly from the |
| 200 |
[prescription given in RL1.4](https://unicode.org/reports/tr18/#Simple_Word_Boundaries) |
| 201 |
but is permissible according to |
| 202 |
[UTS#18 Annex C](https://unicode.org/reports/tr18/#Compatibility_Properties). |
| 203 |
Namely, it is convenient and simpler to have `\w` and `\b` be in sync with |
| 204 |
one another. |
| 205 |
|
| 206 |
Finally, Unicode word boundaries can be disabled, which will cause ASCII word |
| 207 |
boundaries to be used instead. That is, `\b` is a Unicode word boundary while |
| 208 |
`(?-u)\b` is an ASCII-only word boundary. This can occasionally be beneficial |
| 209 |
if performance is important, since the implementation of Unicode word |
| 210 |
boundaries is currently sub-optimal on non-ASCII text. |
| 211 |
|
| 212 |
|
| 213 |
## RL1.5 Simple Loose Matches |
| 214 |
|
| 215 |
[UTS#18 RL1.5](https://unicode.org/reports/tr18/#Simple_Loose_Matches) |
| 216 |
|
| 217 |
The regex crate provides full support for case insensitive matching in |
| 218 |
accordance with RL1.5. That is, it uses the "simple" case folding mapping. The |
| 219 |
"simple" mapping was chosen because of a key convenient property: every |
| 220 |
"simple" mapping is a mapping from exactly one code point to exactly one other |
| 221 |
code point. This makes case insensitive matching of character classes, for |
| 222 |
example, straight-forward to implement. |
| 223 |
|
| 224 |
When case insensitive mode is enabled (e.g., `(?i)[a]` is equivalent to `a|A`), |
| 225 |
then all characters classes are case folded as well. |
| 226 |
|
| 227 |
|
| 228 |
## RL1.6 Line Boundaries |
| 229 |
|
| 230 |
[UTS#18 RL1.6](https://unicode.org/reports/tr18/#Line_Boundaries) |
| 231 |
|
| 232 |
The regex crate only provides support for recognizing the `\n` (`END OF LINE`) |
| 233 |
character as a line boundary by default. One can also opt into treating |
| 234 |
`\r\n|\r|\n` as a line boundary via CRLF mode. This choice was made mostly for |
| 235 |
implementation convenience, and to avoid performance cliffs that Unicode word |
| 236 |
boundaries are subject to. |
| 237 |
|
| 238 |
|
| 239 |
## RL1.7 Code Points |
| 240 |
|
| 241 |
[UTS#18 RL1.7](https://unicode.org/reports/tr18/#Supplementary_Characters) |
| 242 |
|
| 243 |
The regex crate provides full support for Unicode code point matching. Namely, |
| 244 |
the fundamental atom of any match is always a single code point. |
| 245 |
|
| 246 |
Given Rust's strong ties to UTF-8, the following guarantees are also provided: |
| 247 |
|
| 248 |
* All matches are reported on valid UTF-8 code unit boundaries. That is, any |
| 249 |
match range returned by the public regex API is guaranteed to successfully |
| 250 |
slice the string that was searched. |
| 251 |
* By consequence of the above, it is impossible to match surrogode code points. |
| 252 |
No support for UTF-16 is provided, so this is never necessary. |
| 253 |
|
| 254 |
Note that when Unicode mode is disabled, the fundamental atom of matching is |
| 255 |
no longer a code point but a single byte. When Unicode mode is disabled, many |
| 256 |
Unicode features are disabled as well. For example, `(?-u)\pL` is not a valid |
| 257 |
regex but `\pL(?-u)\xFF` (matches any Unicode `Letter` followed by the literal |
| 258 |
byte `\xFF`) is, for example. |