| 1 |
regex |
| 2 |
===== |
| 3 |
This crate provides routines for searching strings for matches of a [regular |
| 4 |
expression] (aka "regex"). The regex syntax supported by this crate is similar |
| 5 |
to other regex engines, but it lacks several features that are not known how to |
| 6 |
implement efficiently. This includes, but is not limited to, look-around and |
| 7 |
backreferences. In exchange, all regex searches in this crate have worst case |
| 8 |
`O(m * n)` time complexity, where `m` is proportional to the size of the regex |
| 9 |
and `n` is proportional to the size of the string being searched. |
| 10 |
|
| 11 |
[regular expression]: https://en.wikipedia.org/wiki/Regular_expression |
| 12 |
|
| 13 |
[](https://github.com/rust-lang/regex/actions) |
| 14 |
[](https://crates.io/crates/regex) |
| 15 |
|
| 16 |
### Documentation |
| 17 |
|
| 18 |
[Module documentation with examples](https://docs.rs/regex). |
| 19 |
The module documentation also includes a comprehensive description of the |
| 20 |
syntax supported. |
| 21 |
|
| 22 |
Documentation with examples for the various matching functions and iterators |
| 23 |
can be found on the |
| 24 |
[`Regex` type](https://docs.rs/regex/*/regex/struct.Regex.html). |
| 25 |
|
| 26 |
### Usage |
| 27 |
|
| 28 |
To bring this crate into your repository, either add `regex` to your |
| 29 |
`Cargo.toml`, or run `cargo add regex`. |
| 30 |
|
| 31 |
Here's a simple example that matches a date in YYYY-MM-DD format and prints the |
| 32 |
year, month and day: |
| 33 |
|
| 34 |
```rust |
| 35 |
use regex::Regex; |
| 36 |
|
| 37 |
fn main() { |
| 38 |
let re = Regex::new(r"(?x) |
| 39 |
(?P<year>\d{4}) # the year |
| 40 |
- |
| 41 |
(?P<month>\d{2}) # the month |
| 42 |
- |
| 43 |
(?P<day>\d{2}) # the day |
| 44 |
").unwrap(); |
| 45 |
|
| 46 |
let caps = re.captures("2010-03-14").unwrap(); |
| 47 |
assert_eq!("2010", &caps["year"]); |
| 48 |
assert_eq!("03", &caps["month"]); |
| 49 |
assert_eq!("14", &caps["day"]); |
| 50 |
} |
| 51 |
``` |
| 52 |
|
| 53 |
If you have lots of dates in text that you'd like to iterate over, then it's |
| 54 |
easy to adapt the above example with an iterator: |
| 55 |
|
| 56 |
```rust |
| 57 |
use regex::Regex; |
| 58 |
|
| 59 |
fn main() { |
| 60 |
let re = Regex::new(r"(\d{4})-(\d{2})-(\d{2})").unwrap(); |
| 61 |
let hay = "On 2010-03-14, foo happened. On 2014-10-14, bar happened."; |
| 62 |
|
| 63 |
let mut dates = vec![]; |
| 64 |
for (_, [year, month, day]) in re.captures_iter(hay).map(|c| c.extract()) { |
| 65 |
dates.push((year, month, day)); |
| 66 |
} |
| 67 |
assert_eq!(dates, vec![ |
| 68 |
("2010", "03", "14"), |
| 69 |
("2014", "10", "14"), |
| 70 |
]); |
| 71 |
} |
| 72 |
``` |
| 73 |
|
| 74 |
### Usage: Avoid compiling the same regex in a loop |
| 75 |
|
| 76 |
It is an anti-pattern to compile the same regular expression in a loop since |
| 77 |
compilation is typically expensive. (It takes anywhere from a few microseconds |
| 78 |
to a few **milliseconds** depending on the size of the regex.) Not only is |
| 79 |
compilation itself expensive, but this also prevents optimizations that reuse |
| 80 |
allocations internally to the matching engines. |
| 81 |
|
| 82 |
In Rust, it can sometimes be a pain to pass regular expressions around if |
| 83 |
they're used from inside a helper function. Instead, we recommend using the |
| 84 |
[`once_cell`](https://crates.io/crates/once_cell) crate to ensure that |
| 85 |
regular expressions are compiled exactly once. For example: |
| 86 |
|
| 87 |
```rust |
| 88 |
use { |
| 89 |
once_cell::sync::Lazy, |
| 90 |
regex::Regex, |
| 91 |
}; |
| 92 |
|
| 93 |
fn some_helper_function(haystack: &str) -> bool { |
| 94 |
static RE: Lazy<Regex> = Lazy::new(|| Regex::new(r"...").unwrap()); |
| 95 |
RE.is_match(haystack) |
| 96 |
} |
| 97 |
|
| 98 |
fn main() { |
| 99 |
assert!(some_helper_function("abc")); |
| 100 |
assert!(!some_helper_function("ac")); |
| 101 |
} |
| 102 |
``` |
| 103 |
|
| 104 |
Specifically, in this example, the regex will be compiled when it is used for |
| 105 |
the first time. On subsequent uses, it will reuse the previous compilation. |
| 106 |
|
| 107 |
### Usage: match regular expressions on `&[u8]` |
| 108 |
|
| 109 |
The main API of this crate (`regex::Regex`) requires the caller to pass a |
| 110 |
`&str` for searching. In Rust, an `&str` is required to be valid UTF-8, which |
| 111 |
means the main API can't be used for searching arbitrary bytes. |
| 112 |
|
| 113 |
To match on arbitrary bytes, use the `regex::bytes::Regex` API. The API is |
| 114 |
identical to the main API, except that it takes an `&[u8]` to search on instead |
| 115 |
of an `&str`. The `&[u8]` APIs also permit disabling Unicode mode in the regex |
| 116 |
even when the pattern would match invalid UTF-8. For example, `(?-u:.)` is |
| 117 |
not allowed in `regex::Regex` but is allowed in `regex::bytes::Regex` since |
| 118 |
`(?-u:.)` matches any byte except for `\n`. Conversely, `.` will match the |
| 119 |
UTF-8 encoding of any Unicode scalar value except for `\n`. |
| 120 |
|
| 121 |
This example shows how to find all null-terminated strings in a slice of bytes: |
| 122 |
|
| 123 |
```rust |
| 124 |
use regex::bytes::Regex; |
| 125 |
|
| 126 |
let re = Regex::new(r"(?-u)(?<cstr>[^\x00]+)\x00").unwrap(); |
| 127 |
let text = b"foo\xFFbar\x00baz\x00"; |
| 128 |
|
| 129 |
// Extract all of the strings without the null terminator from each match. |
| 130 |
// The unwrap is OK here since a match requires the `cstr` capture to match. |
| 131 |
let cstrs: Vec<&[u8]> = |
| 132 |
re.captures_iter(text) |
| 133 |
.map(|c| c.name("cstr").unwrap().as_bytes()) |
| 134 |
.collect(); |
| 135 |
assert_eq!(vec![&b"foo\xFFbar"[..], &b"baz"[..]], cstrs); |
| 136 |
``` |
| 137 |
|
| 138 |
Notice here that the `[^\x00]+` will match any *byte* except for `NUL`, |
| 139 |
including bytes like `\xFF` which are not valid UTF-8. When using the main API, |
| 140 |
`[^\x00]+` would instead match any valid UTF-8 sequence except for `NUL`. |
| 141 |
|
| 142 |
### Usage: match multiple regular expressions simultaneously |
| 143 |
|
| 144 |
This demonstrates how to use a `RegexSet` to match multiple (possibly |
| 145 |
overlapping) regular expressions in a single scan of the search text: |
| 146 |
|
| 147 |
```rust |
| 148 |
use regex::RegexSet; |
| 149 |
|
| 150 |
let set = RegexSet::new(&[ |
| 151 |
r"\w+", |
| 152 |
r"\d+", |
| 153 |
r"\pL+", |
| 154 |
r"foo", |
| 155 |
r"bar", |
| 156 |
r"barfoo", |
| 157 |
r"foobar", |
| 158 |
]).unwrap(); |
| 159 |
|
| 160 |
// Iterate over and collect all of the matches. |
| 161 |
let matches: Vec<_> = set.matches("foobar").into_iter().collect(); |
| 162 |
assert_eq!(matches, vec![0, 2, 3, 4, 6]); |
| 163 |
|
| 164 |
// You can also test whether a particular regex matched: |
| 165 |
let matches = set.matches("foobar"); |
| 166 |
assert!(!matches.matched(5)); |
| 167 |
assert!(matches.matched(6)); |
| 168 |
``` |
| 169 |
|
| 170 |
|
| 171 |
### Usage: regex internals as a library |
| 172 |
|
| 173 |
The [`regex-automata` directory](./regex-automata/) contains a crate that |
| 174 |
exposes all of the internal matching engines used by the `regex` crate. The |
| 175 |
idea is that the `regex` crate exposes a simple API for 99% of use cases, but |
| 176 |
`regex-automata` exposes oodles of customizable behaviors. |
| 177 |
|
| 178 |
[Documentation for `regex-automata`.](https://docs.rs/regex-automata) |
| 179 |
|
| 180 |
|
| 181 |
### Usage: a regular expression parser |
| 182 |
|
| 183 |
This repository contains a crate that provides a well tested regular expression |
| 184 |
parser, abstract syntax and a high-level intermediate representation for |
| 185 |
convenient analysis. It provides no facilities for compilation or execution. |
| 186 |
This may be useful if you're implementing your own regex engine or otherwise |
| 187 |
need to do analysis on the syntax of a regular expression. It is otherwise not |
| 188 |
recommended for general use. |
| 189 |
|
| 190 |
[Documentation for `regex-syntax`.](https://docs.rs/regex-syntax) |
| 191 |
|
| 192 |
|
| 193 |
### Crate features |
| 194 |
|
| 195 |
This crate comes with several features that permit tweaking the trade off |
| 196 |
between binary size, compilation time and runtime performance. Users of this |
| 197 |
crate can selectively disable Unicode tables, or choose from a variety of |
| 198 |
optimizations performed by this crate to disable. |
| 199 |
|
| 200 |
When all of these features are disabled, runtime match performance may be much |
| 201 |
worse, but if you're matching on short strings, or if high performance isn't |
| 202 |
necessary, then such a configuration is perfectly serviceable. To disable |
| 203 |
all such features, use the following `Cargo.toml` dependency configuration: |
| 204 |
|
| 205 |
```toml |
| 206 |
[dependencies.regex] |
| 207 |
version = "1.3" |
| 208 |
default-features = false |
| 209 |
# Unless you have a specific reason not to, it's good sense to enable standard |
| 210 |
# library support. It enables several optimizations and avoids spin locks. It |
| 211 |
# also shouldn't meaningfully impact compile times or binary size. |
| 212 |
features = ["std"] |
| 213 |
``` |
| 214 |
|
| 215 |
This will reduce the dependency tree of `regex` down to two crates: |
| 216 |
`regex-syntax` and `regex-automata`. |
| 217 |
|
| 218 |
The full set of features one can disable are |
| 219 |
[in the "Crate features" section of the documentation](https://docs.rs/regex/1.*/#crate-features). |
| 220 |
|
| 221 |
|
| 222 |
### Performance |
| 223 |
|
| 224 |
One of the goals of this crate is for the regex engine to be "fast." What that |
| 225 |
is a somewhat nebulous goal, it is usually interpreted in one of two ways. |
| 226 |
First, it means that all searches take worst case `O(m * n)` time, where |
| 227 |
`m` is proportional to `len(regex)` and `n` is proportional to `len(haystack)`. |
| 228 |
Second, it means that even aside from the time complexity constraint, regex |
| 229 |
searches are "fast" in practice. |
| 230 |
|
| 231 |
While the first interpretation is pretty unambiguous, the second one remains |
| 232 |
nebulous. While nebulous, it guides this crate's architecture and the sorts of |
| 233 |
the trade offs it makes. For example, here are some general architectural |
| 234 |
statements that follow as a result of the goal to be "fast": |
| 235 |
|
| 236 |
* When given the choice between faster regex searches and faster _Rust compile |
| 237 |
times_, this crate will generally choose faster regex searches. |
| 238 |
* When given the choice between faster regex searches and faster _regex compile |
| 239 |
times_, this crate will generally choose faster regex searches. That is, it is |
| 240 |
generally acceptable for `Regex::new` to get a little slower if it means that |
| 241 |
searches get faster. (This is a somewhat delicate balance to strike, because |
| 242 |
the speed of `Regex::new` needs to remain somewhat reasonable. But this is why |
| 243 |
one should avoid re-compiling the same regex over and over again.) |
| 244 |
* When given the choice between faster regex searches and simpler API |
| 245 |
design, this crate will generally choose faster regex searches. For example, |
| 246 |
if one didn't care about performance, we could like get rid of both of |
| 247 |
the `Regex::is_match` and `Regex::find` APIs and instead just rely on |
| 248 |
`Regex::captures`. |
| 249 |
|
| 250 |
There are perhaps more ways that being "fast" influences things. |
| 251 |
|
| 252 |
While this repository used to provide its own benchmark suite, it has since |
| 253 |
been moved to [rebar](https://github.com/BurntSushi/rebar). The benchmarks are |
| 254 |
quite extensive, and there are many more than what is shown in rebar's README |
| 255 |
(which is just limited to a "curated" set meant to compare performance between |
| 256 |
regex engines). To run all of this crate's benchmarks, first start by cloning |
| 257 |
and installing `rebar`: |
| 258 |
|
| 259 |
```text |
| 260 |
$ git clone https://github.com/BurntSushi/rebar |
| 261 |
$ cd rebar |
| 262 |
$ cargo install --path ./ |
| 263 |
``` |
| 264 |
|
| 265 |
Then build the benchmark harness for just this crate: |
| 266 |
|
| 267 |
```text |
| 268 |
$ rebar build -e '^rust/regex$' |
| 269 |
``` |
| 270 |
|
| 271 |
Run all benchmarks for this crate as tests (each benchmark is executed once to |
| 272 |
ensure it works): |
| 273 |
|
| 274 |
```text |
| 275 |
$ rebar measure -e '^rust/regex$' -t |
| 276 |
``` |
| 277 |
|
| 278 |
Record measurements for all benchmarks and save them to a CSV file: |
| 279 |
|
| 280 |
```text |
| 281 |
$ rebar measure -e '^rust/regex$' | tee results.csv |
| 282 |
``` |
| 283 |
|
| 284 |
Explore benchmark timings: |
| 285 |
|
| 286 |
```text |
| 287 |
$ rebar cmp results.csv |
| 288 |
``` |
| 289 |
|
| 290 |
See the `rebar` documentation for more details on how it works and how to |
| 291 |
compare results with other regex engines. |
| 292 |
|
| 293 |
|
| 294 |
### Hacking |
| 295 |
|
| 296 |
The `regex` crate is, for the most part, a pretty thin wrapper around the |
| 297 |
[`meta::Regex`](https://docs.rs/regex-automata/latest/regex_automata/meta/struct.Regex.html) |
| 298 |
from the |
| 299 |
[`regex-automata` crate](https://docs.rs/regex-automata/latest/regex_automata/). |
| 300 |
Therefore, if you're looking to work on the internals of this crate, you'll |
| 301 |
likely either want to look in `regex-syntax` (for parsing) or `regex-automata` |
| 302 |
(for construction of finite automata and the search routines). |
| 303 |
|
| 304 |
My [blog on regex internals](https://blog.burntsushi.net/regex-internals/) |
| 305 |
goes into more depth. |
| 306 |
|
| 307 |
|
| 308 |
### Minimum Rust version policy |
| 309 |
|
| 310 |
This crate's minimum supported `rustc` version is `1.65.0`. |
| 311 |
|
| 312 |
The policy is that the minimum Rust version required to use this crate can be |
| 313 |
increased in minor version updates. For example, if regex 1.0 requires Rust |
| 314 |
1.20.0, then regex 1.0.z for all values of `z` will also require Rust 1.20.0 or |
| 315 |
newer. However, regex 1.y for `y > 0` may require a newer minimum version of |
| 316 |
Rust. |
| 317 |
|
| 318 |
|
| 319 |
### License |
| 320 |
|
| 321 |
This project is licensed under either of |
| 322 |
|
| 323 |
* Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or |
| 324 |
https://www.apache.org/licenses/LICENSE-2.0) |
| 325 |
* MIT license ([LICENSE-MIT](LICENSE-MIT) or |
| 326 |
https://opensource.org/licenses/MIT) |
| 327 |
|
| 328 |
at your option. |
| 329 |
|
| 330 |
The data in `regex-syntax/src/unicode_tables/` is licensed under the Unicode |
| 331 |
License Agreement |
| 332 |
([LICENSE-UNICODE](https://www.unicode.org/copyright.html#License)). |