Debian Codegraph @

.cargo/vendor/nom/README.md

raw
1 # nom, eating data byte by byte
2
3 [![LICENSE](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
4 [![Join the chat at https://gitter.im/Geal/nom](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/Geal/nom?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
5 [![Build Status](https://github.com/rust-bakery/nom/actions/workflows/ci.yml/badge.svg)](https://github.com/rust-bakery/nom/actions/workflows/ci.yml)
6 [![Coverage Status](https://coveralls.io/repos/github/rust-bakery/nom/badge.svg?branch=main)](https://coveralls.io/github/rust-bakery/nom?branch=main)
7 [![crates.io Version](https://img.shields.io/crates/v/nom.svg)](https://crates.io/crates/nom)
8 [![Minimum rustc version](https://img.shields.io/badge/rustc-1.56.0+-lightgray.svg)](#rust-version-requirements-msrv)
9
10 nom is a parser combinators library written in Rust. Its goal is to provide tools
11 to build safe parsers without compromising the speed or memory consumption. To
12 that end, it uses extensively Rust's *strong typing* and *memory safety* to produce
13 fast and correct parsers, and provides functions, macros and traits to abstract most of the
14 error prone plumbing.
15
16 ![nom logo in CC0 license, by Ange Albertini](https://raw.githubusercontent.com/Geal/nom/main/assets/nom.png)
17
18 *nom will happily take a byte out of your files :)*
19
20 <!-- toc -->
21
22 - [Example](#example)
23 - [Documentation](#documentation)
24 - [Why use nom?](#why-use-nom)
25 - [Binary format parsers](#binary-format-parsers)
26 - [Text format parsers](#text-format-parsers)
27 - [Programming language parsers](#programming-language-parsers)
28 - [Streaming formats](#streaming-formats)
29 - [Parser combinators](#parser-combinators)
30 - [Technical features](#technical-features)
31 - [Rust version requirements](#rust-version-requirements-msrv)
32 - [Installation](#installation)
33 - [Related projects](#related-projects)
34 - [Parsers written with nom](#parsers-written-with-nom)
35 - [Contributors](#contributors)
36
37 <!-- tocstop -->
38
39 ## Example
40
41 [Hexadecimal color](https://developer.mozilla.org/en-US/docs/Web/CSS/color) parser:
42
43 ```rust
44 use nom::{
45 bytes::complete::{tag, take_while_m_n},
46 combinator::map_res,
47 sequence::Tuple,
48 IResult,
49 Parser,
50 };
51
52 #[derive(Debug, PartialEq)]
53 pub struct Color {
54 pub red: u8,
55 pub green: u8,
56 pub blue: u8,
57 }
58
59 fn from_hex(input: &str) -> Result<u8, std::num::ParseIntError> {
60 u8::from_str_radix(input, 16)
61 }
62
63 fn is_hex_digit(c: char) -> bool {
64 c.is_digit(16)
65 }
66
67 fn hex_primary(input: &str) -> IResult<&str, u8> {
68 map_res(
69 take_while_m_n(2, 2, is_hex_digit),
70 from_hex
71 ).parse(input)
72 }
73
74 fn hex_color(input: &str) -> IResult<&str, Color> {
75 let (input, _) = tag("#")(input)?;
76 let (input, (red, green, blue)) = (hex_primary, hex_primary, hex_primary).parse(input)?;
77 Ok((input, Color { red, green, blue }))
78 }
79
80 fn main() {
81 println!("{:?}", hex_color("#2F14DF"))
82 }
83
84 #[test]
85 fn parse_color() {
86 assert_eq!(
87 hex_color("#2F14DF"),
88 Ok((
89 "",
90 Color {
91 red: 47,
92 green: 20,
93 blue: 223,
94 }
95 ))
96 );
97 }
98 ```
99
100 ## Documentation
101
102 - [Reference documentation](https://docs.rs/nom)
103 - [The Nominomicon: A Guide To Using Nom](https://tfpk.github.io/nominomicon/)
104 - [Various design documents and tutorials](https://github.com/rust-bakery/nom/tree/main/doc)
105 - [List of combinators and their behaviour](https://github.com/rust-bakery/nom/blob/main/doc/choosing_a_combinator.md)
106
107 If you need any help developing your parsers, please ping `geal` on IRC (Libera, Geeknode, OFTC), go to `#nom-parsers` on Libera IRC, or on the [Gitter chat room](https://gitter.im/Geal/nom).
108
109 ## Why use nom
110
111 If you want to write:
112
113 ### Binary format parsers
114
115 nom was designed to properly parse binary formats from the beginning. Compared
116 to the usual handwritten C parsers, nom parsers are just as fast, free from
117 buffer overflow vulnerabilities, and handle common patterns for you:
118
119 - [TLV](https://en.wikipedia.org/wiki/Type-length-value)
120 - Bit level parsing
121 - Hexadecimal viewer in the debugging macros for easy data analysis
122 - Streaming parsers for network formats and huge files
123
124 Example projects:
125
126 - [FLV parser](https://github.com/rust-av/flavors)
127 - [Matroska parser](https://github.com/rust-av/matroska)
128 - [tar parser](https://github.com/Keruspe/tar-parser.rs)
129
130 ### Text format parsers
131
132 While nom was made for binary format at first, it soon grew to work just as
133 well with text formats. From line based formats like CSV, to more complex, nested
134 formats such as JSON, nom can manage it, and provides you with useful tools:
135
136 - Fast case insensitive comparison
137 - Recognizers for escaped strings
138 - Regular expressions can be embedded in nom parsers to represent complex character patterns succinctly
139 - Special care has been given to managing non ASCII characters properly
140
141 Example projects:
142
143 - [HTTP proxy](https://github.com/sozu-proxy/sozu/blob/main/lib/src/protocol/h2/parser.rs)
144 - [TOML parser](https://github.com/joelself/tomllib)
145
146 ### Programming language parsers
147
148 While programming language parsers are usually written manually for more
149 flexibility and performance, nom can be (and has been successfully) used
150 as a prototyping parser for a language.
151
152 nom will get you started quickly with powerful custom error types, that you
153 can leverage with [nom_locate](https://github.com/fflorent/nom_locate) to
154 pinpoint the exact line and column of the error. No need for separate
155 tokenizing, lexing and parsing phases: nom can automatically handle whitespace
156 parsing, and construct an AST in place.
157
158 Example projects:
159
160 - [PHP VM](https://github.com/tagua-vm/parser)
161 - [xshade shading language](https://github.com/xshade-lang/xshade)
162
163 ### Streaming formats
164
165 While a lot of formats (and the code handling them) assume that they can fit
166 the complete data in memory, there are formats for which we only get a part
167 of the data at once, like network formats, or huge files.
168 nom has been designed for a correct behaviour with partial data: If there is
169 not enough data to decide, nom will tell you it needs more instead of silently
170 returning a wrong result. Whether your data comes entirely or in chunks, the
171 result should be the same.
172
173 It allows you to build powerful, deterministic state machines for your protocols.
174
175 Example projects:
176
177 - [HTTP proxy](https://github.com/sozu-proxy/sozu/blob/main/lib/src/protocol/h2/parser.rs)
178 - [Using nom with generators](https://github.com/rust-bakery/generator_nom)
179
180 ## Parser combinators
181
182 Parser combinators are an approach to parsers that is very different from
183 software like [lex](https://en.wikipedia.org/wiki/Lex_(software)) and
184 [yacc](https://en.wikipedia.org/wiki/Yacc). Instead of writing the grammar
185 in a separate file and generating the corresponding code, you use very
186 small functions with very specific purpose, like "take 5 bytes", or
187 "recognize the word 'HTTP'", and assemble them in meaningful patterns
188 like "recognize 'HTTP', then a space, then a version".
189 The resulting code is small, and looks like the grammar you would have
190 written with other parser approaches.
191
192 This has a few advantages:
193
194 - The parsers are small and easy to write
195 - The parsers components are easy to reuse (if they're general enough, please add them to nom!)
196 - The parsers components are easy to test separately (unit tests and property-based tests)
197 - The parser combination code looks close to the grammar you would have written
198 - You can build partial parsers, specific to the data you need at the moment, and ignore the rest
199
200 ## Technical features
201
202 nom parsers are for:
203 - [x] **byte-oriented**: The basic type is `&[u8]` and parsers will work as much as possible on byte array slices (but are not limited to them)
204 - [x] **bit-oriented**: nom can address a byte slice as a bit stream
205 - [x] **string-oriented**: The same kind of combinators can apply on UTF-8 strings as well
206 - [x] **zero-copy**: If a parser returns a subset of its input data, it will return a slice of that input, without copying
207 - [x] **streaming**: nom can work on partial data and detect when it needs more data to produce a correct result
208 - [x] **descriptive errors**: The parsers can aggregate a list of error codes with pointers to the incriminated input slice. Those error lists can be pattern matched to provide useful messages.
209 - [x] **custom error types**: You can provide a specific type to improve errors returned by parsers
210 - [x] **safe parsing**: nom leverages Rust's safe memory handling and powerful types, and parsers are routinely fuzzed and tested with real world data. So far, the only flaws found by fuzzing were in code written outside of nom
211 - [x] **speed**: Benchmarks have shown that nom parsers often outperform many parser combinators library like Parsec and attoparsec, some regular expression engines and even handwritten C parsers
212
213 Some benchmarks are available on [GitHub](https://github.com/rust-bakery/parser_benchmarks).
214
215 ## Rust version requirements (MSRV)
216
217 The 7.0 series of nom supports **Rustc version 1.56 or greater**.
218
219 The current policy is that this will only be updated in the next major nom release.
220
221 ## Installation
222
223 nom is available on [crates.io](https://crates.io/crates/nom) and can be included in your Cargo enabled project like this:
224
225 ```toml
226 [dependencies]
227 nom = "7"
228 ```
229
230 There are a few compilation features:
231
232 * `alloc`: (activated by default) if disabled, nom can work in `no_std` builds without memory allocators. If enabled, combinators that allocate (like `many0`) will be available
233 * `std`: (activated by default, activates `alloc` too) if disabled, nom can work in `no_std` builds
234
235 You can configure those features like this:
236
237 ```toml
238 [dependencies.nom]
239 version = "7"
240 default-features = false
241 features = ["alloc"]
242 ```
243
244 # Related projects
245
246 - [Get line and column info in nom's input type](https://github.com/fflorent/nom_locate)
247 - [Using nom as lexer and parser](https://github.com/Rydgel/monkey-rust)
248
249 # Parsers written with nom
250
251 Here is a (non exhaustive) list of known projects using nom:
252
253 - Text file formats: [Ceph Crush](https://github.com/cholcombe973/crushtool),
254 [Cronenberg](https://github.com/ayrat555/cronenberg),
255 [Email](https://github.com/deuxfleurs-org/eml-codec),
256 [XFS Runtime Stats](https://github.com/ChrisMacNaughton/xfs-rs),
257 [CSV](https://github.com/GuillaumeGomez/csv-parser),
258 [FASTA](https://github.com/TianyiShi2001/nom-fasta),
259 [FASTQ](https://github.com/elij/fastq.rs),
260 [INI](https://github.com/rust-bakery/nom/blob/main/tests/ini.rs),
261 [ISO 8601 dates](https://github.com/badboy/iso8601),
262 [libconfig-like configuration file format](https://github.com/filipegoncalves/rust-config),
263 [Web archive](https://github.com/sbeckeriv/warc_nom_parser),
264 [PDB](https://github.com/TianyiShi2001/nom-pdb),
265 [proto files](https://github.com/tafia/protobuf-parser),
266 [Fountain screenplay markup](https://github.com/adamchalmers/fountain-rs),
267 [vimwiki](https://github.com/chipsenkbeil/vimwiki-rs/tree/master/vimwiki) & [vimwiki_macros](https://github.com/chipsenkbeil/vimwiki-rs/tree/master/vimwiki_macros),
268 [Kconfig language](https://github.com/Mcdostone/nom-kconfig), [Askama templates](https://crates.io/crates/askama_parser/)
269 - Programming languages:
270 [PHP](https://github.com/tagua-vm/parser),
271 [Basic Calculator](https://github.com/balajisivaraman/basic_calculator_rs),
272 [GLSL](https://github.com/phaazon/glsl),
273 [Lua](https://github.com/rozbb/nom-lua53),
274 [Python](https://github.com/ProgVal/rust-python-parser),
275 [SQL](https://github.com/ms705/nom-sql),
276 [Elm](https://github.com/cout970/Elm-interpreter),
277 [SystemVerilog](https://github.com/dalance/sv-parser),
278 [Turtle](https://github.com/vandenoever/rome/tree/master/src/io/turtle),
279 [CSML](https://github.com/CSML-by-Clevy/csml-engine/tree/dev/csml_interpreter),
280 [Wasm](https://github.com/fabrizio-m/wasm-nom),
281 [Pseudocode](https://github.com/Gungy2/pseudocod),
282 [Filter for MeiliSearch](https://github.com/meilisearch/meilisearch),
283 [PotterScript](https://github.com/fmiras/potterscript)
284 - Interface definition formats: [Thrift](https://github.com/thehydroimpulse/thrust)
285 - Audio, video and image formats:
286 [GIF](https://github.com/Geal/gif.rs),
287 [MagicaVoxel .vox](https://github.com/dust-engine/dot_vox),
288 [MIDI](https://github.com/derekdreery/nom-midi-rs),
289 [SWF](https://github.com/open-flash/swf-parser),
290 [WAVE](https://github.com/Noise-Labs/wave),
291 [Matroska (MKV)](https://github.com/rust-av/matroska),
292 [Exif/Metadata parser for JPEG/HEIF/HEIC/MOV/MP4](https://github.com/mindeng/nom-exif)
293 - Document formats:
294 [TAR](https://github.com/Keruspe/tar-parser.rs),
295 [GZ](https://github.com/nharward/nom-gzip),
296 [GDSII](https://github.com/erihsu/gds2-io)
297 - Cryptographic formats:
298 [X.509](https://github.com/rusticata/x509-parser)
299 - Network protocol formats:
300 [Bencode](https://github.com/jbaum98/bencode.rs),
301 [D-Bus](https://github.com/toshokan/misato),
302 [DHCP](https://github.com/rusticata/dhcp-parser),
303 [HTTP](https://github.com/sozu-proxy/sozu/tree/main/lib/src/protocol/http),
304 [URI](https://github.com/santifa/rrp/blob/master/src/uri.rs),
305 [IMAP](https://github.com/djc/tokio-imap) ([alt](https://github.com/duesee/imap-codec)),
306 [IRC](https://github.com/Detegr/RBot-parser),
307 [Pcap-NG](https://github.com/richo/pcapng-rs),
308 [Pcap](https://github.com/ithinuel/pcap-rs),
309 [Pcap + PcapNG](https://github.com/rusticata/pcap-parser),
310 [IKEv2](https://github.com/rusticata/ipsec-parser),
311 [NTP](https://github.com/rusticata/ntp-parser),
312 [SNMP](https://github.com/rusticata/snmp-parser),
313 [Kerberos v5](https://github.com/rusticata/kerberos-parser),
314 [DER](https://github.com/rusticata/der-parser),
315 [TLS](https://github.com/rusticata/tls-parser),
316 [V5, V7, V9, IPFIX / Netflow v10](https://github.com/mikemiles-dev/netflow_parser),
317 [GTP](https://github.com/fuerstenau/gorrosion-gtp),
318 [SIP](https://github.com/kurotych/sipcore/tree/master/crates/sipmsg),
319 [SMTP](https://github.com/Ekleog/kannader),
320 [Prometheus](https://github.com/vectordotdev/vector/blob/master/lib/prometheus-parser/src/line.rs),
321 [DNS](https://github.com/adiSuper94/dns-rs)
322 - Language specifications:
323 [BNF](https://github.com/shnewto/bnf)
324 - Misc formats:
325 [Game Boy ROM](https://github.com/MarkMcCaskey/gameboy-rom-parser),
326 [ANT FIT](https://github.com/stadelmanma/fitparse-rs),
327 [Version Numbers](https://github.com/fosskers/rs-versions),
328 [Telcordia/Bellcore SR-4731 SOR OTDR files](https://github.com/JamesHarrison/otdrs),
329 [MySQL binary log](https://github.com/PrivateRookie/boxercrab),
330 [URI](https://github.com/Skasselbard/nom-uri),
331 [Furigana](https://github.com/sachaarbonel/furigana.rs),
332 [Wordle Result](https://github.com/Fyko/wordle-stats/tree/main/parser),
333 [NBT](https://github.com/phoenixr-codes/mcnbt)
334
335 Want to create a new parser using `nom`? A list of not yet implemented formats is available [here](https://github.com/rust-bakery/nom/issues/14).
336
337 Want to add your parser here? Create a pull request for it!
338
339 # Contributors
340
341 nom is the fruit of the work of many contributors over the years, many thanks for your help!
342
343 <a href="https://github.com/rust-bakery/nom/graphs/contributors">
344 <img src="https://contributors-img.web.app/image?repo=rust-bakery/nom" />
345 </a>