| 1 |
Parser for Rust source code |
| 2 |
=========================== |
| 3 |
|
| 4 |
[<img alt="github" src="https://img.shields.io/badge/github-dtolnay/syn-8da0cb?style=for-the-badge&labelColor=555555&logo=github" height="20">](https://github.com/dtolnay/syn) |
| 5 |
[<img alt="crates.io" src="https://img.shields.io/crates/v/syn.svg?style=for-the-badge&color=fc8d62&logo=rust" height="20">](https://crates.io/crates/syn) |
| 6 |
[<img alt="docs.rs" src="https://img.shields.io/badge/docs.rs-syn-66c2a5?style=for-the-badge&labelColor=555555&logo=docs.rs" height="20">](https://docs.rs/syn) |
| 7 |
[<img alt="build status" src="https://img.shields.io/github/actions/workflow/status/dtolnay/syn/ci.yml?branch=master&style=for-the-badge" height="20">](https://github.com/dtolnay/syn/actions?query=branch%3Amaster) |
| 8 |
|
| 9 |
Syn is a parsing library for parsing a stream of Rust tokens into a syntax tree |
| 10 |
of Rust source code. |
| 11 |
|
| 12 |
Currently this library is geared toward use in Rust procedural macros, but |
| 13 |
contains some APIs that may be useful more generally. |
| 14 |
|
| 15 |
- **Data structures** — Syn provides a complete syntax tree that can represent |
| 16 |
any valid Rust source code. The syntax tree is rooted at [`syn::File`] which |
| 17 |
represents a full source file, but there are other entry points that may be |
| 18 |
useful to procedural macros including [`syn::Item`], [`syn::Expr`] and |
| 19 |
[`syn::Type`]. |
| 20 |
|
| 21 |
- **Derives** — Of particular interest to derive macros is [`syn::DeriveInput`] |
| 22 |
which is any of the three legal input items to a derive macro. An example |
| 23 |
below shows using this type in a library that can derive implementations of a |
| 24 |
user-defined trait. |
| 25 |
|
| 26 |
- **Parsing** — Parsing in Syn is built around [parser functions] with the |
| 27 |
signature `fn(ParseStream) -> Result<T>`. Every syntax tree node defined by |
| 28 |
Syn is individually parsable and may be used as a building block for custom |
| 29 |
syntaxes, or you may dream up your own brand new syntax without involving any |
| 30 |
of our syntax tree types. |
| 31 |
|
| 32 |
- **Location information** — Every token parsed by Syn is associated with a |
| 33 |
`Span` that tracks line and column information back to the source of that |
| 34 |
token. These spans allow a procedural macro to display detailed error messages |
| 35 |
pointing to all the right places in the user's code. There is an example of |
| 36 |
this below. |
| 37 |
|
| 38 |
- **Feature flags** — Functionality is aggressively feature gated so your |
| 39 |
procedural macros enable only what they need, and do not pay in compile time |
| 40 |
for all the rest. |
| 41 |
|
| 42 |
[`syn::File`]: https://docs.rs/syn/2.0/syn/struct.File.html |
| 43 |
[`syn::Item`]: https://docs.rs/syn/2.0/syn/enum.Item.html |
| 44 |
[`syn::Expr`]: https://docs.rs/syn/2.0/syn/enum.Expr.html |
| 45 |
[`syn::Type`]: https://docs.rs/syn/2.0/syn/enum.Type.html |
| 46 |
[`syn::DeriveInput`]: https://docs.rs/syn/2.0/syn/struct.DeriveInput.html |
| 47 |
[parser functions]: https://docs.rs/syn/2.0/syn/parse/index.html |
| 48 |
|
| 49 |
[*Release notes*](https://github.com/dtolnay/syn/releases) |
| 50 |
|
| 51 |
<br> |
| 52 |
|
| 53 |
## Resources |
| 54 |
|
| 55 |
The best way to learn about procedural macros is by writing some. Consider |
| 56 |
working through [this procedural macro workshop][workshop] to get familiar with |
| 57 |
the different types of procedural macros. The workshop contains relevant links |
| 58 |
into the Syn documentation as you work through each project. |
| 59 |
|
| 60 |
[workshop]: https://github.com/dtolnay/proc-macro-workshop |
| 61 |
|
| 62 |
<br> |
| 63 |
|
| 64 |
## Example of a derive macro |
| 65 |
|
| 66 |
The canonical derive macro using Syn looks like this. We write an ordinary Rust |
| 67 |
function tagged with a `proc_macro_derive` attribute and the name of the trait |
| 68 |
we are deriving. Any time that derive appears in the user's code, the Rust |
| 69 |
compiler passes their data structure as tokens into our macro. We get to execute |
| 70 |
arbitrary Rust code to figure out what to do with those tokens, then hand some |
| 71 |
tokens back to the compiler to compile into the user's crate. |
| 72 |
|
| 73 |
[`TokenStream`]: https://doc.rust-lang.org/proc_macro/struct.TokenStream.html |
| 74 |
|
| 75 |
```toml |
| 76 |
[dependencies] |
| 77 |
syn = "2.0" |
| 78 |
quote = "1.0" |
| 79 |
|
| 80 |
[lib] |
| 81 |
proc-macro = true |
| 82 |
``` |
| 83 |
|
| 84 |
```rust |
| 85 |
use proc_macro::TokenStream; |
| 86 |
use quote::quote; |
| 87 |
use syn::{parse_macro_input, DeriveInput}; |
| 88 |
|
| 89 |
#[proc_macro_derive(MyMacro)] |
| 90 |
pub fn my_macro(input: TokenStream) -> TokenStream { |
| 91 |
// Parse the input tokens into a syntax tree |
| 92 |
let input = parse_macro_input!(input as DeriveInput); |
| 93 |
|
| 94 |
// Build the output, possibly using quasi-quotation |
| 95 |
let expanded = quote! { |
| 96 |
// ... |
| 97 |
}; |
| 98 |
|
| 99 |
// Hand the output tokens back to the compiler |
| 100 |
TokenStream::from(expanded) |
| 101 |
} |
| 102 |
``` |
| 103 |
|
| 104 |
The [`heapsize`] example directory shows a complete working implementation of a |
| 105 |
derive macro. The example derives a `HeapSize` trait which computes an estimate |
| 106 |
of the amount of heap memory owned by a value. |
| 107 |
|
| 108 |
[`heapsize`]: examples/heapsize |
| 109 |
|
| 110 |
```rust |
| 111 |
pub trait HeapSize { |
| 112 |
/// Total number of bytes of heap memory owned by `self`. |
| 113 |
fn heap_size_of_children(&self) -> usize; |
| 114 |
} |
| 115 |
``` |
| 116 |
|
| 117 |
The derive macro allows users to write `#[derive(HeapSize)]` on data structures |
| 118 |
in their program. |
| 119 |
|
| 120 |
```rust |
| 121 |
#[derive(HeapSize)] |
| 122 |
struct Demo<'a, T: ?Sized> { |
| 123 |
a: Box<T>, |
| 124 |
b: u8, |
| 125 |
c: &'a str, |
| 126 |
d: String, |
| 127 |
} |
| 128 |
``` |
| 129 |
|
| 130 |
<br> |
| 131 |
|
| 132 |
## Spans and error reporting |
| 133 |
|
| 134 |
The token-based procedural macro API provides great control over where the |
| 135 |
compiler's error messages are displayed in user code. Consider the error the |
| 136 |
user sees if one of their field types does not implement `HeapSize`. |
| 137 |
|
| 138 |
```rust |
| 139 |
#[derive(HeapSize)] |
| 140 |
struct Broken { |
| 141 |
ok: String, |
| 142 |
bad: std::thread::Thread, |
| 143 |
} |
| 144 |
``` |
| 145 |
|
| 146 |
By tracking span information all the way through the expansion of a procedural |
| 147 |
macro as shown in the `heapsize` example, token-based macros in Syn are able to |
| 148 |
trigger errors that directly pinpoint the source of the problem. |
| 149 |
|
| 150 |
```console |
| 151 |
error[E0277]: the trait bound `std::thread::Thread: HeapSize` is not satisfied |
| 152 |
--> src/main.rs:7:5 |
| 153 |
| |
| 154 |
7 | bad: std::thread::Thread, |
| 155 |
| ^^^^^^^^^^^^^^^^^^^^^^^^ the trait `HeapSize` is not implemented for `std::thread::Thread` |
| 156 |
``` |
| 157 |
|
| 158 |
<br> |
| 159 |
|
| 160 |
## Parsing a custom syntax |
| 161 |
|
| 162 |
The [`lazy-static`] example directory shows the implementation of a |
| 163 |
`functionlike!(...)` procedural macro in which the input tokens are parsed using |
| 164 |
Syn's parsing API. |
| 165 |
|
| 166 |
[`lazy-static`]: examples/lazy-static |
| 167 |
|
| 168 |
The example reimplements the popular `lazy_static` crate from crates.io as a |
| 169 |
procedural macro. |
| 170 |
|
| 171 |
```rust |
| 172 |
lazy_static! { |
| 173 |
static ref USERNAME: Regex = Regex::new("^[a-z0-9_-]{3,16}$").unwrap(); |
| 174 |
} |
| 175 |
``` |
| 176 |
|
| 177 |
The implementation shows how to trigger custom warnings and error messages on |
| 178 |
the macro input. |
| 179 |
|
| 180 |
```console |
| 181 |
warning: come on, pick a more creative name |
| 182 |
--> src/main.rs:10:16 |
| 183 |
| |
| 184 |
10 | static ref FOO: String = "lazy_static".to_owned(); |
| 185 |
| ^^^ |
| 186 |
``` |
| 187 |
|
| 188 |
<br> |
| 189 |
|
| 190 |
## Testing |
| 191 |
|
| 192 |
When testing macros, we often care not just that the macro can be used |
| 193 |
successfully but also that when the macro is provided with invalid input it |
| 194 |
produces maximally helpful error messages. Consider using the [`trybuild`] crate |
| 195 |
to write tests for errors that are emitted by your macro or errors detected by |
| 196 |
the Rust compiler in the expanded code following misuse of the macro. Such tests |
| 197 |
help avoid regressions from later refactors that mistakenly make an error no |
| 198 |
longer trigger or be less helpful than it used to be. |
| 199 |
|
| 200 |
[`trybuild`]: https://github.com/dtolnay/trybuild |
| 201 |
|
| 202 |
<br> |
| 203 |
|
| 204 |
## Debugging |
| 205 |
|
| 206 |
When developing a procedural macro it can be helpful to look at what the |
| 207 |
generated code looks like. Use `cargo rustc -- -Zunstable-options |
| 208 |
--pretty=expanded` or the [`cargo expand`] subcommand. |
| 209 |
|
| 210 |
[`cargo expand`]: https://github.com/dtolnay/cargo-expand |
| 211 |
|
| 212 |
To show the expanded code for some crate that uses your procedural macro, run |
| 213 |
`cargo expand` from that crate. To show the expanded code for one of your own |
| 214 |
test cases, run `cargo expand --test the_test_case` where the last argument is |
| 215 |
the name of the test file without the `.rs` extension. |
| 216 |
|
| 217 |
This write-up by Brandon W Maister discusses debugging in more detail: |
| 218 |
[Debugging Rust's new Custom Derive system][debugging]. |
| 219 |
|
| 220 |
[debugging]: https://quodlibetor.github.io/posts/debugging-rusts-new-custom-derive-system/ |
| 221 |
|
| 222 |
<br> |
| 223 |
|
| 224 |
## Optional features |
| 225 |
|
| 226 |
Syn puts a lot of functionality behind optional features in order to optimize |
| 227 |
compile time for the most common use cases. The following features are |
| 228 |
available. |
| 229 |
|
| 230 |
- **`derive`** *(enabled by default)* — Data structures for representing the |
| 231 |
possible input to a derive macro, including structs and enums and types. |
| 232 |
- **`full`** — Data structures for representing the syntax tree of all valid |
| 233 |
Rust source code, including items and expressions. |
| 234 |
- **`parsing`** *(enabled by default)* — Ability to parse input tokens into a |
| 235 |
syntax tree node of a chosen type. |
| 236 |
- **`printing`** *(enabled by default)* — Ability to print a syntax tree node as |
| 237 |
tokens of Rust source code. |
| 238 |
- **`visit`** — Trait for traversing a syntax tree. |
| 239 |
- **`visit-mut`** — Trait for traversing and mutating in place a syntax tree. |
| 240 |
- **`fold`** — Trait for transforming an owned syntax tree. |
| 241 |
- **`clone-impls`** *(enabled by default)* — Clone impls for all syntax tree |
| 242 |
types. |
| 243 |
- **`extra-traits`** — Debug, Eq, PartialEq, Hash impls for all syntax tree |
| 244 |
types. |
| 245 |
- **`proc-macro`** *(enabled by default)* — Runtime dependency on the dynamic |
| 246 |
library libproc_macro from rustc toolchain. |
| 247 |
|
| 248 |
<br> |
| 249 |
|
| 250 |
## Proc macro shim |
| 251 |
|
| 252 |
Syn operates on the token representation provided by the [proc-macro2] crate |
| 253 |
from crates.io rather than using the compiler's built in proc-macro crate |
| 254 |
directly. This enables code using Syn to execute outside of the context of a |
| 255 |
procedural macro, such as in unit tests or build.rs, and we avoid needing |
| 256 |
incompatible ecosystems for proc macros vs non-macro use cases. |
| 257 |
|
| 258 |
In general all of your code should be written against proc-macro2 rather than |
| 259 |
proc-macro. The one exception is in the signatures of procedural macro entry |
| 260 |
points, which are required by the language to use `proc_macro::TokenStream`. |
| 261 |
|
| 262 |
The proc-macro2 crate will automatically detect and use the compiler's data |
| 263 |
structures when a procedural macro is active. |
| 264 |
|
| 265 |
[proc-macro2]: https://docs.rs/proc-macro2/1.0/proc_macro2/ |
| 266 |
|
| 267 |
<br> |
| 268 |
|
| 269 |
#### License |
| 270 |
|
| 271 |
<sup> |
| 272 |
Licensed under either of <a href="LICENSE-APACHE">Apache License, Version |
| 273 |
2.0</a> or <a href="LICENSE-MIT">MIT license</a> at your option. |
| 274 |
</sup> |
| 275 |
|
| 276 |
<br> |
| 277 |
|
| 278 |
<sub> |
| 279 |
Unless you explicitly state otherwise, any contribution intentionally submitted |
| 280 |
for inclusion in this crate by you, as defined in the Apache-2.0 license, shall |
| 281 |
be dual licensed as above, without any additional terms or conditions. |
| 282 |
</sub> |