Debian Codegraph @

guide/src/distribution.md

raw
1 # Distribution
2
3 ## Build Wheels
4
5 When building Rust binary or library, it's possible to depend on libraries and symbols only available on the build machine.
6 To ensure the wheels are portable, native python modules on linux must only dynamically link a set of libraries and symbols called manylinux and musllinux.
7 The pypa offers special docker images to ensure compliance with the [manylinux rules](https://peps.python.org/pep-0599/#the-manylinux2014-policy).
8 If you want to publish linux wheels on pypi, **you need to use a manylinux docker image or [build with zig](#use-zig)**.
9
10 The Rust compiler since version 1.64 [requires at least glibc 2.17](https://blog.rust-lang.org/2022/08/01/Increasing-glibc-kernel-requirements.html), so you need to use at least manylinux2014.
11 For publishing, we recommend enforcing the same manylinux version as the image with the manylinux flag, e.g. use `--manylinux 2014` if you are building in `quay.io/pypa/manylinux2014_x86_64`.
12 The [PyO3/maturin-action](https://github.com/PyO3/maturin-action) github action already takes care of this if you set e.g. `manylinux: 2014`.
13
14 If you are publishing to PyPI, you can use `--compatibility pypi` to allow only builds for targets that are accepted by PyPI, and reject builds for unsupported operating systems and architectures.
15
16 maturin can check the generated library for manylinux compliance (an auditwheel reimplementation) and gives the wheel the proper platform tag.
17
18 - If your system's glibc is too new, it will assign the `linux` tag.
19 - If you link other shared libraries, maturin will try to bundle them within the wheel, note that this requires [patchelf](https://github.com/NixOS/patchelf),
20 which can be installed along with maturin from PyPI: `pip install maturin[patchelf]`.
21
22 You can also manually disable those checks and directly use native linux target with `--manylinux off`.
23
24 The [pyo3/maturin](https://ghcr.io/pyo3/maturin) image is based on the manylinux2014 image, and passes arguments to the `maturin` binary. You can use it like this:
25
26 ```
27 docker run --rm -v $(pwd):/io ghcr.io/pyo3/maturin build --release # or other maturin arguments
28 ```
29
30 Note that this image is very basic and only contains python, maturin and stable Rust. If you need additional tools, you can run commands inside the manylinux container.
31 See [konstin/complex-manylinux-maturin-docker](https://github.com/konstin/complex-manylinux-maturin-docker) for a small educational example
32 or [nanoporetech/fast-ctc-decode](https://github.com/nanoporetech/fast-ctc-decode/blob/b226ea0f2b2f4f474eff47349703d57d2ea4801b/.github/workflows/publish.yml) for a real world setup.
33
34 ```
35 Usage: maturin build [OPTIONS] [ARGS]...
36
37 Arguments:
38 [ARGS]...
39 Rustc flags
40
41 Options:
42 --strip
43 Strip the library for minimum file size
44
45 --sdist
46 Build a source distribution
47
48 --compatibility [<compatibility>...]
49 Control platform tags. Use `pypi` to ensure PyPI compatibility, or specify platform-specific
50 tags like `manylinux2014`, `musllinux_1_2`, or `linux`.
51
52 The default is the lowest compatible `manylinux` tag, or plain `linux` if nothing matched
53
54 -i, --interpreter [<INTERPRETER>...]
55 The python versions to build wheels for, given as the executables of interpreters such as `python3.9` or `/usr/bin/python3.8`
56
57 -f, --find-interpreter
58 Find interpreters from the host machine
59
60 -b, --bindings <BINDINGS>
61 Which kind of bindings to use
62
63 [possible values: pyo3, pyo3-ffi, cffi, uniffi, bin]
64
65 -o, --out <OUT>
66 The directory to store the built wheels in. Defaults to a new "wheels" directory in the project's target directory
67
68 --auditwheel <AUDITWHEEL>
69 Audit wheel for manylinux compliance
70
71 Possible values:
72 - repair: Audit and repair wheel for manylinux compliance
73 - check: Check wheel for manylinux compliance, but do not repair
74 - skip: Don't check for manylinux compliance
75
76 --zig
77 For manylinux targets, use zig to ensure compliance for the chosen manylinux version
78
79 Default to manylinux2014/manylinux_2_17 if you do not specify an `--compatibility`
80
81 Make sure you installed zig with `pip install maturin[zig]`
82
83 -q, --quiet
84 Do not print cargo log messages
85
86 --ignore-rust-version
87 Ignore `rust-version` specification in packages
88
89 -v, --verbose...
90 Use verbose output (-vv very verbose/build.rs output)
91
92 --color <WHEN>
93 Coloring: auto, always, never
94
95 --config <KEY=VALUE>
96 Override a configuration value (unstable)
97
98 -Z <FLAG>
99 Unstable (nightly-only) flags to Cargo, see 'cargo -Z help' for details
100
101 --future-incompat-report
102 Outputs a future incompatibility report at the end of the build (unstable)
103
104 --compression-method <COMPRESSION_METHOD>
105 Zip compresson method to use
106
107 [default: deflated]
108
109 Possible values:
110 - deflated: Deflate compression
111 - stored: No compression
112 - zstd: Zstandard compression
113
114 -h, --help
115 Print help (see a summary with '-h')
116
117 Compilation Options:
118 -r, --release
119 Build artifacts in release mode, with optimizations
120
121 -j, --jobs <N>
122 Number of parallel jobs, defaults to # of CPUs
123
124 --profile <PROFILE-NAME>
125 Build artifacts with the specified Cargo profile
126
127 --target <TRIPLE>
128 Build for the target triple
129
130 [env: CARGO_BUILD_TARGET=]
131
132 --target-dir <DIRECTORY>
133 Directory for all generated artifacts
134
135 --timings=<FMTS>
136 Timing output formats (unstable) (comma separated): html, json
137
138 Feature Selection:
139 -F, --features <FEATURES>
140 Space or comma separated list of features to activate
141
142 --all-features
143 Activate all available features
144
145 --no-default-features
146 Do not activate the `default` feature
147
148 Manifest Options:
149 -m, --manifest-path <PATH>
150 Path to Cargo.toml
151
152 --frozen
153 Require Cargo.lock and cache are up to date
154
155 --locked
156 Require Cargo.lock is up to date
157
158 --offline
159 Run without accessing the network
160 ```
161
162 ## Source Distribution
163
164 Maturin supports building through `pyproject.toml`. To use it, create a `pyproject.toml` next to your `Cargo.toml` with the following content:
165
166 ```toml
167 [build-system]
168 requires = ["maturin>=1.0,<2.0"]
169 build-backend = "maturin"
170 ```
171
172 If a `pyproject.toml` with a `[build-system]` entry is present, maturin can build a source distribution of your package when `--sdist` is specified.
173 The source distribution will contain the same files as `cargo package`. To only build a source distribution, use the `maturin sdist` command.
174
175 You can then e.g. install your package with `pip install .`. With `pip install . -v` you can see the output of cargo and maturin.
176
177 You can use the options `compatibility`, `skip-auditwheel`, `bindings`, `strip` and common Cargo build options such as `features` under `[tool.maturin]` the same way you would when running maturin directly.
178 The `bindings` key is required for cffi and bin projects as those can't be automatically detected. Currently, all builds are in release mode (see [this thread](https://discuss.python.org/t/pep-517-debug-vs-release-builds/1924) for details).
179
180 For a non-manylinux build with cffi bindings you could use the following:
181
182 ```toml
183 [build-system]
184 requires = ["maturin>=1.0,<2.0"]
185 build-backend = "maturin"
186
187 [tool.maturin]
188 bindings = "cffi"
189 compatibility = "linux"
190 ```
191
192 `manylinux` option is also accepted as an alias of `compatibility` for backward compatibility with old version of maturin.
193
194 To include arbitrary files in the sdist for use during compilation specify `include` as an array of `path` globs with `format` set to `sdist`:
195
196 ```toml
197 [tool.maturin]
198 include = [{ path = "path/**/*", format = "sdist" }]
199 ```
200
201 ### Cross Compiling
202
203 Maturin has decent cross compilation support for `pyo3` and `bin` bindings,
204 other kind of bindings may work but aren't tested regularly.
205
206 #### Cross-compile to Linux/macOS
207
208 ##### Use Docker
209
210 For manylinux support the [manylinux-cross](https://github.com/rust-cross/manylinux-cross) docker images can be used.
211 And [maturin-action](https://github.com/PyO3/maturin-action) makes it easy to do cross compilation on GitHub Actions.
212
213 ##### Use Zig
214
215 Since v0.12.7 maturin added support for linking with [`zig cc`](https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html),
216 compile for Linux works and is regularly tested on CI, other platforms may also work but aren't tested regularly.
217
218 You can install zig following the [official documentation](https://ziglang.org/download), or install it from PyPI via `pip install ziglang`.
219 Then pass `--zig` to maturin `build` or `publish` commands to use it, for example
220
221 ```bash
222 maturin build --release --target aarch64-unknown-linux-gnu --zig
223 ```
224
225 #### Cross-compile to Windows
226
227 Pyo3 0.16.5 added an experimental feature `generate-import-lib` enables the user to cross compile
228 extension modules for Windows targets without setting the `PYO3_CROSS_LIB_DIR` environment variable
229 or providing any Windows Python library files.
230
231 ```toml
232 [dependencies]
233 pyo3 = { version = "0.25.0", features = ["extension-module", "generate-import-lib"] }
234 ```
235
236 It uses an external [`python3-dll-a`](https://docs.rs/python3-dll-a/latest/python3_dll_a/) crate to
237 generate import libraries for the Python DLL for MinGW-w64 and MSVC compile targets.
238 Note: MSVC targets require LLVM binutils or MSVC build tools to be available on the host system.
239 More specifically, `python3-dll-a` requires `llvm-dlltool` or `lib.exe` executable to be present in `PATH` when targeting `*-pc-windows-msvc`.
240
241 maturin integrates [`cargo-xwin`](https://github.com/messense/cargo-xwin) to enable MSVC targets cross compilation support,
242 it will download and unpack the Microsoft CRT headers and import libraries, and Windows SDK headers and import libraries
243 needed for compiling and linking automatically.
244
245 **By using this to cross compiling to Windows MSVC targets you are consented to accept the license at [https://go.microsoft.com/fwlink/?LinkId=2086102](https://go.microsoft.com/fwlink/?LinkId=2086102)**.
246 (Building on Windows natively does not apply.)
247
248 ## GitHub Actions
249
250 If your project uses GitHub Actions, you can use the `maturin generate-ci` command to generate a GitHub Actions workflow file.
251
252 ```bash
253 mkdir -p .github/workflows
254 maturin generate-ci github > .github/workflows/CI.yml
255 ```
256
257 There are some options to customize the generated workflow file:
258
259 ```
260 Generate CI configuration
261
262 Usage: maturin generate-ci [OPTIONS] <CI>
263
264 Arguments:
265 <CI>
266 CI provider
267
268 Possible values:
269 - github: GitHub
270
271 Options:
272 -m, --manifest-path <PATH>
273 Path to Cargo.toml
274
275 -v, --verbose...
276 Use verbose output.
277
278 * Default: Show build information and `cargo build` output. * `-v`: Use `cargo build -v`.
279 * `-vv`: Show debug logging and use `cargo build -vv`. * `-vvv`: Show trace logging.
280
281 You can configure fine-grained logging using the `RUST_LOG` environment variable.
282 (<https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/struct.EnvFilter.html#directives>)
283
284 -o, --output <PATH>
285 Output path
286
287 [default: -]
288
289 --platform <platform>...
290 Platform support
291
292 [default: linux musllinux windows macos]
293
294 Possible values:
295 - all: All
296 - manylinux: Manylinux
297 - musllinux: Musllinux
298 - windows: Windows
299 - macos: macOS
300 - emscripten: Emscripten
301
302 --pytest
303 Enable pytest
304
305 --zig
306 Use zig to do cross compilation
307
308 --skip-attestation
309 Skip artifact attestation
310
311 -h, --help
312 Print help (see a summary with '-h')
313 ```
314
315 ### Using PyPI's trusted publishing
316
317 By default, the workflow provided by `generate-ci` will publish the release artifacts to PyPI using API token authentication. However, maturin also supports [trusted publishing (OpenID Connect)](https://docs.pypi.org/trusted-publishers/).
318
319 To enable it, modify the `release` action in the generated GitHub workflow file:
320
321 - remove `MATURIN_PYPI_TOKEN` from the `env` section to make maturin use trusted publishing
322 - add `id-token: write` to the action's `permissions` (see [Configuring OpenID Connect in PyPI](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-pypi) from GitHub's documentation).
323 - if `Environment name: pypi` was set in PyPI, add `environment: pypi`
324
325 Make sure to follow the steps listed in [PyPI's documentation](https://docs.pypi.org/trusted-publishers/adding-a-publisher/) to set up your GitHub repository as a trusted publisher in the PyPI project settings before attempting to run the workflow.