Debian Codegraph @

guide/src/import_hook.md

raw
1 # Import Hook
2
3 [maturin_import_hook](https://pypi.org/project/maturin-import-hook/) provides a mechanism to
4 automatically rebuild Maturin projects when they are imported by Python scripts.
5
6 This reduces friction when developing mixed Python/Rust codebases because edits made to Rust components take effect
7 automatically like edits to Python components do. Automatic rebuilding eliminates the possibility of Python code
8 using outdated rust components, which often leads to confusing behaviour.
9
10 The import hook can be configured to activate automatically as the interpreter starts or manually in individual scripts.
11 Only Maturin packages installed editable mode (`maturin develop` or `pip install -e`) are considered by the import hook.
12 When the build is up to date the overhead of the import hook is minimal.
13
14 The hook has some additional features such as importing stand-alone `.rs` files.
15 See [Features](#features) for a complete list.
16
17 ## Installation
18
19 Install the package with:
20 ```shell
21 $ pip install maturin_import_hook
22 ```
23
24 Then install it into the current virtual environment with:
25 ```shell
26 $ python -m maturin_import_hook site install
27 ```
28 This will activate the hook automatically as the interpreter starts.
29 This command only has to be run once per virtual environment.
30 To use `site install`, ensure you have write access to `site-packages`.
31 Using a [virtual environment](https://docs.python.org/3/library/venv.html) is recommended rather than installing into
32 the system interpreter.
33
34 The managed site installation can be removed with:
35 ```shell
36 $ python -m maturin_import_hook site uninstall
37 ```
38
39 To manually activate the import hook only for specific scripts instead of installing into the virtual
40 environment see [Manual Activation](#manual-activation).
41
42 ## Basic Configuration
43
44 The site installation can be customized. For example, to build in release mode:
45 ```shell
46 $ python -m maturin_import_hook site install --args="--release"
47 ```
48 The `--args` option accepts `maturin develop` arguments. For more options see `--help`.
49
50 The site installation can also be edited manually. Use `python -m maturin_import_hook site info` to locate the
51 `sitecustomize.py` file. See [Install Arguments](#install-arguments) for a list of arguments to `install()`.
52
53 To avoid unnecessary rebuilds caused by irrelevant file modifications, create empty `.maturin_hook_ignore` files to
54 manually ignore directories. Several directories and file types such as `target/` and `*.py` are ignored by default.
55 If more customization is needed, see [Custom File Searching](#custom-file-searching).
56
57 ## CLI
58
59 The package provides a CLI for managing the build cache and site installation. For details, run:
60
61 ```shell
62 python -m maturin_import_hook --help
63 ```
64
65 * `site (info | install | uninstall)`
66 * Manage import hook installation in [`sitecustomize.py`](#sitecustomize) of the
67 active environment.
68 * `install` options:
69 * `--force`: Whether to overwrite any existing managed import hook installation.
70 * `--(no-)-project-importer`: Whether to enable the project importer.
71 * `--(no-)-rs-file-importer`: Whether to enable the rs file importer.
72 * `--(no-)-detect-uv`: Whether to automatically detect and use the `--uv` flag.
73 * `--args`: The arguments to pass to `maturin`.
74 * `--user`: whether to install into `usercustomize.py` instead of `sitecustomize.py`.
75 * `cache (info | clear)`
76 * Get info about the size and location of the build cache.
77 * Manage the build cache of the active environment.
78 * `version`
79 * Show version info of the import hook and associated tools. Useful for providing information to bug reports.
80
81
82 ## Features
83
84 The import hook supports:
85
86 * All the binding types and project layouts supported by Maturin.
87 * All [CPython](https://www.python.org/) and [PyPy](https://pypy.org/) versions supported by Maturin.
88 * Windows, Linux and MacOS.
89 * Interactive environments such as [IPython](https://ipython.org/), [Jupyter](https://jupyter.org/) and
90 [Python REPL](https://docs.python.org/3/tutorial/interpreter.html#interactive-mode).
91 * Importing multiple Maturin projects in the same script.
92 * Importing stand-alone `.rs` files that use `PyO3` bindings (see [Import Rust File](#import-rust-file)).
93 * `importlib.reload()` (currently unsupported on Windows).
94 * Detecting edits to path dependencies at any depth.
95 * Multiple environments with different interpreters using isolated build caches.
96 * Concurrent imports/builds from multiple scripts (useful for tools like [`pytest-xdist`](https://pypi.org/project/pytest-xdist/))
97 * Extensibility (see [Advanced Usage](#advanced-usage)).
98
99 ## Manual Activation
100
101 To activate the import hook in a single Python script, call `install()` at the top of the script:
102
103 ```python
104 import maturin_import_hook
105 maturin_import_hook.install() # Must come first. Not active for imports above.
106
107 import my_rust_package # An editable-installed Maturin project.
108
109 import foo.my_rust_script # `foo/my_rust_script.rs` defines a pyo3 module.
110 ```
111
112 The Maturin project importer and the Rust file importer can be used separately:
113
114 ```python
115 from maturin_import_hook import project_importer
116 project_importer.install()
117
118 from maturin_import_hook import rust_file_importer
119 rust_file_importer.install()
120 ```
121
122 The import hook can be configured to control its behaviour (see [Install Arguments](#install-arguments)).
123
124 The import hook is intended for use in development environments and not for
125 production environments, so any calls to `install()` should ideally be removed
126 (or disabled, see [Environment Variables](#environment-variables)) before reaching production.
127 This is another reason why the site installation is convenient.
128
129 The hook remains active across multiple Python modules so a single `install()` at the top of the main module is
130 sometimes sufficient. This can cause problems however, when the main module is not imported or not imported first,
131 such as in tests. `install()` can be called in each module but be aware that each call replaces the previous settings.
132
133 Tip: If you want to use non-defaults across multiple modules you can create a function like:
134 ```python
135 import maturin_import_hook
136
137 _HOOK_INSTALLED = False
138
139 def install_maturin_hook() -> None:
140 global _HOOK_INSTALLED
141 if not _HOOK_INSTALLED:
142 maturin_import_hook.install(
143 ... # your custom configuration here
144 )
145 _HOOK_INSTALLED = True
146 ```
147 Then call `install_maturin_hook()` at the top of each module. This will ensure the custom options are used everywhere
148 without any code duplication.
149
150 ## Import Rust File
151
152 When a `.rs` file is imported, a Maturin project with [pyo3](https://pyo3.rs/) bindings will be created for it in the
153 Maturin build cache and the resulting library will be built and loaded.
154
155 For example, `my_extension.rs`:
156 ```rust
157 use pyo3::prelude::*;
158
159 #[pyfunction]
160 fn double(x: usize) -> usize { x * 2 }
161
162 #[pymodule]
163 fn my_extension(m: &Bound<'_, PyModule>) -> PyResult<()> {
164 m.add_function(wrap_pyfunction!(double, m)?)
165 }
166 ```
167 The `#[pymodule]` must have the same name as the filename.
168 The version of `pyo3` is determined by `maturin new --bindings pyo3`.
169
170
171
172 ## Environment Variables
173
174 The import hook can be disabled by setting `MATURIN_IMPORT_HOOK_ENABLED=0`. This can be used to disable
175 the import hook in production while leaving calls to `install()` in place.
176
177 Build files will be stored in an appropriate place for the current system but can be overridden
178 by setting `MATURIN_BUILD_DIR`. These files can be deleted without causing any issues (unless a build is in progress).
179 The precedence for storing build files is:
180
181 * `MATURIN_BUILD_DIR`
182 * (Each environment will store its cache in a subdirectory of the given path).
183 * `<virtualenv_dir>/maturin_build_cache`
184 * `<system_cache_dir>/maturin_build_cache`
185 * e.g. `~/.cache/maturin_build_cache` on POSIX.
186
187 See the location being used with the CLI: `python -m maturin_import_hook cache info`
188
189 ## Logging
190
191 By default, the `maturin_import_hook` logger does not propagate to the root logger. This is so that `INFO` level
192 messages are shown without having to configure logging (`INFO` level is normally not visible). The import hook
193 also has extensive `DEBUG` level logging that would clutter application logs. So by not propagating, `DEBUG`
194 messages from the import hook are not shown even if the root logger has `DEBUG` level visible.
195
196 If you prefer, `maturin_import_hook.reset_logger()` can be called to undo the default configuration and propagate
197 the messages as normal.
198
199 When debugging issues with the import hook, you should first call `reset_logger()` then configure the root logger
200 to show `DEBUG` messages. You can also run with the environment variable `RUST_LOG=maturin=debug` to get more
201 information from Maturin.
202
203 ```python
204 import logging
205 logging.basicConfig(format='%(asctime)s %(name)s [%(levelname)s] %(message)s', level=logging.DEBUG)
206 import maturin_import_hook
207 maturin_import_hook.reset_logger()
208 maturin_import_hook.install()
209 ```
210
211 ## Background
212
213 ### Import Hook
214
215 An [import hook](https://docs.python.org/3/reference/import.html#import-hooks) is a class added to
216 [sys.meta_path](https://docs.python.org/3/library/sys.html#sys.meta_path) that is invoked by the Python interpreter
217 whenever it encounters an `import` statement (or import via other means). `find_spec()` is called for each hook until
218 one returns a `ModuleSpec` indicating that it found the module. If a hook cannot handle a particular import it
219 returns `None`. Import hooks can be used to create modules on demand (e.g. the `.rs` importer) or trigger
220 side-effects with imports (e.g. the project importer).
221
222 [MaturinProjectImporter](https://github.com/PyO3/maturin-import-hook/blob/main/src/maturin_import_hook/project_importer.py):
223
224 1. Detects when an import corresponds to an editable installed Maturin project.
225 2. Determines if the current build of the project is up to date.
226 3. Rebuilds the project if necessary.
227
228 [MaturinRustFileImporter](https://github.com/PyO3/maturin-import-hook/blob/main/src/maturin_import_hook/rust_file_importer.py):
229
230 1. Searches `sys.path` for an `.rs` file matching the import
231 (e.g. for `import foo.bar`, the hook will look for `foo/bar.rs` at each search path).
232 2. Creates a temporary Maturin project for the `.rs` file or reuses the project if it already exists.
233 3. Rebuilds the project if necessary.
234
235 The above steps are a simplification as supporting `importlib.reload()` requires more complex logic.
236 See [reloading.md](https://github.com/PyO3/maturin-import-hook/blob/main/docs/reloading.md) for more details.
237
238 ### Sitecustomize
239
240 If a [sitecustomize.py](https://docs.python.org/3/library/site.html) file exists in the `site-packages` directory of
241 a Python installation it is loaded automatically by the Python interpreter as it starts unless the `-S` flag is
242 passed to it. This makes `sitecustomize.py` a convenient place to activate the import hook.
243
244
245 ## Advanced Usage
246
247 ### Install Arguments
248
249 The arguments to `install()` are ([source](https://github.com/PyO3/maturin-import-hook/blob/main/src/maturin_import_hook/__init__.py)):
250
251 ```python
252 """
253 enable_project_importer: enable the hook for automatically rebuilding editable installed maturin projects
254
255 enable_rs_file_importer: enable the hook for importing .rs files as though they were regular python modules
256
257 enable_reloading: enable workarounds to allow the extension modules to be reloaded with `importlib.reload()`
258
259 settings: settings corresponding to flags passed to maturin.
260
261 build_dir: where to put the compiled artifacts. defaults to `$MATURIN_BUILD_DIR`,
262 `sys.exec_prefix / 'maturin_build_cache'` or
263 `$HOME/.cache/maturin_build_cache/<interpreter_hash>` in order of preference
264
265 force_rebuild: whether to always rebuild and skip checking whether anything has changed
266
267 lock_timeout_seconds: a lock is required to prevent projects from being built concurrently.
268 If the lock is not released before this timeout is reached the import hook stops waiting and aborts.
269 A value of None means that the import hook will wait for the lock indefinitely.
270
271 show_warnings: whether to show compilation warnings
272
273 file_searcher: an object used to find source and installed project files that are used to determine whether
274 a project has changed and needs to be rebuilt
275
276 enable_automatic_install: whether to install detected packages using the import hook even if they
277 are not already installed into the virtual environment or are installed in non-editable mode.
278 """
279 ```
280 ### Subclassing
281
282 The import hook classes can be subclassed to further customize to specific use cases.
283 For example settings can be configured per-project or loaded from configuration files.
284
285 ```python
286 import sys
287 from pathlib import Path
288 from maturin_import_hook.settings import MaturinSettings
289 from maturin_import_hook.project_importer import MaturinProjectImporter
290
291 class CustomImporter(MaturinProjectImporter):
292 def get_settings(self, module_path: str, source_path: Path) -> MaturinSettings:
293 return MaturinSettings(
294 release=True,
295 strip=True,
296 # ...
297 )
298
299 sys.meta_path.insert(0, CustomImporter())
300 ```
301
302 ### Custom File Searching
303
304 The `file_searcher` argument to `install()` can be used to customize which files should be included/excluded when
305 deciding whether a package needs to be rebuilt:
306
307 ```python
308 from collections.abc import Iterator
309 from pathlib import Path
310 from maturin_import_hook.project_importer import ProjectFileSearcher, install
311
312 class CustomFileSearcher(ProjectFileSearcher):
313 def get_source_paths(
314 self,
315 project_dir: Path,
316 all_path_dependencies: list[Path],
317 installed_package_root: Path,
318 ) -> Iterator[Path]: ...
319
320 def get_installation_paths(self, installed_package_root: Path) -> Iterator[Path]: ...
321
322 install(file_searcher=CustomFileSearcher())
323 ```
324
325 See `maturin_import_hook.project.DefaultProjectFileSearcher` for the default list of include/exclude criteria.
326
327 The class variables of `DefaultProjectFileSearcher` can be edited **before** calling `install()` to make simple
328 customizations.