| 1 |
# Import Hook |
| 2 |
|
| 3 |
[maturin_import_hook](https://pypi.org/project/maturin-import-hook/) provides a mechanism to |
| 4 |
automatically rebuild Maturin projects when they are imported by Python scripts. |
| 5 |
|
| 6 |
This reduces friction when developing mixed Python/Rust codebases because edits made to Rust components take effect |
| 7 |
automatically like edits to Python components do. Automatic rebuilding eliminates the possibility of Python code |
| 8 |
using outdated rust components, which often leads to confusing behaviour. |
| 9 |
|
| 10 |
The import hook can be configured to activate automatically as the interpreter starts or manually in individual scripts. |
| 11 |
Only Maturin packages installed editable mode (`maturin develop` or `pip install -e`) are considered by the import hook. |
| 12 |
When the build is up to date the overhead of the import hook is minimal. |
| 13 |
|
| 14 |
The hook has some additional features such as importing stand-alone `.rs` files. |
| 15 |
See [Features](#features) for a complete list. |
| 16 |
|
| 17 |
## Installation |
| 18 |
|
| 19 |
Install the package with: |
| 20 |
```shell |
| 21 |
$ pip install maturin_import_hook |
| 22 |
``` |
| 23 |
|
| 24 |
Then install it into the current virtual environment with: |
| 25 |
```shell |
| 26 |
$ python -m maturin_import_hook site install |
| 27 |
``` |
| 28 |
This will activate the hook automatically as the interpreter starts. |
| 29 |
This command only has to be run once per virtual environment. |
| 30 |
To use `site install`, ensure you have write access to `site-packages`. |
| 31 |
Using a [virtual environment](https://docs.python.org/3/library/venv.html) is recommended rather than installing into |
| 32 |
the system interpreter. |
| 33 |
|
| 34 |
The managed site installation can be removed with: |
| 35 |
```shell |
| 36 |
$ python -m maturin_import_hook site uninstall |
| 37 |
``` |
| 38 |
|
| 39 |
To manually activate the import hook only for specific scripts instead of installing into the virtual |
| 40 |
environment see [Manual Activation](#manual-activation). |
| 41 |
|
| 42 |
## Basic Configuration |
| 43 |
|
| 44 |
The site installation can be customized. For example, to build in release mode: |
| 45 |
```shell |
| 46 |
$ python -m maturin_import_hook site install --args="--release" |
| 47 |
``` |
| 48 |
The `--args` option accepts `maturin develop` arguments. For more options see `--help`. |
| 49 |
|
| 50 |
The site installation can also be edited manually. Use `python -m maturin_import_hook site info` to locate the |
| 51 |
`sitecustomize.py` file. See [Install Arguments](#install-arguments) for a list of arguments to `install()`. |
| 52 |
|
| 53 |
To avoid unnecessary rebuilds caused by irrelevant file modifications, create empty `.maturin_hook_ignore` files to |
| 54 |
manually ignore directories. Several directories and file types such as `target/` and `*.py` are ignored by default. |
| 55 |
If more customization is needed, see [Custom File Searching](#custom-file-searching). |
| 56 |
|
| 57 |
## CLI |
| 58 |
|
| 59 |
The package provides a CLI for managing the build cache and site installation. For details, run: |
| 60 |
|
| 61 |
```shell |
| 62 |
python -m maturin_import_hook --help |
| 63 |
``` |
| 64 |
|
| 65 |
* `site (info | install | uninstall)` |
| 66 |
* Manage import hook installation in [`sitecustomize.py`](#sitecustomize) of the |
| 67 |
active environment. |
| 68 |
* `install` options: |
| 69 |
* `--force`: Whether to overwrite any existing managed import hook installation. |
| 70 |
* `--(no-)-project-importer`: Whether to enable the project importer. |
| 71 |
* `--(no-)-rs-file-importer`: Whether to enable the rs file importer. |
| 72 |
* `--(no-)-detect-uv`: Whether to automatically detect and use the `--uv` flag. |
| 73 |
* `--args`: The arguments to pass to `maturin`. |
| 74 |
* `--user`: whether to install into `usercustomize.py` instead of `sitecustomize.py`. |
| 75 |
* `cache (info | clear)` |
| 76 |
* Get info about the size and location of the build cache. |
| 77 |
* Manage the build cache of the active environment. |
| 78 |
* `version` |
| 79 |
* Show version info of the import hook and associated tools. Useful for providing information to bug reports. |
| 80 |
|
| 81 |
|
| 82 |
## Features |
| 83 |
|
| 84 |
The import hook supports: |
| 85 |
|
| 86 |
* All the binding types and project layouts supported by Maturin. |
| 87 |
* All [CPython](https://www.python.org/) and [PyPy](https://pypy.org/) versions supported by Maturin. |
| 88 |
* Windows, Linux and MacOS. |
| 89 |
* Interactive environments such as [IPython](https://ipython.org/), [Jupyter](https://jupyter.org/) and |
| 90 |
[Python REPL](https://docs.python.org/3/tutorial/interpreter.html#interactive-mode). |
| 91 |
* Importing multiple Maturin projects in the same script. |
| 92 |
* Importing stand-alone `.rs` files that use `PyO3` bindings (see [Import Rust File](#import-rust-file)). |
| 93 |
* `importlib.reload()` (currently unsupported on Windows). |
| 94 |
* Detecting edits to path dependencies at any depth. |
| 95 |
* Multiple environments with different interpreters using isolated build caches. |
| 96 |
* Concurrent imports/builds from multiple scripts (useful for tools like [`pytest-xdist`](https://pypi.org/project/pytest-xdist/)) |
| 97 |
* Extensibility (see [Advanced Usage](#advanced-usage)). |
| 98 |
|
| 99 |
## Manual Activation |
| 100 |
|
| 101 |
To activate the import hook in a single Python script, call `install()` at the top of the script: |
| 102 |
|
| 103 |
```python |
| 104 |
import maturin_import_hook |
| 105 |
maturin_import_hook.install() # Must come first. Not active for imports above. |
| 106 |
|
| 107 |
import my_rust_package # An editable-installed Maturin project. |
| 108 |
|
| 109 |
import foo.my_rust_script # `foo/my_rust_script.rs` defines a pyo3 module. |
| 110 |
``` |
| 111 |
|
| 112 |
The Maturin project importer and the Rust file importer can be used separately: |
| 113 |
|
| 114 |
```python |
| 115 |
from maturin_import_hook import project_importer |
| 116 |
project_importer.install() |
| 117 |
|
| 118 |
from maturin_import_hook import rust_file_importer |
| 119 |
rust_file_importer.install() |
| 120 |
``` |
| 121 |
|
| 122 |
The import hook can be configured to control its behaviour (see [Install Arguments](#install-arguments)). |
| 123 |
|
| 124 |
The import hook is intended for use in development environments and not for |
| 125 |
production environments, so any calls to `install()` should ideally be removed |
| 126 |
(or disabled, see [Environment Variables](#environment-variables)) before reaching production. |
| 127 |
This is another reason why the site installation is convenient. |
| 128 |
|
| 129 |
The hook remains active across multiple Python modules so a single `install()` at the top of the main module is |
| 130 |
sometimes sufficient. This can cause problems however, when the main module is not imported or not imported first, |
| 131 |
such as in tests. `install()` can be called in each module but be aware that each call replaces the previous settings. |
| 132 |
|
| 133 |
Tip: If you want to use non-defaults across multiple modules you can create a function like: |
| 134 |
```python |
| 135 |
import maturin_import_hook |
| 136 |
|
| 137 |
_HOOK_INSTALLED = False |
| 138 |
|
| 139 |
def install_maturin_hook() -> None: |
| 140 |
global _HOOK_INSTALLED |
| 141 |
if not _HOOK_INSTALLED: |
| 142 |
maturin_import_hook.install( |
| 143 |
... # your custom configuration here |
| 144 |
) |
| 145 |
_HOOK_INSTALLED = True |
| 146 |
``` |
| 147 |
Then call `install_maturin_hook()` at the top of each module. This will ensure the custom options are used everywhere |
| 148 |
without any code duplication. |
| 149 |
|
| 150 |
## Import Rust File |
| 151 |
|
| 152 |
When a `.rs` file is imported, a Maturin project with [pyo3](https://pyo3.rs/) bindings will be created for it in the |
| 153 |
Maturin build cache and the resulting library will be built and loaded. |
| 154 |
|
| 155 |
For example, `my_extension.rs`: |
| 156 |
```rust |
| 157 |
use pyo3::prelude::*; |
| 158 |
|
| 159 |
#[pyfunction] |
| 160 |
fn double(x: usize) -> usize { x * 2 } |
| 161 |
|
| 162 |
#[pymodule] |
| 163 |
fn my_extension(m: &Bound<'_, PyModule>) -> PyResult<()> { |
| 164 |
m.add_function(wrap_pyfunction!(double, m)?) |
| 165 |
} |
| 166 |
``` |
| 167 |
The `#[pymodule]` must have the same name as the filename. |
| 168 |
The version of `pyo3` is determined by `maturin new --bindings pyo3`. |
| 169 |
|
| 170 |
|
| 171 |
|
| 172 |
## Environment Variables |
| 173 |
|
| 174 |
The import hook can be disabled by setting `MATURIN_IMPORT_HOOK_ENABLED=0`. This can be used to disable |
| 175 |
the import hook in production while leaving calls to `install()` in place. |
| 176 |
|
| 177 |
Build files will be stored in an appropriate place for the current system but can be overridden |
| 178 |
by setting `MATURIN_BUILD_DIR`. These files can be deleted without causing any issues (unless a build is in progress). |
| 179 |
The precedence for storing build files is: |
| 180 |
|
| 181 |
* `MATURIN_BUILD_DIR` |
| 182 |
* (Each environment will store its cache in a subdirectory of the given path). |
| 183 |
* `<virtualenv_dir>/maturin_build_cache` |
| 184 |
* `<system_cache_dir>/maturin_build_cache` |
| 185 |
* e.g. `~/.cache/maturin_build_cache` on POSIX. |
| 186 |
|
| 187 |
See the location being used with the CLI: `python -m maturin_import_hook cache info` |
| 188 |
|
| 189 |
## Logging |
| 190 |
|
| 191 |
By default, the `maturin_import_hook` logger does not propagate to the root logger. This is so that `INFO` level |
| 192 |
messages are shown without having to configure logging (`INFO` level is normally not visible). The import hook |
| 193 |
also has extensive `DEBUG` level logging that would clutter application logs. So by not propagating, `DEBUG` |
| 194 |
messages from the import hook are not shown even if the root logger has `DEBUG` level visible. |
| 195 |
|
| 196 |
If you prefer, `maturin_import_hook.reset_logger()` can be called to undo the default configuration and propagate |
| 197 |
the messages as normal. |
| 198 |
|
| 199 |
When debugging issues with the import hook, you should first call `reset_logger()` then configure the root logger |
| 200 |
to show `DEBUG` messages. You can also run with the environment variable `RUST_LOG=maturin=debug` to get more |
| 201 |
information from Maturin. |
| 202 |
|
| 203 |
```python |
| 204 |
import logging |
| 205 |
logging.basicConfig(format='%(asctime)s %(name)s [%(levelname)s] %(message)s', level=logging.DEBUG) |
| 206 |
import maturin_import_hook |
| 207 |
maturin_import_hook.reset_logger() |
| 208 |
maturin_import_hook.install() |
| 209 |
``` |
| 210 |
|
| 211 |
## Background |
| 212 |
|
| 213 |
### Import Hook |
| 214 |
|
| 215 |
An [import hook](https://docs.python.org/3/reference/import.html#import-hooks) is a class added to |
| 216 |
[sys.meta_path](https://docs.python.org/3/library/sys.html#sys.meta_path) that is invoked by the Python interpreter |
| 217 |
whenever it encounters an `import` statement (or import via other means). `find_spec()` is called for each hook until |
| 218 |
one returns a `ModuleSpec` indicating that it found the module. If a hook cannot handle a particular import it |
| 219 |
returns `None`. Import hooks can be used to create modules on demand (e.g. the `.rs` importer) or trigger |
| 220 |
side-effects with imports (e.g. the project importer). |
| 221 |
|
| 222 |
[MaturinProjectImporter](https://github.com/PyO3/maturin-import-hook/blob/main/src/maturin_import_hook/project_importer.py): |
| 223 |
|
| 224 |
1. Detects when an import corresponds to an editable installed Maturin project. |
| 225 |
2. Determines if the current build of the project is up to date. |
| 226 |
3. Rebuilds the project if necessary. |
| 227 |
|
| 228 |
[MaturinRustFileImporter](https://github.com/PyO3/maturin-import-hook/blob/main/src/maturin_import_hook/rust_file_importer.py): |
| 229 |
|
| 230 |
1. Searches `sys.path` for an `.rs` file matching the import |
| 231 |
(e.g. for `import foo.bar`, the hook will look for `foo/bar.rs` at each search path). |
| 232 |
2. Creates a temporary Maturin project for the `.rs` file or reuses the project if it already exists. |
| 233 |
3. Rebuilds the project if necessary. |
| 234 |
|
| 235 |
The above steps are a simplification as supporting `importlib.reload()` requires more complex logic. |
| 236 |
See [reloading.md](https://github.com/PyO3/maturin-import-hook/blob/main/docs/reloading.md) for more details. |
| 237 |
|
| 238 |
### Sitecustomize |
| 239 |
|
| 240 |
If a [sitecustomize.py](https://docs.python.org/3/library/site.html) file exists in the `site-packages` directory of |
| 241 |
a Python installation it is loaded automatically by the Python interpreter as it starts unless the `-S` flag is |
| 242 |
passed to it. This makes `sitecustomize.py` a convenient place to activate the import hook. |
| 243 |
|
| 244 |
|
| 245 |
## Advanced Usage |
| 246 |
|
| 247 |
### Install Arguments |
| 248 |
|
| 249 |
The arguments to `install()` are ([source](https://github.com/PyO3/maturin-import-hook/blob/main/src/maturin_import_hook/__init__.py)): |
| 250 |
|
| 251 |
```python |
| 252 |
""" |
| 253 |
enable_project_importer: enable the hook for automatically rebuilding editable installed maturin projects |
| 254 |
|
| 255 |
enable_rs_file_importer: enable the hook for importing .rs files as though they were regular python modules |
| 256 |
|
| 257 |
enable_reloading: enable workarounds to allow the extension modules to be reloaded with `importlib.reload()` |
| 258 |
|
| 259 |
settings: settings corresponding to flags passed to maturin. |
| 260 |
|
| 261 |
build_dir: where to put the compiled artifacts. defaults to `$MATURIN_BUILD_DIR`, |
| 262 |
`sys.exec_prefix / 'maturin_build_cache'` or |
| 263 |
`$HOME/.cache/maturin_build_cache/<interpreter_hash>` in order of preference |
| 264 |
|
| 265 |
force_rebuild: whether to always rebuild and skip checking whether anything has changed |
| 266 |
|
| 267 |
lock_timeout_seconds: a lock is required to prevent projects from being built concurrently. |
| 268 |
If the lock is not released before this timeout is reached the import hook stops waiting and aborts. |
| 269 |
A value of None means that the import hook will wait for the lock indefinitely. |
| 270 |
|
| 271 |
show_warnings: whether to show compilation warnings |
| 272 |
|
| 273 |
file_searcher: an object used to find source and installed project files that are used to determine whether |
| 274 |
a project has changed and needs to be rebuilt |
| 275 |
|
| 276 |
enable_automatic_install: whether to install detected packages using the import hook even if they |
| 277 |
are not already installed into the virtual environment or are installed in non-editable mode. |
| 278 |
""" |
| 279 |
``` |
| 280 |
### Subclassing |
| 281 |
|
| 282 |
The import hook classes can be subclassed to further customize to specific use cases. |
| 283 |
For example settings can be configured per-project or loaded from configuration files. |
| 284 |
|
| 285 |
```python |
| 286 |
import sys |
| 287 |
from pathlib import Path |
| 288 |
from maturin_import_hook.settings import MaturinSettings |
| 289 |
from maturin_import_hook.project_importer import MaturinProjectImporter |
| 290 |
|
| 291 |
class CustomImporter(MaturinProjectImporter): |
| 292 |
def get_settings(self, module_path: str, source_path: Path) -> MaturinSettings: |
| 293 |
return MaturinSettings( |
| 294 |
release=True, |
| 295 |
strip=True, |
| 296 |
# ... |
| 297 |
) |
| 298 |
|
| 299 |
sys.meta_path.insert(0, CustomImporter()) |
| 300 |
``` |
| 301 |
|
| 302 |
### Custom File Searching |
| 303 |
|
| 304 |
The `file_searcher` argument to `install()` can be used to customize which files should be included/excluded when |
| 305 |
deciding whether a package needs to be rebuilt: |
| 306 |
|
| 307 |
```python |
| 308 |
from collections.abc import Iterator |
| 309 |
from pathlib import Path |
| 310 |
from maturin_import_hook.project_importer import ProjectFileSearcher, install |
| 311 |
|
| 312 |
class CustomFileSearcher(ProjectFileSearcher): |
| 313 |
def get_source_paths( |
| 314 |
self, |
| 315 |
project_dir: Path, |
| 316 |
all_path_dependencies: list[Path], |
| 317 |
installed_package_root: Path, |
| 318 |
) -> Iterator[Path]: ... |
| 319 |
|
| 320 |
def get_installation_paths(self, installed_package_root: Path) -> Iterator[Path]: ... |
| 321 |
|
| 322 |
install(file_searcher=CustomFileSearcher()) |
| 323 |
``` |
| 324 |
|
| 325 |
See `maturin_import_hook.project.DefaultProjectFileSearcher` for the default list of include/exclude criteria. |
| 326 |
|
| 327 |
The class variables of `DefaultProjectFileSearcher` can be edited **before** calling `install()` to make simple |
| 328 |
customizations. |