| 1 |
{ |
| 2 |
"cells": [ |
| 3 |
{ |
| 4 |
"cell_type": "markdown", |
| 5 |
"metadata": {}, |
| 6 |
"source": [ |
| 7 |
"# Compilation\n", |
| 8 |
"\n", |
| 9 |
"Various libraries provide tools for tracing numeric functions and turning the resulting computation into a more efficient, compiled function. Notably:\n", |
| 10 |
"\n", |
| 11 |
"* [``jax.jit``](https://github.com/google/jax)\n", |
| 12 |
"* [``tensorflow.function``](https://www.tensorflow.org/api_docs/python/tf/function)\n", |
| 13 |
"* [``torch.jit.trace``](https://pytorch.org/docs/stable/jit.html)\n", |
| 14 |
"\n", |
| 15 |
" ``autoray`` is obviously very well suited to these since it just dispatches functions to whichever library is doing the tracing - functions written using autoray should be immediately compatible with all of them.\n", |
| 16 |
"\n", |
| 17 |
"**The `autojit` wrapper**\n", |
| 18 |
"\n", |
| 19 |
"Moreover, ``autoray`` also provides a *unified interface* for compiling functions so that the compilation backend can be easily switched or automatically identified:\n", |
| 20 |
"\n", |
| 21 |
"```python\n", |
| 22 |
"from autoray import autojit\n", |
| 23 |
"\n", |
| 24 |
"mgs = autojit(modified_gram_schmidt)\n", |
| 25 |
"```\n", |
| 26 |
"\n", |
| 27 |
"Currently ``autojit`` supports functions with the signature ``fn(*args, **kwargs) -> array`` where both ``args`` and ``kwargs`` can be any nested combination of ``tuple``, ``list`` and ``dict`` objects containings arrays.\n", |
| 28 |
"We can compare different compiled versions of this simply by changing the ``backend`` option:\n", |
| 29 |
"\n", |
| 30 |
"```python\n", |
| 31 |
"x = do(\"random.normal\", size=(50, 50), like='numpy')\n", |
| 32 |
"\n", |
| 33 |
"# first the uncompiled version\n", |
| 34 |
"%%timeit\n", |
| 35 |
"modified_gram_schmidt(x)\n", |
| 36 |
"# 23.5 ms ± 241 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n", |
| 37 |
"\n", |
| 38 |
"# 'python' mode unravels computation into source then uses compile+exec\n", |
| 39 |
"%%timeit\n", |
| 40 |
"mgs(x) # backend='python'\n", |
| 41 |
"# 17.8 ms ± 191 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n", |
| 42 |
"\n", |
| 43 |
"%%timeit\n", |
| 44 |
"mgs(x, backend='torch')\n", |
| 45 |
"# 11.9 ms ± 80.5 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)\n", |
| 46 |
"\n", |
| 47 |
"%%timeit\n", |
| 48 |
"mgs(x, backend='tensorflow')\n", |
| 49 |
"# 1.87 ms ± 441 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)\n", |
| 50 |
"\n", |
| 51 |
"# need to config jax to run on same footing\n", |
| 52 |
"from jax.config import config\n", |
| 53 |
"config.update(\"jax_enable_x64\", True)\n", |
| 54 |
"config.update('jax_platform_name', 'cpu')\n", |
| 55 |
"\n", |
| 56 |
"%%timeit\n", |
| 57 |
"mgs(x, backend='jax')\n", |
| 58 |
"# 226 µs ± 14.8 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)\n", |
| 59 |
"\n", |
| 60 |
"%%timeit\n", |
| 61 |
"do('linalg.qr', x, like='numpy')[0] # appriximately the 'C' version\n", |
| 62 |
"# 156 µs ± 32.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n", |
| 63 |
"```\n", |
| 64 |
"\n", |
| 65 |
"Here you see *(with this very for-loop heavy function)*, that there are significant gains to be made for all the compilations options. Whilst ``jax`` for example achieves fantastic performance, it should be noted the compilation step takes a lot of time and scales badly (super-linearly) with the number of computational nodes." |
| 66 |
] |
| 67 |
} |
| 68 |
], |
| 69 |
"metadata": { |
| 70 |
"kernelspec": { |
| 71 |
"display_name": "numpy", |
| 72 |
"language": "python", |
| 73 |
"name": "python3" |
| 74 |
}, |
| 75 |
"language_info": { |
| 76 |
"codemirror_mode": { |
| 77 |
"name": "ipython", |
| 78 |
"version": 3 |
| 79 |
}, |
| 80 |
"file_extension": ".py", |
| 81 |
"mimetype": "text/x-python", |
| 82 |
"name": "python", |
| 83 |
"nbconvert_exporter": "python", |
| 84 |
"pygments_lexer": "ipython3", |
| 85 |
"version": "3.10.9" |
| 86 |
}, |
| 87 |
"orig_nbformat": 4 |
| 88 |
}, |
| 89 |
"nbformat": 4, |
| 90 |
"nbformat_minor": 2 |
| 91 |
} |