Debian Codegraph @

Platforms/emscripten/README.md

raw
1 # Python WebAssembly (WASM) build
2
3 **WASI support is [tier 2](https://peps.python.org/pep-0011/#tier-2).**
4 **Emscripten support is [tier 3](https://peps.python.org/pep-0011/#tier-3).**
5
6 This directory contains configuration and helpers to facilitate cross
7 compilation of CPython to WebAssembly (WASM). Python supports Emscripten
8 (*wasm32-emscripten*) and WASI (*wasm32-wasi*) targets. Emscripten builds
9 run in modern browsers and JavaScript runtimes like *Node.js*. WASI builds
10 use WASM runtimes such as *wasmtime*.
11
12 Users and developers are encouraged to use the script
13 `Tools/wasm/wasm_build.py`. The tool automates the build process and provides
14 assistance with installation of SDKs, running tests, etc.
15
16 **NOTE**: If you are looking for information that is not directly related to
17 building CPython for WebAssembly (or the resulting build), please see
18 https://github.com/psf/webassembly for more information.
19
20 ## wasm32-emscripten
21
22 ### Build
23
24 To cross compile to the ``wasm32-emscripten`` platform you need
25 [the Emscripten compiler toolchain](https://emscripten.org/),
26 a Python interpreter, and an installation of Node version 18 or newer.
27 Emscripten version 4.0.2 is recommended; newer versions may also work, but all
28 official testing is performed with that version. All commands below are relative
29 to a checkout of the Python repository.
30
31 #### Install [the Emscripten compiler toolchain](https://emscripten.org/docs/getting_started/downloads.html)
32
33 You can install the Emscripten toolchain as follows:
34 ```shell
35 git clone https://github.com/emscripten-core/emsdk.git --depth 1
36 ./emsdk/emsdk install latest
37 ./emsdk/emsdk activate latest
38 ```
39 To add the Emscripten compiler to your path:
40 ```shell
41 source ./emsdk/emsdk_env.sh
42 ```
43 This adds `emcc` and `emconfigure` to your path.
44
45 ##### Optionally: enable ccache for EMSDK
46
47 The ``EM_COMPILER_WRAPPER`` must be set after the EMSDK environment is
48 sourced. Otherwise the source script removes the environment variable.
49
50 ```shell
51 export EM_COMPILER_WRAPPER=ccache
52 ```
53
54 #### Compile and build Python interpreter
55
56 You can use `python Tools/wasm/emscripten` to compile and build targeting
57 Emscripten. You can do everything at once with:
58 ```shell
59 python Tools/wasm/emscripten build
60 ```
61 or you can break it out into four separate steps:
62 ```shell
63 python Tools/wasm/emscripten configure-build-python
64 python Tools/wasm/emscripten make-build-python
65 python Tools/wasm/emscripten make-libffi
66 python Tools/wasm/emscripten configure-host
67 python Tools/wasm/emscripten make-host
68 ```
69 Extra arguments to the configure steps are passed along to configure. For
70 instance, to do a debug build, you can use:
71 ```shell
72 python Tools/wasm/emscripten build --with-py-debug
73 ```
74
75 ### Running from node
76
77 If you want to run the normal Python CLI, you can use `python.sh`. It takes the
78 same options as the normal Python CLI entrypoint, though the REPL does not
79 function and will crash.
80
81 `python.sh` invokes `node_entry.mjs` which imports the Emscripten module for the
82 Python process and starts it up with the appropriate settings. If you wish to
83 make a node application that "embeds" the interpreter instead of acting like the
84 CLI you will need to write your own alternative to `node_entry.mjs`.
85
86
87 ### Running tests
88
89 After building, you can run the full test suite with:
90 ```shell
91 ./cross-build/wasm32-emscripten/build/python/python.sh -m test -uall
92 ```
93 You can run the browser smoke test with:
94 ```shell
95 ./Platforms/emscripten/browser_test/run_test.sh
96 ```
97
98 ### The Web Example
99
100 When building for Emscripten, the web example will be built automatically. It
101 is in the ``web_example`` directory. To run the web example, ``cd`` into the
102 ``web_example`` directory, then run ``python server.py``. This will start a web
103 server; you can then visit ``http://localhost:8000/`` in a browser to see a
104 simple REPL example.
105
106 The web example relies on a bug fix in Emscripten version 3.1.73 so if you build
107 with earlier versions of Emscripten it may not work. The web example uses
108 ``SharedArrayBuffer``. For security reasons browsers only provide
109 ``SharedArrayBuffer`` in secure environments with cross-origin isolation. The
110 webserver must send cross-origin headers and correct MIME types for the
111 JavaScript and WebAssembly files. Otherwise the terminal will fail to load with
112 an error message like ``ReferenceError: SharedArrayBuffer is not defined``. See
113 more information here:
114 https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer#security_requirements
115
116 Note that ``SharedArrayBuffer`` is _not required_ to use Python itself, only the
117 web example. If cross-origin isolation is not appropriate for your use case you
118 may make your own application embedding `python.mjs` which does not use
119 ``SharedArrayBuffer`` and serve it without the cross-origin isolation headers.
120
121 ### Embedding Python in a custom JavaScript application
122
123 You can look at `python.worker.mjs` and `node_entry.mjs` for inspiration. At a
124 minimum you must import ``createEmscriptenModule`` and you need to call
125 ``createEmscriptenModule`` with an appropriate settings object. This settings
126 object will need a prerun hook that installs the Python standard library into
127 the Emscripten file system.
128
129 #### NodeJs
130
131 In Node, you can use the NodeFS to mount the standard library in your native
132 file system into the Emscripten file system:
133 ```js
134 import createEmscriptenModule from "./python.mjs";
135
136 await createEmscriptenModule({
137 preRun(Module) {
138 Module.FS.mount(
139 Module.FS.filesystems.NODEFS,
140 { root: "/path/to/python/stdlib" },
141 "/lib/",
142 );
143 },
144 });
145 ```
146
147 #### Browser
148
149 In the browser, the simplest approach is to put the standard library in a zip
150 file it and install it. With Python 3.14 this could look like:
151 ```js
152 import createEmscriptenModule from "./python.mjs";
153
154 await createEmscriptenModule({
155 async preRun(Module) {
156 Module.FS.mkdirTree("/lib/python3.14/lib-dynload/");
157 Module.addRunDependency("install-stdlib");
158 const resp = await fetch("python3.14.zip");
159 const stdlibBuffer = await resp.arrayBuffer();
160 Module.FS.writeFile(`/lib/python314.zip`, new Uint8Array(stdlibBuffer), {
161 canOwn: true,
162 });
163 Module.removeRunDependency("install-stdlib");
164 },
165 });
166 ```
167
168 ### Limitations and issues
169
170 #### Network stack
171
172 - Python's socket module does not work with Emscripten's emulated POSIX
173 sockets yet. Network modules like ``asyncio``, ``urllib``, ``selectors``,
174 etc. are not available.
175 - Only ``AF_INET`` and ``AF_INET6`` with ``SOCK_STREAM`` (TCP) or
176 ``SOCK_DGRAM`` (UDP) are available. ``AF_UNIX`` is not supported.
177 - ``socketpair`` does not work.
178 - Blocking sockets are not available and non-blocking sockets don't work
179 correctly, e.g. ``socket.accept`` crashes the runtime. ``gethostbyname``
180 does not resolve to a real IP address. IPv6 is not available.
181 - The ``select`` module is limited. ``select.select()`` crashes the runtime
182 due to lack of exectfd support.
183
184 #### processes, signals
185
186 - Processes are not supported. System calls like fork, popen, and subprocess
187 fail with ``ENOSYS`` or ``ENOSUP``.
188 - Signal support is limited. ``signal.alarm``, ``itimer``, ``sigaction``
189 are not available or do not work correctly. ``SIGTERM`` exits the runtime.
190 - Keyboard interrupt (CTRL+C) handling is not implemented yet.
191 - Resource-related functions like ``os.nice`` and most functions of the
192 ``resource`` module are not available.
193
194 #### threading
195
196 - Threading is disabled by default. The ``configure`` option
197 ``--enable-wasm-pthreads`` adds compiler flag ``-pthread`` and
198 linker flags ``-sUSE_PTHREADS -sPROXY_TO_PTHREAD``.
199 - pthread support requires WASM threads and SharedArrayBuffer (bulk memory).
200 The Node.JS runtime keeps a pool of web workers around. Each web worker
201 uses several file descriptors (eventfd, epoll, pipe).
202 - It's not advised to enable threading when building for browsers or with
203 dynamic linking support; there are performance and stability issues.
204
205 #### file system
206
207 - Most user, group, and permission related function and modules are not
208 supported or don't work as expected, e.g.``pwd`` module, ``grp`` module,
209 ``os.setgroups``, ``os.chown``, and so on. ``lchown`` and ``lchmod`` are
210 not available.
211 - ``umask`` is a no-op.
212 - hard links (``os.link``) are not supported.
213 - Offset and iovec I/O functions (e.g. ``os.pread``, ``os.preadv``) are not
214 available.
215 - ``os.mknod`` and ``os.mkfifo``
216 [don't work](https://github.com/emscripten-core/emscripten/issues/16158)
217 and are disabled.
218 - Large file support crashes the runtime and is disabled.
219 - ``mmap`` module is unstable. flush (``msync``) can crash the runtime.
220
221 #### Misc
222
223 - Heap memory and stack size are limited. Recursion or extensive memory
224 consumption can crash Python.
225 - Most stdlib modules with a dependency on external libraries are missing,
226 e.g. ``ctypes``, ``readline``, ``ssl``, and more.
227 - Shared extension modules are not implemented yet. All extension modules
228 are statically linked into the main binary. The experimental configure
229 option ``--enable-wasm-dynamic-linking`` enables dynamic extension
230 support. It's currently known to crash in combination with threading.
231 - glibc extensions for date and time formatting are not available.
232 - ``locales`` module is affected by musl libc issues,
233 [gh-90548](https://github.com/python/cpython/issues/90548).
234 - Python's object allocator ``obmalloc`` is disabled by default.
235 - ``ensurepip`` is not available.
236
237 #### In the browser
238
239 - The interactive shell does not handle copy 'n paste and unicode support
240 well.
241 - The bundled stdlib is limited. Network-related modules,
242 multiprocessing, dbm, tests and similar modules
243 are not shipped. All other modules are bundled as pre-compiled
244 ``pyc`` files.
245 - In-memory file system (MEMFS) is not persistent and limited.
246 - Test modules are built by default. Use ``--disable-test-modules`` to disable
247 building test modules like ``_testcapi``.
248
249 ## WASI (wasm32-wasi)
250
251 See [the devguide on how to build and run for WASI](https://devguide.python.org/getting-started/setup-building/#wasi).
252
253 ## Detecting WebAssembly builds
254
255 ### Python code
256
257 ```python
258 import os, sys
259
260 if sys.platform == "emscripten":
261 # Python on Emscripten
262 ...
263 if sys.platform == "wasi":
264 # Python on WASI
265 ...
266
267 if os.name == "posix":
268 # WASM platforms identify as POSIX-like.
269 # Windows does not provide os.uname().
270 machine = os.uname().machine
271 if machine.startswith("wasm"):
272 # WebAssembly (wasm32, wasm64 potentially in the future)
273 ```
274
275 ```python
276 >>> import os, sys
277 >>> os.uname()
278 posix.uname_result(
279 sysname='Emscripten',
280 nodename='emscripten',
281 release='4.0.2',
282 version='#1',
283 machine='wasm32'
284 )
285 >>> os.name
286 'posix'
287 >>> sys.platform
288 'emscripten'
289 >>> sys._emscripten_info
290 sys._emscripten_info(
291 emscripten_version=(3, 1, 10),
292 runtime='Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0',
293 pthreads=False,
294 shared_memory=False
295 )
296 ```
297
298 ```python
299 >>> sys._emscripten_info
300 sys._emscripten_info(
301 emscripten_version=(3, 1, 19),
302 runtime='Node.js v14.18.2',
303 pthreads=True,
304 shared_memory=True
305 )
306 ```
307
308 ```python
309 >>> import os, sys
310 >>> os.uname()
311 posix.uname_result(
312 sysname='wasi',
313 nodename='(none)',
314 release='0.0.0',
315 version='0.0.0',
316 machine='wasm32'
317 )
318 >>> os.name
319 'posix'
320 >>> sys.platform
321 'wasi'
322 ```
323
324 ### C code
325
326 Emscripten SDK and WASI SDK define several built-in macros. You can dump a
327 full list of built-ins with ``emcc -dM -E - < /dev/null`` and
328 ``/path/to/wasi-sdk/bin/clang -dM -E - < /dev/null``.
329
330 * WebAssembly ``__wasm__`` (also ``__wasm``)
331 * wasm32 ``__wasm32__`` (also ``__wasm32``)
332 * wasm64 ``__wasm64__``
333 * Emscripten ``__EMSCRIPTEN__`` (also ``EMSCRIPTEN``)
334 * Emscripten version ``__EMSCRIPTEN_major__``, ``__EMSCRIPTEN_minor__``, ``__EMSCRIPTEN_tiny__``
335 * WASI ``__wasi__``