Debian Codegraph @

Tools/wasm/README.md

raw
1 # Python WebAssembly (WASM) build
2
3 **WASI support is [tier 2](https://peps.python.org/pep-0011/#tier-2).**
4 **Emscripten is NOT officially supported as of Python 3.13.**
5
6 This directory contains configuration and helpers to facilitate cross
7 compilation of CPython to WebAssembly (WASM). Python supports Emscripten
8 (*wasm32-emscripten*) and WASI (*wasm32-wasi*) targets. Emscripten builds
9 run in modern browsers and JavaScript runtimes like *Node.js*. WASI builds
10 use WASM runtimes such as *wasmtime*.
11
12 Users and developers are encouraged to use the script
13 `Tools/wasm/wasm_build.py`. The tool automates the build process and provides
14 assistance with installation of SDKs, running tests, etc.
15
16 **NOTE**: If you are looking for information that is not directly related to
17 building CPython for WebAssembly (or the resulting build), please see
18 https://github.com/psf/webassembly for more information.
19
20 ## wasm32-emscripten
21
22 ### Build
23
24 For now the build system has two target flavors. The ``Emscripten/browser``
25 target (``--with-emscripten-target=browser``) is optimized for browsers.
26 It comes with a reduced and preloaded stdlib without tests and threading
27 support. The ``Emscripten/node`` target has threading enabled and can
28 access the file system directly.
29
30 Cross compiling to the wasm32-emscripten platform needs the
31 [Emscripten](https://emscripten.org/) SDK and a build Python interpreter.
32 Emscripten 3.1.19 or newer are recommended. All commands below are relative
33 to a repository checkout.
34
35 #### Toolchain
36
37 ##### Container image
38
39 Christian Heimes maintains a container image with Emscripten SDK, Python
40 build dependencies, WASI-SDK, wasmtime, and several additional tools.
41
42 From within your local CPython repo clone, run one of the following commands:
43
44 ```
45 # Fedora, RHEL, CentOS
46 podman run --rm -ti -v $(pwd):/python-wasm/cpython:Z -w /python-wasm/cpython quay.io/tiran/cpythonbuild:emsdk3
47
48 # other
49 docker run --rm -ti -v $(pwd):/python-wasm/cpython -w /python-wasm/cpython quay.io/tiran/cpythonbuild:emsdk3
50 ```
51
52 ##### Manually
53
54 ###### Install [Emscripten SDK](https://emscripten.org/docs/getting_started/downloads.html)
55
56 **NOTE**: Follow the on-screen instructions how to add the SDK to ``PATH``.
57
58 ```shell
59 git clone https://github.com/emscripten-core/emsdk.git /opt/emsdk
60 /opt/emsdk/emsdk install latest
61 /opt/emsdk/emsdk activate latest
62 ```
63
64 ###### Optionally: enable ccache for EMSDK
65
66 The ``EM_COMPILER_WRAPPER`` must be set after the EMSDK environment is
67 sourced. Otherwise the source script removes the environment variable.
68
69 ```
70 . /opt/emsdk/emsdk_env.sh
71 EM_COMPILER_WRAPPER=ccache
72 ```
73
74 ###### Optionally: pre-build and cache static libraries
75
76 Emscripten SDK provides static builds of core libraries without PIC
77 (position-independent code). Python builds with ``dlopen`` support require
78 PIC. To populate the build cache, run:
79
80 ```shell
81 . /opt/emsdk/emsdk_env.sh
82 embuilder build zlib bzip2 MINIMAL_PIC
83 embuilder --pic build zlib bzip2 MINIMAL_PIC
84 ```
85
86
87 ### Compile and build Python interpreter
88
89 From within the container, run the following command:
90
91 ```shell
92 ./Tools/wasm/wasm_build.py build
93 ```
94
95 The command is roughly equivalent to:
96
97 ```shell
98 mkdir -p builddir/build
99 pushd builddir/build
100 ../../configure -C
101 make -j$(nproc)
102 popd
103 ```
104
105 #### Cross-compile to wasm32-emscripten for browser
106
107 ```shell
108 ./Tools/wasm/wasm_build.py emscripten-browser
109 ```
110
111 The command is roughly equivalent to:
112
113 ```shell
114 mkdir -p builddir/emscripten-browser
115 pushd builddir/emscripten-browser
116
117 CONFIG_SITE=../../Tools/wasm/config.site-wasm32-emscripten \
118 emconfigure ../../configure -C \
119 --host=wasm32-unknown-emscripten \
120 --build=$(../../config.guess) \
121 --with-emscripten-target=browser \
122 --with-build-python=$(pwd)/../build/python
123
124 emmake make -j$(nproc)
125 popd
126 ```
127
128 Serve `python.html` with a local webserver and open the file in a browser.
129 Python comes with a minimal web server script that sets necessary HTTP
130 headers like COOP, COEP, and mimetypes. Run the script outside the container
131 and from the root of the CPython checkout.
132
133 ```shell
134 ./Tools/wasm/wasm_webserver.py
135 ```
136
137 and open http://localhost:8000/builddir/emscripten-browser/python.html . This
138 directory structure enables the *C/C++ DevTools Support (DWARF)* to load C
139 and header files with debug builds.
140
141
142 #### Cross compile to wasm32-emscripten for node
143
144 ```shell
145 ./Tools/wasm/wasm_build.py emscripten-node-dl
146 ```
147
148 The command is roughly equivalent to:
149
150 ```shell
151 mkdir -p builddir/emscripten-node-dl
152 pushd builddir/emscripten-node-dl
153
154 CONFIG_SITE=../../Tools/wasm/config.site-wasm32-emscripten \
155 emconfigure ../../configure -C \
156 --host=wasm32-unknown-emscripten \
157 --build=$(../../config.guess) \
158 --with-emscripten-target=node \
159 --enable-wasm-dynamic-linking \
160 --with-build-python=$(pwd)/../build/python
161
162 emmake make -j$(nproc)
163 popd
164 ```
165
166 ```shell
167 node --experimental-wasm-threads --experimental-wasm-bulk-memory --experimental-wasm-bigint builddir/emscripten-node-dl/python.js
168 ```
169
170 (``--experimental-wasm-bigint`` is not needed with recent NodeJS versions)
171
172 ### Limitations and issues
173
174 Emscripten before 3.1.8 has known bugs that can cause memory corruption and
175 resource leaks. 3.1.8 contains several fixes for bugs in date and time
176 functions.
177
178 #### Network stack
179
180 - Python's socket module does not work with Emscripten's emulated POSIX
181 sockets yet. Network modules like ``asyncio``, ``urllib``, ``selectors``,
182 etc. are not available.
183 - Only ``AF_INET`` and ``AF_INET6`` with ``SOCK_STREAM`` (TCP) or
184 ``SOCK_DGRAM`` (UDP) are available. ``AF_UNIX`` is not supported.
185 - ``socketpair`` does not work.
186 - Blocking sockets are not available and non-blocking sockets don't work
187 correctly, e.g. ``socket.accept`` crashes the runtime. ``gethostbyname``
188 does not resolve to a real IP address. IPv6 is not available.
189 - The ``select`` module is limited. ``select.select()`` crashes the runtime
190 due to lack of exectfd support.
191
192 #### processes, signals
193
194 - Processes are not supported. System calls like fork, popen, and subprocess
195 fail with ``ENOSYS`` or ``ENOSUP``.
196 - Signal support is limited. ``signal.alarm``, ``itimer``, ``sigaction``
197 are not available or do not work correctly. ``SIGTERM`` exits the runtime.
198 - Keyboard interrupt (CTRL+C) handling is not implemented yet.
199 - Resource-related functions like ``os.nice`` and most functions of the
200 ``resource`` module are not available.
201
202 #### threading
203
204 - Threading is disabled by default. The ``configure`` option
205 ``--enable-wasm-pthreads`` adds compiler flag ``-pthread`` and
206 linker flags ``-sUSE_PTHREADS -sPROXY_TO_PTHREAD``.
207 - pthread support requires WASM threads and SharedArrayBuffer (bulk memory).
208 The Node.JS runtime keeps a pool of web workers around. Each web worker
209 uses several file descriptors (eventfd, epoll, pipe).
210 - It's not advised to enable threading when building for browsers or with
211 dynamic linking support; there are performance and stability issues.
212
213 #### file system
214
215 - Most user, group, and permission related function and modules are not
216 supported or don't work as expected, e.g.``pwd`` module, ``grp`` module,
217 ``os.setgroups``, ``os.chown``, and so on. ``lchown`` and ``lchmod`` are
218 not available.
219 - ``umask`` is a no-op.
220 - hard links (``os.link``) are not supported.
221 - Offset and iovec I/O functions (e.g. ``os.pread``, ``os.preadv``) are not
222 available.
223 - ``os.mknod`` and ``os.mkfifo``
224 [don't work](https://github.com/emscripten-core/emscripten/issues/16158)
225 and are disabled.
226 - Large file support crashes the runtime and is disabled.
227 - ``mmap`` module is unstable. flush (``msync``) can crash the runtime.
228
229 #### Misc
230
231 - Heap memory and stack size are limited. Recursion or extensive memory
232 consumption can crash Python.
233 - Most stdlib modules with a dependency on external libraries are missing,
234 e.g. ``ctypes``, ``readline``, ``ssl``, and more.
235 - Shared extension modules are not implemented yet. All extension modules
236 are statically linked into the main binary. The experimental configure
237 option ``--enable-wasm-dynamic-linking`` enables dynamic extension
238 support. It's currently known to crash in combination with threading.
239 - glibc extensions for date and time formatting are not available.
240 - ``locales`` module is affected by musl libc issues,
241 [gh-90548](https://github.com/python/cpython/issues/90548).
242 - Python's object allocator ``obmalloc`` is disabled by default.
243 - ``ensurepip`` is not available.
244 - Some ``ctypes`` features like ``c_longlong`` and ``c_longdouble`` may need
245 NodeJS option ``--experimental-wasm-bigint``.
246
247 #### In the browser
248
249 - The interactive shell does not handle copy 'n paste and unicode support
250 well.
251 - The bundled stdlib is limited. Network-related modules,
252 multiprocessing, dbm, tests and similar modules
253 are not shipped. All other modules are bundled as pre-compiled
254 ``pyc`` files.
255 - In-memory file system (MEMFS) is not persistent and limited.
256 - Test modules are disabled by default. Use ``--enable-test-modules`` build
257 test modules like ``_testcapi``.
258
259 ### wasm32-emscripten in node
260
261 Node builds use ``NODERAWFS``.
262
263 - Node RawFS allows direct access to the host file system without need to
264 perform ``FS.mount()`` call.
265
266 ### wasm64-emscripten
267
268 - wasm64 requires recent NodeJS and ``--experimental-wasm-memory64``.
269 - ``EM_JS`` functions must return ``BigInt()``.
270 - ``Py_BuildValue()`` format strings must match size of types. Confusing 32
271 and 64 bits types leads to memory corruption, see
272 [gh-95876](https://github.com/python/cpython/issues/95876) and
273 [gh-95878](https://github.com/python/cpython/issues/95878).
274
275 ### Hosting Python WASM builds
276
277 The simple REPL terminal uses SharedArrayBuffer. For security reasons
278 browsers only provide the feature in secure environments with cross-origin
279 isolation. The webserver must send cross-origin headers and correct MIME types
280 for the JavaScript and WASM files. Otherwise the terminal will fail to load
281 with an error message like ``Browsers disable shared array buffer``.
282
283 #### Apache HTTP .htaccess
284
285 Place a ``.htaccess`` file in the same directory as ``python.wasm``.
286
287 ```
288 # .htaccess
289 Header set Cross-Origin-Opener-Policy same-origin
290 Header set Cross-Origin-Embedder-Policy require-corp
291
292 AddType application/javascript js
293 AddType application/wasm wasm
294
295 <IfModule mod_deflate.c>
296 AddOutputFilterByType DEFLATE text/html application/javascript application/wasm
297 </IfModule>
298 ```
299
300 ## WASI (wasm32-wasi)
301
302 See [the devguide on how to build and run for WASI](https://devguide.python.org/getting-started/setup-building/#wasi).
303
304 ## Detecting WebAssembly builds
305
306 ### Python code
307
308 ```python
309 import os, sys
310
311 if sys.platform == "emscripten":
312 # Python on Emscripten
313 ...
314 if sys.platform == "wasi":
315 # Python on WASI
316 ...
317
318 if os.name == "posix":
319 # WASM platforms identify as POSIX-like.
320 # Windows does not provide os.uname().
321 machine = os.uname().machine
322 if machine.startswith("wasm"):
323 # WebAssembly (wasm32, wasm64 potentially in the future)
324 ```
325
326 ```python
327 >>> import os, sys
328 >>> os.uname()
329 posix.uname_result(
330 sysname='Emscripten',
331 nodename='emscripten',
332 release='3.1.19',
333 version='#1',
334 machine='wasm32'
335 )
336 >>> os.name
337 'posix'
338 >>> sys.platform
339 'emscripten'
340 >>> sys._emscripten_info
341 sys._emscripten_info(
342 emscripten_version=(3, 1, 10),
343 runtime='Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0',
344 pthreads=False,
345 shared_memory=False
346 )
347 ```
348
349 ```python
350 >>> sys._emscripten_info
351 sys._emscripten_info(
352 emscripten_version=(3, 1, 19),
353 runtime='Node.js v14.18.2',
354 pthreads=True,
355 shared_memory=True
356 )
357 ```
358
359 ```python
360 >>> import os, sys
361 >>> os.uname()
362 posix.uname_result(
363 sysname='wasi',
364 nodename='(none)',
365 release='0.0.0',
366 version='0.0.0',
367 machine='wasm32'
368 )
369 >>> os.name
370 'posix'
371 >>> sys.platform
372 'wasi'
373 ```
374
375 ### C code
376
377 Emscripten SDK and WASI SDK define several built-in macros. You can dump a
378 full list of built-ins with ``emcc -dM -E - < /dev/null`` and
379 ``/path/to/wasi-sdk/bin/clang -dM -E - < /dev/null``.
380
381 * WebAssembly ``__wasm__`` (also ``__wasm``)
382 * wasm32 ``__wasm32__`` (also ``__wasm32``)
383 * wasm64 ``__wasm64__``
384 * Emscripten ``__EMSCRIPTEN__`` (also ``EMSCRIPTEN``)
385 * Emscripten version ``__EMSCRIPTEN_major__``, ``__EMSCRIPTEN_minor__``, ``__EMSCRIPTEN_tiny__``
386 * WASI ``__wasi__``