| 1 |
==================== |
| 2 |
Python on iOS README |
| 3 |
==================== |
| 4 |
|
| 5 |
:Authors: |
| 6 |
Russell Keith-Magee (2023-11) |
| 7 |
|
| 8 |
This document provides a quick overview of some iOS specific features in the |
| 9 |
Python distribution. |
| 10 |
|
| 11 |
These instructions are only needed if you're planning to compile Python for iOS |
| 12 |
yourself. Most users should *not* need to do this. If you're looking to |
| 13 |
experiment with writing an iOS app in Python, tools such as `BeeWare's Briefcase |
| 14 |
<https://briefcase.readthedocs.io>`__ and `Kivy's Buildozer |
| 15 |
<https://buildozer.readthedocs.io>`__ will provide a much more approachable |
| 16 |
user experience. |
| 17 |
|
| 18 |
Compilers for building on iOS |
| 19 |
============================= |
| 20 |
|
| 21 |
Building for iOS requires the use of Apple's Xcode tooling. It is strongly |
| 22 |
recommended that you use the most recent stable release of Xcode. This will |
| 23 |
require the use of the most (or second-most) recently released macOS version, |
| 24 |
as Apple does not maintain Xcode for older macOS versions. The Xcode Command |
| 25 |
Line Tools are not sufficient for iOS development; you need a *full* Xcode |
| 26 |
install. |
| 27 |
|
| 28 |
If you want to run your code on the iOS simulator, you'll also need to install |
| 29 |
an iOS Simulator Platform. You should be prompted to select an iOS Simulator |
| 30 |
Platform when you first run Xcode. Alternatively, you can add an iOS Simulator |
| 31 |
Platform by selecting an open the Platforms tab of the Xcode Settings panel. |
| 32 |
|
| 33 |
iOS specific arguments to configure |
| 34 |
=================================== |
| 35 |
|
| 36 |
* ``--enable-framework[=DIR]`` |
| 37 |
|
| 38 |
This argument specifies the location where the Python.framework will be |
| 39 |
installed. If ``DIR`` is not specified, the framework will be installed into |
| 40 |
a subdirectory of the ``iOS/Frameworks`` folder. |
| 41 |
|
| 42 |
This argument *must* be provided when configuring iOS builds. iOS does not |
| 43 |
support non-framework builds. |
| 44 |
|
| 45 |
* ``--with-framework-name=NAME`` |
| 46 |
|
| 47 |
Specify the name for the Python framework; defaults to ``Python``. |
| 48 |
|
| 49 |
.. admonition:: Use this option with care! |
| 50 |
|
| 51 |
Unless you know what you're doing, changing the name of the Python |
| 52 |
framework on iOS is not advised. If you use this option, you won't be able |
| 53 |
to run the ``make testios`` target without making significant manual |
| 54 |
alterations, and you won't be able to use any binary packages unless you |
| 55 |
compile them yourself using your own framework name. |
| 56 |
|
| 57 |
Building Python on iOS |
| 58 |
====================== |
| 59 |
|
| 60 |
ABIs and Architectures |
| 61 |
---------------------- |
| 62 |
|
| 63 |
iOS apps can be deployed on physical devices, and on the iOS simulator. Although |
| 64 |
the API used on these devices is identical, the ABI is different - you need to |
| 65 |
link against different libraries for an iOS device build (``iphoneos``) or an |
| 66 |
iOS simulator build (``iphonesimulator``). |
| 67 |
|
| 68 |
Apple uses the ``XCframework`` format to allow specifying a single dependency |
| 69 |
that supports multiple ABIs. An ``XCframework`` is a wrapper around multiple |
| 70 |
ABI-specific frameworks that share a common API. |
| 71 |
|
| 72 |
iOS can also support different CPU architectures within each ABI. At present, |
| 73 |
there is only a single supported architecture on physical devices - ARM64. |
| 74 |
However, the *simulator* supports 2 architectures - ARM64 (for running on Apple |
| 75 |
Silicon machines), and x86_64 (for running on older Intel-based machines). |
| 76 |
|
| 77 |
To support multiple CPU architectures on a single platform, Apple uses a "fat |
| 78 |
binary" format - a single physical file that contains support for multiple |
| 79 |
architectures. It is possible to compile and use a "thin" single architecture |
| 80 |
version of a binary for testing purposes; however, the "thin" binary will not be |
| 81 |
portable to machines using other architectures. |
| 82 |
|
| 83 |
Building a single-architecture framework |
| 84 |
---------------------------------------- |
| 85 |
|
| 86 |
The Python build system will create a ``Python.framework`` that supports a |
| 87 |
*single* ABI with a *single* architecture. Unlike macOS, iOS does not allow a |
| 88 |
framework to contain non-library content, so the iOS build will produce a |
| 89 |
``bin`` and ``lib`` folder in the same output folder as ``Python.framework``. |
| 90 |
The ``lib`` folder will be needed at runtime to support the Python library. |
| 91 |
|
| 92 |
If you want to use Python in a real iOS project, you need to produce multiple |
| 93 |
``Python.framework`` builds, one for each ABI and architecture. iOS builds of |
| 94 |
Python *must* be constructed as framework builds. To support this, you must |
| 95 |
provide the ``--enable-framework`` flag when configuring the build. The build |
| 96 |
also requires the use of cross-compilation. The minimal commands for building |
| 97 |
Python for the ARM64 iOS simulator will look something like:: |
| 98 |
|
| 99 |
$ export PATH="$(pwd)/iOS/Resources/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin" |
| 100 |
$ ./configure \ |
| 101 |
--enable-framework \ |
| 102 |
--host=arm64-apple-ios-simulator \ |
| 103 |
--build=arm64-apple-darwin \ |
| 104 |
--with-build-python=/path/to/python.exe |
| 105 |
$ make |
| 106 |
$ make install |
| 107 |
|
| 108 |
In this invocation: |
| 109 |
|
| 110 |
* ``iOS/Resources/bin`` has been added to the path, providing some shims for the |
| 111 |
compilers and linkers needed by the build. Xcode requires the use of ``xcrun`` |
| 112 |
to invoke compiler tooling. However, if ``xcrun`` is pre-evaluated and the |
| 113 |
result passed to ``configure``, these results can embed user- and |
| 114 |
version-specific paths into the sysconfig data, which limits the portability |
| 115 |
of the compiled Python. Alternatively, if ``xcrun`` is used *as* the compiler, |
| 116 |
it requires that compiler variables like ``CC`` include spaces, which can |
| 117 |
cause significant problems with many C configuration systems which assume that |
| 118 |
``CC`` will be a single executable. |
| 119 |
|
| 120 |
To work around this problem, the ``iOS/Resources/bin`` folder contains some |
| 121 |
wrapper scripts that present as simple compilers and linkers, but wrap |
| 122 |
underlying calls to ``xcrun``. This allows configure to use a ``CC`` |
| 123 |
definition without spaces, and without user- or version-specific paths, while |
| 124 |
retaining the ability to adapt to the local Xcode install. These scripts are |
| 125 |
included in the ``bin`` directory of an iOS install. |
| 126 |
|
| 127 |
These scripts will, by default, use the currently active Xcode installation. |
| 128 |
If you want to use a different Xcode installation, you can use |
| 129 |
``xcode-select`` to set a new default Xcode globally, or you can use the |
| 130 |
``DEVELOPER_DIR`` environment variable to specify an Xcode install. The |
| 131 |
scripts will use the default ``iphoneos``/``iphonesimulator`` SDK version for |
| 132 |
the select Xcode install; if you want to use a different SDK, you can set the |
| 133 |
``IOS_SDK_VERSION`` environment variable. (e.g, setting |
| 134 |
``IOS_SDK_VERSION=17.1`` would cause the scripts to use the ``iphoneos17.1`` |
| 135 |
and ``iphonesimulator17.1`` SDKs, regardless of the Xcode default.) |
| 136 |
|
| 137 |
The path has also been cleared of any user customizations. A common source of |
| 138 |
bugs is for tools like Homebrew to accidentally leak macOS binaries into an iOS |
| 139 |
build. Resetting the path to a known "bare bones" value is the easiest way to |
| 140 |
avoid these problems. |
| 141 |
|
| 142 |
* ``--host`` is the architecture and ABI that you want to build, in GNU compiler |
| 143 |
triple format. This will be one of: |
| 144 |
|
| 145 |
- ``arm64-apple-ios`` for ARM64 iOS devices. |
| 146 |
- ``arm64-apple-ios-simulator`` for the iOS simulator running on Apple |
| 147 |
Silicon devices. |
| 148 |
- ``x86_64-apple-ios-simulator`` for the iOS simulator running on Intel |
| 149 |
devices. |
| 150 |
|
| 151 |
* ``--build`` is the GNU compiler triple for the machine that will be running |
| 152 |
the compiler. This is one of: |
| 153 |
|
| 154 |
- ``arm64-apple-darwin`` for Apple Silicon devices. |
| 155 |
- ``x86_64-apple-darwin`` for Intel devices. |
| 156 |
|
| 157 |
* ``/path/to/python.exe`` is the path to a Python binary on the machine that |
| 158 |
will be running the compiler. This is needed because the Python compilation |
| 159 |
process involves running some Python code. On a normal desktop build of |
| 160 |
Python, you can compile a python interpreter and then use that interpreter to |
| 161 |
run Python code. However, the binaries produced for iOS won't run on macOS, so |
| 162 |
you need to provide an external Python interpreter. This interpreter must be |
| 163 |
the same version as the Python that is being compiled. To be completely safe, |
| 164 |
this should be the *exact* same commit hash. However, the longer a Python |
| 165 |
release has been stable, the more likely it is that this constraint can be |
| 166 |
relaxed - the same micro version will often be sufficient. |
| 167 |
|
| 168 |
* The ``install`` target for iOS builds is slightly different to other |
| 169 |
platforms. On most platforms, ``make install`` will install the build into |
| 170 |
the final runtime location. This won't be the case for iOS, as the final |
| 171 |
runtime location will be on a physical device. |
| 172 |
|
| 173 |
However, you still need to run the ``install`` target for iOS builds, as it |
| 174 |
performs some final framework assembly steps. The location specified with |
| 175 |
``--enable-framework`` will be the location where ``make install`` will |
| 176 |
assemble the complete iOS framework. This completed framework can then |
| 177 |
be copied and relocated as required. |
| 178 |
|
| 179 |
For a full CPython build, you also need to specify the paths to iOS builds of |
| 180 |
the binary libraries that CPython depends on (XZ, BZip2, LibFFI and OpenSSL). |
| 181 |
This can be done by defining the ``LIBLZMA_CFLAGS``, ``LIBLZMA_LIBS``, |
| 182 |
``BZIP2_CFLAGS``, ``BZIP2_LIBS``, ``LIBFFI_CFLAGS``, and ``LIBFFI_LIBS`` |
| 183 |
environment variables, and the ``--with-openssl`` configure option. Versions of |
| 184 |
these libraries pre-compiled for iOS can be found in `this repository |
| 185 |
<https://github.com/beeware/cpython-apple-source-deps/releases>`__. LibFFI is |
| 186 |
especially important, as many parts of the standard library (including the |
| 187 |
``platform``, ``sysconfig`` and ``webbrowser`` modules) require the use of the |
| 188 |
``ctypes`` module at runtime. |
| 189 |
|
| 190 |
By default, Python will be compiled with an iOS deployment target (i.e., the |
| 191 |
minimum supported iOS version) of 13.0. To specify a different deployment |
| 192 |
target, provide the version number as part of the ``--host`` argument - for |
| 193 |
example, ``--host=arm64-apple-ios15.4-simulator`` would compile an ARM64 |
| 194 |
simulator build with a deployment target of 15.4. |
| 195 |
|
| 196 |
Merge thin frameworks into fat frameworks |
| 197 |
----------------------------------------- |
| 198 |
|
| 199 |
Once you've built a ``Python.framework`` for each ABI and architecture, you |
| 200 |
must produce a "fat" framework for each ABI that contains all the architectures |
| 201 |
for that ABI. |
| 202 |
|
| 203 |
The ``iphoneos`` build only needs to support a single architecture, so it can be |
| 204 |
used without modification. |
| 205 |
|
| 206 |
If you only want to support a single simulator architecture, (e.g., only support |
| 207 |
ARM64 simulators), you can use a single architecture ``Python.framework`` build. |
| 208 |
However, if you want to create ``Python.xcframework`` that supports *all* |
| 209 |
architectures, you'll need to merge the ``iphonesimulator`` builds for ARM64 and |
| 210 |
x86_64 into a single "fat" framework. |
| 211 |
|
| 212 |
The "fat" framework can be constructed by performing a directory merge of the |
| 213 |
content of the two "thin" ``Python.framework`` directories, plus the ``bin`` and |
| 214 |
``lib`` folders for each thin framework. When performing this merge: |
| 215 |
|
| 216 |
* The pure Python standard library content is identical for each architecture, |
| 217 |
except for a handful of platform-specific files (such as the ``sysconfig`` |
| 218 |
module). Ensure that the "fat" framework has the union of all standard library |
| 219 |
files. |
| 220 |
|
| 221 |
* Any binary files in the standard library, plus the main |
| 222 |
``libPython3.X.dylib``, can be merged using the ``lipo`` tool, provide by |
| 223 |
Xcode:: |
| 224 |
|
| 225 |
$ lipo -create -output module.dylib path/to/x86_64/module.dylib path/to/arm64/module.dylib |
| 226 |
|
| 227 |
* The header files will be identical on both architectures, except for |
| 228 |
``pyconfig.h``. Copy all the headers from one platform (say, arm64), rename |
| 229 |
``pyconfig.h`` to ``pyconfig-arm64.h``, and copy the ``pyconfig.h`` for the |
| 230 |
other architecture into the merged header folder as ``pyconfig-x86_64.h``. |
| 231 |
Then copy the ``iOS/Resources/pyconfig.h`` file from the CPython sources into |
| 232 |
the merged headers folder. This will allow the two Python architectures to |
| 233 |
share a common ``pyconfig.h`` header file. |
| 234 |
|
| 235 |
At this point, you should have 2 Python.framework folders - one for ``iphoneos``, |
| 236 |
and one for ``iphonesimulator`` that is a merge of x86+64 and ARM64 content. |
| 237 |
|
| 238 |
Merge frameworks into an XCframework |
| 239 |
------------------------------------ |
| 240 |
|
| 241 |
Now that we have 2 (potentially fat) ABI-specific frameworks, we can merge those |
| 242 |
frameworks into a single ``XCframework``. |
| 243 |
|
| 244 |
The initial skeleton of an ``XCframework`` is built using:: |
| 245 |
|
| 246 |
xcodebuild -create-xcframework -output Python.xcframework -framework path/to/iphoneos/Python.framework -framework path/to/iphonesimulator/Python.framework |
| 247 |
|
| 248 |
Then, copy the ``bin`` and ``lib`` folders into the architecture-specific slices of |
| 249 |
the XCframework:: |
| 250 |
|
| 251 |
cp path/to/iphoneos/bin Python.xcframework/ios-arm64 |
| 252 |
cp path/to/iphoneos/lib Python.xcframework/ios-arm64 |
| 253 |
|
| 254 |
cp path/to/iphonesimulator/bin Python.xcframework/ios-arm64_x86_64-simulator |
| 255 |
cp path/to/iphonesimulator/lib Python.xcframework/ios-arm64_x86_64-simulator |
| 256 |
|
| 257 |
Note that the name of the architecture-specific slice for the simulator will |
| 258 |
depend on the CPU architecture(s) that you build. |
| 259 |
|
| 260 |
You now have a Python.xcframework that can be used in a project. |
| 261 |
|
| 262 |
Testing Python on iOS |
| 263 |
===================== |
| 264 |
|
| 265 |
The ``iOS/testbed`` folder that contains an Xcode project that is able to run |
| 266 |
the iOS test suite. This project converts the Python test suite into a single |
| 267 |
test case in Xcode's XCTest framework. The single XCTest passes if the test |
| 268 |
suite passes. |
| 269 |
|
| 270 |
To run the test suite, configure a Python build for an iOS simulator (i.e., |
| 271 |
``--host=arm64-apple-ios-simulator`` or ``--host=x86_64-apple-ios-simulator`` |
| 272 |
), specifying a framework build (i.e. ``--enable-framework``). Ensure that your |
| 273 |
``PATH`` has been configured to include the ``iOS/Resources/bin`` folder and |
| 274 |
exclude any non-iOS tools, then run:: |
| 275 |
|
| 276 |
$ make all |
| 277 |
$ make install |
| 278 |
$ make testios |
| 279 |
|
| 280 |
This will: |
| 281 |
|
| 282 |
* Build an iOS framework for your chosen architecture; |
| 283 |
* Finalize the single-platform framework; |
| 284 |
* Make a clean copy of the testbed project; |
| 285 |
* Install the Python iOS framework into the copy of the testbed project; and |
| 286 |
* Run the test suite on an "iPhone SE (3rd generation)" simulator. |
| 287 |
|
| 288 |
On success, the test suite will exit and report successful completion of the |
| 289 |
test suite. On a 2022 M1 MacBook Pro, the test suite takes approximately 15 |
| 290 |
minutes to run; a couple of extra minutes is required to compile the testbed |
| 291 |
project, and then boot and prepare the iOS simulator. |
| 292 |
|
| 293 |
Debugging test failures |
| 294 |
----------------------- |
| 295 |
|
| 296 |
Running ``make testios`` generates a standalone version of the ``iOS/testbed`` |
| 297 |
project, and runs the full test suite. It does this using ``iOS/testbed`` |
| 298 |
itself - the folder is an executable module that can be used to create and run |
| 299 |
a clone of the testbed project. |
| 300 |
|
| 301 |
You can generate your own standalone testbed instance by running:: |
| 302 |
|
| 303 |
$ python iOS/testbed clone --framework iOS/Frameworks/arm64-iphonesimulator my-testbed |
| 304 |
|
| 305 |
This invocation assumes that ``iOS/Frameworks/arm64-iphonesimulator`` is the |
| 306 |
path to the iOS simulator framework for your platform (ARM64 in this case); |
| 307 |
``my-testbed`` is the name of the folder for the new testbed clone. |
| 308 |
|
| 309 |
You can then use the ``my-testbed`` folder to run the Python test suite, |
| 310 |
passing in any command line arguments you may require. For example, if you're |
| 311 |
trying to diagnose a failure in the ``os`` module, you might run:: |
| 312 |
|
| 313 |
$ python my-testbed run -- test -W test_os |
| 314 |
|
| 315 |
This is the equivalent of running ``python -m test -W test_os`` on a desktop |
| 316 |
Python build. Any arguments after the ``--`` will be passed to testbed as if |
| 317 |
they were arguments to ``python -m`` on a desktop machine. |
| 318 |
|
| 319 |
Testing in Xcode |
| 320 |
^^^^^^^^^^^^^^^^ |
| 321 |
|
| 322 |
You can also open the testbed project in Xcode by running:: |
| 323 |
|
| 324 |
$ open my-testbed/iOSTestbed.xcodeproj |
| 325 |
|
| 326 |
This will allow you to use the full Xcode suite of tools for debugging. |
| 327 |
|
| 328 |
The arguments used to run the test suite are defined as part of the test plan. |
| 329 |
To modify the test plan, select the test plan node of the project tree (it |
| 330 |
should be the first child of the root node), and select the "Configurations" |
| 331 |
tab. Modify the "Arguments Passed On Launch" value to change the testing |
| 332 |
arguments. |
| 333 |
|
| 334 |
The test plan also disables parallel testing, and specifies the use of the |
| 335 |
``iOSTestbed.lldbinit`` file for providing configuration of the debugger. The |
| 336 |
default debugger configuration disables automatic breakpoints on the |
| 337 |
``SIGINT``, ``SIGUSR1``, ``SIGUSR2``, and ``SIGXFSZ`` signals. |
| 338 |
|
| 339 |
Testing on an iOS device |
| 340 |
^^^^^^^^^^^^^^^^^^^^^^^^ |
| 341 |
|
| 342 |
To test on an iOS device, the app needs to be signed with known developer |
| 343 |
credentials. To obtain these credentials, you must have an iOS Developer |
| 344 |
account, and your Xcode install will need to be logged into your account (see |
| 345 |
the Accounts tab of the Preferences dialog). |
| 346 |
|
| 347 |
Once the project is open, and you're signed into your Apple Developer account, |
| 348 |
select the root node of the project tree (labeled "iOSTestbed"), then the |
| 349 |
"Signing & Capabilities" tab in the details page. Select a development team |
| 350 |
(this will likely be your own name), and plug in a physical device to your |
| 351 |
macOS machine with a USB cable. You should then be able to select your physical |
| 352 |
device from the list of targets in the pulldown in the Xcode titlebar. |