| 1 |
|
| 2 |
=========================== |
| 3 |
Grant Proposal for CFFI 1.0 |
| 4 |
=========================== |
| 5 |
|
| 6 |
*Accepted by the PSF board on April 4, 2015* |
| 7 |
|
| 8 |
This Grant Proposal is to give a boost towards "CFFI 1.0". Two main |
| 9 |
issues with the current CFFI need to be solved: the difficulties of |
| 10 |
installation, and the potentially large time taken at import. |
| 11 |
|
| 12 |
1. The difficulties of installation can be seen from outside by looking |
| 13 |
at various workarounds and 3rd-party documentation that have grown into |
| 14 |
existence. For example, the `setup.py` of projects like cryptography, |
| 15 |
PyNaCl and bcrypt deploys workarounds that are explicitly documented in |
| 16 |
https://caremad.io/2014/11/distributing-a-cffi-project/. |
| 17 |
|
| 18 |
2. The time taken at import is excessive in some cases. For example, |
| 19 |
importing `pygame-cffi` on a Raspberry Pi ARM board takes on the order |
| 20 |
of 10 to 20 seconds (and this is the "fast" case where the compiler |
| 21 |
doesn't need to be invoked any more). |
| 22 |
|
| 23 |
|
| 24 |
Technical Overview |
| 25 |
------------------ |
| 26 |
|
| 27 |
"CFFI" is an existing Python project which complements the ctypes, |
| 28 |
SWIG and Cython approaches to ease writing C Extension Modules for |
| 29 |
Python. It has several advantages over the previous approaches, which |
| 30 |
are presented at the start of the documentation at |
| 31 |
http://cffi.readthedocs.org/en/latest/ . It has been very successful |
| 32 |
so far: http://pypi-ranking.info/alltime records almost 7 million |
| 33 |
downloads (for comparison, the #1 of all packages has almost 36 |
| 34 |
million downloads). CFFI works on any Python >= 2.6, including 3.x, |
| 35 |
as well as on PyPy. |
| 36 |
|
| 37 |
One problem is that while getting started with CFFI is very easy, the |
| 38 |
installation process of a package that uses CFFI has got its rough |
| 39 |
edges. CFFI (at least in its "verify()" mode) is based on calling the |
| 40 |
C compiler to get information about the exact C types, structures, |
| 41 |
argument types to functions, and so on. The C compiler is invoked |
| 42 |
transparently at run-time, and the results cached. A |
| 43 |
correctly-installed package using CFFI should cache the results at |
| 44 |
installation time, but it can be difficult to ensure that no more |
| 45 |
run-time compiler invocation is needed; doing so requires following |
| 46 |
some extra guidelines or understanding some internal details. (The |
| 47 |
problem is particularly acute on Windows where a typical user might |
| 48 |
not have a proper C compiler installed.) |
| 49 |
|
| 50 |
To fix this, we have in mind adding a different CFFI mode (replacing |
| 51 |
"verify()"), while keeping the access to the underlying C library |
| 52 |
unmodified. In this mode, the code containing the cdef() and verify() |
| 53 |
invocations would be moved to a separate Python source file. Running |
| 54 |
that Python file would produce a dynamically-linked library. There |
| 55 |
would be no caching logic involved; you would need to run it |
| 56 |
explicitly during development whenever you made changes to it, to |
| 57 |
re-generate and re-compile the dynamically-linked library. |
| 58 |
|
| 59 |
When distributed, the same file would be run (once) during |
| 60 |
installation. This can be fully automated in setuptools-based |
| 61 |
setup.py files; alternatively, it can be done in distutils-based |
| 62 |
setup.py files by requiring prior manual installation of CFFI itself. |
| 63 |
|
| 64 |
A major difference with the existing verify() approach would be that |
| 65 |
the ``.so/.dll/.dylib`` file would not be immediately loaded into the |
| 66 |
process; you would load it only from the installed program at |
| 67 |
run-time, and get the ``ffi`` and ``lib`` objects in this way (these |
| 68 |
are the two objects that you use so far to access a C library with |
| 69 |
verify()). |
| 70 |
|
| 71 |
Additionally, this would solve another issue: every import of a large |
| 72 |
CFFI-using package takes a while so far. This is caused by CFFI |
| 73 |
needing to parse again the C source code given in the cdef() (adding a |
| 74 |
run-time dependency to the ``pycparser`` and ``ply`` packages). CFFI |
| 75 |
also computes a CRC to know if it can reuse its cache. In the |
| 76 |
proposed change, all the cdef() code would be pre-parsed and stored in |
| 77 |
the dynamically-linked library, and no CRC would be needed. This |
| 78 |
would massively reduce the import times. |
| 79 |
|
| 80 |
|
| 81 |
Grant objective |
| 82 |
--------------- |
| 83 |
|
| 84 |
The objective is to give a boost towards "CFFI 1.0", which needs to have |
| 85 |
the functionalities described above in order to solve the two main |
| 86 |
issues with the current CFFI: the difficulties of installation, and the |
| 87 |
time taken at import. |
| 88 |
|
| 89 |
Included in the objective: the internal refactorings of CFFI that are |
| 90 |
needed to get it done cleanly. The goal is to avoid simply adding |
| 91 |
another layer on top of the old unchanged CFFI. |
| 92 |
|
| 93 |
This work may happen eventually in any case, but support from the PSF |
| 94 |
would help make it happen sooner rather than later. |
| 95 |
|
| 96 |
|
| 97 |
Grant size |
| 98 |
---------- |
| 99 |
|
| 100 |
2'500 US$ for supporting the development time. This would cover 2.5 |
| 101 |
weeks of full-time work at the part-time cost of 25 US$ per hour. |
| 102 |
|
| 103 |
The estimated work time until the CFFI 1.0 release is a bit larger |
| 104 |
than that (I estimate it at roughly 4 weeks), but 2.5 weeks should |
| 105 |
cover all the basics. An extended grant size of 4'000 US$ would be |
| 106 |
appreciated but not required ``:-)`` |
| 107 |
|
| 108 |
|
| 109 |
Grant beneficiaries |
| 110 |
------------------- |
| 111 |
|
| 112 |
Armin Rigo, main author of CFFI, committing 2.5 weeks of full-time |
| 113 |
work. |
| 114 |
|
| 115 |
|
| 116 |
Grant follow-up |
| 117 |
--------------- |
| 118 |
|
| 119 |
I will report on the success of the grant on the CFFI mailing list and |
| 120 |
on the blog I usually post to (the PyPy blog) and mention the PSF as |
| 121 |
providing the grant. The PSF will receive an email pointing to these |
| 122 |
postings once they are out. Moreover a full CFFI 1.0 release should |
| 123 |
follow (likely starting with beta versions); the PSF will receive |
| 124 |
another email pointing to it. |