Debian Codegraph @

doc/misc/grant-cffi-1.0.rst

raw
1
2 ===========================
3 Grant Proposal for CFFI 1.0
4 ===========================
5
6 *Accepted by the PSF board on April 4, 2015*
7
8 This Grant Proposal is to give a boost towards "CFFI 1.0". Two main
9 issues with the current CFFI need to be solved: the difficulties of
10 installation, and the potentially large time taken at import.
11
12 1. The difficulties of installation can be seen from outside by looking
13 at various workarounds and 3rd-party documentation that have grown into
14 existence. For example, the `setup.py` of projects like cryptography,
15 PyNaCl and bcrypt deploys workarounds that are explicitly documented in
16 https://caremad.io/2014/11/distributing-a-cffi-project/.
17
18 2. The time taken at import is excessive in some cases. For example,
19 importing `pygame-cffi` on a Raspberry Pi ARM board takes on the order
20 of 10 to 20 seconds (and this is the "fast" case where the compiler
21 doesn't need to be invoked any more).
22
23
24 Technical Overview
25 ------------------
26
27 "CFFI" is an existing Python project which complements the ctypes,
28 SWIG and Cython approaches to ease writing C Extension Modules for
29 Python. It has several advantages over the previous approaches, which
30 are presented at the start of the documentation at
31 http://cffi.readthedocs.org/en/latest/ . It has been very successful
32 so far: http://pypi-ranking.info/alltime records almost 7 million
33 downloads (for comparison, the #1 of all packages has almost 36
34 million downloads). CFFI works on any Python >= 2.6, including 3.x,
35 as well as on PyPy.
36
37 One problem is that while getting started with CFFI is very easy, the
38 installation process of a package that uses CFFI has got its rough
39 edges. CFFI (at least in its "verify()" mode) is based on calling the
40 C compiler to get information about the exact C types, structures,
41 argument types to functions, and so on. The C compiler is invoked
42 transparently at run-time, and the results cached. A
43 correctly-installed package using CFFI should cache the results at
44 installation time, but it can be difficult to ensure that no more
45 run-time compiler invocation is needed; doing so requires following
46 some extra guidelines or understanding some internal details. (The
47 problem is particularly acute on Windows where a typical user might
48 not have a proper C compiler installed.)
49
50 To fix this, we have in mind adding a different CFFI mode (replacing
51 "verify()"), while keeping the access to the underlying C library
52 unmodified. In this mode, the code containing the cdef() and verify()
53 invocations would be moved to a separate Python source file. Running
54 that Python file would produce a dynamically-linked library. There
55 would be no caching logic involved; you would need to run it
56 explicitly during development whenever you made changes to it, to
57 re-generate and re-compile the dynamically-linked library.
58
59 When distributed, the same file would be run (once) during
60 installation. This can be fully automated in setuptools-based
61 setup.py files; alternatively, it can be done in distutils-based
62 setup.py files by requiring prior manual installation of CFFI itself.
63
64 A major difference with the existing verify() approach would be that
65 the ``.so/.dll/.dylib`` file would not be immediately loaded into the
66 process; you would load it only from the installed program at
67 run-time, and get the ``ffi`` and ``lib`` objects in this way (these
68 are the two objects that you use so far to access a C library with
69 verify()).
70
71 Additionally, this would solve another issue: every import of a large
72 CFFI-using package takes a while so far. This is caused by CFFI
73 needing to parse again the C source code given in the cdef() (adding a
74 run-time dependency to the ``pycparser`` and ``ply`` packages). CFFI
75 also computes a CRC to know if it can reuse its cache. In the
76 proposed change, all the cdef() code would be pre-parsed and stored in
77 the dynamically-linked library, and no CRC would be needed. This
78 would massively reduce the import times.
79
80
81 Grant objective
82 ---------------
83
84 The objective is to give a boost towards "CFFI 1.0", which needs to have
85 the functionalities described above in order to solve the two main
86 issues with the current CFFI: the difficulties of installation, and the
87 time taken at import.
88
89 Included in the objective: the internal refactorings of CFFI that are
90 needed to get it done cleanly. The goal is to avoid simply adding
91 another layer on top of the old unchanged CFFI.
92
93 This work may happen eventually in any case, but support from the PSF
94 would help make it happen sooner rather than later.
95
96
97 Grant size
98 ----------
99
100 2'500 US$ for supporting the development time. This would cover 2.5
101 weeks of full-time work at the part-time cost of 25 US$ per hour.
102
103 The estimated work time until the CFFI 1.0 release is a bit larger
104 than that (I estimate it at roughly 4 weeks), but 2.5 weeks should
105 cover all the basics. An extended grant size of 4'000 US$ would be
106 appreciated but not required ``:-)``
107
108
109 Grant beneficiaries
110 -------------------
111
112 Armin Rigo, main author of CFFI, committing 2.5 weeks of full-time
113 work.
114
115
116 Grant follow-up
117 ---------------
118
119 I will report on the success of the grant on the CFFI mailing list and
120 on the blog I usually post to (the PyPy blog) and mention the PSF as
121 providing the grant. The PSF will receive an email pointing to these
122 postings once they are out. Moreover a full CFFI 1.0 release should
123 follow (likely starting with beta versions); the PSF will receive
124 another email pointing to it.