Debian Codegraph @

gcc/jit/docs/internals/index.rst

raw
1 .. Copyright (C) 2014-2023 Free Software Foundation, Inc.
2 Originally contributed by David Malcolm <[email protected]>
3
4 This is free software: you can redistribute it and/or modify it
5 under the terms of the GNU General Public License as published by
6 the Free Software Foundation, either version 3 of the License, or
7 (at your option) any later version.
8
9 This program is distributed in the hope that it will be useful, but
10 WITHOUT ANY WARRANTY; without even the implied warranty of
11 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
12 General Public License for more details.
13
14 You should have received a copy of the GNU General Public License
15 along with this program. If not, see
16 <https://www.gnu.org/licenses/>.
17
18 Internals
19 =========
20
21 Working on the JIT library
22 --------------------------
23 Having checked out the source code (to "src"), you can configure and build
24 the JIT library like this:
25
26 .. code-block:: bash
27
28 mkdir build
29 mkdir install
30 PREFIX=$(pwd)/install
31 cd build
32 ../src/configure \
33 --enable-host-shared \
34 --enable-languages=jit,c++ \
35 --disable-bootstrap \
36 --enable-checking=release \
37 --prefix=$PREFIX
38 nice make -j4 # altering the "4" to however many cores you have
39
40 This should build a libgccjit.so within jit/build/gcc:
41
42 .. code-block:: console
43
44 [build] $ file gcc/libgccjit.so*
45 gcc/libgccjit.so: symbolic link to `libgccjit.so.0'
46 gcc/libgccjit.so.0: symbolic link to `libgccjit.so.0.0.1'
47 gcc/libgccjit.so.0.0.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
48
49 Here's what those configuration options mean:
50
51 .. option:: --enable-host-shared
52
53 Configuring with this option means that the compiler is built as
54 position-independent code, which incurs a slight performance hit,
55 but it necessary for a shared library.
56
57 .. option:: --enable-languages=jit,c++
58
59 This specifies which frontends to build. The JIT library looks like
60 a frontend to the rest of the code.
61
62 The C++ portion of the JIT test suite requires the C++ frontend to be
63 enabled at configure-time, or you may see errors like this when
64 running the test suite:
65
66 .. code-block:: console
67
68 xgcc: error: /home/david/jit/src/gcc/testsuite/jit.dg/test-quadratic.cc: C++ compiler not installed on this system
69 c++: error trying to exec 'cc1plus': execvp: No such file or directory
70
71 .. option:: --disable-bootstrap
72
73 For hacking on the "jit" subdirectory, performing a full
74 bootstrap can be overkill, since it's unused by a bootstrap. However,
75 when submitting patches, you should remove this option, to ensure that
76 the compiler can still bootstrap itself.
77
78 .. option:: --enable-checking=release
79
80 The compile can perform extensive self-checking as it runs, useful when
81 debugging, but slowing things down.
82
83 For maximum speed, configure with ``--enable-checking=release`` to
84 disable this self-checking.
85
86 Running the test suite
87 ----------------------
88
89 .. code-block:: console
90
91 [build] $ cd gcc
92 [gcc] $ make check-jit RUNTESTFLAGS="-v -v -v"
93
94 A summary of the tests can then be seen in:
95
96 .. code-block:: console
97
98 jit/build/gcc/testsuite/jit/jit.sum
99
100 and detailed logs in:
101
102 .. code-block:: console
103
104 jit/build/gcc/testsuite/jit/jit.log
105
106 The test executables are normally deleted after each test is run. For
107 debugging, they can be preserved by setting :envvar:`PRESERVE_EXECUTABLES`
108 in the environment. If so, they can then be seen as:
109
110 .. code-block:: console
111
112 jit/build/gcc/testsuite/jit/*.exe
113
114 which can be run independently.
115
116 You can compile and run individual tests by passing "jit.exp=TESTNAME" to RUNTESTFLAGS e.g.:
117
118 .. code-block:: console
119
120 [gcc] $ PRESERVE_EXECUTABLES= \
121 make check-jit \
122 RUNTESTFLAGS="-v -v -v jit.exp=test-factorial.c"
123
124 and once a test has been compiled, you can debug it directly:
125
126 .. code-block:: console
127
128 [gcc] $ PATH=.:$PATH \
129 LD_LIBRARY_PATH=. \
130 LIBRARY_PATH=. \
131 gdb --args \
132 testsuite/jit/test-factorial.c.exe
133
134 Running under valgrind
135 **********************
136
137 The jit testsuite detects if :envvar:`RUN_UNDER_VALGRIND` is present in the
138 environment (with any value). If it is present, it runs the test client
139 code under `valgrind <https://valgrind.org>`_,
140 specifcally, the default
141 `memcheck <https://valgrind.org/docs/manual/mc-manual.html>`_
142 tool with
143 `--leak-check=full
144 <https://valgrind.org/docs/manual/mc-manual.html#opt.leak-check>`_.
145
146 It automatically parses the output from valgrind, injecting XFAIL results if
147 any issues are found, or PASS results if the output is clean. The output
148 is saved to ``TESTNAME.exe.valgrind.txt``.
149
150 For example, the following invocation verbosely runs the testcase
151 ``test-sum-of-squares.c`` under valgrind, showing an issue:
152
153 .. code-block:: console
154
155 $ RUN_UNDER_VALGRIND= \
156 make check-jit \
157 RUNTESTFLAGS="-v -v -v jit.exp=test-sum-of-squares.c"
158
159 (...verbose log contains detailed valgrind errors, if any...)
160
161 === jit Summary ===
162
163 # of expected passes 28
164 # of expected failures 2
165
166 $ less testsuite/jit/jit.sum
167 (...other results...)
168 XFAIL: jit.dg/test-sum-of-squares.c: test-sum-of-squares.c.exe.valgrind.txt: definitely lost: 8 bytes in 1 blocks
169 XFAIL: jit.dg/test-sum-of-squares.c: test-sum-of-squares.c.exe.valgrind.txt: unsuppressed errors: 1
170 (...other results...)
171
172 $ less testsuite/jit/test-sum-of-squares.c.exe.valgrind.txt
173 (...shows full valgrind report for this test case...)
174
175 When running under valgrind, it's best to have configured gcc with
176 :option:`--enable-valgrind-annotations`, which automatically suppresses
177 various known false positives.
178
179 Environment variables
180 ---------------------
181 When running client code against a locally-built libgccjit, three
182 environment variables need to be set up:
183
184 .. envvar:: LD_LIBRARY_PATH
185
186 `libgccjit.so` is dynamically linked into client code, so if running
187 against a locally-built library, ``LD_LIBRARY_PATH`` needs to be set
188 up appropriately. The library can be found within the "gcc"
189 subdirectory of the build tree:
190
191 .. code-block:: console
192
193 $ file libgccjit.so*
194 libgccjit.so: symbolic link to `libgccjit.so.0'
195 libgccjit.so.0: symbolic link to `libgccjit.so.0.0.1'
196 libgccjit.so.0.0.1: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, not stripped
197
198 .. envvar:: PATH
199
200 The library uses a driver executable for converting from .s assembler
201 files to .so shared libraries. Specifically, it looks for a name
202 expanded from
203 ``${target_noncanonical}-gcc-${gcc_BASEVER}${exeext}``
204 such as ``x86_64-unknown-linux-gnu-gcc-5.0.0``.
205
206 Hence ``PATH`` needs to include a directory where the library can
207 locate this executable.
208
209 The executable is normally installed to the installation bindir
210 (e.g. /usr/bin), but a copy is also created within the "gcc"
211 subdirectory of the build tree for running the testsuite, and for ease
212 of development.
213
214 .. envvar:: LIBRARY_PATH
215
216 The driver executable invokes the linker, and the latter needs to locate
217 support libraries needed by the generated code, or you will see errors
218 like:
219
220 .. code-block:: console
221
222 ld: cannot find crtbeginS.o: No such file or directory
223 ld: cannot find -lgcc
224 ld: cannot find -lgcc_s
225
226 Hence if running directly from a locally-built copy (without installing),
227 ``LIBRARY_PATH`` needs to contain the "gcc" subdirectory of the build
228 tree.
229
230 For example, to run a binary that uses the library against a non-installed
231 build of the library in LIBGCCJIT_BUILD_DIR you need an invocation of the
232 client code like this, to preprend the dir to each of the environment
233 variables:
234
235 .. code-block:: console
236
237 $ LD_LIBRARY_PATH=$(LIBGCCJIT_BUILD_DIR):$(LD_LIBRARY_PATH) \
238 PATH=$(LIBGCCJIT_BUILD_DIR):$(PATH) \
239 LIBRARY_PATH=$(LIBGCCJIT_BUILD_DIR):$(LIBRARY_PATH) \
240 ./jit-hello-world
241 hello world
242
243 Packaging notes
244 ---------------
245 The configure-time option :option:`--enable-host-shared` is needed when
246 building the jit in order to get position-independent code. This will
247 slow down the regular compiler by a few percent. Hence when packaging gcc
248 with libgccjit, please configure and build twice:
249
250 * once without :option:`--enable-host-shared` for most languages, and
251
252 * once with :option:`--enable-host-shared` for the jit
253
254 For example:
255
256 .. code-block:: bash
257
258 # Configure and build with --enable-host-shared
259 # for the jit:
260 mkdir configuration-for-jit
261 pushd configuration-for-jit
262 $(SRCDIR)/configure \
263 --enable-host-shared \
264 --enable-languages=jit \
265 --prefix=$(DESTDIR)
266 make
267 popd
268
269 # Configure and build *without* --enable-host-shared
270 # for maximum speed:
271 mkdir standard-configuration
272 pushd standard-configuration
273 $(SRCDIR)/configure \
274 --enable-languages=all \
275 --prefix=$(DESTDIR)
276 make
277 popd
278
279 # Both of the above are configured to install to $(DESTDIR)
280 # Install the configuration with --enable-host-shared first
281 # *then* the one without, so that the faster build
282 # of "cc1" et al overwrites the slower build.
283 pushd configuration-for-jit
284 make install
285 popd
286
287 pushd standard-configuration
288 make install
289 popd
290
291 Overview of code structure
292 --------------------------
293
294 The library is implemented in C++.
295
296 * ``libgccjit.cc`` implements the API entrypoints. It performs error
297 checking, then calls into classes of the gcc::jit::recording namespace
298 within ``jit-recording.cc`` and ``jit-recording.h``.
299
300 * The gcc::jit::recording classes (within ``jit-recording.cc`` and
301 ``jit-recording.h``) record the API calls that are made:
302
303 .. literalinclude:: ../../jit-common.h
304 :start-after: /* Recording types. */
305 :end-before: /* End of recording types. */
306 :language: c++
307
308 * When the context is compiled, the gcc::jit::playback classes (within
309 ``jit-playback.cc`` and ``jit-playback.h``) replay the API calls
310 within langhook:parse_file:
311
312 .. literalinclude:: ../../jit-common.h
313 :start-after: /* Playback types. */
314 :end-before: /* End of playback types. */
315 :language: c++
316
317 .. literalinclude:: ../../notes.txt
318 :lines: 1-
319
320 Here is a high-level summary from ``jit-common.h``:
321
322 .. include:: ../../jit-common.h
323 :start-after: This comment is included by the docs.
324 :end-before: End of comment for inclusion in the docs. */
325
326 .. _example-of-log-file:
327
328 Another way to understand the structure of the code is to enable logging,
329 via :c:func:`gcc_jit_context_set_logfile`. Here is an example of a log
330 generated via this call:
331
332 .. literalinclude:: test-hello-world.exe.log.txt
333 :lines: 1-
334
335 Design notes
336 ------------
337 It should not be possible for client code to cause an internal compiler
338 error. If this *does* happen, the root cause should be isolated (perhaps
339 using :c:func:`gcc_jit_context_dump_reproducer_to_file`) and the cause
340 should be rejected via additional checking. The checking ideally should
341 be within the libgccjit API entrypoints in libgccjit.cc, since this is as
342 close as possible to the error; failing that, a good place is within
343 ``recording::context::validate ()`` in jit-recording.cc.
344
345 Submitting patches
346 ------------------
347 Please read the contribution guidelines for gcc at
348 https://gcc.gnu.org/contribute.html.
349
350 Patches for the jit should be sent to both the
351 [email protected] and [email protected] mailing lists,
352 with "jit" and "PATCH" in the Subject line.
353
354 You don't need to do a full bootstrap for code that just touches the
355 ``jit`` and ``testsuite/jit.dg`` subdirectories. However, please run
356 ``make check-jit`` before submitting the patch, and mention the results
357 in your email (along with the host triple that the tests were run on).
358
359 A good patch should contain the information listed in the
360 gcc contribution guide linked to above; for a ``jit`` patch, the patch
361 shold contain:
362
363 * the code itself (for example, a new API entrypoint will typically
364 touch ``libgccjit.h`` and ``.c``, along with support code in
365 ``jit-recording.[ch]`` and ``jit-playback.[ch]`` as appropriate)
366
367 * test coverage
368
369 * documentation for the C API
370
371 * documentation for the C++ API
372
373 A patch that adds new API entrypoints should also contain:
374
375 * a feature macro in ``libgccjit.h`` so that client code that doesn't
376 use a "configure" mechanism can still easily detect the presence of
377 the entrypoint. See e.g. ``LIBGCCJIT_HAVE_SWITCH_STATEMENTS`` (for
378 a category of entrypoints) and
379 ``LIBGCCJIT_HAVE_gcc_jit_context_set_bool_allow_unreachable_blocks``
380 (for an individual entrypoint).
381
382 * a new ABI tag containing the new symbols (in ``libgccjit.map``), so
383 that we can detect client code that uses them
384
385 * Support for :c:func:`gcc_jit_context_dump_reproducer_to_file`. Most
386 jit testcases attempt to dump their contexts to a .c file; ``jit.exp``
387 then sanity-checks the generated c by compiling them (though
388 not running them). A new API entrypoint
389 needs to "know" how to write itself back out to C (by implementing
390 ``gcc::jit::recording::memento::write_reproducer`` for the appropriate
391 ``memento`` subclass).
392
393 * C++ bindings for the new entrypoints (see ``libgccjit++.h``); ideally
394 with test coverage, though the C++ API test coverage is admittedly
395 spotty at the moment
396
397 * documentation for the new C entrypoints
398
399 * documentation for the new C++ entrypoints
400
401 * documentation for the new ABI tag (see ``topics/compatibility.rst``).
402
403 Depending on the patch you can either extend an existing test case, or
404 add a new test case. If you add an entirely new testcase: ``jit.exp``
405 expects jit testcases to begin with ``test-``, or ``test-error-`` (for a
406 testcase that generates an error on a :c:type:`gcc_jit_context`).
407
408 Every new testcase that doesn't generate errors should also touch
409 ``gcc/testsuite/jit.dg/all-non-failing-tests.h``:
410
411 * Testcases that don't generate errors should ideally be added to the
412 ``testcases`` array in that file; this means that, in addition
413 to being run standalone, they also get run within
414 ``test-combination.c`` (which runs all successful tests inside one
415 big :c:type:`gcc_jit_context`), and ``test-threads.c`` (which runs all
416 successful tests in one process, each one running in a different
417 thread on a different :c:type:`gcc_jit_context`).
418
419 .. note::
420
421 Given that exported functions within a :c:type:`gcc_jit_context`
422 must have unique names, and most testcases are run within
423 ``test-combination.c``, this means that every jit-compiled test
424 function typically needs a name that's unique across the entire
425 test suite.
426
427 * Testcases that aren't to be added to the ``testcases`` array should
428 instead add a comment to the file clarifying why they're not in that
429 array. See the file for examples.
430
431 Typically a patch that touches the .rst documentation will also need the
432 texinfo to be regenerated. You can do this with
433 `Sphinx 1.0 <https://sphinx-doc.org/>`_ or later by
434 running ``make texinfo`` within ``SRCDIR/gcc/jit/docs``. Don't do this
435 within the patch sent to the mailing list; it can often be relatively
436 large and inconsequential (e.g. anchor renumbering), rather like generated
437 "configure" changes from configure.ac. You can regenerate it when
438 committing to svn.