Debian Codegraph @

doc/source/devel/ci.rst

raw
1 .. _devel-ci:
2
3 Continuous integration
4 =======================
5
6 This document describes the continuous integration (CI) set up for Modules:
7 what each workflow tests, when it runs, and how to work with the coverage
8 and dependency-update tooling that goes along with it.
9
10 All CI is implemented as `GitHub Actions
11 <https://docs.github.com/en/actions>`_ workflows, defined under
12 :file:`.github/workflows/`. Every workflow that builds and tests Modules
13 follows the same overall shape: install system packages for the target
14 platform, ``./configure`` with a job-specific option combination, ``make``,
15 run the testsuite (see :doc:`testsuite`), ``make install``, re-run the
16 testsuite against the installed tree, then ``make uninstall``.
17
18 Overview
19 --------
20
21 ``linux_tests.yaml``
22 Build/testsuite across Tcl versions, ``configure`` option
23 combinations and Linux distributions. Runs on push, pull request.
24 ``macos_tests.yaml``
25 Build/testsuite on macOS. Runs on push, pull request.
26 ``windows_tests.yaml``
27 Build/testsuite on native Windows, Cygwin and MSYS2. Runs on push,
28 pull request.
29 ``lint_tests.yaml``
30 Static analysis (Nagelfar, ShellCheck). Runs on push, pull request.
31 ``completion_tests.yaml``
32 Interactive shell Tab-completion tests. Runs on push, pull request.
33 ``differential_shellcheck.yml``
34 ShellCheck diff annotations on the changed lines only. Runs on pull
35 request (to ``main``).
36 ``easybuild_tests.yaml``
37 Downstream integration: EasyBuild framework test suite against
38 Modules. Runs on push, pull request.
39 ``scorecard.yml``
40 `OSSF Scorecard`_ supply-chain security analysis. Runs on push (to
41 ``main``), weekly, manual.
42 ``cflite.yml``
43 `ClusterFuzzLite`_ fuzzing of the optional C helper library
44 (:file:`lib/`): a short pass on pull requests touching :file:`lib/`
45 or :file:`.clusterfuzzlite/`, and a longer batch pass bi-weekly/manual.
46
47 All of the above live under :file:`.github/workflows/`.
48
49 .. _OSSF Scorecard: https://github.com/ossf/scorecard
50 .. _ClusterFuzzLite: https://google.github.io/clusterfuzzlite/
51
52 Every build/test workflow (all except :file:`differential_shellcheck.yml`,
53 :file:`scorecard.yml` and :file:`cflite.yml`) triggers on ``push`` and
54 ``pull_request``, both restricted to ``main`` and ``v*.*.x`` branches.
55
56 Build/test workflows
57 ---------------------
58
59 Linux (:file:`linux_tests.yaml`)
60 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
61
62 The largest and most detailed workflow, made of one job per Tcl
63 version/environment combination:
64
65 ``tcl86-nolibtclenvmodules``, ``tcl85-nolibtclenvmodules``
66 Tcl 8.6 and Tcl 8.5 (the latter built from source) with the optional C
67 helper library (see ``lib/`` in the repository layout) disabled via
68 ``--disable-libtclenvmodules``.
69 ``tcl85-2``, ``tcl86``, ``tcl85``, ``tcl91``, ``tcl90``
70 Further Tcl 8.5/8.6/9.0/9.1 combinations (Tcl 9.0/9.1 built from
71 source), each passing a different, deliberately varied set of
72 ``./configure`` options (pager, color, versioning, quarantine,
73 icase, domain name detection, multilib, siteconfig, locked configs,
74 ...). Spreading option coverage across jobs this way, rather than
75 repeating the same ``configure`` line everywhere, is what exercises
76 the option surface described in :doc:`add-new-config-option`; when
77 adding a new option, consider whether an existing job's option set is
78 the natural place to add it. ``tcl91`` additionally checks out the
79 repository into a working directory whose name contains ``+`` and
80 other shell-special characters, to catch path-quoting regressions.
81 ``el8``, ``el9``
82 Full build, test, install and RPM/SRPM packaging inside
83 ``rockylinux:8``/``almalinux:9`` containers, exercising the RPM
84 packaging path (``make srpm rpm``) that the other jobs don't cover.
85
86 Most jobs also set ``COVERAGE: y`` (see `Coverage`_ below) and run under
87 ``xvfb-run`` where the testsuite needs an X11 display (X resource
88 handling tests, see :doc:`testsuite`).
89
90 macOS (:file:`macos_tests.yaml`)
91 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
92
93 A single ``macos`` job on ``macos-15`` (Apple Silicon), using Tcl/Tk
94 installed via Homebrew. It additionally compiles the ``testutil-*.c``
95 helper libraries used by the testsuite as ``dylib``\ s
96 (``clang -dynamiclib``, universal ``arm64``/``arm64e``) to exercise
97 ``DYLD``-based library insertion, the macOS equivalent of the
98 ``LD_PRELOAD`` quarantine tests run elsewhere.
99
100 Windows (:file:`windows_tests.yaml`)
101 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
102
103 Four jobs on ``windows-2022``, covering the different ways Modules can run
104 on Windows:
105
106 ``native-cmd``
107 Native Windows Tcl (installed from the `BAWT Tcl distribution
108 <https://www.bawt.tcl3d.org/>`_), built via ``make dist-win`` and
109 installed/tested/uninstalled through the generated
110 :file:`INSTALL.bat`/:file:`TESTINSTALL.bat`/:file:`UNINSTALL.bat`
111 scripts, driven from a ``cmd`` shell. Before installing, the inherited
112 ``PATH`` is padded past the legacy 1024-character limit of the
113 ``setx`` tool (mimicking a Visual Studio Developer Prompt) to guard
114 against the persisted system ``PATH`` getting silently truncated.
115 ``native-pwsh``
116 Same native Windows install, but through the PowerShell variant
117 (:file:`INSTALL_PWSH.bat`/:file:`TESTINSTALL_PWSH.ps1`).
118 ``cygwin``
119 Build and test inside a `Cygwin <https://www.cygwin.com/>`_
120 environment, using the regular ``./configure``/``make``/
121 :file:`script/mt` flow like the Unix workflows.
122 ``msys``
123 Same, inside an `MSYS2 <https://www.msys2.org/>`_ ``MSYS`` environment.
124
125 Lint (:file:`lint_tests.yaml`)
126 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
127
128 Runs the ``lint`` DejaGnu tool (Nagelfar for Tcl files, ShellCheck for
129 sh/bash/ksh scripts) via ``script/mt lint``, i.e. the same static analysis
130 described as the ``lint`` tool in :doc:`testsuite`. On failure it uploads
131 :file:`lint.log` as a build artifact.
132
133 Completion (:file:`completion_tests.yaml`)
134 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
135
136 Runs the ``completion`` DejaGnu tool (real, Expect-driven Tab-key presses
137 against the built :file:`init/bash_completion` script) via
138 ``script/mt comp``, i.e. the same interactive shell Tab-completion testing
139 described as the ``completion`` tool in :doc:`testsuite`. Only needs the
140 baseline packages (``bash``, ``expect-dev``, ``dejagnu``) rather than the
141 full multi-shell set installed by :file:`linux_tests.yaml`, since the
142 ``completion.00-init`` layout currently covers bash only. On failure it
143 uploads :file:`completion.log` as a build artifact.
144
145 Differential ShellCheck (:file:`differential_shellcheck.yml`)
146 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
147
148 Runs only on pull requests targeting ``main`` (not on ``push``). Unlike the
149 full-repository ShellCheck run in the ``lint`` tool, this uses the
150 `redhat-plumbers-in-action/differential-shellcheck
151 <https://github.com/redhat-plumbers-in-action/differential-shellcheck>`_
152 action to lint only the lines actually changed by the pull request and post
153 results as inline PR review annotations and GitHub code-scanning alerts
154 (requires the extra ``security-events: write`` and ``pull-requests: write``
155 permissions granted to this job only).
156
157 EasyBuild integration (:file:`easybuild_tests.yaml`)
158 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
159
160 Builds and installs Modules, then clones the latest tagged release of the
161 `EasyBuild framework <https://github.com/easybuilders/easybuild-framework>`_
162 (a major downstream consumer of Modules) and runs *its* own test suite
163 (``python -m test.framework.suite``) configured to use the
164 just-built Modules as its modules tool. This is the one workflow that
165 verifies Modules against a real downstream project rather than against its
166 own testsuite, catching regressions the in-repo testsuite wouldn't see.
167
168 Scorecard analysis (:file:`scorecard.yml`)
169 -------------------------------------------
170
171 Runs `OSSF Scorecard`_, a supply-chain security posture checker, on every
172 push to ``main``, weekly (Saturdays at 02:30 UTC), and on manual dispatch.
173 Results are published to the public OSSF API and uploaded as a SARIF file
174 to GitHub's Code scanning. This is unrelated to build/test coverage; it
175 checks repository practices (branch protection, pinned dependencies, ...).
176 See ``.github/security-insights.yml`` for how this feeds into the project's
177 OpenSSF Best Practices self-assessment.
178
179 Fuzzing (:file:`cflite.yml`)
180 ----------------------------
181
182 Runs `ClusterFuzzLite`_, a lightweight continuous-fuzzing setup built on
183 OSS-Fuzz's libFuzzer/sanitizer tooling, against the optional C helper
184 library in :file:`lib/` (see ``lib/envmodules.c`` in the repository
185 layout). This is what satisfies the `OSSF Scorecard`_ "Fuzzing" check
186 above: for C/C++ projects Scorecard only recognizes OSS-Fuzz registration
187 or the presence of a :file:`.clusterfuzzlite/Dockerfile`, not arbitrary
188 libFuzzer harnesses on their own.
189
190 :file:`.clusterfuzzlite/` at the repository root holds the fuzzing setup:
191
192 ``project.yaml``, ``Dockerfile``, ``build.sh``
193 Standard ClusterFuzzLite build integration files. ``build.sh``
194 regenerates :file:`lib/config.h` through ``lib/configure`` (with
195 ``--disable-shared --disable-stubs``, since the fuzz targets link
196 against libtcl directly rather than through the Tcl stub mechanism
197 the shipped extension uses) and compiles ``envmodules.c`` together
198 with each fuzz target.
199 ``fuzz_parsedatetimearg.c``, ``fuzz_readfile.c``, ``fuzz_getfilesindirectory.c``
200 One target per :file:`lib/envmodules.c` entry point that parses
201 externally-influenced input: date/time argument strings, file
202 content (including the ``#%Module`` magic-cookie check), and
203 directory-entry names, respectively. ``Envmodules_InitStateUsernameObjCmd``,
204 ``Envmodules_InitStateUsergroupsObjCmd`` and
205 ``Envmodules_InitStateClockSecondsObjCmd`` take no Tcl-level
206 arguments and only query process/OS state (uid, group list, current
207 time), so there is no fuzzer-mutable input for them and no fuzz
208 target is provided.
209
210 :file:`cflite.yml` has two jobs, gated on the triggering event: ``PR`` runs
211 a short AddressSanitizer/UndefinedBehaviorSanitizer fuzzing pass, in
212 ``code-change`` mode (reporting only issues newly introduced by the pull
213 request's diff), on pull requests touching :file:`lib/` or
214 :file:`.clusterfuzzlite/`. ``BatchFuzzing`` runs the same targets for a
215 full hour, bi-weekly and on manual dispatch, to catch issues that need a
216 deeper corpus to reach.
217
218 Coverage
219 --------
220
221 Most build/test jobs set ``COVERAGE: y``, which (as described in
222 :doc:`testsuite`) runs the testsuite against a Nagelfar-instrumented
223 :file:`modulecmd.tcl` and produces marked-up :file:`tcl/*.tcl_m` files.
224 After the testsuite runs, each job converts that coverage data
225 (``script/nglfar2ccov`` per Tcl file, plus ``gcov`` for the optional C
226 helper library in :file:`lib/`) and uploads it via the
227 ``codecov/codecov-action``. :file:`codecov.yml` at the repository root
228 configures Codecov to wait for 7 builds before notifying (so the PR check
229 reflects combined coverage from all the Linux/macOS jobs that report it,
230 not just the first one to finish) and to fail the ``project`` coverage
231 status if the CI run itself failed.
232
233 Dependency updates
234 -------------------
235
236 :file:`.github/dependabot.yml` configures `Dependabot
237 <https://docs.github.com/en/code-security/dependabot>`_ to check the
238 ``github-actions`` ecosystem (i.e. the ``uses:`` action references in
239 :file:`.github/workflows/`) weekly and open pull requests to bump pinned
240 versions, keeping the actions used by all of the above workflows current.
241 It also checks the ``docker`` ecosystem in :file:`.clusterfuzzlite/`,
242 keeping the digest-pinned ``FROM`` line in
243 :file:`.clusterfuzzlite/Dockerfile` current.
244
245 .. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: