| 1 |
.. _devel-ci: |
| 2 |
|
| 3 |
Continuous integration |
| 4 |
======================= |
| 5 |
|
| 6 |
This document describes the continuous integration (CI) set up for Modules: |
| 7 |
what each workflow tests, when it runs, and how to work with the coverage |
| 8 |
and dependency-update tooling that goes along with it. |
| 9 |
|
| 10 |
All CI is implemented as `GitHub Actions |
| 11 |
<https://docs.github.com/en/actions>`_ workflows, defined under |
| 12 |
:file:`.github/workflows/`. Every workflow that builds and tests Modules |
| 13 |
follows the same overall shape: install system packages for the target |
| 14 |
platform, ``./configure`` with a job-specific option combination, ``make``, |
| 15 |
run the testsuite (see :doc:`testsuite`), ``make install``, re-run the |
| 16 |
testsuite against the installed tree, then ``make uninstall``. |
| 17 |
|
| 18 |
Overview |
| 19 |
-------- |
| 20 |
|
| 21 |
``linux_tests.yaml`` |
| 22 |
Build/testsuite across Tcl versions, ``configure`` option |
| 23 |
combinations and Linux distributions. Runs on push, pull request. |
| 24 |
``macos_tests.yaml`` |
| 25 |
Build/testsuite on macOS. Runs on push, pull request. |
| 26 |
``windows_tests.yaml`` |
| 27 |
Build/testsuite on native Windows, Cygwin and MSYS2. Runs on push, |
| 28 |
pull request. |
| 29 |
``lint_tests.yaml`` |
| 30 |
Static analysis (Nagelfar, ShellCheck). Runs on push, pull request. |
| 31 |
``completion_tests.yaml`` |
| 32 |
Interactive shell Tab-completion tests. Runs on push, pull request. |
| 33 |
``differential_shellcheck.yml`` |
| 34 |
ShellCheck diff annotations on the changed lines only. Runs on pull |
| 35 |
request (to ``main``). |
| 36 |
``easybuild_tests.yaml`` |
| 37 |
Downstream integration: EasyBuild framework test suite against |
| 38 |
Modules. Runs on push, pull request. |
| 39 |
``scorecard.yml`` |
| 40 |
`OSSF Scorecard`_ supply-chain security analysis. Runs on push (to |
| 41 |
``main``), weekly, manual. |
| 42 |
``cflite.yml`` |
| 43 |
`ClusterFuzzLite`_ fuzzing of the optional C helper library |
| 44 |
(:file:`lib/`): a short pass on pull requests touching :file:`lib/` |
| 45 |
or :file:`.clusterfuzzlite/`, and a longer batch pass bi-weekly/manual. |
| 46 |
|
| 47 |
All of the above live under :file:`.github/workflows/`. |
| 48 |
|
| 49 |
.. _OSSF Scorecard: https://github.com/ossf/scorecard |
| 50 |
.. _ClusterFuzzLite: https://google.github.io/clusterfuzzlite/ |
| 51 |
|
| 52 |
Every build/test workflow (all except :file:`differential_shellcheck.yml`, |
| 53 |
:file:`scorecard.yml` and :file:`cflite.yml`) triggers on ``push`` and |
| 54 |
``pull_request``, both restricted to ``main`` and ``v*.*.x`` branches. |
| 55 |
|
| 56 |
Build/test workflows |
| 57 |
--------------------- |
| 58 |
|
| 59 |
Linux (:file:`linux_tests.yaml`) |
| 60 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 61 |
|
| 62 |
The largest and most detailed workflow, made of one job per Tcl |
| 63 |
version/environment combination: |
| 64 |
|
| 65 |
``tcl86-nolibtclenvmodules``, ``tcl85-nolibtclenvmodules`` |
| 66 |
Tcl 8.6 and Tcl 8.5 (the latter built from source) with the optional C |
| 67 |
helper library (see ``lib/`` in the repository layout) disabled via |
| 68 |
``--disable-libtclenvmodules``. |
| 69 |
``tcl85-2``, ``tcl86``, ``tcl85``, ``tcl91``, ``tcl90`` |
| 70 |
Further Tcl 8.5/8.6/9.0/9.1 combinations (Tcl 9.0/9.1 built from |
| 71 |
source), each passing a different, deliberately varied set of |
| 72 |
``./configure`` options (pager, color, versioning, quarantine, |
| 73 |
icase, domain name detection, multilib, siteconfig, locked configs, |
| 74 |
...). Spreading option coverage across jobs this way, rather than |
| 75 |
repeating the same ``configure`` line everywhere, is what exercises |
| 76 |
the option surface described in :doc:`add-new-config-option`; when |
| 77 |
adding a new option, consider whether an existing job's option set is |
| 78 |
the natural place to add it. ``tcl91`` additionally checks out the |
| 79 |
repository into a working directory whose name contains ``+`` and |
| 80 |
other shell-special characters, to catch path-quoting regressions. |
| 81 |
``el8``, ``el9`` |
| 82 |
Full build, test, install and RPM/SRPM packaging inside |
| 83 |
``rockylinux:8``/``almalinux:9`` containers, exercising the RPM |
| 84 |
packaging path (``make srpm rpm``) that the other jobs don't cover. |
| 85 |
|
| 86 |
Most jobs also set ``COVERAGE: y`` (see `Coverage`_ below) and run under |
| 87 |
``xvfb-run`` where the testsuite needs an X11 display (X resource |
| 88 |
handling tests, see :doc:`testsuite`). |
| 89 |
|
| 90 |
macOS (:file:`macos_tests.yaml`) |
| 91 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 92 |
|
| 93 |
A single ``macos`` job on ``macos-15`` (Apple Silicon), using Tcl/Tk |
| 94 |
installed via Homebrew. It additionally compiles the ``testutil-*.c`` |
| 95 |
helper libraries used by the testsuite as ``dylib``\ s |
| 96 |
(``clang -dynamiclib``, universal ``arm64``/``arm64e``) to exercise |
| 97 |
``DYLD``-based library insertion, the macOS equivalent of the |
| 98 |
``LD_PRELOAD`` quarantine tests run elsewhere. |
| 99 |
|
| 100 |
Windows (:file:`windows_tests.yaml`) |
| 101 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 102 |
|
| 103 |
Four jobs on ``windows-2022``, covering the different ways Modules can run |
| 104 |
on Windows: |
| 105 |
|
| 106 |
``native-cmd`` |
| 107 |
Native Windows Tcl (installed from the `BAWT Tcl distribution |
| 108 |
<https://www.bawt.tcl3d.org/>`_), built via ``make dist-win`` and |
| 109 |
installed/tested/uninstalled through the generated |
| 110 |
:file:`INSTALL.bat`/:file:`TESTINSTALL.bat`/:file:`UNINSTALL.bat` |
| 111 |
scripts, driven from a ``cmd`` shell. Before installing, the inherited |
| 112 |
``PATH`` is padded past the legacy 1024-character limit of the |
| 113 |
``setx`` tool (mimicking a Visual Studio Developer Prompt) to guard |
| 114 |
against the persisted system ``PATH`` getting silently truncated. |
| 115 |
``native-pwsh`` |
| 116 |
Same native Windows install, but through the PowerShell variant |
| 117 |
(:file:`INSTALL_PWSH.bat`/:file:`TESTINSTALL_PWSH.ps1`). |
| 118 |
``cygwin`` |
| 119 |
Build and test inside a `Cygwin <https://www.cygwin.com/>`_ |
| 120 |
environment, using the regular ``./configure``/``make``/ |
| 121 |
:file:`script/mt` flow like the Unix workflows. |
| 122 |
``msys`` |
| 123 |
Same, inside an `MSYS2 <https://www.msys2.org/>`_ ``MSYS`` environment. |
| 124 |
|
| 125 |
Lint (:file:`lint_tests.yaml`) |
| 126 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 127 |
|
| 128 |
Runs the ``lint`` DejaGnu tool (Nagelfar for Tcl files, ShellCheck for |
| 129 |
sh/bash/ksh scripts) via ``script/mt lint``, i.e. the same static analysis |
| 130 |
described as the ``lint`` tool in :doc:`testsuite`. On failure it uploads |
| 131 |
:file:`lint.log` as a build artifact. |
| 132 |
|
| 133 |
Completion (:file:`completion_tests.yaml`) |
| 134 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 135 |
|
| 136 |
Runs the ``completion`` DejaGnu tool (real, Expect-driven Tab-key presses |
| 137 |
against the built :file:`init/bash_completion` script) via |
| 138 |
``script/mt comp``, i.e. the same interactive shell Tab-completion testing |
| 139 |
described as the ``completion`` tool in :doc:`testsuite`. Only needs the |
| 140 |
baseline packages (``bash``, ``expect-dev``, ``dejagnu``) rather than the |
| 141 |
full multi-shell set installed by :file:`linux_tests.yaml`, since the |
| 142 |
``completion.00-init`` layout currently covers bash only. On failure it |
| 143 |
uploads :file:`completion.log` as a build artifact. |
| 144 |
|
| 145 |
Differential ShellCheck (:file:`differential_shellcheck.yml`) |
| 146 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 147 |
|
| 148 |
Runs only on pull requests targeting ``main`` (not on ``push``). Unlike the |
| 149 |
full-repository ShellCheck run in the ``lint`` tool, this uses the |
| 150 |
`redhat-plumbers-in-action/differential-shellcheck |
| 151 |
<https://github.com/redhat-plumbers-in-action/differential-shellcheck>`_ |
| 152 |
action to lint only the lines actually changed by the pull request and post |
| 153 |
results as inline PR review annotations and GitHub code-scanning alerts |
| 154 |
(requires the extra ``security-events: write`` and ``pull-requests: write`` |
| 155 |
permissions granted to this job only). |
| 156 |
|
| 157 |
EasyBuild integration (:file:`easybuild_tests.yaml`) |
| 158 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 159 |
|
| 160 |
Builds and installs Modules, then clones the latest tagged release of the |
| 161 |
`EasyBuild framework <https://github.com/easybuilders/easybuild-framework>`_ |
| 162 |
(a major downstream consumer of Modules) and runs *its* own test suite |
| 163 |
(``python -m test.framework.suite``) configured to use the |
| 164 |
just-built Modules as its modules tool. This is the one workflow that |
| 165 |
verifies Modules against a real downstream project rather than against its |
| 166 |
own testsuite, catching regressions the in-repo testsuite wouldn't see. |
| 167 |
|
| 168 |
Scorecard analysis (:file:`scorecard.yml`) |
| 169 |
------------------------------------------- |
| 170 |
|
| 171 |
Runs `OSSF Scorecard`_, a supply-chain security posture checker, on every |
| 172 |
push to ``main``, weekly (Saturdays at 02:30 UTC), and on manual dispatch. |
| 173 |
Results are published to the public OSSF API and uploaded as a SARIF file |
| 174 |
to GitHub's Code scanning. This is unrelated to build/test coverage; it |
| 175 |
checks repository practices (branch protection, pinned dependencies, ...). |
| 176 |
See ``.github/security-insights.yml`` for how this feeds into the project's |
| 177 |
OpenSSF Best Practices self-assessment. |
| 178 |
|
| 179 |
Fuzzing (:file:`cflite.yml`) |
| 180 |
---------------------------- |
| 181 |
|
| 182 |
Runs `ClusterFuzzLite`_, a lightweight continuous-fuzzing setup built on |
| 183 |
OSS-Fuzz's libFuzzer/sanitizer tooling, against the optional C helper |
| 184 |
library in :file:`lib/` (see ``lib/envmodules.c`` in the repository |
| 185 |
layout). This is what satisfies the `OSSF Scorecard`_ "Fuzzing" check |
| 186 |
above: for C/C++ projects Scorecard only recognizes OSS-Fuzz registration |
| 187 |
or the presence of a :file:`.clusterfuzzlite/Dockerfile`, not arbitrary |
| 188 |
libFuzzer harnesses on their own. |
| 189 |
|
| 190 |
:file:`.clusterfuzzlite/` at the repository root holds the fuzzing setup: |
| 191 |
|
| 192 |
``project.yaml``, ``Dockerfile``, ``build.sh`` |
| 193 |
Standard ClusterFuzzLite build integration files. ``build.sh`` |
| 194 |
regenerates :file:`lib/config.h` through ``lib/configure`` (with |
| 195 |
``--disable-shared --disable-stubs``, since the fuzz targets link |
| 196 |
against libtcl directly rather than through the Tcl stub mechanism |
| 197 |
the shipped extension uses) and compiles ``envmodules.c`` together |
| 198 |
with each fuzz target. |
| 199 |
``fuzz_parsedatetimearg.c``, ``fuzz_readfile.c``, ``fuzz_getfilesindirectory.c`` |
| 200 |
One target per :file:`lib/envmodules.c` entry point that parses |
| 201 |
externally-influenced input: date/time argument strings, file |
| 202 |
content (including the ``#%Module`` magic-cookie check), and |
| 203 |
directory-entry names, respectively. ``Envmodules_InitStateUsernameObjCmd``, |
| 204 |
``Envmodules_InitStateUsergroupsObjCmd`` and |
| 205 |
``Envmodules_InitStateClockSecondsObjCmd`` take no Tcl-level |
| 206 |
arguments and only query process/OS state (uid, group list, current |
| 207 |
time), so there is no fuzzer-mutable input for them and no fuzz |
| 208 |
target is provided. |
| 209 |
|
| 210 |
:file:`cflite.yml` has two jobs, gated on the triggering event: ``PR`` runs |
| 211 |
a short AddressSanitizer/UndefinedBehaviorSanitizer fuzzing pass, in |
| 212 |
``code-change`` mode (reporting only issues newly introduced by the pull |
| 213 |
request's diff), on pull requests touching :file:`lib/` or |
| 214 |
:file:`.clusterfuzzlite/`. ``BatchFuzzing`` runs the same targets for a |
| 215 |
full hour, bi-weekly and on manual dispatch, to catch issues that need a |
| 216 |
deeper corpus to reach. |
| 217 |
|
| 218 |
Coverage |
| 219 |
-------- |
| 220 |
|
| 221 |
Most build/test jobs set ``COVERAGE: y``, which (as described in |
| 222 |
:doc:`testsuite`) runs the testsuite against a Nagelfar-instrumented |
| 223 |
:file:`modulecmd.tcl` and produces marked-up :file:`tcl/*.tcl_m` files. |
| 224 |
After the testsuite runs, each job converts that coverage data |
| 225 |
(``script/nglfar2ccov`` per Tcl file, plus ``gcov`` for the optional C |
| 226 |
helper library in :file:`lib/`) and uploads it via the |
| 227 |
``codecov/codecov-action``. :file:`codecov.yml` at the repository root |
| 228 |
configures Codecov to wait for 7 builds before notifying (so the PR check |
| 229 |
reflects combined coverage from all the Linux/macOS jobs that report it, |
| 230 |
not just the first one to finish) and to fail the ``project`` coverage |
| 231 |
status if the CI run itself failed. |
| 232 |
|
| 233 |
Dependency updates |
| 234 |
------------------- |
| 235 |
|
| 236 |
:file:`.github/dependabot.yml` configures `Dependabot |
| 237 |
<https://docs.github.com/en/code-security/dependabot>`_ to check the |
| 238 |
``github-actions`` ecosystem (i.e. the ``uses:`` action references in |
| 239 |
:file:`.github/workflows/`) weekly and open pull requests to bump pinned |
| 240 |
versions, keeping the actions used by all of the above workflows current. |
| 241 |
It also checks the ``docker`` ecosystem in :file:`.clusterfuzzlite/`, |
| 242 |
keeping the digest-pinned ``FROM`` line in |
| 243 |
:file:`.clusterfuzzlite/Dockerfile` current. |
| 244 |
|
| 245 |
.. vim:set tabstop=2 shiftwidth=2 expandtab autoindent: |