| 1 |
.. -*- mode: rst-mode -*- |
| 2 |
.. |
| 3 |
.. Version number is filled in automatically. |
| 4 |
|
| 5 |
.. |version| replace:: 0.72 |
| 6 |
|
| 7 |
================================================== |
| 8 |
BTest - A Generic Driver for Powerful System Tests |
| 9 |
================================================== |
| 10 |
|
| 11 |
BTest is a powerful framework for writing system tests. Freely |
| 12 |
borrowing some ideas from other packages, its main objective is to |
| 13 |
provide an easy-to-use, straightforward driver for a suite of |
| 14 |
shell-based tests. Each test consists of a set of command lines that |
| 15 |
will be executed, and success is determined based on their exit |
| 16 |
codes. ``btest`` comes with some additional tools that can be used |
| 17 |
within such tests to robustly compare output against a previously established |
| 18 |
baseline. |
| 19 |
|
| 20 |
This document describes BTest |version|. See the ``CHANGES`` |
| 21 |
file in the source tree for version history. |
| 22 |
|
| 23 |
.. contents:: |
| 24 |
|
| 25 |
Prerequisites |
| 26 |
============= |
| 27 |
|
| 28 |
BTest has the following prerequisites: |
| 29 |
|
| 30 |
- Python version >= 3.5 (older versions may work, but are not |
| 31 |
well-tested). |
| 32 |
|
| 33 |
- Bash (note that on FreeBSD and Alpine Linux, bash is not installed |
| 34 |
by default). |
| 35 |
|
| 36 |
BTest has the following optional prerequisites to enable additional |
| 37 |
functionality: |
| 38 |
|
| 39 |
- Sphinx. |
| 40 |
|
| 41 |
- perf (Linux only). Note that on Debian/Ubuntu, you also need to install |
| 42 |
the "linux-tools" package. |
| 43 |
|
| 44 |
Download and Installation |
| 45 |
========================= |
| 46 |
|
| 47 |
Installation is simple and standard via ``pip``:: |
| 48 |
|
| 49 |
> pip install btest |
| 50 |
|
| 51 |
Alternatively, you can download a tarball `from PyPI <https://pypi.org/project/btest/#files>`_ |
| 52 |
and install locally:: |
| 53 |
|
| 54 |
> tar xzvf btest-*.tar.gz |
| 55 |
> cd btest-* |
| 56 |
> python3 setup.py install |
| 57 |
|
| 58 |
The same approach also works on a local git clone of the source tree, |
| 59 |
located at https://github.com/zeek/btest. |
| 60 |
|
| 61 |
Each will install a few scripts: ``btest`` is the main driver program, |
| 62 |
and there are a number of further helper scripts that we discuss below |
| 63 |
(including ``btest-diff``, which is a tool for comparing output to a |
| 64 |
previously established baseline). |
| 65 |
|
| 66 |
.. _running btest: |
| 67 |
|
| 68 |
Running BTest |
| 69 |
============= |
| 70 |
|
| 71 |
A BTest testsuite consists of one or more "btests", executed by the |
| 72 |
``btest`` driver. Btests are plain text files in which ``btest`` |
| 73 |
identifies keywords with corresponding arguments that tell it what to |
| 74 |
do. BTest is *not* a language; it recognizes keywords in any text |
| 75 |
file, including when embedded in other scripting languages. A common |
| 76 |
idiom in BTest is to use keywords to process the btest file via a |
| 77 |
particular command, often a script interpreter. This approach feels |
| 78 |
unusal at first, but lends BTest much of its flexibility: btest |
| 79 |
files can contain pretty much anything, as long as ``btest`` |
| 80 |
identifies keywords in it. |
| 81 |
|
| 82 |
``btest`` requires a `configuration file`_. With it, you can run |
| 83 |
``btest`` on an existing testsuite in several ways: |
| 84 |
|
| 85 |
- Point it at directories containing btests:: |
| 86 |
|
| 87 |
> btest ./testsuite/ |
| 88 |
|
| 89 |
- Use the config file to enumerate directories to scan for tests, |
| 90 |
via the ``TestDirs`` `option`_:: |
| 91 |
|
| 92 |
> btest |
| 93 |
|
| 94 |
- Run btests selectively, by pointing ``btest`` at a specific test file:: |
| 95 |
|
| 96 |
> btest ./testsuite/my.test |
| 97 |
|
| 98 |
More detail on this when we cover `test selection`_. |
| 99 |
|
| 100 |
Writing a Test |
| 101 |
============== |
| 102 |
|
| 103 |
First Steps |
| 104 |
----------- |
| 105 |
|
| 106 |
In the most simple case, ``btest`` simply executes a set of command |
| 107 |
lines, each of which must be prefixed with the ``@TEST-EXEC:`` |
| 108 |
keyword:: |
| 109 |
|
| 110 |
> cat examples/t1 |
| 111 |
@TEST-EXEC: echo "Foo" | grep -q Foo |
| 112 |
@TEST-EXEC: test -d . |
| 113 |
> btest examples/t1 |
| 114 |
examples.t1 ... ok |
| 115 |
|
| 116 |
The test passes as both command lines return success. If one of them |
| 117 |
didn't, that would be reported:: |
| 118 |
|
| 119 |
> cat examples/t2 |
| 120 |
@TEST-EXEC: echo "Foo" | grep -q Foo |
| 121 |
@TEST-EXEC: test -d DOESNOTEXIST |
| 122 |
> btest examples/t2 |
| 123 |
examples.t2 ... failed |
| 124 |
|
| 125 |
Usually you will just run all tests found in a directory:: |
| 126 |
|
| 127 |
> btest examples |
| 128 |
examples.t1 ... ok |
| 129 |
examples.t2 ... failed |
| 130 |
1 test failed |
| 131 |
|
| 132 |
The file containing the test can simultaneously act as *its input*. Let's |
| 133 |
say we want to verify a shell script:: |
| 134 |
|
| 135 |
> cat examples/t3.sh |
| 136 |
# @TEST-EXEC: sh %INPUT |
| 137 |
ls /etc | grep -q passwd |
| 138 |
> btest examples/t3.sh |
| 139 |
examples.t3 ... ok |
| 140 |
|
| 141 |
Here, ``btest`` executes (something similar to) ``sh |
| 142 |
examples/t3.sh``, and then checks the return value as usual. The |
| 143 |
example also shows that the ``@TEST-EXEC`` keyword can appear |
| 144 |
anywhere, in particular inside the comment section of another |
| 145 |
language. |
| 146 |
|
| 147 |
Using Baselines |
| 148 |
--------------- |
| 149 |
|
| 150 |
Now, let's say we want to verify the output of a program, making sure |
| 151 |
that it matches our expectations---a common use case for BTest. To do |
| 152 |
this, we rely on BTest's built-in support for test baselines. These |
| 153 |
baselines record prior output of a test, adding support for |
| 154 |
abstracting away brittle details such as ever-changing timestamps or |
| 155 |
home directories. BTest comes with tooling to establish, update, and |
| 156 |
verify baselines, and to plug in "`canonifiers`_": scripts that |
| 157 |
abstract, or "normalize", troublesome detail from a baseline. |
| 158 |
|
| 159 |
In our test, we first add a command line that produces the output we |
| 160 |
want to check, and then run ``btest-diff`` to make sure it matches the |
| 161 |
previously recorded baseline. ``btest-diff`` is itself just a script |
| 162 |
that returns success if the output matches a pre-recorded baseline |
| 163 |
after applying any required normalizations. |
| 164 |
|
| 165 |
In the following example, we use an awk script as a fancy way to print all |
| 166 |
file names starting with a dot in the user's home directory. We |
| 167 |
write that list into a file called ``dots`` and then check whether |
| 168 |
its content matches what we know from last time:: |
| 169 |
|
| 170 |
> cat examples/t4.awk |
| 171 |
# @TEST-EXEC: ls -a $HOME | awk -f %INPUT >dots |
| 172 |
# @TEST-EXEC: btest-diff dots |
| 173 |
/^\.+/ { print $1 } |
| 174 |
|
| 175 |
Note that each test gets its own little sandbox directory when run, |
| 176 |
so by creating a file like ``dots``, you aren't cluttering up |
| 177 |
anything. |
| 178 |
|
| 179 |
The first time we run this test, we need to record a baseline. The |
| 180 |
``btest`` command includes a baseline-update mode, set via ``-U``, |
| 181 |
that achieves this:: |
| 182 |
|
| 183 |
> btest -U examples/t4.awk |
| 184 |
|
| 185 |
``btest-diff`` recognizes this update mode via an environment variable |
| 186 |
set by ``btest``, and records the ``dots`` file in a separate baseline |
| 187 |
folder. With this baseline in place, modifications to the output now |
| 188 |
trigger a test failure:: |
| 189 |
|
| 190 |
> btest examples/t4.awk |
| 191 |
examples.t4 ... ok |
| 192 |
> touch ~/.NEWDOTFILE |
| 193 |
> btest examples/t4.awk |
| 194 |
examples.t4 ... failed |
| 195 |
1 test failed |
| 196 |
|
| 197 |
If we want to see what exactly changed in ``dots`` to trigger the |
| 198 |
failure, ``btest`` allows us to record the discrepancies via a |
| 199 |
*diagnostics* mode that records them in a file called ``.diag``:: |
| 200 |
|
| 201 |
> btest -d examples/t4.awk |
| 202 |
examples.t4 ... failed |
| 203 |
% 'btest-diff dots' failed unexpectedly (exit code 1) |
| 204 |
% cat .diag |
| 205 |
== File =============================== |
| 206 |
[... current dots file ...] |
| 207 |
== Diff =============================== |
| 208 |
--- /Users/robin/work/binpacpp/btest/Baseline/examples.t4/dots |
| 209 |
2010-10-28 20:11:11.000000000 -0700 |
| 210 |
+++ dots 2010-10-28 20:12:30.000000000 -0700 |
| 211 |
@@ -4,6 +4,7 @@ |
| 212 |
.CFUserTextEncoding |
| 213 |
.DS_Store |
| 214 |
.MacOSX |
| 215 |
+.NEWDOTFILE |
| 216 |
.Rhistory |
| 217 |
.Trash |
| 218 |
.Xauthority |
| 219 |
======================================= |
| 220 |
|
| 221 |
% cat .stderr |
| 222 |
[... if any of the commands had printed something to stderr, that would follow here ...] |
| 223 |
|
| 224 |
Once we delete the new file, the test passes again:: |
| 225 |
|
| 226 |
> rm ~/.NEWDOTFILE |
| 227 |
> btest -d examples/t4.awk |
| 228 |
examples.t4 ... ok |
| 229 |
|
| 230 |
That's the essence of the functionality the ``btest`` package |
| 231 |
provides. This example did not use canonifiers. We cover these, |
| 232 |
and a number of additional options that extend or modify this basic |
| 233 |
approach, in the following sections. |
| 234 |
|
| 235 |
Reference |
| 236 |
========= |
| 237 |
|
| 238 |
Command Line Usage |
| 239 |
------------------ |
| 240 |
|
| 241 |
``btest`` must be started with a list of tests and/or directories |
| 242 |
given on the command line. In the latter case, the default is to |
| 243 |
recursively scan the directories and assume all files found to be |
| 244 |
tests to perform. It is however possible to exclude specific files and |
| 245 |
directories by specifying a suitable `configuration file`_. |
| 246 |
|
| 247 |
``btest`` returns exit code 0 if all tests have successfully passed, |
| 248 |
and 1 otherwise. Exit code 1 can also result in case of other errors. |
| 249 |
|
| 250 |
``btest`` accepts the following options: |
| 251 |
|
| 252 |
-a ALTERNATIVE, --alternative=ALTERNATIVE |
| 253 |
Activates an alternative_ configuration defined in the |
| 254 |
configuration file. Multiple alternatives can be given as a |
| 255 |
comma-separated list (in this case, all specified tests are run |
| 256 |
once for each specified alternative). If ``ALTERNATIVE`` is ``-`` |
| 257 |
that refers to running with the standard setup, which can be used |
| 258 |
to run tests both with and without alternatives by giving both. |
| 259 |
|
| 260 |
-A, --show-all |
| 261 |
Shows an output line for all tests that were run (this includes tests |
| 262 |
that passed, failed, or were skipped), rather than only failed tests. |
| 263 |
Note that this option has no effect when stdout is not a TTY |
| 264 |
(because all tests are shown in that case). |
| 265 |
|
| 266 |
-b, --brief |
| 267 |
Does not output *anything* for tests which pass. If all tests |
| 268 |
pass, there will not be any output at all except final summary |
| 269 |
information. |
| 270 |
|
| 271 |
-c CONFIG, --config=CONFIG |
| 272 |
Specifies an alternative `configuration file`_ to use. If not |
| 273 |
specified, the default is to use a file called ``btest.cfg`` |
| 274 |
if found in the current directory. An alternative way to specify |
| 275 |
a different config file is with the ``BTEST_CFG`` environment |
| 276 |
variable (however, the command-line option overrides ``BTEST_CFG``). |
| 277 |
|
| 278 |
-d, --diagnostics |
| 279 |
Reports diagnostics for all failed tests. The diagnostics |
| 280 |
include the command line that failed, its output to standard |
| 281 |
error, and potential additional information recorded by the |
| 282 |
command line for diagnostic purposes (see `@TEST-EXEC`_ |
| 283 |
below). In the case of ``btest-diff``, the latter is the |
| 284 |
``diff`` between baseline and actual output. |
| 285 |
|
| 286 |
-D, --diagnostics-all |
| 287 |
Reports diagnostics for all tests, including those which pass. |
| 288 |
|
| 289 |
-f DIAGFILE, --file-diagnostics=DIAGFILE |
| 290 |
Writes diagnostics for all failed tests into the given file. |
| 291 |
If the file already exists, it will be overwritten. |
| 292 |
|
| 293 |
-g GROUPS, --groups=GROUPS |
| 294 |
Runs only tests assigned to the given test groups, see |
| 295 |
`@TEST-GROUP`_. Multiple groups can be given as a |
| 296 |
comma-separated list. Specifying groups with a leading ``-`` |
| 297 |
leads to all tests to run that are *not* not part of them. |
| 298 |
Specifying a sole ``-`` as a group name selects all tests that |
| 299 |
do not belong to any group. (Note that if you combine these |
| 300 |
variants to create ambiguous situations, it's left |
| 301 |
undefined which tests will end up running). |
| 302 |
|
| 303 |
-j THREADS, --jobs=THREADS |
| 304 |
Runs up to the given number of tests in parallel. If no number |
| 305 |
is given, BTest substitutes the number of available CPU cores |
| 306 |
as reported by the OS. |
| 307 |
|
| 308 |
By default, BTest assumes that all tests can be executed |
| 309 |
concurrently without further constraints. One can however |
| 310 |
ensure serialization of subsets by assigning them to the same |
| 311 |
serialization set, see `@TEST-SERIALIZE`_. |
| 312 |
|
| 313 |
-q, --quiet |
| 314 |
Suppress information output other than about failed tests. |
| 315 |
If all tests pass, there will not be any output at all. |
| 316 |
|
| 317 |
-r, --rerun |
| 318 |
Runs only tests that failed last time. After each execution |
| 319 |
(except when updating baselines), BTest generates a state file |
| 320 |
that records the tests that have failed. Using this option on |
| 321 |
the next run then reads that file back in and limits execution |
| 322 |
to those tests found in there. |
| 323 |
|
| 324 |
-R FORMAT, --documentation=FORMAT |
| 325 |
Generates a reference of all tests and prints that to standard |
| 326 |
output. The output can be of two types, specified by |
| 327 |
``FORMAT``: ``rst`` prints reStructuredText, and ``md`` prints |
| 328 |
Markdown. In the output each test includes the documentation |
| 329 |
string that's defined for it through ``@TEST-DOC``. |
| 330 |
|
| 331 |
-t, --tmp-keep |
| 332 |
Does not delete any temporary files created for running the |
| 333 |
tests (including their outputs). By default, the temporary |
| 334 |
files for a test will be located in ``.tmp/<test>/``, where |
| 335 |
``<test>`` is the relative path of the test file with all slashes |
| 336 |
replaced with dots and the file extension removed (e.g., the files |
| 337 |
for ``example/t3.sh`` will be in ``.tmp/example.t3``). |
| 338 |
|
| 339 |
-T, --update-times |
| 340 |
Record new `timing`_ baselines for the current host for tests that |
| 341 |
have `@TEST-MEASURE-TIME`_. Tests are run as normal except that |
| 342 |
the timing measurements are recorded as the new baseline instead |
| 343 |
of being compared to a previous baseline. |
| 344 |
|
| 345 |
--trace-file=TRACEFILE |
| 346 |
Record test execution timings in Chrome tracing format to the given |
| 347 |
file. If the file exists already, it is overwritten. The file can be |
| 348 |
loaded in Chrome-based browsers at `<about:tracing>`_, or converted to |
| 349 |
standalone HTML with `trace2html <https://pypi.org/project/trace2html/>`_. |
| 350 |
|
| 351 |
-U, --update-baseline |
| 352 |
Records a new baseline for all ``btest-diff`` commands found |
| 353 |
in any of the specified tests. To do this, all tests are run |
| 354 |
as normal except that when ``btest-diff`` is executed, it |
| 355 |
does not compute a diff but instead considers the given file |
| 356 |
to be authoritative and records it as the version to compare |
| 357 |
with in future runs. |
| 358 |
|
| 359 |
-u, --update-interactive |
| 360 |
Each time a ``btest-diff`` command fails in any tests that are |
| 361 |
run, ``btest`` will stop and ask whether or not the user wants to |
| 362 |
record a new baseline. |
| 363 |
|
| 364 |
-v, --verbose |
| 365 |
Shows all test command lines as they are executed. |
| 366 |
|
| 367 |
-w, --wait |
| 368 |
Interactively waits for ``<enter>`` after showing diagnostics |
| 369 |
for a test. |
| 370 |
|
| 371 |
-x FILE, --xml=FILE |
| 372 |
Records test results in JUnit XML format to the given file. |
| 373 |
If the file exists already, it is overwritten. |
| 374 |
|
| 375 |
-z RETRIES, --retries=RETRIES |
| 376 |
Retry any failed tests up to this many times to determine if |
| 377 |
they are unstable. |
| 378 |
|
| 379 |
.. _configuration file: configuration_ |
| 380 |
.. _configuration: |
| 381 |
|
| 382 |
Configuration |
| 383 |
------------- |
| 384 |
|
| 385 |
Specifics of ``btest``'s execution can be tuned with a configuration |
| 386 |
file, which by default is ``btest.cfg`` if that's found in the |
| 387 |
current directory. It can alternatively be specified with the |
| 388 |
``--config`` command line option, or a ``BTEST_CFG`` environment |
| 389 |
variable. The configuration file is |
| 390 |
"INI-style", and an example comes with the distribution, see |
| 391 |
``btest.cfg.example``. A configuration file has one main section, |
| 392 |
``btest``, that defines most options; as well as an optional section |
| 393 |
for defining `environment variables`_ and further optional sections |
| 394 |
for defining alternatives_. |
| 395 |
|
| 396 |
Note that all paths specified in the configuration file are relative |
| 397 |
to ``btest``'s *base directory*. The base directory is either the |
| 398 |
one where the configuration file is located if such is given/found, |
| 399 |
or the current working directory if not. One can also override it |
| 400 |
explicitly by setting the environment variable ``BTEST_TEST_BASE``. |
| 401 |
When setting values for |
| 402 |
configuration options, the absolute path to the base directory is |
| 403 |
available by using the macro ``%(testbase)s`` (the weird syntax is |
| 404 |
due to Python's ``ConfigParser`` class). |
| 405 |
|
| 406 |
Furthermore, all values can use standard "backtick-syntax" to |
| 407 |
include the output of external commands (e.g., xyz=`\echo test\`). |
| 408 |
Note that the backtick expansion is performed after any ``%(..)`` |
| 409 |
have already been replaced (including within the backticks). |
| 410 |
|
| 411 |
.. _option: `options`_ |
| 412 |
.. _options: |
| 413 |
|
| 414 |
Options |
| 415 |
~~~~~~~ |
| 416 |
|
| 417 |
The following options can be set in the ``btest`` section of the |
| 418 |
configuration file: |
| 419 |
|
| 420 |
``BaselineDir`` |
| 421 |
One or more directories where to store the baseline files for |
| 422 |
``btest-diff`` (note that the actual baseline files will be placed |
| 423 |
into test-specific subdirectories of this directory). By default, |
| 424 |
this is set to ``%(testbase)s/Baseline``. |
| 425 |
|
| 426 |
If multiple directories are to be used, they must be separated by |
| 427 |
colons. ``btest-diff`` will then search them for baseline files in |
| 428 |
order when looking for a baseline to compare against. When |
| 429 |
updating a baseline, it will always store the new version inside |
| 430 |
the first directory. Using multiple directories is most useful in |
| 431 |
combination with alternatives_ to support alternate executions |
| 432 |
where some tests produce expected differences in their output. |
| 433 |
|
| 434 |
This option can also be set through an environment variable |
| 435 |
``BTEST_BASELINE_DIR``. |
| 436 |
|
| 437 |
``CommandPrefix`` |
| 438 |
Changes the naming of all ``btest`` commands by replacing the |
| 439 |
``@TEST-`` prefix with a custom string. For example, with |
| 440 |
``CommandPrefix=$TEST-``, the ``@TEST-EXEC`` command becomes |
| 441 |
``$TEST-EXEC``. |
| 442 |
|
| 443 |
``Finalizer`` |
| 444 |
A command that will be executed each time any test has |
| 445 |
successfully run. It runs in the same directory as the test itself |
| 446 |
and receives the name of the test as its only argument. The return |
| 447 |
value indicates whether the test should indeed be considered |
| 448 |
successful. By default, there's no finalizer set. |
| 449 |
|
| 450 |
``IgnoreDirs`` |
| 451 |
A space-separated list of relative directory names to ignore |
| 452 |
when scanning test directories recursively. Default is empty. |
| 453 |
|
| 454 |
An alternative way to ignore a directory is placing a file |
| 455 |
``.btest-ignore`` in it. |
| 456 |
|
| 457 |
``IgnoreFiles`` |
| 458 |
A space-separated list of filename globs matching files to |
| 459 |
ignore when scanning given test directories recursively. |
| 460 |
Default is empty. |
| 461 |
|
| 462 |
An alternative way to ignore a file is by placing ``@TEST-IGNORE`` |
| 463 |
in it. |
| 464 |
|
| 465 |
``Initializer`` |
| 466 |
A command that will be executed before each test. It runs in |
| 467 |
the same directory as the test itself will and receives the name |
| 468 |
of the test as its only argument. The return value indicates whether |
| 469 |
the test should continue; if false, the test will be considered |
| 470 |
failed. By default, there's no initializer set. |
| 471 |
|
| 472 |
``MinVersion`` |
| 473 |
On occasion, you'll want to ensure that the version of ``btest`` |
| 474 |
running your testsuite includes a particular feature. By setting |
| 475 |
this value to a given version number (as reported by ``btest |
| 476 |
--version``), ``btest`` installations older than this version will |
| 477 |
fail test execution with exit code 1 and a corresponding error |
| 478 |
message on stderr. |
| 479 |
|
| 480 |
``PartFinalizer`` |
| 481 |
A command that will be executed each time a test *part* has |
| 482 |
successfully run. This operates similarly to ``Finalizer`` except |
| 483 |
that it runs after each test part rather than only at completion |
| 484 |
of the full test. See `parts`_ for more about test parts. |
| 485 |
|
| 486 |
``PartInitializer`` |
| 487 |
A command that will be executed before each test *part*. This operates |
| 488 |
similarly to ``Initializer`` except that it runs at the beginning of any |
| 489 |
test part that BTest runs. See `parts`_ for more about test parts. |
| 490 |
|
| 491 |
Since a failing test part aborts execution of the test, part initializers do |
| 492 |
not run for any subsequent skipped parts. |
| 493 |
|
| 494 |
``PartTeardown`` |
| 495 |
A command that will run after any test *part* that has run, regardless |
| 496 |
of failure or success of the part. This operates similarly to ``Teardown`` |
| 497 |
except it applies to test `parts`_ instead of the full test. |
| 498 |
|
| 499 |
Since a failing test part aborts execution of the test, part teardowns do |
| 500 |
not run for any subsequent skipped parts. |
| 501 |
|
| 502 |
``PerfPath`` |
| 503 |
Specifies a path to the ``perf`` tool, which is used on Linux to |
| 504 |
measure the execution times of tests. By default, BTest searches |
| 505 |
for ``perf`` in ``PATH``. |
| 506 |
|
| 507 |
``PortRange`` |
| 508 |
Specifies a port range like "10000-11000" to use in conjunction with |
| 509 |
``@TEST-PORT`` commands. Port assignments will be restricted to this |
| 510 |
range. The default range is "1024-65535". |
| 511 |
|
| 512 |
``StateFile`` |
| 513 |
The name of the state file to record the names of failing tests. Default is |
| 514 |
``.btest.failed.dat``. |
| 515 |
|
| 516 |
``Teardown`` |
| 517 |
A command that will be executed each time any test has run, regardless of |
| 518 |
whether that test succeeded. Conceptually, it pairs with an ``Initializer`` |
| 519 |
that sets up test infrastructure that requires tear-down at the end of the |
| 520 |
test. It runs in the same directory as the test itself and receives the name |
| 521 |
of the test as its only argument. There's no default teardown command. |
| 522 |
|
| 523 |
Teardown commands may return a non-zero exit code, which fails the |
| 524 |
corresponding test. Succeeding teardown commands do not override an |
| 525 |
otherwise failing test; such tests will still fail. |
| 526 |
|
| 527 |
To allow teardown routines to reason about the preceding tests, they |
| 528 |
receive two additional environment variables: |
| 529 |
|
| 530 |
``TEST_FAILED`` |
| 531 |
This variable is defined (to 1) when the test has failed, and absent |
| 532 |
otherwise. |
| 533 |
|
| 534 |
``TEST_LAST_RETCODE`` |
| 535 |
This variable contains the numeric exit code of the last command |
| 536 |
run prior to teardown. |
| 537 |
|
| 538 |
``TestDirs`` |
| 539 |
A space-separated list of directories to search for tests. If |
| 540 |
defined, one doesn't need to specify any tests on the command |
| 541 |
line. |
| 542 |
|
| 543 |
``TimingBaselineDir`` |
| 544 |
A directory where to store the host-specific `timing`_ baseline |
| 545 |
files. By default, this is set to |
| 546 |
``%(testbase)s/Baseline/_Timing``. |
| 547 |
|
| 548 |
``TimingDeltaPerc`` |
| 549 |
A value defining the `timing`_ deviation percentage that's tolerated |
| 550 |
for a test before it's considered failed. Default is 1.0 (which means |
| 551 |
a 1.0% deviation is tolerated by default). |
| 552 |
|
| 553 |
``TmpDir`` |
| 554 |
A directory where to create temporary files when running tests. |
| 555 |
By default, this is set to ``%(testbase)s/.tmp``. |
| 556 |
|
| 557 |
.. _environment variables: |
| 558 |
|
| 559 |
Environment Variables |
| 560 |
~~~~~~~~~~~~~~~~~~~~~ |
| 561 |
|
| 562 |
A special section ``environment`` defines environment variables that |
| 563 |
will be propagated to all tests:: |
| 564 |
|
| 565 |
[environment] |
| 566 |
CFLAGS=-O3 |
| 567 |
PATH=%(testbase)s/bin:%(default_path)s |
| 568 |
|
| 569 |
Note how ``PATH`` can be adjusted to include local scripts: the |
| 570 |
example above prefixes it with a local ``bin/`` directory inside the |
| 571 |
base directory, using the predefined ``default_path`` macro to refer |
| 572 |
to the ``PATH`` as it is set by default. |
| 573 |
|
| 574 |
Furthermore, by setting ``PATH`` to include the ``btest`` |
| 575 |
distribution directory, one could skip the installation of the |
| 576 |
``btest`` package. |
| 577 |
|
| 578 |
.. _alternative: alternatives_ |
| 579 |
.. _alternatives: |
| 580 |
|
| 581 |
Alternatives |
| 582 |
~~~~~~~~~~~~ |
| 583 |
|
| 584 |
BTest can run a set of tests with different settings than it would |
| 585 |
normally use by specifying an *alternative* configuration. Currently, |
| 586 |
three things can be adjusted: |
| 587 |
|
| 588 |
- Further environment variables can be set that will then be |
| 589 |
available to all the commands that a test executes. |
| 590 |
|
| 591 |
- *Filters* can modify an input file before a test uses it. |
| 592 |
|
| 593 |
- *Substitutions* can modify command lines executed as part of a |
| 594 |
test. |
| 595 |
|
| 596 |
We discuss the three separately in the following. All of them are |
| 597 |
defined by adding sections ``[<type>-<name>]`` where ``<type>`` |
| 598 |
corresponds to the type of adjustment being made and ``<name>`` is the |
| 599 |
name of the alternative. Once at least one section is defined for a |
| 600 |
name, that alternative can be enabled by BTest's ``--alternative`` |
| 601 |
flag. |
| 602 |
|
| 603 |
Environment Variables |
| 604 |
^^^^^^^^^^^^^^^^^^^^^ |
| 605 |
|
| 606 |
An alternative can add further environment variables by defining an |
| 607 |
``[environment-<name>]`` section:: |
| 608 |
|
| 609 |
[environment-myalternative] |
| 610 |
CFLAGS=-O3 |
| 611 |
|
| 612 |
Running ``btest`` with ``--alternative=myalternative`` will now make |
| 613 |
the ``CFLAGS`` environment variable available to all commands |
| 614 |
executed. |
| 615 |
|
| 616 |
As a special case, one can override two specific environment |
| 617 |
variables---``BTEST_TEST_BASE`` and ``BTEST_BASELINE_DIR``---inside an |
| 618 |
alternative's environment section to have them not only be passed on |
| 619 |
to child processes, but also apply to the ``btest`` process itself. |
| 620 |
That way, one can switch to a different base and baseline directories |
| 621 |
for an alternative. |
| 622 |
|
| 623 |
.. _filters: |
| 624 |
|
| 625 |
Filters |
| 626 |
^^^^^^^ |
| 627 |
|
| 628 |
Filters are a transparent way to adapt the input to a specific test |
| 629 |
command before it is executed. A filter is defined by adding a section |
| 630 |
``[filter-<name>]`` to the configuration file. This section must have |
| 631 |
exactly one entry, and the name of that entry is interpreted as the |
| 632 |
name of a command whose input is to be filtered. The value of that |
| 633 |
entry is the name of a filter script that will be run with two |
| 634 |
arguments representing input and output files, respectively. Example:: |
| 635 |
|
| 636 |
[filter-myalternative] |
| 637 |
cat=%(testbase)s/bin/filter-cat |
| 638 |
|
| 639 |
Once the filter is activated by running ``btest`` with |
| 640 |
``--alternative=myalternative``, every time a ``@TEST-EXEC: cat |
| 641 |
%INPUT`` is found, ``btest`` will first execute (something similar to) |
| 642 |
``%(testbase)s/bin/filter-cat %INPUT out.tmp``, and then subsequently |
| 643 |
``cat out.tmp`` (i.e., the original command but with the filtered |
| 644 |
output). In the simplest case, the filter could be a no-op in the |
| 645 |
form ``cp $1 $2``. |
| 646 |
|
| 647 |
|
| 648 |
**NOTE:** There are a few limitations to the filter concept currently: |
| 649 |
|
| 650 |
* Filters are *always* fed with ``%INPUT`` as their first |
| 651 |
argument. We should add a way to filter other files as well. |
| 652 |
|
| 653 |
* Filtered commands are only recognized if they are directly |
| 654 |
starting the command line. For example, ``@TEST-EXEC: ls | cat |
| 655 |
>output`` would not trigger the example filter above. |
| 656 |
|
| 657 |
* Filters are only executed for ``@TEST-EXEC``, not for |
| 658 |
``@TEST-EXEC-FAIL``. |
| 659 |
|
| 660 |
.. _substitution: |
| 661 |
|
| 662 |
Substitutions |
| 663 |
^^^^^^^^^^^^^ |
| 664 |
|
| 665 |
Substitutions are similar to filters, yet they do not adapt the input |
| 666 |
but the command line being executed. A substitution is defined by |
| 667 |
adding a section ``[substitution-<name>]`` to the configuration file. |
| 668 |
For each entry in this section, the entry's name specifies the |
| 669 |
command that is to be replaced with something else given as its value. |
| 670 |
Example:: |
| 671 |
|
| 672 |
[substitution-myalternative] |
| 673 |
gcc=gcc -O2 |
| 674 |
|
| 675 |
Once the substitution is activated by running ``btest`` with |
| 676 |
``--alternative=myalternative``, every time a ``@TEST-EXEC`` executes |
| 677 |
``gcc``, that is replaced with ``gcc -O2``. The replacement is simple |
| 678 |
string substitution so it works not only with commands but anything |
| 679 |
found on the command line; it however only replaces full words, not |
| 680 |
subparts of words. |
| 681 |
|
| 682 |
Supported Keywords |
| 683 |
------------------ |
| 684 |
|
| 685 |
``btest`` scans a test file for lines containing keywords that |
| 686 |
trigger certain functionality. It knows the following keywords: |
| 687 |
|
| 688 |
``@TEST-ALTERNATIVE: <alternative>`` |
| 689 |
Runs this test only for the given alternative (see alternative_). |
| 690 |
If ``<alternative>`` is ``default``, the test executes when BTest runs |
| 691 |
with no alternative given (which however is the default anyway). |
| 692 |
|
| 693 |
``@TEST-COPY-FILE: <file>`` |
| 694 |
Copy the given file into the test's directory before the test is |
| 695 |
run. If ``<file>`` is a relative path, it's interpreted relative |
| 696 |
to the BTest's base directory. Environment variables in ``<file>`` |
| 697 |
will be replaced if enclosed in ``${..}``. This command can be |
| 698 |
given multiple times. |
| 699 |
|
| 700 |
``@TEST-DOC: <docstring>`` |
| 701 |
Associates a documentation string with the test. These strings |
| 702 |
get included into the output of the ``--documentation`` option. |
| 703 |
|
| 704 |
.. _@TEST-EXEC: |
| 705 |
|
| 706 |
``@TEST-EXEC: <cmdline>`` |
| 707 |
Executes the given command line and aborts the test if it |
| 708 |
returns an error code other than zero. The ``<cmdline>`` is |
| 709 |
passed to the shell and thus can be a pipeline, use redirection, |
| 710 |
and any environment variables specified in ``<cmdline>`` will be |
| 711 |
expanded, etc. |
| 712 |
|
| 713 |
When running a test, the current working directory for all |
| 714 |
command lines will be set to a temporary sandbox (and will be |
| 715 |
deleted later). |
| 716 |
|
| 717 |
There are two macros that can be used in ``<cmdline>``: |
| 718 |
``%INPUT`` will be replaced with the full pathname of the file defining |
| 719 |
the test (this file is in a temporary sandbox directory and is a copy |
| 720 |
of the original test file); and ``%DIR`` will be replaced with the full |
| 721 |
pathname of the directory where the test file is located (note that |
| 722 |
this is the directory where the original test file is located, not |
| 723 |
the directory where the ``%INPUT`` file is located). The latter can |
| 724 |
be used to reference further files also located there. |
| 725 |
|
| 726 |
In addition to environment variables defined in the |
| 727 |
configuration file, there are further ones that are passed into |
| 728 |
the commands: |
| 729 |
|
| 730 |
``TEST_BASE`` |
| 731 |
The BTest base directory, i.e., the directory where |
| 732 |
``btest.cfg`` is located. |
| 733 |
|
| 734 |
``TEST_BASELINE`` |
| 735 |
A list of directories where the command can save permanent |
| 736 |
information across ``btest`` runs. (This is where |
| 737 |
``btest-diff`` stores its baseline in ``UPDATE`` mode.) |
| 738 |
|
| 739 |
Multiple entries are separated by colons. If more than one |
| 740 |
entry is given, semantics should be to search them in order. |
| 741 |
(This is where ``btest-diff`` stores its baseline in |
| 742 |
``UPDATE`` mode.) |
| 743 |
|
| 744 |
``TEST_DIAGNOSTICS`` |
| 745 |
A file where further diagnostic information can be saved |
| 746 |
in case a command fails (this is also where ``btest-diff`` |
| 747 |
stores its diff). If this file exists, then the |
| 748 |
``--diagnostics-all`` or ``--diagnostics`` options will show |
| 749 |
this file (for the latter option, only if a command fails). |
| 750 |
|
| 751 |
``TEST_MODE`` |
| 752 |
This is normally set to ``TEST``, but will be ``UPDATE`` |
| 753 |
if ``btest`` is run with ``--update-baseline``, or |
| 754 |
``UPDATE_INTERACTIVE`` if run with ``--update-interactive``. |
| 755 |
|
| 756 |
``TEST_NAME`` |
| 757 |
The name of the currently executing test. |
| 758 |
|
| 759 |
``TEST_PART`` |
| 760 |
The test part number (see `parts`_ for more about test parts). |
| 761 |
|
| 762 |
**NOTE:** |
| 763 |
|
| 764 |
If a command returns the special exit code 100, the test is |
| 765 |
considered failed, however subsequent test commands within the |
| 766 |
current test are still run. ``btest-diff`` uses this special |
| 767 |
exit code to indicate that no baseline has yet been established. |
| 768 |
|
| 769 |
If a command returns the special exit code 200, the test is |
| 770 |
considered failed and all further tests are aborted. |
| 771 |
``btest-diff`` uses this special exit code when ``btest`` is run |
| 772 |
with the ``--update-interactive`` option and the user chooses to |
| 773 |
abort the tests when prompted to record a new baseline. |
| 774 |
|
| 775 |
``TEST_VERBOSE`` |
| 776 |
The path of a file where the test can record further |
| 777 |
information about its execution that will be included with |
| 778 |
BTest's ``--verbose`` output. This is for further tracking |
| 779 |
the execution of commands and should generally generate |
| 780 |
output that follows a line-based structure. |
| 781 |
|
| 782 |
|
| 783 |
``@TEST-EXEC-FAIL: <cmdline>`` |
| 784 |
Like ``@TEST-EXEC``, except that this expects the command to |
| 785 |
*fail*, i.e., the test is aborted when the return code is zero. |
| 786 |
|
| 787 |
.. _@TEST-GROUP: |
| 788 |
|
| 789 |
``@TEST-GROUP: <group>`` |
| 790 |
Assigns the test to a group of name ``<group>``. By using option |
| 791 |
``-g`` one can limit execution to all tests that belong to a given |
| 792 |
group (or a set of groups). |
| 793 |
|
| 794 |
``@TEST-IGNORE`` |
| 795 |
This is used to indicate that this file should be skipped (i.e., no |
| 796 |
test commands in this file will be executed). An alternative way to |
| 797 |
ignore files is by using the ``IgnoreFiles`` option in the btest |
| 798 |
configuration file. |
| 799 |
|
| 800 |
``@TEST-KNOWN-FAILURE`` |
| 801 |
Marks a test as known to currently fail. This only changes BTest's |
| 802 |
output, which upon failure will indicate that that is expected; it |
| 803 |
won't change the test's processing otherwise. The keyword doesn't |
| 804 |
take any arguments but one could add a descriptive text, as in :: |
| 805 |
|
| 806 |
.. @TEST-KNOWN-FAILURE: We know this fails because .... |
| 807 |
|
| 808 |
.. _@TEST-MEASURE-TIME: |
| 809 |
|
| 810 |
``@TEST-MEASURE-TIME`` |
| 811 |
Measures execution time for this test and compares it to a |
| 812 |
previously established `timing`_ baseline. If it deviates significantly, |
| 813 |
the test will be considered failed. |
| 814 |
|
| 815 |
``@TEST-NOT-ALTERNATIVE: <alternative>`` |
| 816 |
Ignores this test for the given alternative (see alternative_). |
| 817 |
If ``<alternative>`` is ``default``, the test is ignored if BTest runs |
| 818 |
with no alternative given. |
| 819 |
|
| 820 |
.. _@TEST-PORT: |
| 821 |
|
| 822 |
``@TEST-PORT: <env>`` |
| 823 |
Assign an available TCP port number to an environment variable |
| 824 |
that is accessible from the running test process. ``<env>`` is an |
| 825 |
arbitrary user-chosen string that will be set to the next available |
| 826 |
TCP port number. Availability is based on checking successful |
| 827 |
binding of the port on IPv4 INADDR_ANY and also restricted to the |
| 828 |
range specified by the ``PortRange`` option. IPv6 is not supported. |
| 829 |
Note that using the ``-j`` option to parallelize execution will |
| 830 |
work such that unique/available port numbers are assigned between |
| 831 |
concurrent tests, however there is still a potential race condition |
| 832 |
for external processes to claim a port before the test actually |
| 833 |
runs and claims it for itself. |
| 834 |
|
| 835 |
``@TEST-REQUIRES: <cmdline>`` |
| 836 |
Defines a condition that must be met for the test to be executed. |
| 837 |
The given command line will be run before any of the actual test |
| 838 |
commands, and it must return success for the test to continue. If |
| 839 |
it does not return success, the rest of the test will be skipped |
| 840 |
but doing so will not be considered a failure of the test. This allows to |
| 841 |
write conditional tests that may not always make sense to run, depending |
| 842 |
on whether external constraints are satisfied or not (say, whether |
| 843 |
a particular library is available). Multiple requirements may be |
| 844 |
specified and then all must be met for the test to continue. |
| 845 |
|
| 846 |
.. _@TEST-SERIALIZE: |
| 847 |
|
| 848 |
``@TEST-SERIALIZE: <set>`` |
| 849 |
When using option ``-j`` to parallelize execution, all tests that |
| 850 |
specify the same serialization set are guaranteed to run |
| 851 |
sequentially. ``<set>`` is an arbitrary user-chosen string. |
| 852 |
|
| 853 |
``@TEST-START-FILE <file>`` |
| 854 |
This is used to include an additional input file for a test |
| 855 |
right inside the test file. All lines following the keyword line |
| 856 |
will be written into the given file until a line containing |
| 857 |
``@TEST-END-FILE`` is found. The lines containing ``@TEST-START-FILE`` |
| 858 |
and ``@TEST-END-FILE``, and all lines in between, will be removed from |
| 859 |
the test's %INPUT. Example:: |
| 860 |
|
| 861 |
> cat examples/t6.sh |
| 862 |
# @TEST-EXEC: awk -f %INPUT <foo.dat >output |
| 863 |
# @TEST-EXEC: btest-diff output |
| 864 |
|
| 865 |
{ lines += 1; } |
| 866 |
END { print lines; } |
| 867 |
|
| 868 |
@TEST-START-FILE foo.dat |
| 869 |
1 |
| 870 |
2 |
| 871 |
3 |
| 872 |
@TEST-END-FILE |
| 873 |
|
| 874 |
> btest -D examples/t6.sh |
| 875 |
examples.t6 ... ok |
| 876 |
% cat .diag |
| 877 |
== File =============================== |
| 878 |
3 |
| 879 |
|
| 880 |
Multiple such files can be defined within a single test. |
| 881 |
|
| 882 |
Note that this is only one way to use further input files. |
| 883 |
Another is to store a file in the same directory as the test |
| 884 |
itself, making sure it's ignored via ``IgnoreFiles``, and then |
| 885 |
refer to it via ``%DIR/<name>``. |
| 886 |
|
| 887 |
``@TEST-START-NEXT`` |
| 888 |
This keyword lets you define multiple test inputs in the |
| 889 |
same file, all executing with the same command lines. See |
| 890 |
`defining multiple tests in one file`_ for details. |
| 891 |
|
| 892 |
.. _test selection: `selecting tests`_ |
| 893 |
.. _selecting tests: |
| 894 |
|
| 895 |
Selecting Tests |
| 896 |
=============== |
| 897 |
|
| 898 |
Internally, ``btest`` uses logical names for tests, abstracting input |
| 899 |
files. Those names result from substituting path separators with dots, |
| 900 |
ignoring btest file suffixes, and potentially adding additional |
| 901 |
labeling. ``btest`` does this only for tests within the ``TestDirs`` |
| 902 |
directories given in the `configuration file`. |
| 903 |
|
| 904 |
In addition to the invocations covered in `Running BTest`_, you can |
| 905 |
use logical names when telling ``btest`` which tests to run. For |
| 906 |
example, instead of saying :: |
| 907 |
|
| 908 |
> btest testsuite/foo.sh |
| 909 |
|
| 910 |
you can use:: |
| 911 |
|
| 912 |
> btest testsuite.foo |
| 913 |
|
| 914 |
This distinction rarely matters, but it's something to be aware of |
| 915 |
when `defining multiple tests in one file`_, which we cover next. |
| 916 |
|
| 917 |
.. _more than one test: `defining multiple tests in one file`_ |
| 918 |
.. _defining multiple tests in one file: |
| 919 |
|
| 920 |
Defining Multiple Tests in one File |
| 921 |
=================================== |
| 922 |
|
| 923 |
On occasion you want to use the same constellation of keywords on a |
| 924 |
set of input files. BTest supports this via the ``@TEST-START-NEXT`` |
| 925 |
keyword. When ``btest`` encounters this keyword, it initially |
| 926 |
considers the input file to end at that point, and runs all |
| 927 |
``@TEST-EXEC-*`` with an ``%INPUT`` truncated accordingly. |
| 928 |
Afterwards, it creates a *new* ``%INPUT`` with everything *following* |
| 929 |
the ``@TEST-START-NEXT`` marker, running the *same* commands |
| 930 |
again. (It ignores any ``@TEST-EXEC-*`` lines later in the file.) |
| 931 |
|
| 932 |
The effect is that a single file can define multiple tests that the |
| 933 |
``btest`` output will enumerate:: |
| 934 |
|
| 935 |
> cat examples/t5.sh |
| 936 |
# @TEST-EXEC: cat %INPUT | wc -c >output |
| 937 |
# @TEST-EXEC: btest-diff output |
| 938 |
|
| 939 |
This is the first test input in this file. |
| 940 |
|
| 941 |
# @TEST-START-NEXT |
| 942 |
|
| 943 |
... and the second. |
| 944 |
|
| 945 |
> ./btest -D examples/t5.sh |
| 946 |
examples.t5 ... ok |
| 947 |
% cat .diag |
| 948 |
== File =============================== |
| 949 |
119 |
| 950 |
[...] |
| 951 |
|
| 952 |
examples.t5-2 ... ok |
| 953 |
% cat .diag |
| 954 |
== File =============================== |
| 955 |
22 |
| 956 |
[...] |
| 957 |
|
| 958 |
``btest`` automatically generates the ``-<n>`` suffix for each of the tests. |
| 959 |
|
| 960 |
**NOTE:** It matters how you name tests when running them |
| 961 |
individually. When you specify the btest file ("``examples/t5.sh``"), |
| 962 |
``btest`` will run all of the contained tests. When you use the |
| 963 |
logical name, ``btest`` will run only that specific test: in the |
| 964 |
above scenario, ``examples.t5`` runs only the first test defined |
| 965 |
in the file, while ``examples.t5-2`` only runs the second. This |
| 966 |
also applies to baseline updates. |
| 967 |
|
| 968 |
.. _parts: `splitting tests into parts`_ |
| 969 |
.. _splitting tests into parts: |
| 970 |
|
| 971 |
Splitting Tests into Parts |
| 972 |
========================== |
| 973 |
|
| 974 |
One can also split a single test across multiple files by adding a |
| 975 |
numerical ``#<n>`` postfix to their names, where each ``<n>`` |
| 976 |
represents a separate part of the test. ``btest`` will combine all of |
| 977 |
a test's parts in numerical order and execute them subsequently within |
| 978 |
the same sandbox. Example:: |
| 979 |
|
| 980 |
> cat examples/t7.sh#1 |
| 981 |
# @TEST-EXEC: echo Part 1 - %INPUT >>output |
| 982 |
|
| 983 |
> cat examples/t7.sh#2 |
| 984 |
# @TEST-EXEC: echo Part 2 - %INPUT >>output |
| 985 |
|
| 986 |
> cat examples/t7.sh#3 |
| 987 |
# @TEST-EXEC: btest-diff output |
| 988 |
|
| 989 |
> btest -D examples/t7.sh |
| 990 |
examples.t7 ... ok |
| 991 |
% cat .diag |
| 992 |
== File =============================== |
| 993 |
Part 1 - /Users/robin/bro/docs/aux/btest/.tmp/examples.t7/t7.sh#1 |
| 994 |
Part 2 - /Users/robin/bro/docs/aux/btest/.tmp/examples.t7/t7.sh#2 |
| 995 |
|
| 996 |
Note how ``output`` contains the output of both ``t7.sh#1`` and ``t7.sh#2``, |
| 997 |
however in each case ``%INPUT`` refers to the corresponding part. For |
| 998 |
the first part of a test, one can also omit the ``#1`` postfix in the filename. |
| 999 |
|
| 1000 |
.. _canonifiers: `canonifying diffs`_ |
| 1001 |
.. _canonifying diffs: |
| 1002 |
|
| 1003 |
Canonifying Diffs |
| 1004 |
================= |
| 1005 |
|
| 1006 |
``btest-diff`` has the capability to filter its input through an |
| 1007 |
additional script before it compares the current version with the |
| 1008 |
baseline. This can be useful if certain elements in an output are |
| 1009 |
*expected* to change (e.g., timestamps). The filter can then |
| 1010 |
remove/replace these with something consistent. To enable such |
| 1011 |
canonification, set the environment variable |
| 1012 |
``TEST_DIFF_CANONIFIER`` to a script reading the original version |
| 1013 |
from stdin and writing the canonified version to stdout. |
| 1014 |
For examples of canonifier scripts, take a look at those `used in the |
| 1015 |
Zeek distribution <https://github.com/zeek/zeek/tree/master/testing/scripts/>`_. |
| 1016 |
|
| 1017 |
**NOTE:** ``btest-diff`` passes both the pre-recorded baseline and |
| 1018 |
the fresh test output through any canonifiers before comparing |
| 1019 |
their contents. BTest version 0.63 introduced two changes in |
| 1020 |
``btest-diff``'s baseline handling: |
| 1021 |
|
| 1022 |
* ``btest-diff`` now records baselines in canonicalized form. The |
| 1023 |
benefit here is that by canonicalizing upon recording, you can |
| 1024 |
use ``btest -U`` more freely, keeping expected noise out of |
| 1025 |
revision control. The downside is that updates to canonifiers |
| 1026 |
require a refresh of the baselines. |
| 1027 |
|
| 1028 |
* ``btest-diff`` now prefixes the baselines with a header that |
| 1029 |
warns against manual modification, and knows to exclude that |
| 1030 |
header from comparison. We recommend only ever updating |
| 1031 |
baselines via ``btest -U`` (or its interactive sibling, ``-u``). |
| 1032 |
|
| 1033 |
Once you use canonicalized baselines in your project, it's a good |
| 1034 |
idea to use ``MinVersion = 0.63`` in your btest.cfg to avoid the |
| 1035 |
use of older ``btest`` installations. Since these are unaware of |
| 1036 |
the new baseline header and repeated application of canonifiers |
| 1037 |
may cause unexpected alterations to already-canonified baselines, |
| 1038 |
using such versions will likely cause test failures. |
| 1039 |
|
| 1040 |
Binary Data in Baselines |
| 1041 |
======================== |
| 1042 |
|
| 1043 |
``btest`` baselines usually consist of text files, i.e. content that |
| 1044 |
mostly makes sense to process line by line. It's possible to use |
| 1045 |
binary data as well, though. For such data, ``btest-diff`` supports a |
| 1046 |
binary mode in which it will treat the baselines as binary "blobs". In |
| 1047 |
this mode, it will compare test output to baselines for byte-by-byte |
| 1048 |
equality only, it will never apply any canonifiers, and it will leave |
| 1049 |
the test output untouched during baseline updates. |
| 1050 |
|
| 1051 |
To use binary mode, invoke ``btest-diff`` with the ``--binary`` flag. |
| 1052 |
|
| 1053 |
Running Processes in the Background |
| 1054 |
=================================== |
| 1055 |
|
| 1056 |
Sometimes processes need to be spawned in the background for a test, |
| 1057 |
in particular if multiple processes need to cooperate in some fashion. |
| 1058 |
``btest`` comes with two helper scripts to make life easier in such a |
| 1059 |
situation: |
| 1060 |
|
| 1061 |
``btest-bg-run <tag> <cmdline>`` |
| 1062 |
This is a script that runs ``<cmdline>`` in the background, i.e., |
| 1063 |
it's like using ``cmdline &`` in a shell script. Test execution |
| 1064 |
continues immediately with the next command. Note that the spawned |
| 1065 |
command is *not* run in the current directory, but instead in a |
| 1066 |
newly created sub-directory called ``<tag>``. This allows |
| 1067 |
spawning multiple instances of the same process without needing to |
| 1068 |
worry about conflicting outputs. If you want to access a command's |
| 1069 |
output later, like with ``btest-diff``, use ``<tag>/foo.log`` to |
| 1070 |
access it. |
| 1071 |
|
| 1072 |
``btest-bg-wait [-k] <timeout>`` |
| 1073 |
This script waits for all processes previously spawned via |
| 1074 |
``btest-bg-run`` to finish. If any of them exits with a non-zero |
| 1075 |
return code, ``btest-bg-wait`` does so as well, indicating a |
| 1076 |
failed test. ``<timeout>`` is mandatory and gives the maximum |
| 1077 |
number of seconds to wait for any of the processes to terminate. |
| 1078 |
If any process hasn't done so when the timeout expires, it will be |
| 1079 |
killed and the test is considered to be failed as long as ``-k`` |
| 1080 |
is not given. If ``-k`` is given, pending processes are still |
| 1081 |
killed but the test continues normally, i.e., non-termination is |
| 1082 |
not considered a failure in this case. This script also collects |
| 1083 |
the processes' stdout and stderr outputs for diagnostics output. |
| 1084 |
|
| 1085 |
|
| 1086 |
.. _progress: |
| 1087 |
|
| 1088 |
Displaying Progress |
| 1089 |
=================== |
| 1090 |
|
| 1091 |
For long-running tests it can be helpful to display progress messages |
| 1092 |
during their execution so that one sees where the test is currently |
| 1093 |
at. There's a helper script, `btest-progress`, to facilitate that. The |
| 1094 |
script receives a custom message as its sole argument. When executed |
| 1095 |
while a test is running, ``btest`` will display that message in real-time |
| 1096 |
in its standard and verbose outputs. |
| 1097 |
|
| 1098 |
Example usage:: |
| 1099 |
|
| 1100 |
# @TEST-EXEC: bash %INPUT |
| 1101 |
|
| 1102 |
btest-progress Stage 1 |
| 1103 |
sleep 1 |
| 1104 |
btest-progress Stage 2 |
| 1105 |
sleep 1 |
| 1106 |
btest-progress Stage 3 |
| 1107 |
sleep 1 |
| 1108 |
|
| 1109 |
When the tests execute, ``btest`` will then show these three messages |
| 1110 |
successively. By default, ``btest-progress`` also prints the messages |
| 1111 |
to the test's standard output and standard error. That can be suppressed by |
| 1112 |
adding an option ``-q`` to the invocation. |
| 1113 |
|
| 1114 |
.. _timing: `timing execution`_ |
| 1115 |
.. _timing execution: |
| 1116 |
|
| 1117 |
Timing Execution |
| 1118 |
================ |
| 1119 |
|
| 1120 |
``btest`` can time execution of tests and report significant |
| 1121 |
deviations from past runs. As execution time is inherently |
| 1122 |
system-specific it keeps separate per-host timing baselines for that. |
| 1123 |
Furthermore, as time measurements tend to make sense only for |
| 1124 |
individual, usually longer running tests, they are activated on per |
| 1125 |
test basis by adding a `@TEST-MEASURE-TIME`_ directive. The test |
| 1126 |
will then execute as usual yet also record the duration for which it |
| 1127 |
executes. After the timing baselines are created (with the ``--update-times`` |
| 1128 |
option), further runs on the same host will compare their times against that |
| 1129 |
baseline and declare a test failed if it deviates by more than, by |
| 1130 |
default, 1%. (To tune the behaviour, look at the ``Timing*`` `options`_.) |
| 1131 |
If a test requests measurement but BTest can't find a timing baseline |
| 1132 |
or the necessary tools to perform timing measurements, then it will |
| 1133 |
ignore the request. |
| 1134 |
|
| 1135 |
As timing for a test can deviate quite a bit even on the same host, |
| 1136 |
BTest does not actually measure *time* but the number of CPU |
| 1137 |
instructions that a test executes, which tends to be more stable. |
| 1138 |
That however requires the right tools to be in place. On Linux, BTest |
| 1139 |
leverages `perf <https://perf.wiki.kernel.org>`_. By default, BTest |
| 1140 |
will search for ``perf`` in the ``PATH``; you can specify a different |
| 1141 |
path to the binary by setting ``PerfPath`` in ``btest.cfg``. |
| 1142 |
|
| 1143 |
|
| 1144 |
Integration with Sphinx |
| 1145 |
======================= |
| 1146 |
|
| 1147 |
``btest`` comes with an extension module for the documentation framework |
| 1148 |
`Sphinx <http://sphinx.pocoo.org>`_. The extension module provides two |
| 1149 |
new directives called ``btest`` and ``btest-include``. The ``btest`` |
| 1150 |
directive allows writing a test directly inside a Sphinx document, and |
| 1151 |
then the output from the test's command is included in the generated |
| 1152 |
documentation. The ``btest-include`` directive allows for literal text |
| 1153 |
from another file to be included in the generated documentation. |
| 1154 |
The tests from both directives can also be run externally and will catch |
| 1155 |
if any changes to the included content occur. The following walks |
| 1156 |
through setting this up. |
| 1157 |
|
| 1158 |
Configuration |
| 1159 |
------------- |
| 1160 |
|
| 1161 |
First, you need to tell Sphinx a base directory for the ``btest`` |
| 1162 |
configuration as well as a directory in there where to store tests |
| 1163 |
it extracts from the Sphinx documentation. Typically, you'd just |
| 1164 |
create a new subdirectory ``tests`` in the Sphinx project for the |
| 1165 |
``btest`` setup and then store the tests in there in, e.g., |
| 1166 |
``doc/``:: |
| 1167 |
|
| 1168 |
> cd <sphinx-root> |
| 1169 |
> mkdir tests |
| 1170 |
> mkdir tests/doc |
| 1171 |
|
| 1172 |
Then add the following to your Sphinx ``conf.py``:: |
| 1173 |
|
| 1174 |
extensions += ["btest-sphinx"] |
| 1175 |
btest_base="tests" # Relative to Sphinx-root. |
| 1176 |
btest_tests="doc" # Relative to btest_base. |
| 1177 |
|
| 1178 |
Next, create a ``btest.cfg`` in ``tests/`` as usual and add |
| 1179 |
``doc/`` to the ``TestDirs`` option. Also, add a finalizer to ``btest.cfg``:: |
| 1180 |
|
| 1181 |
[btest] |
| 1182 |
... |
| 1183 |
PartFinalizer=btest-diff-rst |
| 1184 |
|
| 1185 |
|
| 1186 |
Including a Test into a Sphinx Document |
| 1187 |
--------------------------------------- |
| 1188 |
|
| 1189 |
The ``btest`` extension provides a new directive to include a test |
| 1190 |
inside a Sphinx document:: |
| 1191 |
|
| 1192 |
|
| 1193 |
.. btest:: <test-name> |
| 1194 |
|
| 1195 |
<test content> |
| 1196 |
|
| 1197 |
Here, ``<test-name>`` is a custom name for the test; it will be |
| 1198 |
stored in ``btest_tests`` under that name (with a file extension of |
| 1199 |
``.btest``). ``<test content>`` is just a standard test as you would |
| 1200 |
normally put into one of the ``TestDirs``. Example:: |
| 1201 |
|
| 1202 |
|
| 1203 |
.. btest:: just-a-test |
| 1204 |
|
| 1205 |
@TEST-EXEC: expr 2 + 2 |
| 1206 |
|
| 1207 |
When you now run Sphinx, it will (1) store the test content into |
| 1208 |
``tests/doc/just-a-test.btest`` (assuming the above path layout), and (2) |
| 1209 |
execute the test by running ``btest`` on it. You can then run |
| 1210 |
``btest`` manually in ``tests/`` as well and it will execute the test |
| 1211 |
just as it would in a standard setup. If a test fails when Sphinx runs |
| 1212 |
it, there will be a corresponding error and include the diagnostic output |
| 1213 |
into the document. |
| 1214 |
|
| 1215 |
By default, nothing else will be included into the generated |
| 1216 |
documentation, i.e., the above test will just turn into an empty text |
| 1217 |
block. However, ``btest`` comes with a set of scripts that you can use |
| 1218 |
to specify content to be included. As a simple example, |
| 1219 |
``btest-rst-cmd <cmdline>`` will execute a command and (if it |
| 1220 |
succeeds) include both the command line and the standard output into |
| 1221 |
the documentation. Example:: |
| 1222 |
|
| 1223 |
.. btest:: another-test |
| 1224 |
|
| 1225 |
@TEST-EXEC: btest-rst-cmd echo Hello, world! |
| 1226 |
|
| 1227 |
When running Sphinx, this will render as:: |
| 1228 |
|
| 1229 |
# echo Hello, world! |
| 1230 |
Hello, world! |
| 1231 |
|
| 1232 |
|
| 1233 |
The same ``<test-name>`` can be used multiple times, in which case |
| 1234 |
each entry will become one part of a joint test. ``btest`` will |
| 1235 |
execute all parts subsequently within a single sandbox, and earlier |
| 1236 |
results will thus be available to later parts. |
| 1237 |
|
| 1238 |
When running ``btest`` manually in ``tests/``, the ``PartFinalizer`` we |
| 1239 |
added to ``btest.cfg`` (see above) compares the generated reST code |
| 1240 |
with a previously established baseline, just like ``btest-diff`` does |
| 1241 |
with files. To establish the initial baseline, run ``btest -u``, like |
| 1242 |
you would with ``btest-diff``. |
| 1243 |
|
| 1244 |
Scripts |
| 1245 |
------- |
| 1246 |
|
| 1247 |
The following Sphinx support scripts come with ``btest``: |
| 1248 |
|
| 1249 |
``btest-rst-cmd [options] <cmdline>`` |
| 1250 |
|
| 1251 |
By default, this executes ``<cmdline>`` and includes both the |
| 1252 |
command line itself and its standard output into the generated |
| 1253 |
documentation (but only if the command line succeeds). |
| 1254 |
See above for an example. |
| 1255 |
|
| 1256 |
This script provides the following options: |
| 1257 |
|
| 1258 |
-c ALTERNATIVE_CMDLINE |
| 1259 |
Show ``ALTERNATIVE_CMDLINE`` in the generated |
| 1260 |
documentation instead of the one actually executed. (It |
| 1261 |
still runs the ``<cmdline>`` given outside the option.) |
| 1262 |
|
| 1263 |
-d |
| 1264 |
Do not actually execute ``<cmdline>``; just format it for |
| 1265 |
the generated documentation and include no further output. |
| 1266 |
|
| 1267 |
-f FILTER_CMD |
| 1268 |
Pipe the command line's output through ``FILTER_CMD`` |
| 1269 |
before including. If ``-r`` is given, it filters the |
| 1270 |
file's content instead of stdout. |
| 1271 |
|
| 1272 |
-n N |
| 1273 |
Include only ``N`` lines of output, adding a ``[...]`` marker if |
| 1274 |
there's more. |
| 1275 |
|
| 1276 |
-o |
| 1277 |
Do not include the executed command into the generated |
| 1278 |
documentation, just its output. |
| 1279 |
|
| 1280 |
-r FILE |
| 1281 |
Insert ``FILE`` into output instead of stdout. The ``FILE`` must |
| 1282 |
be created by a previous ``@TEST-EXEC`` or ``@TEST-COPY-FILE``. |
| 1283 |
|
| 1284 |
|
| 1285 |
``btest-rst-include [options] <file>`` |
| 1286 |
|
| 1287 |
Includes ``<file>`` inside a code block. The ``<file>`` must be created |
| 1288 |
by a previous ``@TEST-EXEC`` or ``@TEST-COPY-FILE``. |
| 1289 |
|
| 1290 |
This script provides the following options: |
| 1291 |
|
| 1292 |
-n N |
| 1293 |
Include only ``N`` lines of output, adding a ``[...]`` marker if |
| 1294 |
there's more. |
| 1295 |
|
| 1296 |
|
| 1297 |
``btest-rst-pipe <cmdline>`` |
| 1298 |
|
| 1299 |
Executes ``<cmdline>``, includes its standard output inside a code |
| 1300 |
block (but only if the command line succeeds). Note that |
| 1301 |
this script does not include the command line itself into the code |
| 1302 |
block, just the output. |
| 1303 |
|
| 1304 |
**NOTE:** All these scripts can be run directly from the command |
| 1305 |
line to show the reST code they generate. |
| 1306 |
|
| 1307 |
**NOTE:** ``btest-rst-cmd`` can do everything the other scripts |
| 1308 |
provide if you give it the right options. In fact, the other |
| 1309 |
scripts are provided just for convenience and leverage |
| 1310 |
``btest-rst-cmd`` internally. |
| 1311 |
|
| 1312 |
Including Literal Text |
| 1313 |
---------------------- |
| 1314 |
|
| 1315 |
The ``btest`` Sphinx extension module also provides a directive |
| 1316 |
``btest-include`` that functions like ``literalinclude`` (including all |
| 1317 |
its options) but also creates a test checking the included content for |
| 1318 |
changes. As one further extension, the directive expands environment |
| 1319 |
variables of the form ``${var}`` in its argument. Example:: |
| 1320 |
|
| 1321 |
.. btest-include:: ${var}/path/to/file |
| 1322 |
|
| 1323 |
When you now run Sphinx, it will automatically generate a test |
| 1324 |
file in the directory specified by the ``btest_tests`` variable in |
| 1325 |
the Sphinx ``conf.py`` configuration file. In this example, the filename |
| 1326 |
would be ``include-path_to_file.btest`` (it automatically adds a prefix of |
| 1327 |
"include-" and a file extension of ".btest"). When you run |
| 1328 |
the tests externally, the tests generated by the ``btest-include`` |
| 1329 |
directive will check if any of the included content has changed (you'll |
| 1330 |
first need to run ``btest -u`` to establish the initial baseline). |
| 1331 |
|
| 1332 |
|
| 1333 |
License |
| 1334 |
======= |
| 1335 |
|
| 1336 |
BTest is open-source under a BSD license. |