Debian Codegraph @

README

raw
1 .. -*- mode: rst-mode -*-
2 ..
3 .. Version number is filled in automatically.
4
5 .. |version| replace:: 0.72
6
7 ==================================================
8 BTest - A Generic Driver for Powerful System Tests
9 ==================================================
10
11 BTest is a powerful framework for writing system tests. Freely
12 borrowing some ideas from other packages, its main objective is to
13 provide an easy-to-use, straightforward driver for a suite of
14 shell-based tests. Each test consists of a set of command lines that
15 will be executed, and success is determined based on their exit
16 codes. ``btest`` comes with some additional tools that can be used
17 within such tests to robustly compare output against a previously established
18 baseline.
19
20 This document describes BTest |version|. See the ``CHANGES``
21 file in the source tree for version history.
22
23 .. contents::
24
25 Prerequisites
26 =============
27
28 BTest has the following prerequisites:
29
30 - Python version >= 3.5 (older versions may work, but are not
31 well-tested).
32
33 - Bash (note that on FreeBSD and Alpine Linux, bash is not installed
34 by default).
35
36 BTest has the following optional prerequisites to enable additional
37 functionality:
38
39 - Sphinx.
40
41 - perf (Linux only). Note that on Debian/Ubuntu, you also need to install
42 the "linux-tools" package.
43
44 Download and Installation
45 =========================
46
47 Installation is simple and standard via ``pip``::
48
49 > pip install btest
50
51 Alternatively, you can download a tarball `from PyPI <https://pypi.org/project/btest/#files>`_
52 and install locally::
53
54 > tar xzvf btest-*.tar.gz
55 > cd btest-*
56 > python3 setup.py install
57
58 The same approach also works on a local git clone of the source tree,
59 located at https://github.com/zeek/btest.
60
61 Each will install a few scripts: ``btest`` is the main driver program,
62 and there are a number of further helper scripts that we discuss below
63 (including ``btest-diff``, which is a tool for comparing output to a
64 previously established baseline).
65
66 .. _running btest:
67
68 Running BTest
69 =============
70
71 A BTest testsuite consists of one or more "btests", executed by the
72 ``btest`` driver. Btests are plain text files in which ``btest``
73 identifies keywords with corresponding arguments that tell it what to
74 do. BTest is *not* a language; it recognizes keywords in any text
75 file, including when embedded in other scripting languages. A common
76 idiom in BTest is to use keywords to process the btest file via a
77 particular command, often a script interpreter. This approach feels
78 unusal at first, but lends BTest much of its flexibility: btest
79 files can contain pretty much anything, as long as ``btest``
80 identifies keywords in it.
81
82 ``btest`` requires a `configuration file`_. With it, you can run
83 ``btest`` on an existing testsuite in several ways:
84
85 - Point it at directories containing btests::
86
87 > btest ./testsuite/
88
89 - Use the config file to enumerate directories to scan for tests,
90 via the ``TestDirs`` `option`_::
91
92 > btest
93
94 - Run btests selectively, by pointing ``btest`` at a specific test file::
95
96 > btest ./testsuite/my.test
97
98 More detail on this when we cover `test selection`_.
99
100 Writing a Test
101 ==============
102
103 First Steps
104 -----------
105
106 In the most simple case, ``btest`` simply executes a set of command
107 lines, each of which must be prefixed with the ``@TEST-EXEC:``
108 keyword::
109
110 > cat examples/t1
111 @TEST-EXEC: echo "Foo" | grep -q Foo
112 @TEST-EXEC: test -d .
113 > btest examples/t1
114 examples.t1 ... ok
115
116 The test passes as both command lines return success. If one of them
117 didn't, that would be reported::
118
119 > cat examples/t2
120 @TEST-EXEC: echo "Foo" | grep -q Foo
121 @TEST-EXEC: test -d DOESNOTEXIST
122 > btest examples/t2
123 examples.t2 ... failed
124
125 Usually you will just run all tests found in a directory::
126
127 > btest examples
128 examples.t1 ... ok
129 examples.t2 ... failed
130 1 test failed
131
132 The file containing the test can simultaneously act as *its input*. Let's
133 say we want to verify a shell script::
134
135 > cat examples/t3.sh
136 # @TEST-EXEC: sh %INPUT
137 ls /etc | grep -q passwd
138 > btest examples/t3.sh
139 examples.t3 ... ok
140
141 Here, ``btest`` executes (something similar to) ``sh
142 examples/t3.sh``, and then checks the return value as usual. The
143 example also shows that the ``@TEST-EXEC`` keyword can appear
144 anywhere, in particular inside the comment section of another
145 language.
146
147 Using Baselines
148 ---------------
149
150 Now, let's say we want to verify the output of a program, making sure
151 that it matches our expectations---a common use case for BTest. To do
152 this, we rely on BTest's built-in support for test baselines. These
153 baselines record prior output of a test, adding support for
154 abstracting away brittle details such as ever-changing timestamps or
155 home directories. BTest comes with tooling to establish, update, and
156 verify baselines, and to plug in "`canonifiers`_": scripts that
157 abstract, or "normalize", troublesome detail from a baseline.
158
159 In our test, we first add a command line that produces the output we
160 want to check, and then run ``btest-diff`` to make sure it matches the
161 previously recorded baseline. ``btest-diff`` is itself just a script
162 that returns success if the output matches a pre-recorded baseline
163 after applying any required normalizations.
164
165 In the following example, we use an awk script as a fancy way to print all
166 file names starting with a dot in the user's home directory. We
167 write that list into a file called ``dots`` and then check whether
168 its content matches what we know from last time::
169
170 > cat examples/t4.awk
171 # @TEST-EXEC: ls -a $HOME | awk -f %INPUT >dots
172 # @TEST-EXEC: btest-diff dots
173 /^\.+/ { print $1 }
174
175 Note that each test gets its own little sandbox directory when run,
176 so by creating a file like ``dots``, you aren't cluttering up
177 anything.
178
179 The first time we run this test, we need to record a baseline. The
180 ``btest`` command includes a baseline-update mode, set via ``-U``,
181 that achieves this::
182
183 > btest -U examples/t4.awk
184
185 ``btest-diff`` recognizes this update mode via an environment variable
186 set by ``btest``, and records the ``dots`` file in a separate baseline
187 folder. With this baseline in place, modifications to the output now
188 trigger a test failure::
189
190 > btest examples/t4.awk
191 examples.t4 ... ok
192 > touch ~/.NEWDOTFILE
193 > btest examples/t4.awk
194 examples.t4 ... failed
195 1 test failed
196
197 If we want to see what exactly changed in ``dots`` to trigger the
198 failure, ``btest`` allows us to record the discrepancies via a
199 *diagnostics* mode that records them in a file called ``.diag``::
200
201 > btest -d examples/t4.awk
202 examples.t4 ... failed
203 % 'btest-diff dots' failed unexpectedly (exit code 1)
204 % cat .diag
205 == File ===============================
206 [... current dots file ...]
207 == Diff ===============================
208 --- /Users/robin/work/binpacpp/btest/Baseline/examples.t4/dots
209 2010-10-28 20:11:11.000000000 -0700
210 +++ dots 2010-10-28 20:12:30.000000000 -0700
211 @@ -4,6 +4,7 @@
212 .CFUserTextEncoding
213 .DS_Store
214 .MacOSX
215 +.NEWDOTFILE
216 .Rhistory
217 .Trash
218 .Xauthority
219 =======================================
220
221 % cat .stderr
222 [... if any of the commands had printed something to stderr, that would follow here ...]
223
224 Once we delete the new file, the test passes again::
225
226 > rm ~/.NEWDOTFILE
227 > btest -d examples/t4.awk
228 examples.t4 ... ok
229
230 That's the essence of the functionality the ``btest`` package
231 provides. This example did not use canonifiers. We cover these,
232 and a number of additional options that extend or modify this basic
233 approach, in the following sections.
234
235 Reference
236 =========
237
238 Command Line Usage
239 ------------------
240
241 ``btest`` must be started with a list of tests and/or directories
242 given on the command line. In the latter case, the default is to
243 recursively scan the directories and assume all files found to be
244 tests to perform. It is however possible to exclude specific files and
245 directories by specifying a suitable `configuration file`_.
246
247 ``btest`` returns exit code 0 if all tests have successfully passed,
248 and 1 otherwise. Exit code 1 can also result in case of other errors.
249
250 ``btest`` accepts the following options:
251
252 -a ALTERNATIVE, --alternative=ALTERNATIVE
253 Activates an alternative_ configuration defined in the
254 configuration file. Multiple alternatives can be given as a
255 comma-separated list (in this case, all specified tests are run
256 once for each specified alternative). If ``ALTERNATIVE`` is ``-``
257 that refers to running with the standard setup, which can be used
258 to run tests both with and without alternatives by giving both.
259
260 -A, --show-all
261 Shows an output line for all tests that were run (this includes tests
262 that passed, failed, or were skipped), rather than only failed tests.
263 Note that this option has no effect when stdout is not a TTY
264 (because all tests are shown in that case).
265
266 -b, --brief
267 Does not output *anything* for tests which pass. If all tests
268 pass, there will not be any output at all except final summary
269 information.
270
271 -c CONFIG, --config=CONFIG
272 Specifies an alternative `configuration file`_ to use. If not
273 specified, the default is to use a file called ``btest.cfg``
274 if found in the current directory. An alternative way to specify
275 a different config file is with the ``BTEST_CFG`` environment
276 variable (however, the command-line option overrides ``BTEST_CFG``).
277
278 -d, --diagnostics
279 Reports diagnostics for all failed tests. The diagnostics
280 include the command line that failed, its output to standard
281 error, and potential additional information recorded by the
282 command line for diagnostic purposes (see `@TEST-EXEC`_
283 below). In the case of ``btest-diff``, the latter is the
284 ``diff`` between baseline and actual output.
285
286 -D, --diagnostics-all
287 Reports diagnostics for all tests, including those which pass.
288
289 -f DIAGFILE, --file-diagnostics=DIAGFILE
290 Writes diagnostics for all failed tests into the given file.
291 If the file already exists, it will be overwritten.
292
293 -g GROUPS, --groups=GROUPS
294 Runs only tests assigned to the given test groups, see
295 `@TEST-GROUP`_. Multiple groups can be given as a
296 comma-separated list. Specifying groups with a leading ``-``
297 leads to all tests to run that are *not* not part of them.
298 Specifying a sole ``-`` as a group name selects all tests that
299 do not belong to any group. (Note that if you combine these
300 variants to create ambiguous situations, it's left
301 undefined which tests will end up running).
302
303 -j THREADS, --jobs=THREADS
304 Runs up to the given number of tests in parallel. If no number
305 is given, BTest substitutes the number of available CPU cores
306 as reported by the OS.
307
308 By default, BTest assumes that all tests can be executed
309 concurrently without further constraints. One can however
310 ensure serialization of subsets by assigning them to the same
311 serialization set, see `@TEST-SERIALIZE`_.
312
313 -q, --quiet
314 Suppress information output other than about failed tests.
315 If all tests pass, there will not be any output at all.
316
317 -r, --rerun
318 Runs only tests that failed last time. After each execution
319 (except when updating baselines), BTest generates a state file
320 that records the tests that have failed. Using this option on
321 the next run then reads that file back in and limits execution
322 to those tests found in there.
323
324 -R FORMAT, --documentation=FORMAT
325 Generates a reference of all tests and prints that to standard
326 output. The output can be of two types, specified by
327 ``FORMAT``: ``rst`` prints reStructuredText, and ``md`` prints
328 Markdown. In the output each test includes the documentation
329 string that's defined for it through ``@TEST-DOC``.
330
331 -t, --tmp-keep
332 Does not delete any temporary files created for running the
333 tests (including their outputs). By default, the temporary
334 files for a test will be located in ``.tmp/<test>/``, where
335 ``<test>`` is the relative path of the test file with all slashes
336 replaced with dots and the file extension removed (e.g., the files
337 for ``example/t3.sh`` will be in ``.tmp/example.t3``).
338
339 -T, --update-times
340 Record new `timing`_ baselines for the current host for tests that
341 have `@TEST-MEASURE-TIME`_. Tests are run as normal except that
342 the timing measurements are recorded as the new baseline instead
343 of being compared to a previous baseline.
344
345 --trace-file=TRACEFILE
346 Record test execution timings in Chrome tracing format to the given
347 file. If the file exists already, it is overwritten. The file can be
348 loaded in Chrome-based browsers at `<about:tracing>`_, or converted to
349 standalone HTML with `trace2html <https://pypi.org/project/trace2html/>`_.
350
351 -U, --update-baseline
352 Records a new baseline for all ``btest-diff`` commands found
353 in any of the specified tests. To do this, all tests are run
354 as normal except that when ``btest-diff`` is executed, it
355 does not compute a diff but instead considers the given file
356 to be authoritative and records it as the version to compare
357 with in future runs.
358
359 -u, --update-interactive
360 Each time a ``btest-diff`` command fails in any tests that are
361 run, ``btest`` will stop and ask whether or not the user wants to
362 record a new baseline.
363
364 -v, --verbose
365 Shows all test command lines as they are executed.
366
367 -w, --wait
368 Interactively waits for ``<enter>`` after showing diagnostics
369 for a test.
370
371 -x FILE, --xml=FILE
372 Records test results in JUnit XML format to the given file.
373 If the file exists already, it is overwritten.
374
375 -z RETRIES, --retries=RETRIES
376 Retry any failed tests up to this many times to determine if
377 they are unstable.
378
379 .. _configuration file: configuration_
380 .. _configuration:
381
382 Configuration
383 -------------
384
385 Specifics of ``btest``'s execution can be tuned with a configuration
386 file, which by default is ``btest.cfg`` if that's found in the
387 current directory. It can alternatively be specified with the
388 ``--config`` command line option, or a ``BTEST_CFG`` environment
389 variable. The configuration file is
390 "INI-style", and an example comes with the distribution, see
391 ``btest.cfg.example``. A configuration file has one main section,
392 ``btest``, that defines most options; as well as an optional section
393 for defining `environment variables`_ and further optional sections
394 for defining alternatives_.
395
396 Note that all paths specified in the configuration file are relative
397 to ``btest``'s *base directory*. The base directory is either the
398 one where the configuration file is located if such is given/found,
399 or the current working directory if not. One can also override it
400 explicitly by setting the environment variable ``BTEST_TEST_BASE``.
401 When setting values for
402 configuration options, the absolute path to the base directory is
403 available by using the macro ``%(testbase)s`` (the weird syntax is
404 due to Python's ``ConfigParser`` class).
405
406 Furthermore, all values can use standard "backtick-syntax" to
407 include the output of external commands (e.g., xyz=`\echo test\`).
408 Note that the backtick expansion is performed after any ``%(..)``
409 have already been replaced (including within the backticks).
410
411 .. _option: `options`_
412 .. _options:
413
414 Options
415 ~~~~~~~
416
417 The following options can be set in the ``btest`` section of the
418 configuration file:
419
420 ``BaselineDir``
421 One or more directories where to store the baseline files for
422 ``btest-diff`` (note that the actual baseline files will be placed
423 into test-specific subdirectories of this directory). By default,
424 this is set to ``%(testbase)s/Baseline``.
425
426 If multiple directories are to be used, they must be separated by
427 colons. ``btest-diff`` will then search them for baseline files in
428 order when looking for a baseline to compare against. When
429 updating a baseline, it will always store the new version inside
430 the first directory. Using multiple directories is most useful in
431 combination with alternatives_ to support alternate executions
432 where some tests produce expected differences in their output.
433
434 This option can also be set through an environment variable
435 ``BTEST_BASELINE_DIR``.
436
437 ``CommandPrefix``
438 Changes the naming of all ``btest`` commands by replacing the
439 ``@TEST-`` prefix with a custom string. For example, with
440 ``CommandPrefix=$TEST-``, the ``@TEST-EXEC`` command becomes
441 ``$TEST-EXEC``.
442
443 ``Finalizer``
444 A command that will be executed each time any test has
445 successfully run. It runs in the same directory as the test itself
446 and receives the name of the test as its only argument. The return
447 value indicates whether the test should indeed be considered
448 successful. By default, there's no finalizer set.
449
450 ``IgnoreDirs``
451 A space-separated list of relative directory names to ignore
452 when scanning test directories recursively. Default is empty.
453
454 An alternative way to ignore a directory is placing a file
455 ``.btest-ignore`` in it.
456
457 ``IgnoreFiles``
458 A space-separated list of filename globs matching files to
459 ignore when scanning given test directories recursively.
460 Default is empty.
461
462 An alternative way to ignore a file is by placing ``@TEST-IGNORE``
463 in it.
464
465 ``Initializer``
466 A command that will be executed before each test. It runs in
467 the same directory as the test itself will and receives the name
468 of the test as its only argument. The return value indicates whether
469 the test should continue; if false, the test will be considered
470 failed. By default, there's no initializer set.
471
472 ``MinVersion``
473 On occasion, you'll want to ensure that the version of ``btest``
474 running your testsuite includes a particular feature. By setting
475 this value to a given version number (as reported by ``btest
476 --version``), ``btest`` installations older than this version will
477 fail test execution with exit code 1 and a corresponding error
478 message on stderr.
479
480 ``PartFinalizer``
481 A command that will be executed each time a test *part* has
482 successfully run. This operates similarly to ``Finalizer`` except
483 that it runs after each test part rather than only at completion
484 of the full test. See `parts`_ for more about test parts.
485
486 ``PartInitializer``
487 A command that will be executed before each test *part*. This operates
488 similarly to ``Initializer`` except that it runs at the beginning of any
489 test part that BTest runs. See `parts`_ for more about test parts.
490
491 Since a failing test part aborts execution of the test, part initializers do
492 not run for any subsequent skipped parts.
493
494 ``PartTeardown``
495 A command that will run after any test *part* that has run, regardless
496 of failure or success of the part. This operates similarly to ``Teardown``
497 except it applies to test `parts`_ instead of the full test.
498
499 Since a failing test part aborts execution of the test, part teardowns do
500 not run for any subsequent skipped parts.
501
502 ``PerfPath``
503 Specifies a path to the ``perf`` tool, which is used on Linux to
504 measure the execution times of tests. By default, BTest searches
505 for ``perf`` in ``PATH``.
506
507 ``PortRange``
508 Specifies a port range like "10000-11000" to use in conjunction with
509 ``@TEST-PORT`` commands. Port assignments will be restricted to this
510 range. The default range is "1024-65535".
511
512 ``StateFile``
513 The name of the state file to record the names of failing tests. Default is
514 ``.btest.failed.dat``.
515
516 ``Teardown``
517 A command that will be executed each time any test has run, regardless of
518 whether that test succeeded. Conceptually, it pairs with an ``Initializer``
519 that sets up test infrastructure that requires tear-down at the end of the
520 test. It runs in the same directory as the test itself and receives the name
521 of the test as its only argument. There's no default teardown command.
522
523 Teardown commands may return a non-zero exit code, which fails the
524 corresponding test. Succeeding teardown commands do not override an
525 otherwise failing test; such tests will still fail.
526
527 To allow teardown routines to reason about the preceding tests, they
528 receive two additional environment variables:
529
530 ``TEST_FAILED``
531 This variable is defined (to 1) when the test has failed, and absent
532 otherwise.
533
534 ``TEST_LAST_RETCODE``
535 This variable contains the numeric exit code of the last command
536 run prior to teardown.
537
538 ``TestDirs``
539 A space-separated list of directories to search for tests. If
540 defined, one doesn't need to specify any tests on the command
541 line.
542
543 ``TimingBaselineDir``
544 A directory where to store the host-specific `timing`_ baseline
545 files. By default, this is set to
546 ``%(testbase)s/Baseline/_Timing``.
547
548 ``TimingDeltaPerc``
549 A value defining the `timing`_ deviation percentage that's tolerated
550 for a test before it's considered failed. Default is 1.0 (which means
551 a 1.0% deviation is tolerated by default).
552
553 ``TmpDir``
554 A directory where to create temporary files when running tests.
555 By default, this is set to ``%(testbase)s/.tmp``.
556
557 .. _environment variables:
558
559 Environment Variables
560 ~~~~~~~~~~~~~~~~~~~~~
561
562 A special section ``environment`` defines environment variables that
563 will be propagated to all tests::
564
565 [environment]
566 CFLAGS=-O3
567 PATH=%(testbase)s/bin:%(default_path)s
568
569 Note how ``PATH`` can be adjusted to include local scripts: the
570 example above prefixes it with a local ``bin/`` directory inside the
571 base directory, using the predefined ``default_path`` macro to refer
572 to the ``PATH`` as it is set by default.
573
574 Furthermore, by setting ``PATH`` to include the ``btest``
575 distribution directory, one could skip the installation of the
576 ``btest`` package.
577
578 .. _alternative: alternatives_
579 .. _alternatives:
580
581 Alternatives
582 ~~~~~~~~~~~~
583
584 BTest can run a set of tests with different settings than it would
585 normally use by specifying an *alternative* configuration. Currently,
586 three things can be adjusted:
587
588 - Further environment variables can be set that will then be
589 available to all the commands that a test executes.
590
591 - *Filters* can modify an input file before a test uses it.
592
593 - *Substitutions* can modify command lines executed as part of a
594 test.
595
596 We discuss the three separately in the following. All of them are
597 defined by adding sections ``[<type>-<name>]`` where ``<type>``
598 corresponds to the type of adjustment being made and ``<name>`` is the
599 name of the alternative. Once at least one section is defined for a
600 name, that alternative can be enabled by BTest's ``--alternative``
601 flag.
602
603 Environment Variables
604 ^^^^^^^^^^^^^^^^^^^^^
605
606 An alternative can add further environment variables by defining an
607 ``[environment-<name>]`` section::
608
609 [environment-myalternative]
610 CFLAGS=-O3
611
612 Running ``btest`` with ``--alternative=myalternative`` will now make
613 the ``CFLAGS`` environment variable available to all commands
614 executed.
615
616 As a special case, one can override two specific environment
617 variables---``BTEST_TEST_BASE`` and ``BTEST_BASELINE_DIR``---inside an
618 alternative's environment section to have them not only be passed on
619 to child processes, but also apply to the ``btest`` process itself.
620 That way, one can switch to a different base and baseline directories
621 for an alternative.
622
623 .. _filters:
624
625 Filters
626 ^^^^^^^
627
628 Filters are a transparent way to adapt the input to a specific test
629 command before it is executed. A filter is defined by adding a section
630 ``[filter-<name>]`` to the configuration file. This section must have
631 exactly one entry, and the name of that entry is interpreted as the
632 name of a command whose input is to be filtered. The value of that
633 entry is the name of a filter script that will be run with two
634 arguments representing input and output files, respectively. Example::
635
636 [filter-myalternative]
637 cat=%(testbase)s/bin/filter-cat
638
639 Once the filter is activated by running ``btest`` with
640 ``--alternative=myalternative``, every time a ``@TEST-EXEC: cat
641 %INPUT`` is found, ``btest`` will first execute (something similar to)
642 ``%(testbase)s/bin/filter-cat %INPUT out.tmp``, and then subsequently
643 ``cat out.tmp`` (i.e., the original command but with the filtered
644 output). In the simplest case, the filter could be a no-op in the
645 form ``cp $1 $2``.
646
647
648 **NOTE:** There are a few limitations to the filter concept currently:
649
650 * Filters are *always* fed with ``%INPUT`` as their first
651 argument. We should add a way to filter other files as well.
652
653 * Filtered commands are only recognized if they are directly
654 starting the command line. For example, ``@TEST-EXEC: ls | cat
655 >output`` would not trigger the example filter above.
656
657 * Filters are only executed for ``@TEST-EXEC``, not for
658 ``@TEST-EXEC-FAIL``.
659
660 .. _substitution:
661
662 Substitutions
663 ^^^^^^^^^^^^^
664
665 Substitutions are similar to filters, yet they do not adapt the input
666 but the command line being executed. A substitution is defined by
667 adding a section ``[substitution-<name>]`` to the configuration file.
668 For each entry in this section, the entry's name specifies the
669 command that is to be replaced with something else given as its value.
670 Example::
671
672 [substitution-myalternative]
673 gcc=gcc -O2
674
675 Once the substitution is activated by running ``btest`` with
676 ``--alternative=myalternative``, every time a ``@TEST-EXEC`` executes
677 ``gcc``, that is replaced with ``gcc -O2``. The replacement is simple
678 string substitution so it works not only with commands but anything
679 found on the command line; it however only replaces full words, not
680 subparts of words.
681
682 Supported Keywords
683 ------------------
684
685 ``btest`` scans a test file for lines containing keywords that
686 trigger certain functionality. It knows the following keywords:
687
688 ``@TEST-ALTERNATIVE: <alternative>``
689 Runs this test only for the given alternative (see alternative_).
690 If ``<alternative>`` is ``default``, the test executes when BTest runs
691 with no alternative given (which however is the default anyway).
692
693 ``@TEST-COPY-FILE: <file>``
694 Copy the given file into the test's directory before the test is
695 run. If ``<file>`` is a relative path, it's interpreted relative
696 to the BTest's base directory. Environment variables in ``<file>``
697 will be replaced if enclosed in ``${..}``. This command can be
698 given multiple times.
699
700 ``@TEST-DOC: <docstring>``
701 Associates a documentation string with the test. These strings
702 get included into the output of the ``--documentation`` option.
703
704 .. _@TEST-EXEC:
705
706 ``@TEST-EXEC: <cmdline>``
707 Executes the given command line and aborts the test if it
708 returns an error code other than zero. The ``<cmdline>`` is
709 passed to the shell and thus can be a pipeline, use redirection,
710 and any environment variables specified in ``<cmdline>`` will be
711 expanded, etc.
712
713 When running a test, the current working directory for all
714 command lines will be set to a temporary sandbox (and will be
715 deleted later).
716
717 There are two macros that can be used in ``<cmdline>``:
718 ``%INPUT`` will be replaced with the full pathname of the file defining
719 the test (this file is in a temporary sandbox directory and is a copy
720 of the original test file); and ``%DIR`` will be replaced with the full
721 pathname of the directory where the test file is located (note that
722 this is the directory where the original test file is located, not
723 the directory where the ``%INPUT`` file is located). The latter can
724 be used to reference further files also located there.
725
726 In addition to environment variables defined in the
727 configuration file, there are further ones that are passed into
728 the commands:
729
730 ``TEST_BASE``
731 The BTest base directory, i.e., the directory where
732 ``btest.cfg`` is located.
733
734 ``TEST_BASELINE``
735 A list of directories where the command can save permanent
736 information across ``btest`` runs. (This is where
737 ``btest-diff`` stores its baseline in ``UPDATE`` mode.)
738
739 Multiple entries are separated by colons. If more than one
740 entry is given, semantics should be to search them in order.
741 (This is where ``btest-diff`` stores its baseline in
742 ``UPDATE`` mode.)
743
744 ``TEST_DIAGNOSTICS``
745 A file where further diagnostic information can be saved
746 in case a command fails (this is also where ``btest-diff``
747 stores its diff). If this file exists, then the
748 ``--diagnostics-all`` or ``--diagnostics`` options will show
749 this file (for the latter option, only if a command fails).
750
751 ``TEST_MODE``
752 This is normally set to ``TEST``, but will be ``UPDATE``
753 if ``btest`` is run with ``--update-baseline``, or
754 ``UPDATE_INTERACTIVE`` if run with ``--update-interactive``.
755
756 ``TEST_NAME``
757 The name of the currently executing test.
758
759 ``TEST_PART``
760 The test part number (see `parts`_ for more about test parts).
761
762 **NOTE:**
763
764 If a command returns the special exit code 100, the test is
765 considered failed, however subsequent test commands within the
766 current test are still run. ``btest-diff`` uses this special
767 exit code to indicate that no baseline has yet been established.
768
769 If a command returns the special exit code 200, the test is
770 considered failed and all further tests are aborted.
771 ``btest-diff`` uses this special exit code when ``btest`` is run
772 with the ``--update-interactive`` option and the user chooses to
773 abort the tests when prompted to record a new baseline.
774
775 ``TEST_VERBOSE``
776 The path of a file where the test can record further
777 information about its execution that will be included with
778 BTest's ``--verbose`` output. This is for further tracking
779 the execution of commands and should generally generate
780 output that follows a line-based structure.
781
782
783 ``@TEST-EXEC-FAIL: <cmdline>``
784 Like ``@TEST-EXEC``, except that this expects the command to
785 *fail*, i.e., the test is aborted when the return code is zero.
786
787 .. _@TEST-GROUP:
788
789 ``@TEST-GROUP: <group>``
790 Assigns the test to a group of name ``<group>``. By using option
791 ``-g`` one can limit execution to all tests that belong to a given
792 group (or a set of groups).
793
794 ``@TEST-IGNORE``
795 This is used to indicate that this file should be skipped (i.e., no
796 test commands in this file will be executed). An alternative way to
797 ignore files is by using the ``IgnoreFiles`` option in the btest
798 configuration file.
799
800 ``@TEST-KNOWN-FAILURE``
801 Marks a test as known to currently fail. This only changes BTest's
802 output, which upon failure will indicate that that is expected; it
803 won't change the test's processing otherwise. The keyword doesn't
804 take any arguments but one could add a descriptive text, as in ::
805
806 .. @TEST-KNOWN-FAILURE: We know this fails because ....
807
808 .. _@TEST-MEASURE-TIME:
809
810 ``@TEST-MEASURE-TIME``
811 Measures execution time for this test and compares it to a
812 previously established `timing`_ baseline. If it deviates significantly,
813 the test will be considered failed.
814
815 ``@TEST-NOT-ALTERNATIVE: <alternative>``
816 Ignores this test for the given alternative (see alternative_).
817 If ``<alternative>`` is ``default``, the test is ignored if BTest runs
818 with no alternative given.
819
820 .. _@TEST-PORT:
821
822 ``@TEST-PORT: <env>``
823 Assign an available TCP port number to an environment variable
824 that is accessible from the running test process. ``<env>`` is an
825 arbitrary user-chosen string that will be set to the next available
826 TCP port number. Availability is based on checking successful
827 binding of the port on IPv4 INADDR_ANY and also restricted to the
828 range specified by the ``PortRange`` option. IPv6 is not supported.
829 Note that using the ``-j`` option to parallelize execution will
830 work such that unique/available port numbers are assigned between
831 concurrent tests, however there is still a potential race condition
832 for external processes to claim a port before the test actually
833 runs and claims it for itself.
834
835 ``@TEST-REQUIRES: <cmdline>``
836 Defines a condition that must be met for the test to be executed.
837 The given command line will be run before any of the actual test
838 commands, and it must return success for the test to continue. If
839 it does not return success, the rest of the test will be skipped
840 but doing so will not be considered a failure of the test. This allows to
841 write conditional tests that may not always make sense to run, depending
842 on whether external constraints are satisfied or not (say, whether
843 a particular library is available). Multiple requirements may be
844 specified and then all must be met for the test to continue.
845
846 .. _@TEST-SERIALIZE:
847
848 ``@TEST-SERIALIZE: <set>``
849 When using option ``-j`` to parallelize execution, all tests that
850 specify the same serialization set are guaranteed to run
851 sequentially. ``<set>`` is an arbitrary user-chosen string.
852
853 ``@TEST-START-FILE <file>``
854 This is used to include an additional input file for a test
855 right inside the test file. All lines following the keyword line
856 will be written into the given file until a line containing
857 ``@TEST-END-FILE`` is found. The lines containing ``@TEST-START-FILE``
858 and ``@TEST-END-FILE``, and all lines in between, will be removed from
859 the test's %INPUT. Example::
860
861 > cat examples/t6.sh
862 # @TEST-EXEC: awk -f %INPUT <foo.dat >output
863 # @TEST-EXEC: btest-diff output
864
865 { lines += 1; }
866 END { print lines; }
867
868 @TEST-START-FILE foo.dat
869 1
870 2
871 3
872 @TEST-END-FILE
873
874 > btest -D examples/t6.sh
875 examples.t6 ... ok
876 % cat .diag
877 == File ===============================
878 3
879
880 Multiple such files can be defined within a single test.
881
882 Note that this is only one way to use further input files.
883 Another is to store a file in the same directory as the test
884 itself, making sure it's ignored via ``IgnoreFiles``, and then
885 refer to it via ``%DIR/<name>``.
886
887 ``@TEST-START-NEXT``
888 This keyword lets you define multiple test inputs in the
889 same file, all executing with the same command lines. See
890 `defining multiple tests in one file`_ for details.
891
892 .. _test selection: `selecting tests`_
893 .. _selecting tests:
894
895 Selecting Tests
896 ===============
897
898 Internally, ``btest`` uses logical names for tests, abstracting input
899 files. Those names result from substituting path separators with dots,
900 ignoring btest file suffixes, and potentially adding additional
901 labeling. ``btest`` does this only for tests within the ``TestDirs``
902 directories given in the `configuration file`.
903
904 In addition to the invocations covered in `Running BTest`_, you can
905 use logical names when telling ``btest`` which tests to run. For
906 example, instead of saying ::
907
908 > btest testsuite/foo.sh
909
910 you can use::
911
912 > btest testsuite.foo
913
914 This distinction rarely matters, but it's something to be aware of
915 when `defining multiple tests in one file`_, which we cover next.
916
917 .. _more than one test: `defining multiple tests in one file`_
918 .. _defining multiple tests in one file:
919
920 Defining Multiple Tests in one File
921 ===================================
922
923 On occasion you want to use the same constellation of keywords on a
924 set of input files. BTest supports this via the ``@TEST-START-NEXT``
925 keyword. When ``btest`` encounters this keyword, it initially
926 considers the input file to end at that point, and runs all
927 ``@TEST-EXEC-*`` with an ``%INPUT`` truncated accordingly.
928 Afterwards, it creates a *new* ``%INPUT`` with everything *following*
929 the ``@TEST-START-NEXT`` marker, running the *same* commands
930 again. (It ignores any ``@TEST-EXEC-*`` lines later in the file.)
931
932 The effect is that a single file can define multiple tests that the
933 ``btest`` output will enumerate::
934
935 > cat examples/t5.sh
936 # @TEST-EXEC: cat %INPUT | wc -c >output
937 # @TEST-EXEC: btest-diff output
938
939 This is the first test input in this file.
940
941 # @TEST-START-NEXT
942
943 ... and the second.
944
945 > ./btest -D examples/t5.sh
946 examples.t5 ... ok
947 % cat .diag
948 == File ===============================
949 119
950 [...]
951
952 examples.t5-2 ... ok
953 % cat .diag
954 == File ===============================
955 22
956 [...]
957
958 ``btest`` automatically generates the ``-<n>`` suffix for each of the tests.
959
960 **NOTE:** It matters how you name tests when running them
961 individually. When you specify the btest file ("``examples/t5.sh``"),
962 ``btest`` will run all of the contained tests. When you use the
963 logical name, ``btest`` will run only that specific test: in the
964 above scenario, ``examples.t5`` runs only the first test defined
965 in the file, while ``examples.t5-2`` only runs the second. This
966 also applies to baseline updates.
967
968 .. _parts: `splitting tests into parts`_
969 .. _splitting tests into parts:
970
971 Splitting Tests into Parts
972 ==========================
973
974 One can also split a single test across multiple files by adding a
975 numerical ``#<n>`` postfix to their names, where each ``<n>``
976 represents a separate part of the test. ``btest`` will combine all of
977 a test's parts in numerical order and execute them subsequently within
978 the same sandbox. Example::
979
980 > cat examples/t7.sh#1
981 # @TEST-EXEC: echo Part 1 - %INPUT >>output
982
983 > cat examples/t7.sh#2
984 # @TEST-EXEC: echo Part 2 - %INPUT >>output
985
986 > cat examples/t7.sh#3
987 # @TEST-EXEC: btest-diff output
988
989 > btest -D examples/t7.sh
990 examples.t7 ... ok
991 % cat .diag
992 == File ===============================
993 Part 1 - /Users/robin/bro/docs/aux/btest/.tmp/examples.t7/t7.sh#1
994 Part 2 - /Users/robin/bro/docs/aux/btest/.tmp/examples.t7/t7.sh#2
995
996 Note how ``output`` contains the output of both ``t7.sh#1`` and ``t7.sh#2``,
997 however in each case ``%INPUT`` refers to the corresponding part. For
998 the first part of a test, one can also omit the ``#1`` postfix in the filename.
999
1000 .. _canonifiers: `canonifying diffs`_
1001 .. _canonifying diffs:
1002
1003 Canonifying Diffs
1004 =================
1005
1006 ``btest-diff`` has the capability to filter its input through an
1007 additional script before it compares the current version with the
1008 baseline. This can be useful if certain elements in an output are
1009 *expected* to change (e.g., timestamps). The filter can then
1010 remove/replace these with something consistent. To enable such
1011 canonification, set the environment variable
1012 ``TEST_DIFF_CANONIFIER`` to a script reading the original version
1013 from stdin and writing the canonified version to stdout.
1014 For examples of canonifier scripts, take a look at those `used in the
1015 Zeek distribution <https://github.com/zeek/zeek/tree/master/testing/scripts/>`_.
1016
1017 **NOTE:** ``btest-diff`` passes both the pre-recorded baseline and
1018 the fresh test output through any canonifiers before comparing
1019 their contents. BTest version 0.63 introduced two changes in
1020 ``btest-diff``'s baseline handling:
1021
1022 * ``btest-diff`` now records baselines in canonicalized form. The
1023 benefit here is that by canonicalizing upon recording, you can
1024 use ``btest -U`` more freely, keeping expected noise out of
1025 revision control. The downside is that updates to canonifiers
1026 require a refresh of the baselines.
1027
1028 * ``btest-diff`` now prefixes the baselines with a header that
1029 warns against manual modification, and knows to exclude that
1030 header from comparison. We recommend only ever updating
1031 baselines via ``btest -U`` (or its interactive sibling, ``-u``).
1032
1033 Once you use canonicalized baselines in your project, it's a good
1034 idea to use ``MinVersion = 0.63`` in your btest.cfg to avoid the
1035 use of older ``btest`` installations. Since these are unaware of
1036 the new baseline header and repeated application of canonifiers
1037 may cause unexpected alterations to already-canonified baselines,
1038 using such versions will likely cause test failures.
1039
1040 Binary Data in Baselines
1041 ========================
1042
1043 ``btest`` baselines usually consist of text files, i.e. content that
1044 mostly makes sense to process line by line. It's possible to use
1045 binary data as well, though. For such data, ``btest-diff`` supports a
1046 binary mode in which it will treat the baselines as binary "blobs". In
1047 this mode, it will compare test output to baselines for byte-by-byte
1048 equality only, it will never apply any canonifiers, and it will leave
1049 the test output untouched during baseline updates.
1050
1051 To use binary mode, invoke ``btest-diff`` with the ``--binary`` flag.
1052
1053 Running Processes in the Background
1054 ===================================
1055
1056 Sometimes processes need to be spawned in the background for a test,
1057 in particular if multiple processes need to cooperate in some fashion.
1058 ``btest`` comes with two helper scripts to make life easier in such a
1059 situation:
1060
1061 ``btest-bg-run <tag> <cmdline>``
1062 This is a script that runs ``<cmdline>`` in the background, i.e.,
1063 it's like using ``cmdline &`` in a shell script. Test execution
1064 continues immediately with the next command. Note that the spawned
1065 command is *not* run in the current directory, but instead in a
1066 newly created sub-directory called ``<tag>``. This allows
1067 spawning multiple instances of the same process without needing to
1068 worry about conflicting outputs. If you want to access a command's
1069 output later, like with ``btest-diff``, use ``<tag>/foo.log`` to
1070 access it.
1071
1072 ``btest-bg-wait [-k] <timeout>``
1073 This script waits for all processes previously spawned via
1074 ``btest-bg-run`` to finish. If any of them exits with a non-zero
1075 return code, ``btest-bg-wait`` does so as well, indicating a
1076 failed test. ``<timeout>`` is mandatory and gives the maximum
1077 number of seconds to wait for any of the processes to terminate.
1078 If any process hasn't done so when the timeout expires, it will be
1079 killed and the test is considered to be failed as long as ``-k``
1080 is not given. If ``-k`` is given, pending processes are still
1081 killed but the test continues normally, i.e., non-termination is
1082 not considered a failure in this case. This script also collects
1083 the processes' stdout and stderr outputs for diagnostics output.
1084
1085
1086 .. _progress:
1087
1088 Displaying Progress
1089 ===================
1090
1091 For long-running tests it can be helpful to display progress messages
1092 during their execution so that one sees where the test is currently
1093 at. There's a helper script, `btest-progress`, to facilitate that. The
1094 script receives a custom message as its sole argument. When executed
1095 while a test is running, ``btest`` will display that message in real-time
1096 in its standard and verbose outputs.
1097
1098 Example usage::
1099
1100 # @TEST-EXEC: bash %INPUT
1101
1102 btest-progress Stage 1
1103 sleep 1
1104 btest-progress Stage 2
1105 sleep 1
1106 btest-progress Stage 3
1107 sleep 1
1108
1109 When the tests execute, ``btest`` will then show these three messages
1110 successively. By default, ``btest-progress`` also prints the messages
1111 to the test's standard output and standard error. That can be suppressed by
1112 adding an option ``-q`` to the invocation.
1113
1114 .. _timing: `timing execution`_
1115 .. _timing execution:
1116
1117 Timing Execution
1118 ================
1119
1120 ``btest`` can time execution of tests and report significant
1121 deviations from past runs. As execution time is inherently
1122 system-specific it keeps separate per-host timing baselines for that.
1123 Furthermore, as time measurements tend to make sense only for
1124 individual, usually longer running tests, they are activated on per
1125 test basis by adding a `@TEST-MEASURE-TIME`_ directive. The test
1126 will then execute as usual yet also record the duration for which it
1127 executes. After the timing baselines are created (with the ``--update-times``
1128 option), further runs on the same host will compare their times against that
1129 baseline and declare a test failed if it deviates by more than, by
1130 default, 1%. (To tune the behaviour, look at the ``Timing*`` `options`_.)
1131 If a test requests measurement but BTest can't find a timing baseline
1132 or the necessary tools to perform timing measurements, then it will
1133 ignore the request.
1134
1135 As timing for a test can deviate quite a bit even on the same host,
1136 BTest does not actually measure *time* but the number of CPU
1137 instructions that a test executes, which tends to be more stable.
1138 That however requires the right tools to be in place. On Linux, BTest
1139 leverages `perf <https://perf.wiki.kernel.org>`_. By default, BTest
1140 will search for ``perf`` in the ``PATH``; you can specify a different
1141 path to the binary by setting ``PerfPath`` in ``btest.cfg``.
1142
1143
1144 Integration with Sphinx
1145 =======================
1146
1147 ``btest`` comes with an extension module for the documentation framework
1148 `Sphinx <http://sphinx.pocoo.org>`_. The extension module provides two
1149 new directives called ``btest`` and ``btest-include``. The ``btest``
1150 directive allows writing a test directly inside a Sphinx document, and
1151 then the output from the test's command is included in the generated
1152 documentation. The ``btest-include`` directive allows for literal text
1153 from another file to be included in the generated documentation.
1154 The tests from both directives can also be run externally and will catch
1155 if any changes to the included content occur. The following walks
1156 through setting this up.
1157
1158 Configuration
1159 -------------
1160
1161 First, you need to tell Sphinx a base directory for the ``btest``
1162 configuration as well as a directory in there where to store tests
1163 it extracts from the Sphinx documentation. Typically, you'd just
1164 create a new subdirectory ``tests`` in the Sphinx project for the
1165 ``btest`` setup and then store the tests in there in, e.g.,
1166 ``doc/``::
1167
1168 > cd <sphinx-root>
1169 > mkdir tests
1170 > mkdir tests/doc
1171
1172 Then add the following to your Sphinx ``conf.py``::
1173
1174 extensions += ["btest-sphinx"]
1175 btest_base="tests" # Relative to Sphinx-root.
1176 btest_tests="doc" # Relative to btest_base.
1177
1178 Next, create a ``btest.cfg`` in ``tests/`` as usual and add
1179 ``doc/`` to the ``TestDirs`` option. Also, add a finalizer to ``btest.cfg``::
1180
1181 [btest]
1182 ...
1183 PartFinalizer=btest-diff-rst
1184
1185
1186 Including a Test into a Sphinx Document
1187 ---------------------------------------
1188
1189 The ``btest`` extension provides a new directive to include a test
1190 inside a Sphinx document::
1191
1192
1193 .. btest:: <test-name>
1194
1195 <test content>
1196
1197 Here, ``<test-name>`` is a custom name for the test; it will be
1198 stored in ``btest_tests`` under that name (with a file extension of
1199 ``.btest``). ``<test content>`` is just a standard test as you would
1200 normally put into one of the ``TestDirs``. Example::
1201
1202
1203 .. btest:: just-a-test
1204
1205 @TEST-EXEC: expr 2 + 2
1206
1207 When you now run Sphinx, it will (1) store the test content into
1208 ``tests/doc/just-a-test.btest`` (assuming the above path layout), and (2)
1209 execute the test by running ``btest`` on it. You can then run
1210 ``btest`` manually in ``tests/`` as well and it will execute the test
1211 just as it would in a standard setup. If a test fails when Sphinx runs
1212 it, there will be a corresponding error and include the diagnostic output
1213 into the document.
1214
1215 By default, nothing else will be included into the generated
1216 documentation, i.e., the above test will just turn into an empty text
1217 block. However, ``btest`` comes with a set of scripts that you can use
1218 to specify content to be included. As a simple example,
1219 ``btest-rst-cmd <cmdline>`` will execute a command and (if it
1220 succeeds) include both the command line and the standard output into
1221 the documentation. Example::
1222
1223 .. btest:: another-test
1224
1225 @TEST-EXEC: btest-rst-cmd echo Hello, world!
1226
1227 When running Sphinx, this will render as::
1228
1229 # echo Hello, world!
1230 Hello, world!
1231
1232
1233 The same ``<test-name>`` can be used multiple times, in which case
1234 each entry will become one part of a joint test. ``btest`` will
1235 execute all parts subsequently within a single sandbox, and earlier
1236 results will thus be available to later parts.
1237
1238 When running ``btest`` manually in ``tests/``, the ``PartFinalizer`` we
1239 added to ``btest.cfg`` (see above) compares the generated reST code
1240 with a previously established baseline, just like ``btest-diff`` does
1241 with files. To establish the initial baseline, run ``btest -u``, like
1242 you would with ``btest-diff``.
1243
1244 Scripts
1245 -------
1246
1247 The following Sphinx support scripts come with ``btest``:
1248
1249 ``btest-rst-cmd [options] <cmdline>``
1250
1251 By default, this executes ``<cmdline>`` and includes both the
1252 command line itself and its standard output into the generated
1253 documentation (but only if the command line succeeds).
1254 See above for an example.
1255
1256 This script provides the following options:
1257
1258 -c ALTERNATIVE_CMDLINE
1259 Show ``ALTERNATIVE_CMDLINE`` in the generated
1260 documentation instead of the one actually executed. (It
1261 still runs the ``<cmdline>`` given outside the option.)
1262
1263 -d
1264 Do not actually execute ``<cmdline>``; just format it for
1265 the generated documentation and include no further output.
1266
1267 -f FILTER_CMD
1268 Pipe the command line's output through ``FILTER_CMD``
1269 before including. If ``-r`` is given, it filters the
1270 file's content instead of stdout.
1271
1272 -n N
1273 Include only ``N`` lines of output, adding a ``[...]`` marker if
1274 there's more.
1275
1276 -o
1277 Do not include the executed command into the generated
1278 documentation, just its output.
1279
1280 -r FILE
1281 Insert ``FILE`` into output instead of stdout. The ``FILE`` must
1282 be created by a previous ``@TEST-EXEC`` or ``@TEST-COPY-FILE``.
1283
1284
1285 ``btest-rst-include [options] <file>``
1286
1287 Includes ``<file>`` inside a code block. The ``<file>`` must be created
1288 by a previous ``@TEST-EXEC`` or ``@TEST-COPY-FILE``.
1289
1290 This script provides the following options:
1291
1292 -n N
1293 Include only ``N`` lines of output, adding a ``[...]`` marker if
1294 there's more.
1295
1296
1297 ``btest-rst-pipe <cmdline>``
1298
1299 Executes ``<cmdline>``, includes its standard output inside a code
1300 block (but only if the command line succeeds). Note that
1301 this script does not include the command line itself into the code
1302 block, just the output.
1303
1304 **NOTE:** All these scripts can be run directly from the command
1305 line to show the reST code they generate.
1306
1307 **NOTE:** ``btest-rst-cmd`` can do everything the other scripts
1308 provide if you give it the right options. In fact, the other
1309 scripts are provided just for convenience and leverage
1310 ``btest-rst-cmd`` internally.
1311
1312 Including Literal Text
1313 ----------------------
1314
1315 The ``btest`` Sphinx extension module also provides a directive
1316 ``btest-include`` that functions like ``literalinclude`` (including all
1317 its options) but also creates a test checking the included content for
1318 changes. As one further extension, the directive expands environment
1319 variables of the form ``${var}`` in its argument. Example::
1320
1321 .. btest-include:: ${var}/path/to/file
1322
1323 When you now run Sphinx, it will automatically generate a test
1324 file in the directory specified by the ``btest_tests`` variable in
1325 the Sphinx ``conf.py`` configuration file. In this example, the filename
1326 would be ``include-path_to_file.btest`` (it automatically adds a prefix of
1327 "include-" and a file extension of ".btest"). When you run
1328 the tests externally, the tests generated by the ``btest-include``
1329 directive will check if any of the included content has changed (you'll
1330 first need to run ``btest -u`` to establish the initial baseline).
1331
1332
1333 License
1334 =======
1335
1336 BTest is open-source under a BSD license.