| 1 |
Writing TAP Tests |
| 2 |
|
| 3 |
Introduction |
| 4 |
|
| 5 |
This is a guide for users of the C TAP Harness package or similar |
| 6 |
TAP-based test harnesses explaining how to write tests. If your |
| 7 |
package uses C TAP Harness as the test suite driver, you may want to |
| 8 |
copy this document to an appropriate file name in your test suite as |
| 9 |
documentation for contributors. |
| 10 |
|
| 11 |
About TAP |
| 12 |
|
| 13 |
TAP is the Test Anything Protocol, a protocol for communication |
| 14 |
between test cases and a test harness. This is the protocol used by |
| 15 |
Perl for its internal test suite and for nearly all Perl modules, |
| 16 |
since it's the format used by the build tools for Perl modules to run |
| 17 |
tests and report their results. |
| 18 |
|
| 19 |
A TAP-based test suite works with a somewhat different set of |
| 20 |
assumptions than an xUnit test suite. In TAP, each test case is a |
| 21 |
separate program. That program, when run, must produce output in the |
| 22 |
following format: |
| 23 |
|
| 24 |
1..4 |
| 25 |
ok 1 - the first test |
| 26 |
ok 2 |
| 27 |
# a diagnostic, ignored by the harness |
| 28 |
not ok 3 - a failing test |
| 29 |
ok 4 # skip a skipped test |
| 30 |
|
| 31 |
The output should all go to standard output. The first line specifies |
| 32 |
the number of tests to be run, and then each test produces output that |
| 33 |
looks like either "ok <n>" or "not ok <n>" depending on whether the |
| 34 |
test succeeded or failed. Additional information about the test can |
| 35 |
be provided after the "ok <n>" or "not ok <n>", but is optional. |
| 36 |
Additional diagnostics and information can be provided in lines |
| 37 |
beginning with a "#". |
| 38 |
|
| 39 |
Processing directives are supported after the "ok <n>" or "not ok <n>" |
| 40 |
and start with a "#". The main one of interest is "# skip" which says |
| 41 |
that the test was skipped rather than successful and optionally gives |
| 42 |
the reason. Also supported is "# todo", which normally annotates a |
| 43 |
failing test and indicates that test is expected to fail, optionally |
| 44 |
providing a reason for why. |
| 45 |
|
| 46 |
There are three more special cases. First, the initial line stating |
| 47 |
the number of tests to run, called the plan, may appear at the end of |
| 48 |
the output instead of the beginning. This can be useful if the number |
| 49 |
of tests to run is not known in advance. Second, a plan in the form: |
| 50 |
|
| 51 |
1..0 # skip entire test case skipped |
| 52 |
|
| 53 |
can be given instead, which indicates that this entire test case has |
| 54 |
been skipped (generally because it depends on facilities or optional |
| 55 |
configuration which is not present). Finally, if the test case |
| 56 |
encounters a fatal error, it should print the text: |
| 57 |
|
| 58 |
Bail out! |
| 59 |
|
| 60 |
on standard output, optionally followed by an error message, and then |
| 61 |
exit. This tells the harness that the test aborted unexpectedly. |
| 62 |
|
| 63 |
The exit status of a successful test case should always be 0. The |
| 64 |
harness will report the test as "dubious" if all the tests appeared to |
| 65 |
succeed but it exited with a non-zero status. |
| 66 |
|
| 67 |
Writing TAP Tests |
| 68 |
|
| 69 |
Environment |
| 70 |
|
| 71 |
One of the special features of C TAP Harness is the environment that |
| 72 |
it sets up for your test cases. If your test program is called under |
| 73 |
the runtests driver, the environment variables C_TAP_SOURCE and |
| 74 |
C_TAP_BUILD will be set to the top of the test directory in the source |
| 75 |
tree and the top of the build tree, respectively. You can use those |
| 76 |
environment variables to locate additional test data, programs and |
| 77 |
libraries built as part of your software build, and other supporting |
| 78 |
information needed by tests. |
| 79 |
|
| 80 |
The C and shell TAP libraries support a test_file_path() function, |
| 81 |
which looks for a file under the build tree and then under the source |
| 82 |
tree, using the C_TAP_BUILD and C_TAP_SOURCE environment variables, |
| 83 |
and return the full path to the file. This can be used to locate |
| 84 |
supporting data files. They also support a test_tmpdir() function |
| 85 |
that returns a directory that can be used for temporary files during |
| 86 |
tests. |
| 87 |
|
| 88 |
Perl |
| 89 |
|
| 90 |
Since TAP is the native test framework for Perl, writing TAP tests in |
| 91 |
Perl is very easy and extremely well-supported. If you've never |
| 92 |
written tests in Perl before, start by reading the documentation for |
| 93 |
Test::Tutorial and Test::Simple, which walks you through the basics, |
| 94 |
including the TAP output syntax. Then, the best Perl module to use |
| 95 |
for serious testing is Test::More, which provides a lot of additional |
| 96 |
functions over Test::Simple including support for skipping tests, |
| 97 |
bailing out, and not planning tests in advance. See the documentation |
| 98 |
of Test::More for all the details and lots of examples. |
| 99 |
|
| 100 |
C TAP Harness can run Perl test scripts directly and interpret the |
| 101 |
results correctly, and similarly the Perl Test::Harness module and |
| 102 |
prove command can run TAP tests written in other languages using, for |
| 103 |
example, the TAP library that comes with C TAP Harness. You can, if |
| 104 |
you wish, use the library that comes with C TAP Harness but use prove |
| 105 |
instead of runtests for running the test suite. |
| 106 |
|
| 107 |
C |
| 108 |
|
| 109 |
C TAP Harness provides a basic TAP library that takes away most of the |
| 110 |
pain of writing TAP test cases in C. A C test case should start with |
| 111 |
a call to plan(), passing in the number of tests to run. Then, each |
| 112 |
test should use is_int(), is_string(), is_double(), or is_hex() as |
| 113 |
appropriate to compare expected and seen values, or ok() to do a |
| 114 |
simpler boolean test. The is_*() functions take expected and seen |
| 115 |
values and then a printf-style format string explaining the test |
| 116 |
(which may be NULL). ok() takes a boolean and then the printf-style |
| 117 |
string. |
| 118 |
|
| 119 |
Here's a complete example test program that uses the C TAP library: |
| 120 |
|
| 121 |
#include <stddef.h> |
| 122 |
#include <tap/basic.h> |
| 123 |
|
| 124 |
int |
| 125 |
main(void) |
| 126 |
{ |
| 127 |
plan(4); |
| 128 |
|
| 129 |
ok(1, "the first test"); |
| 130 |
is_int(42, 42, NULL); |
| 131 |
diag("a diagnostic, ignored by the harness"); |
| 132 |
ok(0, "a failing test"); |
| 133 |
skip("a skipped test"); |
| 134 |
|
| 135 |
return 0; |
| 136 |
} |
| 137 |
|
| 138 |
This test program produces the output shown above in the section on |
| 139 |
TAP and demonstrates most of the functions. The other functions of |
| 140 |
interest are sysdiag() (like diag() but adds strerror() results), |
| 141 |
bail() and sysbail() for fatal errors, skip_block() to skip a whole |
| 142 |
block of tests, and skip_all() which is called instead of plan() to |
| 143 |
skip an entire test case. |
| 144 |
|
| 145 |
The C TAP library also provides plan_lazy(), which can be called |
| 146 |
instead of plan(). If plan_lazy() is called, the library will keep |
| 147 |
track of how many test results are reported and will print out the |
| 148 |
plan at the end of execution of the program. This should normally be |
| 149 |
avoided since the test may appear to be successful even if it exits |
| 150 |
prematurely, but it can make writing tests easier in some |
| 151 |
circumstances. |
| 152 |
|
| 153 |
Complete API documentation for the basic C TAP library that comes with |
| 154 |
C TAP Harness is available at: |
| 155 |
|
| 156 |
<https://www.eyrie.org/~eagle/software/c-tap-harness/> |
| 157 |
|
| 158 |
It's common to need additional test functions and utility functions |
| 159 |
for your C tests, particularly if you have to set up and tear down a |
| 160 |
test environment for your test programs, and it's useful to have them |
| 161 |
all in the libtap library so that you only have to link your test |
| 162 |
programs with one library. Rather than editing tap/basic.c and |
| 163 |
tap/basic.h to add those additional functions, add additional *.c and |
| 164 |
*.h files into the tap directory with the function implementations and |
| 165 |
prototypes, and then add those additional objects to the library. |
| 166 |
That way, you can update tap/basic.c and tap/basic.h from subsequent |
| 167 |
releases of C TAP Harness without having to merge changes with your |
| 168 |
own code. |
| 169 |
|
| 170 |
Libraries of additional useful TAP test functions are available in |
| 171 |
rra-c-util at: |
| 172 |
|
| 173 |
<https://www.eyrie.org/~eagle/software/rra-c-util/> |
| 174 |
|
| 175 |
Some of the code there is particularly useful when testing programs |
| 176 |
that require Kerberos keys. |
| 177 |
|
| 178 |
If you implement new test functions that compare an expected and seen |
| 179 |
value, it's best to name them is_<something> and take the expected |
| 180 |
value, the seen value, and then a printf-style format string and |
| 181 |
possible arguments to match the calling convention of the functions |
| 182 |
provided by C TAP Harness. |
| 183 |
|
| 184 |
Shell |
| 185 |
|
| 186 |
C TAP Harness provides a library of shell functions to make it easier |
| 187 |
to write TAP tests in shell. That library includes much of the same |
| 188 |
functionality as the C TAP library, but takes its parameters in a |
| 189 |
somewhat different order to make better use of shell features. |
| 190 |
|
| 191 |
The libtap.sh file should be installed in a directory named tap in |
| 192 |
your test suite area. It can then be loaded by tests written in shell |
| 193 |
using the environment set up by runtests with: |
| 194 |
|
| 195 |
. "$C_TAP_SOURCE"/tap/libtap.sh |
| 196 |
|
| 197 |
Here is a complete test case written in shell which produces the same |
| 198 |
output as the TAP sample above: |
| 199 |
|
| 200 |
#!/bin/sh |
| 201 |
|
| 202 |
. "$C_TAP_SOURCE"/tap/libtap.sh |
| 203 |
cd "$C_TAP_BUILD" |
| 204 |
|
| 205 |
plan 4 |
| 206 |
ok 'the first test' true |
| 207 |
ok '' [ 42 -eq 42 ] |
| 208 |
diag a diagnostic, ignored by the harness |
| 209 |
ok '' false |
| 210 |
skip 'a skipped test' |
| 211 |
|
| 212 |
The shell framework doesn't provide the is_* functions, so you'll use |
| 213 |
the ok function more. It takes a string describing the text and then |
| 214 |
treats all of its remaining arguments as a condition, evaluated the |
| 215 |
same way as the arguments to the "if" statement. If that condition |
| 216 |
evaluates to true, the test passes; otherwise, the test fails. |
| 217 |
|
| 218 |
The plan, plan_lazy, diag, and bail functions work the same as with |
| 219 |
the C library. skip takes a string and skips the next test with that |
| 220 |
explanation. skip_block takes a count and a string and skips that |
| 221 |
many tests with that explanation. skip_all takes an optional reason |
| 222 |
and skips the entire test case. |
| 223 |
|
| 224 |
Since it's common for shell programs to want to test the output of |
| 225 |
commands, there's an additional function ok_program provided by the |
| 226 |
shell test library. It takes the test description string, the |
| 227 |
expected exit status, the expected program output, and then treats the |
| 228 |
rest of its arguments as the program to run. That program is run with |
| 229 |
standard error and standard output combined, and then its exit status |
| 230 |
and output are tested against the provided values. |
| 231 |
|
| 232 |
A utility function, strip_colon_error, is provided that runs the |
| 233 |
command given as its arguments and strips text following a colon and a |
| 234 |
space from the output (unless there is no whitespace on the line |
| 235 |
before the colon and the space, normally indicating a prefix of the |
| 236 |
program name). This function can be used to wrap commands that are |
| 237 |
expected to fail with output that has a system- or locale-specific |
| 238 |
error message appended, such as the output of strerror(). |
| 239 |
|
| 240 |
License |
| 241 |
|
| 242 |
This file is part of the documentation of C TAP Harness, which can be |
| 243 |
found at <https://www.eyrie.org/~eagle/software/c-tap-harness/>. |
| 244 |
|
| 245 |
Copyright 2010, 2016 Russ Allbery <[email protected]> |
| 246 |
|
| 247 |
Copying and distribution of this file, with or without modification, |
| 248 |
are permitted in any medium without royalty provided the copyright |
| 249 |
notice and this notice are preserved. This file is offered as-is, |
| 250 |
without any warranty. |
| 251 |
|
| 252 |
SPDX-License-Identifier: FSFAP |