| 1 |
.\" Copyright (c) 2007 The NetBSD Foundation, Inc. |
| 2 |
.\" All rights reserved. |
| 3 |
.\" |
| 4 |
.\" Redistribution and use in source and binary forms, with or without |
| 5 |
.\" modification, are permitted provided that the following conditions |
| 6 |
.\" are met: |
| 7 |
.\" 1. Redistributions of source code must retain the above copyright |
| 8 |
.\" notice, this list of conditions and the following disclaimer. |
| 9 |
.\" 2. Redistributions in binary form must reproduce the above copyright |
| 10 |
.\" notice, this list of conditions and the following disclaimer in the |
| 11 |
.\" documentation and/or other materials provided with the distribution. |
| 12 |
.\" |
| 13 |
.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND |
| 14 |
.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, |
| 15 |
.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF |
| 16 |
.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. |
| 17 |
.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY |
| 18 |
.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL |
| 19 |
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE |
| 20 |
.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS |
| 21 |
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER |
| 22 |
.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR |
| 23 |
.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN |
| 24 |
.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
| 25 |
.Dd July 3, 2015 |
| 26 |
.Dt ATF-TEST-CASE 4 |
| 27 |
.Os |
| 28 |
.Sh NAME |
| 29 |
.Nm atf-test-case |
| 30 |
.Nd generic description of test cases |
| 31 |
.Sh DESCRIPTION |
| 32 |
A |
| 33 |
.Em test case |
| 34 |
is a piece of code that stress-tests a specific feature of the software. |
| 35 |
This feature is typically self-contained enough, either in the amount of |
| 36 |
code that implements it or in the general idea that describes it, to |
| 37 |
warrant its independent testing. |
| 38 |
Given this, test cases are very fine-grained, but they attempt to group |
| 39 |
similar smaller tests which are semantically related. |
| 40 |
.Pp |
| 41 |
A test case is defined by three components regardless of the language it is |
| 42 |
implemented in: a header, a body and a cleanup routine. |
| 43 |
The |
| 44 |
.Em header |
| 45 |
is, basically, a declarative piece of code that defines several |
| 46 |
properties to describe what the test case does and how it behaves. |
| 47 |
In other words: it defines the test case's |
| 48 |
.Em meta-data , |
| 49 |
further described in the |
| 50 |
.Sx Meta-data |
| 51 |
section. |
| 52 |
The |
| 53 |
.Em body |
| 54 |
is the test case itself. |
| 55 |
It executes all actions needed to reproduce the test, and checks for |
| 56 |
failures. |
| 57 |
This body is only executed if the abstract conditions specified by the |
| 58 |
header are met. |
| 59 |
The |
| 60 |
.Em cleanup |
| 61 |
routine is a piece of code always executed after the body, regardless of |
| 62 |
the exit status of the test case. |
| 63 |
It can be used to undo side-effects of the test case. |
| 64 |
Note that almost all side-effects of a test case are automatically cleaned |
| 65 |
up by the library; this is explained in more detail in the rest of this |
| 66 |
document. |
| 67 |
.Pp |
| 68 |
It is extremely important to keep the separation between a test case's |
| 69 |
header and body well-defined, because the header is |
| 70 |
.Em always |
| 71 |
parsed, whereas the body is only executed when the conditions defined in |
| 72 |
the header are met and when the user specifies that test case. |
| 73 |
.Pp |
| 74 |
At last, test cases are always contained into test programs. |
| 75 |
The test programs act as a front-end to them, providing a consistent |
| 76 |
interface to the user and several APIs to ease their implementation. |
| 77 |
.Ss Results |
| 78 |
Upon termination, a test case reports a status and, optionally, a textual |
| 79 |
reason describing why the test reported such status. |
| 80 |
The caller must ensure that the test case really performed the task that its |
| 81 |
status describes, as the test program may be bogus and therefore providing a |
| 82 |
misleading result, e.g., providing a result that indicates success but the |
| 83 |
error code of the program says otherwise. |
| 84 |
.Pp |
| 85 |
The possible exit status of a test case are one of the following: |
| 86 |
.Bl -tag -width expectedXfailureXX |
| 87 |
.It expected_death |
| 88 |
The test case expects to terminate abruptly. |
| 89 |
.It expected_exit |
| 90 |
The test case expects to exit cleanly. |
| 91 |
.It expected_failure |
| 92 |
The test case expects to exit with a controller fatal/non-fatal failure. |
| 93 |
If this happens, the test program exits with a success error code. |
| 94 |
.It expected_signal |
| 95 |
The test case expects to receive a signal that makes it terminate. |
| 96 |
.It expected_timeout |
| 97 |
The test case expects to execute for longer than its timeout. |
| 98 |
.It passed |
| 99 |
The test case was executed successfully. |
| 100 |
The test program exits with a success error code. |
| 101 |
.It skipped |
| 102 |
The test case could not be executed because some preconditions were not |
| 103 |
met. |
| 104 |
This is not a failure because it can typically be resolved by adjusting |
| 105 |
the system to meet the necessary conditions. |
| 106 |
This is always accompanied by a |
| 107 |
.Em reason , |
| 108 |
a message describing why the test was skipped. |
| 109 |
The test program exits with a success error code. |
| 110 |
.It failed |
| 111 |
An error appeared during the execution of the test case. |
| 112 |
This is always accompanied by a |
| 113 |
.Em reason , |
| 114 |
a message describing why the test failed. |
| 115 |
The test program exits with a failure error code. |
| 116 |
.El |
| 117 |
.Pp |
| 118 |
The usefulness of the |
| 119 |
.Sq expected_* |
| 120 |
results comes when writing test cases that verify known failures caused, |
| 121 |
in general, due to programming errors (aka bugs). |
| 122 |
Whenever the faulty condition that the |
| 123 |
.Sq expected_* |
| 124 |
result is trying to cover is fixed, then the test case will be reported as |
| 125 |
.Sq failed |
| 126 |
and the developer will have to adjust it to match its new condition. |
| 127 |
.Pp |
| 128 |
It is important to note that all |
| 129 |
.Sq expected_* |
| 130 |
results are only provided as a |
| 131 |
.Em hint |
| 132 |
to the caller; the caller must verify that the test case did actually terminate |
| 133 |
as the expected condition says. |
| 134 |
.Ss Input/output |
| 135 |
Test cases are free to print whatever they want to their |
| 136 |
.Xr stdout 4 |
| 137 |
and |
| 138 |
.Xr stderr 4 |
| 139 |
file descriptors. |
| 140 |
They are, in fact, encouraged to print status information as they execute |
| 141 |
to keep the user informed of their actions. |
| 142 |
This is specially important for long test cases. |
| 143 |
.Pp |
| 144 |
Test cases will log their results to an auxiliary file, which is then |
| 145 |
collected by the test program they are contained in. |
| 146 |
The developer need not care about this as long as he uses the correct |
| 147 |
APIs to implement the test cases. |
| 148 |
.Pp |
| 149 |
The standard input of the test cases is unconditionally connected to |
| 150 |
.Sq /dev/zero . |
| 151 |
.Ss Meta-data |
| 152 |
The following metadata properties can be exposed via the test case's head: |
| 153 |
.Bl -tag -width requireXmachineXX |
| 154 |
.It descr |
| 155 |
Type: textual. |
| 156 |
Required. |
| 157 |
.Pp |
| 158 |
A brief textual description of the test case's purpose. |
| 159 |
Will be shown to the user in reports. |
| 160 |
Also good for documentation purposes. |
| 161 |
.It has.cleanup |
| 162 |
Type: boolean. |
| 163 |
Optional. |
| 164 |
.Pp |
| 165 |
If set to true, specifies that the test case has a cleanup routine that has |
| 166 |
to be executed by the runtime engine during the cleanup phase of the execution. |
| 167 |
This property is automatically set by the framework when defining a test case |
| 168 |
with a cleanup routine, so it should never be set by hand. |
| 169 |
.It ident |
| 170 |
Type: textual. |
| 171 |
Required. |
| 172 |
.Pp |
| 173 |
The test case's identifier. |
| 174 |
Must be unique inside the test program and should be short but descriptive. |
| 175 |
.It require.arch |
| 176 |
Type: textual. |
| 177 |
Optional. |
| 178 |
.Pp |
| 179 |
A whitespace separated list of architectures that the test case can be run |
| 180 |
under without causing errors due to an architecture mismatch. |
| 181 |
.It require.config |
| 182 |
Type: textual. |
| 183 |
Optional. |
| 184 |
.Pp |
| 185 |
A whitespace separated list of configuration variables that must be defined |
| 186 |
to execute the test case. |
| 187 |
If any of the required variables is not defined, the test case is |
| 188 |
.Em skipped . |
| 189 |
.It require.diskspace |
| 190 |
Type: integer. |
| 191 |
Optional. |
| 192 |
Specifies the minimum amount of available disk space needed by the test. |
| 193 |
The value can have a size suffix such as |
| 194 |
.Sq K , |
| 195 |
.Sq M , |
| 196 |
.Sq G |
| 197 |
or |
| 198 |
.Sq T |
| 199 |
to make the amount of bytes easier to type and read. |
| 200 |
.It require.files |
| 201 |
Type: textual. |
| 202 |
Optional. |
| 203 |
.Pp |
| 204 |
A whitespace separated list of files that must be present to execute the |
| 205 |
test case. |
| 206 |
The names of these files must be absolute paths. |
| 207 |
If any of the required files is not found, the test case is |
| 208 |
.Em skipped . |
| 209 |
.It require.machine |
| 210 |
Type: textual. |
| 211 |
Optional. |
| 212 |
.Pp |
| 213 |
A whitespace separated list of machine types that the test case can be run |
| 214 |
under without causing errors due to a machine type mismatch. |
| 215 |
.It require.memory |
| 216 |
Type: integer. |
| 217 |
Optional. |
| 218 |
Specifies the minimum amount of physical memory needed by the test. |
| 219 |
The value can have a size suffix such as |
| 220 |
.Sq K , |
| 221 |
.Sq M , |
| 222 |
.Sq G |
| 223 |
or |
| 224 |
.Sq T |
| 225 |
to make the amount of bytes easier to type and read. |
| 226 |
.It require.progs |
| 227 |
Type: textual. |
| 228 |
Optional. |
| 229 |
.Pp |
| 230 |
A whitespace separated list of programs that must be present to execute |
| 231 |
the test case. |
| 232 |
These can be given as plain names, in which case they are looked in the |
| 233 |
user's |
| 234 |
.Ev PATH , |
| 235 |
or as absolute paths. |
| 236 |
If any of the required programs is not found, the test case is |
| 237 |
.Em skipped . |
| 238 |
.It require.user |
| 239 |
Type: textual. |
| 240 |
Optional. |
| 241 |
.Pp |
| 242 |
The required privileges to execute the test case. |
| 243 |
Can be one of |
| 244 |
.Sq root |
| 245 |
or |
| 246 |
.Sq unprivileged . |
| 247 |
.Pp |
| 248 |
If the test case is running as a regular user and this property is |
| 249 |
.Sq root , |
| 250 |
the test case is |
| 251 |
.Em skipped . |
| 252 |
.Pp |
| 253 |
If the test case is running as root and this property is |
| 254 |
.Sq unprivileged , |
| 255 |
the runtime engine will automatically drop the privileges if the |
| 256 |
.Sq unprivileged-user |
| 257 |
configuration property is set; otherwise the test case is |
| 258 |
.Em skipped . |
| 259 |
.It timeout |
| 260 |
Type: integral. |
| 261 |
Optional; defaults to |
| 262 |
.Sq 300 . |
| 263 |
.Pp |
| 264 |
Specifies the maximum amount of time the test case can run. |
| 265 |
This is particularly useful because some tests can stall either because they |
| 266 |
are incorrectly coded or because they trigger an anomalous behavior of the |
| 267 |
program. |
| 268 |
It is not acceptable for these tests to stall the whole execution of the |
| 269 |
test program. |
| 270 |
.Pp |
| 271 |
Can optionally be set to zero, in which case the test case has no run-time |
| 272 |
limit. |
| 273 |
This is discouraged. |
| 274 |
.It X- Ns Sq NAME |
| 275 |
Type: textual. |
| 276 |
Optional. |
| 277 |
.Pp |
| 278 |
A user-defined property named |
| 279 |
.Sq NAME . |
| 280 |
These properties are free form, have no special meaning within ATF, and can |
| 281 |
be specified at will by the test case. |
| 282 |
The runtime engine should propagate these properties from the test case to |
| 283 |
the end user so that the end user can rely on custom properties for test case |
| 284 |
tagging and classification. |
| 285 |
.El |
| 286 |
.Ss Environment |
| 287 |
Every time a test case is executed, several environment variables are |
| 288 |
cleared or reseted to sane values to ensure they do not make the test fail |
| 289 |
due to unexpected conditions. |
| 290 |
These variables are: |
| 291 |
.Bl -tag -width LCXMESSAGESXX |
| 292 |
.It Ev HOME |
| 293 |
Set to the work directory's path. |
| 294 |
.It Ev LANG |
| 295 |
Undefined. |
| 296 |
.It Ev LC_ALL |
| 297 |
Undefined. |
| 298 |
.It Ev LC_COLLATE |
| 299 |
Undefined. |
| 300 |
.It Ev LC_CTYPE |
| 301 |
Undefined. |
| 302 |
.It Ev LC_MESSAGES |
| 303 |
Undefined. |
| 304 |
.It Ev LC_MONETARY |
| 305 |
Undefined. |
| 306 |
.It Ev LC_NUMERIC |
| 307 |
Undefined. |
| 308 |
.It Ev LC_TIME |
| 309 |
Undefined. |
| 310 |
.It Ev TZ |
| 311 |
Hardcoded to |
| 312 |
.Sq UTC . |
| 313 |
.El |
| 314 |
.Ss Work directories |
| 315 |
The test program always creates a temporary directory |
| 316 |
and switches to it before running the test case's body. |
| 317 |
This way the test case is free to modify its current directory as it |
| 318 |
wishes, and the runtime engine will be able to clean it up later on in a |
| 319 |
safe way, removing any traces of its execution from the system. |
| 320 |
To do so, the runtime engine will perform a recursive removal of the work |
| 321 |
directory without crossing mount points; if a mount point is found, the |
| 322 |
file system will be unmounted (if possible). |
| 323 |
.Ss File creation mode mask (umask) |
| 324 |
Test cases are always executed with a file creation mode mask (umask) of |
| 325 |
.Sq 0022 . |
| 326 |
The test case's code is free to change this during execution. |
| 327 |
.Sh SEE ALSO |
| 328 |
.Xr atf-test-program 1 |