| 1 |
--- |
| 2 |
short-description: Meson's own unit-test system |
| 3 |
... |
| 4 |
|
| 5 |
# Unit tests |
| 6 |
|
| 7 |
Meson comes with a fully functional unit test system. To use it simply |
| 8 |
build an executable and then use it in a test. |
| 9 |
|
| 10 |
```meson |
| 11 |
e = executable('prog', 'testprog.c') |
| 12 |
test('name of test', e) |
| 13 |
``` |
| 14 |
|
| 15 |
You can add as many tests as you want. They are run with the command `meson |
| 16 |
test`. |
| 17 |
|
| 18 |
Meson captures the output of all tests and writes it in the log file |
| 19 |
`meson-logs/testlog.txt`. |
| 20 |
|
| 21 |
## Test parameters |
| 22 |
|
| 23 |
Some tests require the use of command line arguments or environment |
| 24 |
variables. These are simple to define. |
| 25 |
|
| 26 |
```meson |
| 27 |
test('command line test', exe, args : ['first', 'second']) |
| 28 |
test('envvar test', exe2, env : ['key1=value1', 'key2=value2']) |
| 29 |
``` |
| 30 |
|
| 31 |
Note how you need to specify multiple values as an array. |
| 32 |
|
| 33 |
### MALLOC_PERTURB_ |
| 34 |
|
| 35 |
By default, environment variable |
| 36 |
[`MALLOC_PERTURB_`](http://man7.org/linux/man-pages/man3/mallopt.3.html) is |
| 37 |
set to a random value between 1..255. This can help find memory leaks on |
| 38 |
configurations using glibc, including with non-GCC compilers. This feature |
| 39 |
can be disabled as discussed in [[test]]. |
| 40 |
|
| 41 |
### ASAN_OPTIONS, UBSAN_OPTIONS, and MSAN_OPTIONS |
| 42 |
|
| 43 |
By default, the environment variables `ASAN_OPTIONS`, `UBSAN_OPTIONS`, and |
| 44 |
`MSAN_OPTIONS` are set to enable aborting on detected violations and to give a |
| 45 |
backtrace. This feature can be disabled as discussed in [[test]]. |
| 46 |
|
| 47 |
## Coverage |
| 48 |
|
| 49 |
If you enable coverage measurements by giving Meson the command line |
| 50 |
flag `-Db_coverage=true`, you can generate coverage reports after |
| 51 |
running the tests (running the tests is required to gather the list of |
| 52 |
functions that get called). Meson will autodetect what coverage |
| 53 |
generator tools you have installed and will generate the corresponding |
| 54 |
targets. These targets are `coverage-xml` and `coverage-text` which |
| 55 |
are both provided by [Gcovr](http://gcovr.com) (version 3.3 or higher) |
| 56 |
`coverage-sonarqube` which is provided by [Gcovr](http://gcovr.com) (version 4.2 or higher) |
| 57 |
and `coverage-html`, which requires |
| 58 |
[lcov](https://github.com/linux-test-project/lcov) and |
| 59 |
[GenHTML](https://linux.die.net/man/1/genhtml) or |
| 60 |
[Gcovr](http://gcovr.com). As a convenience, a high-level `coverage` |
| 61 |
target is also generated which will produce all 3 coverage report |
| 62 |
types, if possible. |
| 63 |
|
| 64 |
The output of these commands is written to the log directory `meson-logs` in |
| 65 |
your build directory. |
| 66 |
|
| 67 |
## Parallelism |
| 68 |
|
| 69 |
To reduce test times, Meson will by default run multiple unit tests in |
| 70 |
parallel. It is common to have some tests which cannot be run in |
| 71 |
parallel because they require unique hold on some resource such as a |
| 72 |
file or a D-Bus name. You have to specify these tests with a keyword |
| 73 |
argument. |
| 74 |
|
| 75 |
```meson |
| 76 |
test('unique test', t, is_parallel : false) |
| 77 |
``` |
| 78 |
|
| 79 |
Meson will then make sure that no other unit test is running at the |
| 80 |
same time. Non-parallel tests take longer to run so it is recommended |
| 81 |
that you write your unit tests to be parallel executable whenever |
| 82 |
possible. |
| 83 |
|
| 84 |
By default Meson uses as many concurrent processes as there are cores |
| 85 |
on the test machine. You can override this with the environment |
| 86 |
variable `MESON_TESTTHREADS` or, *since 1.7.0*, `MESON_NUM_PROCESSES`: |
| 87 |
|
| 88 |
```console |
| 89 |
$ MESON_NUM_PROCESSES=5 meson test |
| 90 |
``` |
| 91 |
|
| 92 |
Setting `MESON_NUM_PROCESSES` to 0 enables the default behavior (core |
| 93 |
count), whereas setting an invalid value results in setting the job |
| 94 |
count to 1. |
| 95 |
|
| 96 |
If both environment variables are present, `MESON_NUM_PROCESSES` prevails. |
| 97 |
|
| 98 |
## Priorities |
| 99 |
|
| 100 |
*(added in version 0.52.0)* |
| 101 |
|
| 102 |
Tests can be assigned a priority that determines when a test is |
| 103 |
*started*. Tests with higher priority are started first, tests with |
| 104 |
lower priority started later. The default priority is 0, Meson makes |
| 105 |
no guarantee on the ordering of tests with identical priority. |
| 106 |
|
| 107 |
```meson |
| 108 |
test('started second', t, priority : 0) |
| 109 |
test('started third', t, priority : -50) |
| 110 |
test('started first', t, priority : 1000) |
| 111 |
``` |
| 112 |
|
| 113 |
Note that the test priority only affects the starting order of tests |
| 114 |
and subsequent tests are affected by how long it takes previous tests |
| 115 |
to complete. It is thus possible that a higher-priority test is still |
| 116 |
running when lower-priority tests with a shorter runtime have |
| 117 |
completed. |
| 118 |
|
| 119 |
## Skipped tests and hard errors |
| 120 |
|
| 121 |
Sometimes a test can only determine at runtime that it cannot be run. |
| 122 |
|
| 123 |
For the default `exitcode` testing protocol, the GNU standard approach |
| 124 |
in this case is to exit the program with error code 77. Meson will |
| 125 |
detect this and report these tests as skipped rather than failed. This |
| 126 |
behavior was added in version 0.37.0. |
| 127 |
|
| 128 |
For TAP-based tests, skipped tests should print a single line starting |
| 129 |
with `1..0 # SKIP`. |
| 130 |
|
| 131 |
In addition, sometimes a test fails set up so that it should fail even |
| 132 |
if it is marked as an expected failure. The GNU standard approach in |
| 133 |
this case is to exit the program with error code 99. Again, Meson will |
| 134 |
detect this and report these tests as `ERROR`, ignoring the setting of |
| 135 |
`should_fail`. This behavior was added in version 0.50.0. |
| 136 |
|
| 137 |
## Testing tool |
| 138 |
|
| 139 |
The goal of the Meson test tool is to provide a simple way to run |
| 140 |
tests in a variety of different ways. The tool is designed to be run |
| 141 |
in the build directory. |
| 142 |
|
| 143 |
The simplest thing to do is just to run all tests. |
| 144 |
|
| 145 |
```console |
| 146 |
$ meson test |
| 147 |
``` |
| 148 |
|
| 149 |
### Run subsets of tests |
| 150 |
|
| 151 |
For clarity, consider the meson.build containing: |
| 152 |
|
| 153 |
```meson |
| 154 |
|
| 155 |
test('A', ..., suite: 'foo') |
| 156 |
test('B', ..., suite: ['foo', 'bar']) |
| 157 |
test('C', ..., suite: 'bar') |
| 158 |
test('D', ..., suite: 'baz') |
| 159 |
|
| 160 |
``` |
| 161 |
|
| 162 |
Specify test(s) by name like: |
| 163 |
|
| 164 |
```console |
| 165 |
$ meson test A D |
| 166 |
``` |
| 167 |
|
| 168 |
You can run tests from specific (sub)project: |
| 169 |
|
| 170 |
```console |
| 171 |
$ meson test (sub)project_name: |
| 172 |
``` |
| 173 |
|
| 174 |
or a specific test in a specific project: |
| 175 |
|
| 176 |
```console |
| 177 |
$ meson test (sub)project_name:test_name |
| 178 |
``` |
| 179 |
|
| 180 |
Since version *1.2.0*, you can use wildcards in project |
| 181 |
and test names. For instance, to run all tests beginning with |
| 182 |
"foo" and all tests from projects beginning with "bar": |
| 183 |
|
| 184 |
```console |
| 185 |
$ meson test "foo*" "bar*:" |
| 186 |
``` |
| 187 |
|
| 188 |
|
| 189 |
Tests belonging to a suite `suite` can be run as follows |
| 190 |
|
| 191 |
```console |
| 192 |
$ meson test --suite (sub)project_name:suite |
| 193 |
``` |
| 194 |
|
| 195 |
Since version *0.46*, `(sub)project_name` can be omitted if it is the |
| 196 |
top-level project. |
| 197 |
|
| 198 |
Multiple suites are specified like: |
| 199 |
|
| 200 |
```console |
| 201 |
$ meson test --suite foo --suite bar |
| 202 |
``` |
| 203 |
|
| 204 |
NOTE: If you choose to specify both suite(s) and specific test |
| 205 |
name(s), the test name(s) must be contained in the suite(s). This |
| 206 |
however is redundant-- it would be more useful to specify either |
| 207 |
specific test names or suite(s). |
| 208 |
|
| 209 |
### Other test options |
| 210 |
|
| 211 |
Sometimes you need to run the tests multiple times, which is done like this: |
| 212 |
|
| 213 |
```console |
| 214 |
$ meson test --repeat=10 |
| 215 |
``` |
| 216 |
|
| 217 |
Meson will set the `MESON_TEST_ITERATION` environment variable to the |
| 218 |
current iteration of the test *(added 1.5.0)*. |
| 219 |
|
| 220 |
Invoking tests via a helper executable such as Valgrind can be done with the |
| 221 |
`--wrap` argument |
| 222 |
|
| 223 |
```console |
| 224 |
$ meson test --wrap=valgrind testname |
| 225 |
``` |
| 226 |
|
| 227 |
Arguments to the wrapper binary can be given like this: |
| 228 |
|
| 229 |
```console |
| 230 |
$ meson test --wrap='valgrind --tool=helgrind' testname |
| 231 |
``` |
| 232 |
|
| 233 |
Meson also supports running the tests under GDB. Just doing this: |
| 234 |
|
| 235 |
```console |
| 236 |
$ meson test --gdb testname |
| 237 |
``` |
| 238 |
|
| 239 |
Meson will launch `gdb` all set up to run the test. Just type `run` in |
| 240 |
the GDB command prompt to start the program. |
| 241 |
|
| 242 |
The second use case is a test that segfaults only rarely. In this case |
| 243 |
you can invoke the following command: |
| 244 |
|
| 245 |
```console |
| 246 |
$ meson test --gdb --repeat=10000 testname |
| 247 |
``` |
| 248 |
|
| 249 |
This runs the test up to 10 000 times under GDB automatically. If the |
| 250 |
program crashes, GDB will halt and the user can debug the application. |
| 251 |
Note that testing timeouts are disabled in this case so `meson test` |
| 252 |
will not kill `gdb` while the developer is still debugging it. The |
| 253 |
downside is that if the test binary freezes, the test runner will wait |
| 254 |
forever. |
| 255 |
|
| 256 |
Sometimes, the GDB binary is not in the PATH variable or the user |
| 257 |
wants to use a GDB replacement. Therefore, the invoked GDB program can |
| 258 |
be specified *(added 0.52.0)*: |
| 259 |
|
| 260 |
```console |
| 261 |
$ meson test --gdb --gdb-path /path/to/gdb testname |
| 262 |
``` |
| 263 |
|
| 264 |
```console |
| 265 |
$ meson test --print-errorlogs |
| 266 |
``` |
| 267 |
|
| 268 |
Running tests interactively can be done with the `--interactive` option. |
| 269 |
`meson test --interactive` invokes tests with stdout, stdin and stderr |
| 270 |
connected directly to the calling terminal. This can be useful if your test is |
| 271 |
an integration test running in a container or virtual machine where a debug |
| 272 |
shell is spawned if it fails *(added 1.5.0)*: |
| 273 |
|
| 274 |
```console |
| 275 |
$ meson test --interactive testname |
| 276 |
``` |
| 277 |
|
| 278 |
Meson will report the output produced by the failing tests along with |
| 279 |
other useful information as the environmental variables. This is |
| 280 |
useful, for example, when you run the tests on Travis-CI, Jenkins and |
| 281 |
the like. |
| 282 |
|
| 283 |
By default, the output from tests will be limited to the last 100 lines. The |
| 284 |
maximum number of lines to show can be configured with the `--max-lines` option |
| 285 |
*(added 1.5.0)*: |
| 286 |
|
| 287 |
```console |
| 288 |
$ meson test --max-lines=1000 testname |
| 289 |
``` |
| 290 |
|
| 291 |
**Timeout** |
| 292 |
|
| 293 |
In the test case options, the `timeout` option is specified in a number of seconds. |
| 294 |
|
| 295 |
To disable timeout in test cases, add `timeout: 0` or a negative value to allow |
| 296 |
infinite duration for the test case to complete. |
| 297 |
|
| 298 |
For running tests, you can specify a command line argument for overriding the |
| 299 |
timeout as well: |
| 300 |
|
| 301 |
```console |
| 302 |
$ meson test --timeout-multiplier 0 |
| 303 |
``` |
| 304 |
|
| 305 |
For further information see the command line help of Meson by running |
| 306 |
`meson test -h`. |
| 307 |
|
| 308 |
## Legacy notes |
| 309 |
|
| 310 |
If `meson test` does not work for you, you likely have a old version |
| 311 |
of Meson. In that case you should call `mesontest` instead. If |
| 312 |
`mesontest` doesn't work either you have a very old version prior to |
| 313 |
0.37.0 and should upgrade. |
| 314 |
|
| 315 |
## Test outputs |
| 316 |
|
| 317 |
Meson will write several different files with detailed results of |
| 318 |
running tests. These will be written into $builddir/meson-logs/ |
| 319 |
|
| 320 |
### testlog.json |
| 321 |
|
| 322 |
This is not a proper json file, but a file containing one valid json |
| 323 |
object per line. This is file is designed so each line is streamed out |
| 324 |
as each test is run, so it can be read as a stream while the test |
| 325 |
harness is running |
| 326 |
|
| 327 |
### testlog.junit.xml |
| 328 |
|
| 329 |
This is a valid JUnit XML description of all tests run. It is not |
| 330 |
streamed out, and is written only once all tests complete running. |
| 331 |
|
| 332 |
When tests use the `tap` protocol each test will be recorded as a |
| 333 |
testsuite container, with each case named by the number of the result. |
| 334 |
|
| 335 |
When tests use the `gtest` protocol Meson will inject arguments to the |
| 336 |
test to generate its own JUnit XML, which Meson will include as part |
| 337 |
of this XML file. |
| 338 |
|
| 339 |
*New in 0.55.0* |