Debian Codegraph @

subprojects/meson-docs/docs/markdown/Unit-tests.md

raw
1 ---
2 short-description: Meson's own unit-test system
3 ...
4
5 # Unit tests
6
7 Meson comes with a fully functional unit test system. To use it simply
8 build an executable and then use it in a test.
9
10 ```meson
11 e = executable('prog', 'testprog.c')
12 test('name of test', e)
13 ```
14
15 You can add as many tests as you want. They are run with the command `meson
16 test`.
17
18 Meson captures the output of all tests and writes it in the log file
19 `meson-logs/testlog.txt`.
20
21 ## Test parameters
22
23 Some tests require the use of command line arguments or environment
24 variables. These are simple to define.
25
26 ```meson
27 test('command line test', exe, args : ['first', 'second'])
28 test('envvar test', exe2, env : ['key1=value1', 'key2=value2'])
29 ```
30
31 Note how you need to specify multiple values as an array.
32
33 ### MALLOC_PERTURB_
34
35 By default, environment variable
36 [`MALLOC_PERTURB_`](http://man7.org/linux/man-pages/man3/mallopt.3.html) is
37 set to a random value between 1..255. This can help find memory leaks on
38 configurations using glibc, including with non-GCC compilers. This feature
39 can be disabled as discussed in [[test]].
40
41 ### ASAN_OPTIONS, UBSAN_OPTIONS, and MSAN_OPTIONS
42
43 By default, the environment variables `ASAN_OPTIONS`, `UBSAN_OPTIONS`, and
44 `MSAN_OPTIONS` are set to enable aborting on detected violations and to give a
45 backtrace. This feature can be disabled as discussed in [[test]].
46
47 ## Coverage
48
49 If you enable coverage measurements by giving Meson the command line
50 flag `-Db_coverage=true`, you can generate coverage reports after
51 running the tests (running the tests is required to gather the list of
52 functions that get called). Meson will autodetect what coverage
53 generator tools you have installed and will generate the corresponding
54 targets. These targets are `coverage-xml` and `coverage-text` which
55 are both provided by [Gcovr](http://gcovr.com) (version 3.3 or higher)
56 `coverage-sonarqube` which is provided by [Gcovr](http://gcovr.com) (version 4.2 or higher)
57 and `coverage-html`, which requires
58 [lcov](https://github.com/linux-test-project/lcov) and
59 [GenHTML](https://linux.die.net/man/1/genhtml) or
60 [Gcovr](http://gcovr.com). As a convenience, a high-level `coverage`
61 target is also generated which will produce all 3 coverage report
62 types, if possible.
63
64 The output of these commands is written to the log directory `meson-logs` in
65 your build directory.
66
67 ## Parallelism
68
69 To reduce test times, Meson will by default run multiple unit tests in
70 parallel. It is common to have some tests which cannot be run in
71 parallel because they require unique hold on some resource such as a
72 file or a D-Bus name. You have to specify these tests with a keyword
73 argument.
74
75 ```meson
76 test('unique test', t, is_parallel : false)
77 ```
78
79 Meson will then make sure that no other unit test is running at the
80 same time. Non-parallel tests take longer to run so it is recommended
81 that you write your unit tests to be parallel executable whenever
82 possible.
83
84 By default Meson uses as many concurrent processes as there are cores
85 on the test machine. You can override this with the environment
86 variable `MESON_TESTTHREADS` or, *since 1.7.0*, `MESON_NUM_PROCESSES`:
87
88 ```console
89 $ MESON_NUM_PROCESSES=5 meson test
90 ```
91
92 Setting `MESON_NUM_PROCESSES` to 0 enables the default behavior (core
93 count), whereas setting an invalid value results in setting the job
94 count to 1.
95
96 If both environment variables are present, `MESON_NUM_PROCESSES` prevails.
97
98 ## Priorities
99
100 *(added in version 0.52.0)*
101
102 Tests can be assigned a priority that determines when a test is
103 *started*. Tests with higher priority are started first, tests with
104 lower priority started later. The default priority is 0, Meson makes
105 no guarantee on the ordering of tests with identical priority.
106
107 ```meson
108 test('started second', t, priority : 0)
109 test('started third', t, priority : -50)
110 test('started first', t, priority : 1000)
111 ```
112
113 Note that the test priority only affects the starting order of tests
114 and subsequent tests are affected by how long it takes previous tests
115 to complete. It is thus possible that a higher-priority test is still
116 running when lower-priority tests with a shorter runtime have
117 completed.
118
119 ## Skipped tests and hard errors
120
121 Sometimes a test can only determine at runtime that it cannot be run.
122
123 For the default `exitcode` testing protocol, the GNU standard approach
124 in this case is to exit the program with error code 77. Meson will
125 detect this and report these tests as skipped rather than failed. This
126 behavior was added in version 0.37.0.
127
128 For TAP-based tests, skipped tests should print a single line starting
129 with `1..0 # SKIP`.
130
131 In addition, sometimes a test fails set up so that it should fail even
132 if it is marked as an expected failure. The GNU standard approach in
133 this case is to exit the program with error code 99. Again, Meson will
134 detect this and report these tests as `ERROR`, ignoring the setting of
135 `should_fail`. This behavior was added in version 0.50.0.
136
137 ## Testing tool
138
139 The goal of the Meson test tool is to provide a simple way to run
140 tests in a variety of different ways. The tool is designed to be run
141 in the build directory.
142
143 The simplest thing to do is just to run all tests.
144
145 ```console
146 $ meson test
147 ```
148
149 ### Run subsets of tests
150
151 For clarity, consider the meson.build containing:
152
153 ```meson
154
155 test('A', ..., suite: 'foo')
156 test('B', ..., suite: ['foo', 'bar'])
157 test('C', ..., suite: 'bar')
158 test('D', ..., suite: 'baz')
159
160 ```
161
162 Specify test(s) by name like:
163
164 ```console
165 $ meson test A D
166 ```
167
168 You can run tests from specific (sub)project:
169
170 ```console
171 $ meson test (sub)project_name:
172 ```
173
174 or a specific test in a specific project:
175
176 ```console
177 $ meson test (sub)project_name:test_name
178 ```
179
180 Since version *1.2.0*, you can use wildcards in project
181 and test names. For instance, to run all tests beginning with
182 "foo" and all tests from projects beginning with "bar":
183
184 ```console
185 $ meson test "foo*" "bar*:"
186 ```
187
188
189 Tests belonging to a suite `suite` can be run as follows
190
191 ```console
192 $ meson test --suite (sub)project_name:suite
193 ```
194
195 Since version *0.46*, `(sub)project_name` can be omitted if it is the
196 top-level project.
197
198 Multiple suites are specified like:
199
200 ```console
201 $ meson test --suite foo --suite bar
202 ```
203
204 NOTE: If you choose to specify both suite(s) and specific test
205 name(s), the test name(s) must be contained in the suite(s). This
206 however is redundant-- it would be more useful to specify either
207 specific test names or suite(s).
208
209 ### Other test options
210
211 Sometimes you need to run the tests multiple times, which is done like this:
212
213 ```console
214 $ meson test --repeat=10
215 ```
216
217 Meson will set the `MESON_TEST_ITERATION` environment variable to the
218 current iteration of the test *(added 1.5.0)*.
219
220 Invoking tests via a helper executable such as Valgrind can be done with the
221 `--wrap` argument
222
223 ```console
224 $ meson test --wrap=valgrind testname
225 ```
226
227 Arguments to the wrapper binary can be given like this:
228
229 ```console
230 $ meson test --wrap='valgrind --tool=helgrind' testname
231 ```
232
233 Meson also supports running the tests under GDB. Just doing this:
234
235 ```console
236 $ meson test --gdb testname
237 ```
238
239 Meson will launch `gdb` all set up to run the test. Just type `run` in
240 the GDB command prompt to start the program.
241
242 The second use case is a test that segfaults only rarely. In this case
243 you can invoke the following command:
244
245 ```console
246 $ meson test --gdb --repeat=10000 testname
247 ```
248
249 This runs the test up to 10 000 times under GDB automatically. If the
250 program crashes, GDB will halt and the user can debug the application.
251 Note that testing timeouts are disabled in this case so `meson test`
252 will not kill `gdb` while the developer is still debugging it. The
253 downside is that if the test binary freezes, the test runner will wait
254 forever.
255
256 Sometimes, the GDB binary is not in the PATH variable or the user
257 wants to use a GDB replacement. Therefore, the invoked GDB program can
258 be specified *(added 0.52.0)*:
259
260 ```console
261 $ meson test --gdb --gdb-path /path/to/gdb testname
262 ```
263
264 ```console
265 $ meson test --print-errorlogs
266 ```
267
268 Running tests interactively can be done with the `--interactive` option.
269 `meson test --interactive` invokes tests with stdout, stdin and stderr
270 connected directly to the calling terminal. This can be useful if your test is
271 an integration test running in a container or virtual machine where a debug
272 shell is spawned if it fails *(added 1.5.0)*:
273
274 ```console
275 $ meson test --interactive testname
276 ```
277
278 Meson will report the output produced by the failing tests along with
279 other useful information as the environmental variables. This is
280 useful, for example, when you run the tests on Travis-CI, Jenkins and
281 the like.
282
283 By default, the output from tests will be limited to the last 100 lines. The
284 maximum number of lines to show can be configured with the `--max-lines` option
285 *(added 1.5.0)*:
286
287 ```console
288 $ meson test --max-lines=1000 testname
289 ```
290
291 **Timeout**
292
293 In the test case options, the `timeout` option is specified in a number of seconds.
294
295 To disable timeout in test cases, add `timeout: 0` or a negative value to allow
296 infinite duration for the test case to complete.
297
298 For running tests, you can specify a command line argument for overriding the
299 timeout as well:
300
301 ```console
302 $ meson test --timeout-multiplier 0
303 ```
304
305 For further information see the command line help of Meson by running
306 `meson test -h`.
307
308 ## Legacy notes
309
310 If `meson test` does not work for you, you likely have a old version
311 of Meson. In that case you should call `mesontest` instead. If
312 `mesontest` doesn't work either you have a very old version prior to
313 0.37.0 and should upgrade.
314
315 ## Test outputs
316
317 Meson will write several different files with detailed results of
318 running tests. These will be written into $builddir/meson-logs/
319
320 ### testlog.json
321
322 This is not a proper json file, but a file containing one valid json
323 object per line. This is file is designed so each line is streamed out
324 as each test is run, so it can be read as a stream while the test
325 harness is running
326
327 ### testlog.junit.xml
328
329 This is a valid JUnit XML description of all tests run. It is not
330 streamed out, and is written only once all tests complete running.
331
332 When tests use the `tap` protocol each test will be recorded as a
333 testsuite container, with each case named by the number of the result.
334
335 When tests use the `gtest` protocol Meson will inject arguments to the
336 test to generate its own JUnit XML, which Meson will include as part
337 of this XML file.
338
339 *New in 0.55.0*