| 1 |
.. _imap-developer-unit-tests: |
| 2 |
|
| 3 |
.. Note: This document was converted from the original by Nic Bernstein |
| 4 |
(Onlight). Any formatting mistakes are my fault and not the |
| 5 |
original author's. |
| 6 |
|
| 7 |
Unit Tests |
| 8 |
========== |
| 9 |
|
| 10 |
Table of Contents |
| 11 |
----------------- |
| 12 |
|
| 13 |
- `1. Introduction <#introduction>`__ |
| 14 |
- `2. What Is A Unit Test? <#what-is-a-unit-test>`__ |
| 15 |
- `3. Running The Tests <#running-the-tests>`__ |
| 16 |
|
| 17 |
- `3.1. Setting Up The Machine <#setting-up-the-machine>`__ |
| 18 |
- `3.2 Configure Script <#configure-scripts>`__ |
| 19 |
- `3.3 Make <#running-the-tests>`__ |
| 20 |
- `3.4 Using Valgrind <#using-valgrind>`__ |
| 21 |
- `3.5 The Tests Are Failing <#the-tests-are-failing>`__ |
| 22 |
- `3.6 Debugging A Test <#debugging-a-test>`__ |
| 23 |
|
| 24 |
- `4. Adding Your Own Tests <#adding-your-own-tests>`__ |
| 25 |
|
| 26 |
- `4.1 Where To Put Your Tests <#where-to-put-your-tests>`__ |
| 27 |
- `4.1 Adding A New Suite <#adding-a-new-suite>`__ |
| 28 |
- `4.2 Adding A Test To A Suite <#adding-a-test-to-a-suite>`__ |
| 29 |
- `4.3 Suite Init And Cleanup <#suite-init-and-cleanup>`__ |
| 30 |
|
| 31 |
1. Introduction |
| 32 |
--------------- |
| 33 |
|
| 34 |
Recently, a set of regression unit tests has been added to Cyrus. This |
| 35 |
document explains the purpose implementation of those unit tests, and |
| 36 |
gives an example of how to add more unit tests (because there are never |
| 37 |
enough unit tests!). |
| 38 |
|
| 39 |
2. What Is A Unit Test? |
| 40 |
----------------------- |
| 41 |
|
| 42 |
The `definition on Wikipedia <http://en.wikipedia.org/wiki/Unit_test>`__ |
| 43 |
sheds some light: |
| 44 |
|
| 45 |
...\ **unit testing** is a method by which individual units of |
| 46 |
source code are tested to determine if they are fit for use. A unit |
| 47 |
is the smallest testable part of an application. |
| 48 |
|
| 49 |
In other words, unit testing is about verifying that small pieces of |
| 50 |
code, like individual functions, modules, or classes, work in isolation. |
| 51 |
It is **not** about testing the system as a whole. |
| 52 |
|
| 53 |
The tests implemented here are also **regression tests**, which in |
| 54 |
`Wikipedia's words <http://en.wikipedia.org/wiki/Regression_testing>`__ |
| 55 |
means: |
| 56 |
|
| 57 |
**Regression testing** is any type of software testing that seeks to |
| 58 |
uncover software errors after changes to the program (e.g. bugfixes |
| 59 |
or new functionality) have been made, by retesting the program. The |
| 60 |
intent of regression testing is to assure that a change, such as a |
| 61 |
bugfix, did not introduce new bugs. |
| 62 |
|
| 63 |
In other words, the tests are designed to be easy to run and to work out |
| 64 |
fully automatically whether they have passed or failed, so that they can |
| 65 |
be run usefully by people who didn't write them. |
| 66 |
|
| 67 |
3. Running The Tests |
| 68 |
-------------------- |
| 69 |
|
| 70 |
This section takes you through the process of running Cyrus' unit tests. |
| 71 |
|
| 72 |
3.1. Setting Up The Machine |
| 73 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 74 |
|
| 75 |
Cyrus' unit tests are all located in a new directory, |
| 76 |
``cyrus-imapd/cunit/``. They're written in C, like the remainder of |
| 77 |
Cyrus, and use the `CUnit library from |
| 78 |
SourceForge <http://cunit.sourceforge.net/>`__, with some home grown |
| 79 |
wrappers and other improvements to make our lives easier. |
| 80 |
|
| 81 |
Your first step is step is to ensure that the CUnit library (including |
| 82 |
the headers) is installed. Some modern operating systems already have |
| 83 |
CUnit, for example on Ubuntu you can just do: |
| 84 |
|
| 85 |
:: |
| 86 |
|
| 87 |
me@ubuntu> sudo apt-get install libcunit1-dev |
| 88 |
|
| 89 |
Alternately, you can download the CUnit source, build it and install it. |
| 90 |
It's not a complicated or difficult library, this shouldn't take long. |
| 91 |
When you've done, install it in ``/usr/include`` and ``/usr/lib``. |
| 92 |
|
| 93 |
3.2 Configure Script |
| 94 |
~~~~~~~~~~~~~~~~~~~~ |
| 95 |
|
| 96 |
Because of the dependency on the CUnit library, the tests are disabled |
| 97 |
by default; this means you need enable them with an option to the |
| 98 |
``configure`` script: |
| 99 |
|
| 100 |
:: |
| 101 |
|
| 102 |
me@mybox> ./configure --enable-unit-tests |
| 103 |
... |
| 104 |
checking for CU\_initialize\_registry in -lcunit... yes |
| 105 |
checking CUnit/CUnit.h usability... yes |
| 106 |
checking CUnit/CUnit.h presence... yes |
| 107 |
checking for CUnit/CUnit.h... yes |
| 108 |
... |
| 109 |
|
| 110 |
3.3 Make |
| 111 |
~~~~~~~~ |
| 112 |
|
| 113 |
First you need to build Cyrus itself, using the traditional ``all:`` |
| 114 |
target. |
| 115 |
|
| 116 |
:: |
| 117 |
|
| 118 |
me@mybox> make all |
| 119 |
... |
| 120 |
|
| 121 |
Then, use the new ``check:`` target to build and run the unit tests. |
| 122 |
|
| 123 |
:: |
| 124 |
|
| 125 |
me@mybox> make check |
| 126 |
cd . && /bin/bash /home/me/cyrus-imapd/missing --run automake-1.11 --foreign Makefile |
| 127 |
cd . && /bin/bash ./config.status Makefile depfiles (a) |
| 128 |
config.status: creating Makefile |
| 129 |
config.status: executing depfiles commands |
| 130 |
... |
| 131 |
make[3]: Entering directory \`/home/me/cyrus-imapd' |
| 132 |
make[3]: \`sieve/test' is up to date. |
| 133 |
cunit/cunit.pl --project cunit/default.cunit --generate-wrapper cunit/mboxname.testc (b) |
| 134 |
gcc -DHAVE\_CONFIG\_H ... -c -o cunit/mboxname.o cunit/mboxname.testc-cunit.c |
| 135 |
rm -f cunit/mboxname.testc-cunit.c |
| 136 |
/bin/bash ./libtool --tag=CC --mode=link gcc -fPIC -g -O2 -o |
| 137 |
cunit/unit cunit/unit.o ... lib/libcyrus\_min.la ... (c) |
| 138 |
... |
| 139 |
Running unit tests (d) |
| 140 |
|
| 141 |
CUnit - A Unit testing framework for C - Version 2.1-0 |
| 142 |
http://cunit.sourceforge.net/ |
| 143 |
|
| 144 |
... |
| 145 |
Suite: mboxname (e) |
| 146 |
Test: dir\_hash\_c ... passed |
| 147 |
Test: to\_parts ... passed |
| 148 |
Test: to\_userid ... passed |
| 149 |
Test: to\_usermbox ... passed |
| 150 |
... |
| 151 |
--Run Summary: Type Total Ran Passed Failed (f) |
| 152 |
suites 34 34 n/a 0 |
| 153 |
tests 323 323 323 0 |
| 154 |
asserts 1079745 1079745 1079745 0 |
| 155 |
make[1]: Leaving directory `/home/me/cyrus-imapd/cunit' |
| 156 |
|
| 157 |
Let's take a closer look at what's happening here. |
| 158 |
|
| 159 |
(a) |
| 160 |
The ``check:`` target causes automake to re-run itself. This is |
| 161 |
normal automake behaviour. Note that the older build system used to |
| 162 |
run make recursively in sub-directories, the newer automake-based |
| 163 |
system builds everything from the top directory. |
| 164 |
(b) |
| 165 |
The ``cunit/`` directory contains a number of C source files |
| 166 |
(called, for reasons too complicated to explain here, |
| 167 |
*whatever*.testc) with test code in them. For each of those, a small |
| 168 |
wrapper C source file is generated and then compiled into an object |
| 169 |
file. |
| 170 |
(c) |
| 171 |
Finally, all the compiled object files are linked into an |
| 172 |
executable, with a ``main()`` routine from ``unit.c``, and a number |
| 173 |
of libraries and object files from other parts of the Cyrus tree. |
| 174 |
(d) |
| 175 |
The resulting executable is then run. |
| 176 |
(e) |
| 177 |
The test executable runs all the built tests one by one, telling us |
| 178 |
which ones passed and which ones failed as it runs them. You can |
| 179 |
also run it manually with the name of a test as an argument, and it |
| 180 |
will run only the named test. |
| 181 |
(f) |
| 182 |
At the end, the text executable prints a summary of how many tests |
| 183 |
it ran and how many passed and failed. The key thing to look at here |
| 184 |
is the rightmost column, it should be all zero. |
| 185 |
|
| 186 |
3.4 Using Valgrind |
| 187 |
~~~~~~~~~~~~~~~~~~ |
| 188 |
|
| 189 |
Some failure modes are subtle, and cannot be detected in the C code |
| 190 |
itself; this is where `the Valgrind program <http://valgrind.org/>`__ |
| 191 |
comes in very handy. It detects buffer overruns and memory leaks and |
| 192 |
various other kinds of subtle errors. |
| 193 |
|
| 194 |
To run the unit tests with Valgrind, use the new ``valgrind:`` target. |
| 195 |
|
| 196 |
:: |
| 197 |
|
| 198 |
me@mybox> make valgrind |
| 199 |
... |
| 200 |
valgrind --tool=memcheck --leak-check=full ./unit -v (a) |
| 201 |
==2999== Memcheck, a memory error detector |
| 202 |
==2999== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. |
| 203 |
==2999== Using Valgrind-3.6.0.SVN-Debian and LibVEX; [...] |
| 204 |
==2999== Command: ./unit -v |
| 205 |
==2999== |
| 206 |
... |
| 207 |
--Run Summary: Type Total Ran Passed Failed (b) |
| 208 |
suites 9 9 n/a 0 |
| 209 |
tests 51 51 50 1 |
| 210 |
asserts 474 474 473 1 |
| 211 |
... |
| 212 |
==2999== HEAP SUMMARY: (c) |
| 213 |
==2999== in use at exit: 4,489 bytes in 134 blocks |
| 214 |
==2999== total heap usage: 715 allocs, 581 frees, 352,763 bytes allocated |
| 215 |
==2999== |
| 216 |
==2999== 4 bytes in 1 blocks are definitely lost in loss record 3 of 50 |
| 217 |
==2999== at 0x4C2815C: malloc (vg_replace_malloc.c:236) |
| 218 |
==2999== by 0x44A0CA: xmalloc (xmalloc.c:57) |
| 219 |
==2999== by 0x4399D8: strconcat (util.c:631) |
| 220 |
==2999== by 0x40C059: test_uncast_null (strconcat.c:51) |
| 221 |
==2999== by 0x61B32A9: ??? (in /usr/lib/libcunit.so.1.0.1) |
| 222 |
==2999== by 0x61B36ED: ??? (in /usr/lib/libcunit.so.1.0.1) |
| 223 |
==2999== by 0x61B3827: CU_run_all_tests (in /usr/lib/libcunit.so.1.0.1) |
| 224 |
==2999== by 0x4066CC: run_tests (unit.c:144) |
| 225 |
==2999== by 0x406806: main (unit.c:283) |
| 226 |
==2999== |
| 227 |
... |
| 228 |
|
| 229 |
Here's an explanation of what's happening in the example. |
| 230 |
|
| 231 |
(a) |
| 232 |
The test executable is run as before, but using the ``valgrind`` |
| 233 |
program. The first thing we see is Valgrind's banner message. |
| 234 |
(b) |
| 235 |
The test executable proceeds as normal and eventually emits it's run |
| 236 |
summary, then exits. |
| 237 |
(c) |
| 238 |
After the test executable exits, Valgrind checks for memory leaks |
| 239 |
and prints both a summary of all leaks and a stack trace showing |
| 240 |
where each block of leaked memory was allocated. |
| 241 |
|
| 242 |
I'd just like to say that I love Valgrind and I think it's immensely |
| 243 |
useful. I would have made running the tests under Valgrind the only |
| 244 |
option for the ``check:`` target, except that Valgrind is not available |
| 245 |
on all of Cyrus' supported platforms. |
| 246 |
|
| 247 |
3.5 The Tests Are Failing |
| 248 |
~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 249 |
|
| 250 |
So you've noticed that some of the tests are failing. Let me make the |
| 251 |
guiding principle of unit testing as clear as possible: **THE UNIT TESTS |
| 252 |
SHOULD NOT FAIL**. All of the tests are designed to pass all the time, |
| 253 |
in everyone's environment. The unit tests are run automatically every |
| 254 |
twelve hours on the Cyrus `Continuous Integration |
| 255 |
server <http://ci.cyrusimap.org/>`__, and a failing test fails the whole |
| 256 |
build and makes people unhappy. |
| 257 |
|
| 258 |
There are a few rules which you should follow to help us all get the |
| 259 |
most benefit out of unit testing |
| 260 |
|
| 261 |
- If you see a test failing, investigate it. |
| 262 |
- If you can't investigate, complain on the mailing list or raise a bug |
| 263 |
so that somebody else can investigate. |
| 264 |
- When writing tests, write them to work in all environments and all |
| 265 |
combinations of ``configure`` script options. It's ok to have a test |
| 266 |
which is empty in some circumstances; it's not ok to have a test that |
| 267 |
fails. |
| 268 |
- When adding code, write new tests for the new code. |
| 269 |
- When modifying code, write new tests for the new behaviour. |
| 270 |
- When looking at old code, also take a look at the `coverage |
| 271 |
report <http://ci.cyrusimap.org/job/cyrus-imapd-master/887/cobertura/>`__ |
| 272 |
and consider writing tests for the existing code. |
| 273 |
|
| 274 |
3.6 Debugging A Test |
| 275 |
-------------------- |
| 276 |
|
| 277 |
With the new Cyrus build system, the file ``cunit/unit`` is no longer an |
| 278 |
executable, it's a shell script which sets up some environment variables |
| 279 |
before running the real executable which is hidden away. This makes |
| 280 |
debugging a failing test somewhat challenging. The solution is: |
| 281 |
|
| 282 |
:: |
| 283 |
|
| 284 |
me@mybox> ( cd cunit ; ../libtool --mode=execute gdb --args unit -t crc32 ) |
| 285 |
... |
| 286 |
Reading symbols from /home/me/cyrus-imapd/cunit/.libs/lt-unit...done. |
| 287 |
(gdb) list crc32.testc:1 |
| 288 |
1 /* Unit test for lib/crc32.c */ |
| 289 |
2 #include "cunit/cyrunit.h" |
| 290 |
3 #include "crc32.h" |
| 291 |
... |
| 292 |
(gdb) break test_map |
| 293 |
Breakpoint 1 at 0x44a2f8: file ./cunit/crc32.testc, line 11. |
| 294 |
(gdb) run |
| 295 |
Starting program: /home/me/cyrus-imapd/cunit/.libs/lt-unit -t -v crc32 |
| 296 |
[Thread debugging using libthread_db enabled] |
| 297 |
|
| 298 |
CUnit - A Unit testing framework for C - Version 2.1-0 |
| 299 |
http://cunit.sourceforge.net/ |
| 300 |
|
| 301 |
Suite: crc32 |
| 302 |
Test: map ... |
| 303 |
Breakpoint 1, test_map () at ./cunit/crc32.testc:11 |
| 304 |
11 c = crc32_map(TEXT, sizeof(TEXT)-1); |
| 305 |
(gdb) |
| 306 |
|
| 307 |
|
| 308 |
Note the **-t** option. This turns off test timeouts, which is very |
| 309 |
useful for manual debugging. |
| 310 |
|
| 311 |
4. Adding Your Own Tests |
| 312 |
------------------------ |
| 313 |
|
| 314 |
Adding your own tests is quite simple. Here's how. |
| 315 |
|
| 316 |
4.1 Where To Put Your Tests |
| 317 |
--------------------------- |
| 318 |
|
| 319 |
The unit test code in Cyrus is contained in a set of C source files in |
| 320 |
the ``cunit`` directory. For reasons too complex to go into here, these |
| 321 |
are named *whatever*.testc instead of the more usual *whatever*.c. If |
| 322 |
you look closely, you will see that each of those C source files maps to |
| 323 |
a "Suite" in CUnit parlance. For example, ``cunit/glob.testc`` is listed |
| 324 |
as the Suite "glob" in CUnit's runtime output. |
| 325 |
|
| 326 |
Typically, each Suite tests a single module or a related set of |
| 327 |
functions; for example, ``cunit/glob.testc`` contains tests for the glob |
| 328 |
module in ``lib/glob.c``. |
| 329 |
|
| 330 |
So, if you want to add a new test for a module which already has some |
| 331 |
existing tests, the sensible thing to do is to `add a new test to the |
| 332 |
existing suite <#adding-a-test-to-a-suite>`__. Otherwise, you'll need to |
| 333 |
`add a new Suite <#adding-a-new-suite>`__. |
| 334 |
|
| 335 |
4.1 Adding A New Suite |
| 336 |
---------------------- |
| 337 |
|
| 338 |
Each Suite is a single C source file in the ``cunit/`` directory. Your |
| 339 |
first step is to create a new C source file. For this example, you'll |
| 340 |
create a new Suite to test the CRC32 routines which live in |
| 341 |
``lib/crc32.c``. |
| 342 |
|
| 343 |
:: |
| 344 |
|
| 345 |
me@mybox> vi cunit/crc32.testc |
| 346 |
... |
| 347 |
|
| 348 |
The file should contain something like this. |
| 349 |
|
| 350 |
:: |
| 351 |
|
| 352 |
/* Unit test for lib/crc32.c */ |
| 353 |
#include "cunit/unit.h" (a) |
| 354 |
#include "lib/crc32.h" (b) |
| 355 |
|
| 356 |
static void test_map(void) (c) |
| 357 |
{ |
| 358 |
static const char TEXT[] = "lorem ipsum"; (d) |
| 359 |
static uint32_t CRC32 = 0x0; |
| 360 |
uint32_t c; (e) |
| 361 |
|
| 362 |
c = crc32_map(TEXT, sizeof(TEXT)-1); (f) |
| 363 |
CU_ASSERT_EQUAL(c, CRC32); (g) |
| 364 |
} |
| 365 |
|
| 366 |
Here's an explanation of what all these bits are for. |
| 367 |
|
| 368 |
(a) |
| 369 |
You need to include the header ``"cunit/unit.h"``, which is a thin |
| 370 |
Cyrus wrapper around the CUnit's library's header, |
| 371 |
``<CUnit/CUnit.h>`` with some extra conveniences. |
| 372 |
(b) |
| 373 |
You should also include any headers you need for declarations of the |
| 374 |
functions which you'll be testing. |
| 375 |
(c) |
| 376 |
You need to have at least one function which looks like this: it |
| 377 |
takes no arguments, returns void, and is named ``test_whatever``. It |
| 378 |
may be ``static`` or ``extern``, but I recommend ``static``. |
| 379 |
Functions with this signature are automatically discovered in the |
| 380 |
source code by the Cyrus unit test infrastructure, so all you have |
| 381 |
to do is write the function. Later, a CUnit test named "whatever" |
| 382 |
will be created automatically for your ``test_whatever`` function. |
| 383 |
(d) |
| 384 |
Here's a good place to define the test inputs and expected outputs. |
| 385 |
Note that for this example you have no idea of the actual correct |
| 386 |
output. The right thing to do there is to manually calculate the |
| 387 |
expected result from first principles, or to use a different piece |
| 388 |
of software which you believe to be working. For this example, let's |
| 389 |
just use a known incorrect value and see what happens. |
| 390 |
(e) |
| 391 |
Here's a good place for local variables you need during the test. |
| 392 |
(f) |
| 393 |
Call the function under test (``crc32_map()`` in this example) with |
| 394 |
known inputs, and capture the results in a local variable ``c``. |
| 395 |
(g) |
| 396 |
Compare the actual result in ``c`` with the expected result in |
| 397 |
``CRC32``. The ``CU_ASSERT_EQUAL()`` macro checks that it's two |
| 398 |
arguments are equal (using an integer comparison), and if they're |
| 399 |
different it prints a message and records a failure. Note that |
| 400 |
unlike the libc ``assert()`` macro, control will continue even if |
| 401 |
the assert fails. The CUnit library provides a whole family of |
| 402 |
similar macros, see `the online CUnit |
| 403 |
documentation <http://cunit.sourceforge.net/doc/writing_tests.html#assertions>`__ |
| 404 |
for more details. |
| 405 |
|
| 406 |
Now you need to tell the Cyrus build system about your new Suite. |
| 407 |
|
| 408 |
:: |
| 409 |
|
| 410 |
me@mybox> vi Makefile.am |
| 411 |
... |
| 412 |
|
| 413 |
You need to add the filename of your new test to the definition of the |
| 414 |
``cunit_TESTS`` variable. |
| 415 |
|
| 416 |
:: |
| 417 |
|
| 418 |
cunit_TESTS = \ |
| 419 |
cunit/aaa-db.testc \ |
| 420 |
cunit/annotate.testc \ |
| 421 |
cunit/backend.testc \ |
| 422 |
cunit/binhex.testc \ |
| 423 |
cunit/bitvector.testc \ |
| 424 |
cunit/buf.testc \ |
| 425 |
cunit/byteorder.testc \ |
| 426 |
cunit/charset.testc \ |
| 427 |
cunit/crc32.testc \ |
| 428 |
cunit/dlist.testc \ |
| 429 |
cunit/duplicate.testc \ |
| 430 |
|
| 431 |
At this point you should be able to just rebuild and rerun using **make |
| 432 |
check**. You can also just rebuild without rerunning by using the |
| 433 |
command **make cunit/unit**. |
| 434 |
|
| 435 |
:: |
| 436 |
|
| 437 |
me@mybox> make check |
| 438 |
... |
| 439 |
../cunit/cunit.pl [...] --add-sources [...] crc32.testc |
| 440 |
... |
| 441 |
../cunit/cunit.pl [...] --generate-wrapper crc32.testc |
| 442 |
gcc -c [...] -g -O2 .cunit-crc32.c |
| 443 |
gcc [...] -o unit [...] .cunit-crc32.o ... |
| 444 |
Running unit tests |
| 445 |
|
| 446 |
CUnit - A Unit testing framework for C - Version 2.1-0 |
| 447 |
http://cunit.sourceforge.net/ |
| 448 |
|
| 449 |
... |
| 450 |
Suite: crc32 |
| 451 |
Test: map ... FAILED |
| 452 |
1. crc32.testc:12 - CU_ASSERT_EQUAL(c=1926722702,CRC32=0) |
| 453 |
|
| 454 |
Note how the test failure told us which in source file and at what line |
| 455 |
number the failure occurred, and what the actual and expected values |
| 456 |
were. Let's go and fix that up now. |
| 457 |
|
| 458 |
:: |
| 459 |
|
| 460 |
static const char TEXT[] = "lorem ipsum"; |
| 461 |
static uint32\_t CRC32 = 0x72d7748e; |
| 462 |
|
| 463 |
Re-run ``make check`` and you'll see your test being rebuilt and rerun, |
| 464 |
and this time passing. |
| 465 |
|
| 466 |
:: |
| 467 |
|
| 468 |
me@mybox> make check |
| 469 |
... |
| 470 |
../cunit/cunit.pl [...] --generate-wrapper crc32.testc |
| 471 |
gcc -c [...] -g -O2 .cunit-crc32.c |
| 472 |
gcc [...] -o unit [...] .cunit-crc32.o |
| 473 |
... |
| 474 |
Running unit tests |
| 475 |
|
| 476 |
CUnit - A Unit testing framework for C - Version 2.1-0 |
| 477 |
http://cunit.sourceforge.net/ |
| 478 |
|
| 479 |
... |
| 480 |
Suite: crc32 |
| 481 |
Test: map ... passed |
| 482 |
|
| 483 |
4.2 Adding A Test To A Suite |
| 484 |
---------------------------- |
| 485 |
|
| 486 |
Adding a new test to an existing test is easy: all you have to do is add |
| 487 |
a new function to an existing C source file in the ``cunit/`` directory. |
| 488 |
As an example, let's add a test for the ``crc_iovec()`` function. |
| 489 |
|
| 490 |
:: |
| 491 |
|
| 492 |
me@mybox> vi cunit/crc32.testc |
| 493 |
... |
| 494 |
|
| 495 |
static void test_iovec(void) (a) |
| 496 |
{ |
| 497 |
static const char TEXT1[] = "lorem"; (b) |
| 498 |
static const char TEXT2[] = " ipsum"; |
| 499 |
static uint32_t CRC32 = 0x72d7748e; |
| 500 |
uint32_t c; (c) |
| 501 |
struct iovec iov[2]; |
| 502 |
|
| 503 |
memset(iov, 0, sizeof(iov)); (d) |
| 504 |
iov[0].iov_base = TEXT1; |
| 505 |
iov[0].iov_len = sizeof(TEXT1)-1; |
| 506 |
iov[1].iov_base = TEXT2; |
| 507 |
iov[1].iov_len = sizeof(TEXT2)-1; |
| 508 |
|
| 509 |
c = crc32_iovec(iov, 2); (e) |
| 510 |
CU_ASSERT_EQUAL(c, CRC32); (f) |
| 511 |
} |
| 512 |
|
| 513 |
Here's an explanation of what all these bits are for. |
| 514 |
|
| 515 |
(a) |
| 516 |
Your new test function should look like this: it takes no arguments, |
| 517 |
returns void, and is named ``test_whatever``. It may be ``static`` |
| 518 |
or ``extern``, but I recommend ``static``. Functions with this |
| 519 |
signature are automatically discovered in the source code by the |
| 520 |
Cyrus unit test infrastructure, so all you have to do is write the |
| 521 |
function. Later, a CUnit test named "whatever" will be created |
| 522 |
automatically for your ``test_whatever`` function. Note that the |
| 523 |
opening curly brace must be on the next line or the unit test |
| 524 |
infrastructure will not find the function. |
| 525 |
(b) |
| 526 |
Here's a good place to define the test inputs and expected outputs. |
| 527 |
(c) |
| 528 |
Here's a good place for local variables you need during the test. |
| 529 |
(d) |
| 530 |
Here you set up the input conditions for the function under test. |
| 531 |
(e) |
| 532 |
Call the function under test with your known inputs, and capture the |
| 533 |
results in a local variable, here ``c``. |
| 534 |
(f) |
| 535 |
Compare the actual result in ``c`` with the expected result in |
| 536 |
``CRC32``. The ``CU_ASSERT_EQUAL()`` macro checks that it's two |
| 537 |
arguments are equal (using an integer comparison), and if they're |
| 538 |
different it prints a message and records a failure. Note that |
| 539 |
unlike the libc ``assert()`` macro, control will continue even if |
| 540 |
the assert fails. The CUnit library provides a whole family of |
| 541 |
similar macros, see `the online CUnit |
| 542 |
documentation <http://cunit.sourceforge.net/doc/writing_tests.html#assertions>`__ |
| 543 |
for more details. |
| 544 |
|
| 545 |
Now run ``make check`` and you'll see your test being built and run. |
| 546 |
|
| 547 |
:: |
| 548 |
|
| 549 |
me@mybox> make check |
| 550 |
... |
| 551 |
../cunit/cunit.pl [...] --generate-wrapper crc32.testc |
| 552 |
gcc -c [...] -g -O2 .cunit-crc32.c |
| 553 |
gcc [...] -o unit [...] .cunit-crc32.o |
| 554 |
... |
| 555 |
Running unit tests |
| 556 |
|
| 557 |
|
| 558 |
CUnit - A Unit testing framework for C - Version 2.1-0 |
| 559 |
http://cunit.sourceforge.net/ |
| 560 |
|
| 561 |
... |
| 562 |
Suite: crc32 |
| 563 |
Test: map ... passed |
| 564 |
Test: iovec ... passed |
| 565 |
|
| 566 |
4.3 Suite Setup And Teardown |
| 567 |
---------------------------- |
| 568 |
|
| 569 |
Sometimes the behaviour of the functions under test depend on external |
| 570 |
influences such as environment variables, global variables, or the |
| 571 |
presence of certain files. |
| 572 |
|
| 573 |
These kinds of functions need special treatment to ensure that their |
| 574 |
behaviour is locked down during the running of your tests. Otherwise, |
| 575 |
all sorts of strange behaviour may confuse the results of the tests. For |
| 576 |
example, a test might succeed the first time it's run in a given |
| 577 |
directory and fail the next time. Or a test might succeed when run by |
| 578 |
the author of the test but fail when run by another user. |
| 579 |
|
| 580 |
CUnit provides a special arrangement which helps you in such cases: the |
| 581 |
suite initialisation and cleanup functions. These are two functions that |
| 582 |
you write and which live in the suite source. They are called from CUnit |
| 583 |
respectively before any of the tests in the suite is run, and after all |
| 584 |
tests from that suite are run. |
| 585 |
|
| 586 |
Here's how to use them. The suite setup function should set up any |
| 587 |
global state that the functions under test rely on, in such a way that |
| 588 |
their state is predictable and always the same no matter who runs the |
| 589 |
test or when or how many times. Similarly the suite teardown function |
| 590 |
should clean up any state which might possibly interfere with other test |
| 591 |
suites. Note that some suites will need a setup function but not |
| 592 |
necessarily a teardown function. |
| 593 |
|
| 594 |
Adding these functions is very easy: you just write functions of the |
| 595 |
appropriate signature (names, arguments and return type) and the Cyrus |
| 596 |
unit test infrastructure will automatically discover them and arrange |
| 597 |
for them to be called. The functions should look like (actual example |
| 598 |
taken from ``cunit/mboxname.testc``) this: |
| 599 |
|
| 600 |
:: |
| 601 |
|
| 602 |
static enum enum_value old_config_virtdomains; |
| 603 |
|
| 604 |
static int set_up(void) |
| 605 |
{ |
| 606 |
old_config_virtdomains = config_virtdomains; |
| 607 |
config_virtdomains = IMAP_ENUM_VIRTDOMAINS_ON; |
| 608 |
return 0; |
| 609 |
} |
| 610 |
|
| 611 |
static int tear_down(void) |
| 612 |
{ |
| 613 |
config_virtdomains = old_config_virtdomains; |
| 614 |
return 0; |
| 615 |
} |
| 616 |
|
| 617 |
|
| 618 |
The functions should return 0 on success, and non-zero on error. They |
| 619 |
must not call any ``CU_*`` functions or macros. |
| 620 |
|
| 621 |
Good luck and good testing! |