Debian Codegraph @

doc/html/_sources/imap/developer/unit-tests.rst.txt

raw
1 .. _imap-developer-unit-tests:
2
3 .. Note: This document was converted from the original by Nic Bernstein
4 (Onlight). Any formatting mistakes are my fault and not the
5 original author's.
6
7 Unit Tests
8 ==========
9
10 Table of Contents
11 -----------------
12
13 - `1. Introduction <#introduction>`__
14 - `2. What Is A Unit Test? <#what-is-a-unit-test>`__
15 - `3. Running The Tests <#running-the-tests>`__
16
17 - `3.1. Setting Up The Machine <#setting-up-the-machine>`__
18 - `3.2 Configure Script <#configure-scripts>`__
19 - `3.3 Make <#running-the-tests>`__
20 - `3.4 Using Valgrind <#using-valgrind>`__
21 - `3.5 The Tests Are Failing <#the-tests-are-failing>`__
22 - `3.6 Debugging A Test <#debugging-a-test>`__
23
24 - `4. Adding Your Own Tests <#adding-your-own-tests>`__
25
26 - `4.1 Where To Put Your Tests <#where-to-put-your-tests>`__
27 - `4.1 Adding A New Suite <#adding-a-new-suite>`__
28 - `4.2 Adding A Test To A Suite <#adding-a-test-to-a-suite>`__
29 - `4.3 Suite Init And Cleanup <#suite-init-and-cleanup>`__
30
31 1. Introduction
32 ---------------
33
34 Recently, a set of regression unit tests has been added to Cyrus. This
35 document explains the purpose implementation of those unit tests, and
36 gives an example of how to add more unit tests (because there are never
37 enough unit tests!).
38
39 2. What Is A Unit Test?
40 -----------------------
41
42 The `definition on Wikipedia <http://en.wikipedia.org/wiki/Unit_test>`__
43 sheds some light:
44
45 ...\ **unit testing** is a method by which individual units of
46 source code are tested to determine if they are fit for use. A unit
47 is the smallest testable part of an application.
48
49 In other words, unit testing is about verifying that small pieces of
50 code, like individual functions, modules, or classes, work in isolation.
51 It is **not** about testing the system as a whole.
52
53 The tests implemented here are also **regression tests**, which in
54 `Wikipedia's words <http://en.wikipedia.org/wiki/Regression_testing>`__
55 means:
56
57 **Regression testing** is any type of software testing that seeks to
58 uncover software errors after changes to the program (e.g. bugfixes
59 or new functionality) have been made, by retesting the program. The
60 intent of regression testing is to assure that a change, such as a
61 bugfix, did not introduce new bugs.
62
63 In other words, the tests are designed to be easy to run and to work out
64 fully automatically whether they have passed or failed, so that they can
65 be run usefully by people who didn't write them.
66
67 3. Running The Tests
68 --------------------
69
70 This section takes you through the process of running Cyrus' unit tests.
71
72 3.1. Setting Up The Machine
73 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
74
75 Cyrus' unit tests are all located in a new directory,
76 ``cyrus-imapd/cunit/``. They're written in C, like the remainder of
77 Cyrus, and use the `CUnit library from
78 SourceForge <http://cunit.sourceforge.net/>`__, with some home grown
79 wrappers and other improvements to make our lives easier.
80
81 Your first step is step is to ensure that the CUnit library (including
82 the headers) is installed. Some modern operating systems already have
83 CUnit, for example on Ubuntu you can just do:
84
85 ::
86
87 me@ubuntu> sudo apt-get install libcunit1-dev
88
89 Alternately, you can download the CUnit source, build it and install it.
90 It's not a complicated or difficult library, this shouldn't take long.
91 When you've done, install it in ``/usr/include`` and ``/usr/lib``.
92
93 3.2 Configure Script
94 ~~~~~~~~~~~~~~~~~~~~
95
96 Because of the dependency on the CUnit library, the tests are disabled
97 by default; this means you need enable them with an option to the
98 ``configure`` script:
99
100 ::
101
102 me@mybox> ./configure --enable-unit-tests
103 ...
104 checking for CU\_initialize\_registry in -lcunit... yes
105 checking CUnit/CUnit.h usability... yes
106 checking CUnit/CUnit.h presence... yes
107 checking for CUnit/CUnit.h... yes
108 ...
109
110 3.3 Make
111 ~~~~~~~~
112
113 First you need to build Cyrus itself, using the traditional ``all:``
114 target.
115
116 ::
117
118 me@mybox> make all
119 ...
120
121 Then, use the new ``check:`` target to build and run the unit tests.
122
123 ::
124
125 me@mybox> make check
126 cd . && /bin/bash /home/me/cyrus-imapd/missing --run automake-1.11 --foreign Makefile
127 cd . && /bin/bash ./config.status Makefile depfiles (a)
128 config.status: creating Makefile
129 config.status: executing depfiles commands
130 ...
131 make[3]: Entering directory \`/home/me/cyrus-imapd'
132 make[3]: \`sieve/test' is up to date.
133 cunit/cunit.pl --project cunit/default.cunit --generate-wrapper cunit/mboxname.testc (b)
134 gcc -DHAVE\_CONFIG\_H ... -c -o cunit/mboxname.o cunit/mboxname.testc-cunit.c
135 rm -f cunit/mboxname.testc-cunit.c
136 /bin/bash ./libtool --tag=CC --mode=link gcc -fPIC -g -O2 -o
137 cunit/unit cunit/unit.o ... lib/libcyrus\_min.la ... (c)
138 ...
139 Running unit tests (d)
140
141 CUnit - A Unit testing framework for C - Version 2.1-0
142 http://cunit.sourceforge.net/
143
144 ...
145 Suite: mboxname (e)
146 Test: dir\_hash\_c ... passed
147 Test: to\_parts ... passed
148 Test: to\_userid ... passed
149 Test: to\_usermbox ... passed
150 ...
151 --Run Summary: Type Total Ran Passed Failed (f)
152 suites 34 34 n/a 0
153 tests 323 323 323 0
154 asserts 1079745 1079745 1079745 0
155 make[1]: Leaving directory `/home/me/cyrus-imapd/cunit'
156
157 Let's take a closer look at what's happening here.
158
159 (a)
160 The ``check:`` target causes automake to re-run itself. This is
161 normal automake behaviour. Note that the older build system used to
162 run make recursively in sub-directories, the newer automake-based
163 system builds everything from the top directory.
164 (b)
165 The ``cunit/`` directory contains a number of C source files
166 (called, for reasons too complicated to explain here,
167 *whatever*.testc) with test code in them. For each of those, a small
168 wrapper C source file is generated and then compiled into an object
169 file.
170 (c)
171 Finally, all the compiled object files are linked into an
172 executable, with a ``main()`` routine from ``unit.c``, and a number
173 of libraries and object files from other parts of the Cyrus tree.
174 (d)
175 The resulting executable is then run.
176 (e)
177 The test executable runs all the built tests one by one, telling us
178 which ones passed and which ones failed as it runs them. You can
179 also run it manually with the name of a test as an argument, and it
180 will run only the named test.
181 (f)
182 At the end, the text executable prints a summary of how many tests
183 it ran and how many passed and failed. The key thing to look at here
184 is the rightmost column, it should be all zero.
185
186 3.4 Using Valgrind
187 ~~~~~~~~~~~~~~~~~~
188
189 Some failure modes are subtle, and cannot be detected in the C code
190 itself; this is where `the Valgrind program <http://valgrind.org/>`__
191 comes in very handy. It detects buffer overruns and memory leaks and
192 various other kinds of subtle errors.
193
194 To run the unit tests with Valgrind, use the new ``valgrind:`` target.
195
196 ::
197
198 me@mybox> make valgrind
199 ...
200 valgrind --tool=memcheck --leak-check=full ./unit -v (a)
201 ==2999== Memcheck, a memory error detector
202 ==2999== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
203 ==2999== Using Valgrind-3.6.0.SVN-Debian and LibVEX; [...]
204 ==2999== Command: ./unit -v
205 ==2999==
206 ...
207 --Run Summary: Type Total Ran Passed Failed (b)
208 suites 9 9 n/a 0
209 tests 51 51 50 1
210 asserts 474 474 473 1
211 ...
212 ==2999== HEAP SUMMARY: (c)
213 ==2999== in use at exit: 4,489 bytes in 134 blocks
214 ==2999== total heap usage: 715 allocs, 581 frees, 352,763 bytes allocated
215 ==2999==
216 ==2999== 4 bytes in 1 blocks are definitely lost in loss record 3 of 50
217 ==2999== at 0x4C2815C: malloc (vg_replace_malloc.c:236)
218 ==2999== by 0x44A0CA: xmalloc (xmalloc.c:57)
219 ==2999== by 0x4399D8: strconcat (util.c:631)
220 ==2999== by 0x40C059: test_uncast_null (strconcat.c:51)
221 ==2999== by 0x61B32A9: ??? (in /usr/lib/libcunit.so.1.0.1)
222 ==2999== by 0x61B36ED: ??? (in /usr/lib/libcunit.so.1.0.1)
223 ==2999== by 0x61B3827: CU_run_all_tests (in /usr/lib/libcunit.so.1.0.1)
224 ==2999== by 0x4066CC: run_tests (unit.c:144)
225 ==2999== by 0x406806: main (unit.c:283)
226 ==2999==
227 ...
228
229 Here's an explanation of what's happening in the example.
230
231 (a)
232 The test executable is run as before, but using the ``valgrind``
233 program. The first thing we see is Valgrind's banner message.
234 (b)
235 The test executable proceeds as normal and eventually emits it's run
236 summary, then exits.
237 (c)
238 After the test executable exits, Valgrind checks for memory leaks
239 and prints both a summary of all leaks and a stack trace showing
240 where each block of leaked memory was allocated.
241
242 I'd just like to say that I love Valgrind and I think it's immensely
243 useful. I would have made running the tests under Valgrind the only
244 option for the ``check:`` target, except that Valgrind is not available
245 on all of Cyrus' supported platforms.
246
247 3.5 The Tests Are Failing
248 ~~~~~~~~~~~~~~~~~~~~~~~~~
249
250 So you've noticed that some of the tests are failing. Let me make the
251 guiding principle of unit testing as clear as possible: **THE UNIT TESTS
252 SHOULD NOT FAIL**. All of the tests are designed to pass all the time,
253 in everyone's environment. The unit tests are run automatically every
254 twelve hours on the Cyrus `Continuous Integration
255 server <http://ci.cyrusimap.org/>`__, and a failing test fails the whole
256 build and makes people unhappy.
257
258 There are a few rules which you should follow to help us all get the
259 most benefit out of unit testing
260
261 - If you see a test failing, investigate it.
262 - If you can't investigate, complain on the mailing list or raise a bug
263 so that somebody else can investigate.
264 - When writing tests, write them to work in all environments and all
265 combinations of ``configure`` script options. It's ok to have a test
266 which is empty in some circumstances; it's not ok to have a test that
267 fails.
268 - When adding code, write new tests for the new code.
269 - When modifying code, write new tests for the new behaviour.
270 - When looking at old code, also take a look at the `coverage
271 report <http://ci.cyrusimap.org/job/cyrus-imapd-master/887/cobertura/>`__
272 and consider writing tests for the existing code.
273
274 3.6 Debugging A Test
275 --------------------
276
277 With the new Cyrus build system, the file ``cunit/unit`` is no longer an
278 executable, it's a shell script which sets up some environment variables
279 before running the real executable which is hidden away. This makes
280 debugging a failing test somewhat challenging. The solution is:
281
282 ::
283
284 me@mybox> ( cd cunit ; ../libtool --mode=execute gdb --args unit -t crc32 )
285 ...
286 Reading symbols from /home/me/cyrus-imapd/cunit/.libs/lt-unit...done.
287 (gdb) list crc32.testc:1
288 1 /* Unit test for lib/crc32.c */
289 2 #include "cunit/cyrunit.h"
290 3 #include "crc32.h"
291 ...
292 (gdb) break test_map
293 Breakpoint 1 at 0x44a2f8: file ./cunit/crc32.testc, line 11.
294 (gdb) run
295 Starting program: /home/me/cyrus-imapd/cunit/.libs/lt-unit -t -v crc32
296 [Thread debugging using libthread_db enabled]
297
298 CUnit - A Unit testing framework for C - Version 2.1-0
299 http://cunit.sourceforge.net/
300
301 Suite: crc32
302 Test: map ...
303 Breakpoint 1, test_map () at ./cunit/crc32.testc:11
304 11 c = crc32_map(TEXT, sizeof(TEXT)-1);
305 (gdb)
306
307
308 Note the **-t** option. This turns off test timeouts, which is very
309 useful for manual debugging.
310
311 4. Adding Your Own Tests
312 ------------------------
313
314 Adding your own tests is quite simple. Here's how.
315
316 4.1 Where To Put Your Tests
317 ---------------------------
318
319 The unit test code in Cyrus is contained in a set of C source files in
320 the ``cunit`` directory. For reasons too complex to go into here, these
321 are named *whatever*.testc instead of the more usual *whatever*.c. If
322 you look closely, you will see that each of those C source files maps to
323 a "Suite" in CUnit parlance. For example, ``cunit/glob.testc`` is listed
324 as the Suite "glob" in CUnit's runtime output.
325
326 Typically, each Suite tests a single module or a related set of
327 functions; for example, ``cunit/glob.testc`` contains tests for the glob
328 module in ``lib/glob.c``.
329
330 So, if you want to add a new test for a module which already has some
331 existing tests, the sensible thing to do is to `add a new test to the
332 existing suite <#adding-a-test-to-a-suite>`__. Otherwise, you'll need to
333 `add a new Suite <#adding-a-new-suite>`__.
334
335 4.1 Adding A New Suite
336 ----------------------
337
338 Each Suite is a single C source file in the ``cunit/`` directory. Your
339 first step is to create a new C source file. For this example, you'll
340 create a new Suite to test the CRC32 routines which live in
341 ``lib/crc32.c``.
342
343 ::
344
345 me@mybox> vi cunit/crc32.testc
346 ...
347
348 The file should contain something like this.
349
350 ::
351
352 /* Unit test for lib/crc32.c */
353 #include "cunit/unit.h" (a)
354 #include "lib/crc32.h" (b)
355
356 static void test_map(void) (c)
357 {
358 static const char TEXT[] = "lorem ipsum"; (d)
359 static uint32_t CRC32 = 0x0;
360 uint32_t c; (e)
361
362 c = crc32_map(TEXT, sizeof(TEXT)-1); (f)
363 CU_ASSERT_EQUAL(c, CRC32); (g)
364 }
365
366 Here's an explanation of what all these bits are for.
367
368 (a)
369 You need to include the header ``"cunit/unit.h"``, which is a thin
370 Cyrus wrapper around the CUnit's library's header,
371 ``<CUnit/CUnit.h>`` with some extra conveniences.
372 (b)
373 You should also include any headers you need for declarations of the
374 functions which you'll be testing.
375 (c)
376 You need to have at least one function which looks like this: it
377 takes no arguments, returns void, and is named ``test_whatever``. It
378 may be ``static`` or ``extern``, but I recommend ``static``.
379 Functions with this signature are automatically discovered in the
380 source code by the Cyrus unit test infrastructure, so all you have
381 to do is write the function. Later, a CUnit test named "whatever"
382 will be created automatically for your ``test_whatever`` function.
383 (d)
384 Here's a good place to define the test inputs and expected outputs.
385 Note that for this example you have no idea of the actual correct
386 output. The right thing to do there is to manually calculate the
387 expected result from first principles, or to use a different piece
388 of software which you believe to be working. For this example, let's
389 just use a known incorrect value and see what happens.
390 (e)
391 Here's a good place for local variables you need during the test.
392 (f)
393 Call the function under test (``crc32_map()`` in this example) with
394 known inputs, and capture the results in a local variable ``c``.
395 (g)
396 Compare the actual result in ``c`` with the expected result in
397 ``CRC32``. The ``CU_ASSERT_EQUAL()`` macro checks that it's two
398 arguments are equal (using an integer comparison), and if they're
399 different it prints a message and records a failure. Note that
400 unlike the libc ``assert()`` macro, control will continue even if
401 the assert fails. The CUnit library provides a whole family of
402 similar macros, see `the online CUnit
403 documentation <http://cunit.sourceforge.net/doc/writing_tests.html#assertions>`__
404 for more details.
405
406 Now you need to tell the Cyrus build system about your new Suite.
407
408 ::
409
410 me@mybox> vi Makefile.am
411 ...
412
413 You need to add the filename of your new test to the definition of the
414 ``cunit_TESTS`` variable.
415
416 ::
417
418 cunit_TESTS = \
419 cunit/aaa-db.testc \
420 cunit/annotate.testc \
421 cunit/backend.testc \
422 cunit/binhex.testc \
423 cunit/bitvector.testc \
424 cunit/buf.testc \
425 cunit/byteorder.testc \
426 cunit/charset.testc \
427 cunit/crc32.testc \
428 cunit/dlist.testc \
429 cunit/duplicate.testc \
430
431 At this point you should be able to just rebuild and rerun using **make
432 check**. You can also just rebuild without rerunning by using the
433 command **make cunit/unit**.
434
435 ::
436
437 me@mybox> make check
438 ...
439 ../cunit/cunit.pl [...] --add-sources [...] crc32.testc
440 ...
441 ../cunit/cunit.pl [...] --generate-wrapper crc32.testc
442 gcc -c [...] -g -O2 .cunit-crc32.c
443 gcc [...] -o unit [...] .cunit-crc32.o ...
444 Running unit tests
445
446 CUnit - A Unit testing framework for C - Version 2.1-0
447 http://cunit.sourceforge.net/
448
449 ...
450 Suite: crc32
451 Test: map ... FAILED
452 1. crc32.testc:12 - CU_ASSERT_EQUAL(c=1926722702,CRC32=0)
453
454 Note how the test failure told us which in source file and at what line
455 number the failure occurred, and what the actual and expected values
456 were. Let's go and fix that up now.
457
458 ::
459
460 static const char TEXT[] = "lorem ipsum";
461 static uint32\_t CRC32 = 0x72d7748e;
462
463 Re-run ``make check`` and you'll see your test being rebuilt and rerun,
464 and this time passing.
465
466 ::
467
468 me@mybox> make check
469 ...
470 ../cunit/cunit.pl [...] --generate-wrapper crc32.testc
471 gcc -c [...] -g -O2 .cunit-crc32.c
472 gcc [...] -o unit [...] .cunit-crc32.o
473 ...
474 Running unit tests
475
476 CUnit - A Unit testing framework for C - Version 2.1-0
477 http://cunit.sourceforge.net/
478
479 ...
480 Suite: crc32
481 Test: map ... passed
482
483 4.2 Adding A Test To A Suite
484 ----------------------------
485
486 Adding a new test to an existing test is easy: all you have to do is add
487 a new function to an existing C source file in the ``cunit/`` directory.
488 As an example, let's add a test for the ``crc_iovec()`` function.
489
490 ::
491
492 me@mybox> vi cunit/crc32.testc
493 ...
494
495 static void test_iovec(void) (a)
496 {
497 static const char TEXT1[] = "lorem"; (b)
498 static const char TEXT2[] = " ipsum";
499 static uint32_t CRC32 = 0x72d7748e;
500 uint32_t c; (c)
501 struct iovec iov[2];
502
503 memset(iov, 0, sizeof(iov)); (d)
504 iov[0].iov_base = TEXT1;
505 iov[0].iov_len = sizeof(TEXT1)-1;
506 iov[1].iov_base = TEXT2;
507 iov[1].iov_len = sizeof(TEXT2)-1;
508
509 c = crc32_iovec(iov, 2); (e)
510 CU_ASSERT_EQUAL(c, CRC32); (f)
511 }
512
513 Here's an explanation of what all these bits are for.
514
515 (a)
516 Your new test function should look like this: it takes no arguments,
517 returns void, and is named ``test_whatever``. It may be ``static``
518 or ``extern``, but I recommend ``static``. Functions with this
519 signature are automatically discovered in the source code by the
520 Cyrus unit test infrastructure, so all you have to do is write the
521 function. Later, a CUnit test named "whatever" will be created
522 automatically for your ``test_whatever`` function. Note that the
523 opening curly brace must be on the next line or the unit test
524 infrastructure will not find the function.
525 (b)
526 Here's a good place to define the test inputs and expected outputs.
527 (c)
528 Here's a good place for local variables you need during the test.
529 (d)
530 Here you set up the input conditions for the function under test.
531 (e)
532 Call the function under test with your known inputs, and capture the
533 results in a local variable, here ``c``.
534 (f)
535 Compare the actual result in ``c`` with the expected result in
536 ``CRC32``. The ``CU_ASSERT_EQUAL()`` macro checks that it's two
537 arguments are equal (using an integer comparison), and if they're
538 different it prints a message and records a failure. Note that
539 unlike the libc ``assert()`` macro, control will continue even if
540 the assert fails. The CUnit library provides a whole family of
541 similar macros, see `the online CUnit
542 documentation <http://cunit.sourceforge.net/doc/writing_tests.html#assertions>`__
543 for more details.
544
545 Now run ``make check`` and you'll see your test being built and run.
546
547 ::
548
549 me@mybox> make check
550 ...
551 ../cunit/cunit.pl [...] --generate-wrapper crc32.testc
552 gcc -c [...] -g -O2 .cunit-crc32.c
553 gcc [...] -o unit [...] .cunit-crc32.o
554 ...
555 Running unit tests
556
557
558 CUnit - A Unit testing framework for C - Version 2.1-0
559 http://cunit.sourceforge.net/
560
561 ...
562 Suite: crc32
563 Test: map ... passed
564 Test: iovec ... passed
565
566 4.3 Suite Setup And Teardown
567 ----------------------------
568
569 Sometimes the behaviour of the functions under test depend on external
570 influences such as environment variables, global variables, or the
571 presence of certain files.
572
573 These kinds of functions need special treatment to ensure that their
574 behaviour is locked down during the running of your tests. Otherwise,
575 all sorts of strange behaviour may confuse the results of the tests. For
576 example, a test might succeed the first time it's run in a given
577 directory and fail the next time. Or a test might succeed when run by
578 the author of the test but fail when run by another user.
579
580 CUnit provides a special arrangement which helps you in such cases: the
581 suite initialisation and cleanup functions. These are two functions that
582 you write and which live in the suite source. They are called from CUnit
583 respectively before any of the tests in the suite is run, and after all
584 tests from that suite are run.
585
586 Here's how to use them. The suite setup function should set up any
587 global state that the functions under test rely on, in such a way that
588 their state is predictable and always the same no matter who runs the
589 test or when or how many times. Similarly the suite teardown function
590 should clean up any state which might possibly interfere with other test
591 suites. Note that some suites will need a setup function but not
592 necessarily a teardown function.
593
594 Adding these functions is very easy: you just write functions of the
595 appropriate signature (names, arguments and return type) and the Cyrus
596 unit test infrastructure will automatically discover them and arrange
597 for them to be called. The functions should look like (actual example
598 taken from ``cunit/mboxname.testc``) this:
599
600 ::
601
602 static enum enum_value old_config_virtdomains;
603
604 static int set_up(void)
605 {
606 old_config_virtdomains = config_virtdomains;
607 config_virtdomains = IMAP_ENUM_VIRTDOMAINS_ON;
608 return 0;
609 }
610
611 static int tear_down(void)
612 {
613 config_virtdomains = old_config_virtdomains;
614 return 0;
615 }
616
617
618 The functions should return 0 on success, and non-zero on error. They
619 must not call any ``CU_*`` functions or macros.
620
621 Good luck and good testing!