Debian Codegraph @

manuals/en/developers/regression.tex

raw
1 \chapter{\bacula{} Regression Testing}
2 \label{blb:TheChapterStart8}
3 \index{Testing!\bacula{} Regression}
4 \index{\bacula{} Regression Testing}
5
6 \section{Setting up Regession Testing}
7 \index{Setting up Regession Testing}
8
9 This document is intended mostly for developers who wish to ensure that their
10 changes to \bacula{} don't introduce bugs in the base code. However, you
11 don't need to be a developer to run the regression scripts, and we
12 recommend them before putting your system into production, and before each
13 upgrade, especially if you build from source code. They are
14 simply shell scripts that drive \bacula{} through bconsole and then typically
15 compare the input and output with \btool{diff}.
16
17 You can find the existing regression scripts in the \bacula{} developer's
18 \btool{git} repository on SourceForge. We strongly recommend that you
19 \textbf{clone} the repository because afterwards, you can easily get pull the
20 updates that have been made.
21
22 To get started, we recommend that you create a directory named
23 \bdirectoryname{bacula}, under which you will put the current source code and the current
24 set of regression scripts. Below, we will describe how to set this up.
25
26 The top level directory that we call \bdirectoryname{bacula} can be named anything you
27 want. Note, all the standard regression scripts run as non-root and can be
28 run on the same machine as a production \bacula{} system (the developers run
29 it this way).
30
31 To create the directory structure for the current trunk and to
32 clone the repository, do the following (note, we assume you
33 are working in your home directory in a non-root account):
34
35 \footnotesize
36 \begin{bVerbatim}
37 git clone http://git.bacula.org/bacula.git bacula
38 \end{bVerbatim}
39
40 This will create the directory \bdirectoryname{bacula} and populate it with
41 three directories: \bdirectoryname{bacula}, \bdirectoryname{gui}, and \bdirectoryname{regress}.
42 \bdirectoryname{bacula} contains the \bacula{} source code; \bdirectoryname{gui} contains
43 certain \acs{GUI} programs that you will not need, and \bdirectoryname{regress} contains
44 all the regression scripts. The above should be needed only
45 once. Thereafter to update to the latest code, you do:
46
47 \begin{bVerbatim}
48 cd bacula
49 git pull
50 \end{bVerbatim}
51
52 \smallskip
53
54 There are two different aspects of regression testing that this document will
55 discuss: 1. Running the Regression Script, 2. Writing a Regression test.
56
57 \section{Running the Regression Script}
58 \index{Running the Regression Script}
59 \index{Script!Running the Regression}
60
61 There are a number of different tests that may be run, such as: the standard
62 set that uses disk Volumes and runs under any userid; a small set of tests
63 that write to tape; another set of tests where you must be root to run them.
64 Normally, I run all my tests as non-root and very rarely run the root
65 tests. The tests vary in length, and running the full tests including disk
66 based testing, tape based testing, autochanger based testing, and multiple
67 drive autochanger based testing can take 3 or 4 hours.
68
69 \subsection{Setting the Configuration Parameters}
70 \index{Setting the Configuration Parameters}
71 \index{Parameters!Setting the Configuration}
72
73 There is nothing you need to change in the source directory.
74
75 To begin:
76
77 \begin{bVerbatim}
78 cd bacula/regress
79 \end{bVerbatim}
80
81
82 The
83 very first time you are going to run the regression scripts, you will
84 need to create a custom config file for your system.
85 We suggest that you start by:
86
87 \begin{bVerbatim}
88 cp prototype.conf config
89 \end{bVerbatim}
90
91 Then you can edit the \bfilename{config} file directly.
92
93 \begin{bVerbatim}
94 # Where to get the source to be tested
95 BACULA_SOURCE="${HOME}/bacula/bacula"
96
97 # Where to send email !!!!! Change me !!!!!!!
98 [email protected]
99 SMTP_HOST="localhost"
100
101 TAPE_DRIVE="/dev/nst0"
102 # if you don't have an autochanger set AUTOCHANGER to /dev/null
103 AUTOCHANGER="/dev/sg0"
104 # For two drive tests -- set to /dev/null if you do not have it
105 TAPE_DRIVE1="/dev/null"
106
107 # This must be the path to the autochanger including its name
108 AUTOCHANGER_PATH="/usr/sbin/mtx"
109
110 # Set what backend to use "postresql" "mysql" or "sqlite3"
111 DBTYPE="postgresql"
112
113 # Set your database here
114 #WHICHDB="--with-${DBTYPE}=${SQLITE3_DIR}"
115 WHICHDB="--with-${DBTYPE}"
116
117 # Set this to "--with-tcp-wrappers" or "--without-tcp-wrappers"
118 TCPWRAPPERS="--with-tcp-wrappers"
119
120 # Set this to "" to disable OpenSSL support, "--with-openssl=yes"
121 # to enable it, or provide the path to the OpenSSL installation,
122 # eg "--with-openssl=/usr/local"
123 OPENSSL="--with-openssl"
124
125 # You may put your real host name here, but localhost or 127.0.0.1
126 # is valid also and it has the advantage that it works on a
127 # non-networked machine
128 HOST="localhost"
129 \end{bVerbatim}
130
131 \begin{bdescription}
132 \item [BACULA\_SOURCE] should be the full path to the \bacula{} source code
133 that you wish to test. It will be loaded configured, compiled, and
134 installed with the "make setup" command, which needs to be done only
135 once each time you change the source code.
136
137 \item [EMAIL] should be your email addres. Please remember to change this
138 or I will get a flood of unwanted messages. You may or may not want to see
139 these emails. In my case, I don't need them so I direct it to the bit bucket.
140
141 \item [SMTP\_HOST] defines where your \acs{SMTP} server is.
142
143
144 \item [TAPE\_DRIVE] is the full path to your tape drive. The base set of
145 regression tests do not use a tape, so this is only important if you want to
146 run the full tests. Set this to \bdirectoryname{/dev/null} if you do not have a tape drive.
147
148 \item [TAPE\_DRIVE1] is the full path to your second tape drive, if
149 have one. The base set of
150 regression tests do not use a tape, so this is only important if you want to
151 run the full two drive tests. Set this to \bdirectoryname{/dev/null} if you do not have a
152 second tape drive.
153
154 \item [AUTOCHANGER] is the name of your autochanger control device. Set this to
155 \bdirectoryname{/dev/null} if you do not have an autochanger.
156
157 \item [AUTOCHANGER\_PATH] is the full path including the program name for
158 your autochanger program (normally \mtx{}). Leave the default value if you
159 do not have one.
160
161 \item [TCPWRAPPERS] defines whether or not you want the \btool{./configure}
162 to be performed with \texttt{tcpwrappers} enabled.
163
164 \item [OPENSSL] used to enable/disable \acs{SSL} support for \bacula{}
165 communications and data encryption.
166
167 \item [HOST] is the hostname that it will use when building the
168 scripts. The \bacula{} daemons will be named \bbracket{HOST}-dir, \bbracket{HOST}-fd,
169 \ldots{} It is also the name of the \texttt{HOST} machine that to connect to the
170 daemons by the network. Hence the name should either be your real
171 hostname (with an appropriate \acs{DNS} or \bfilename{/etc/hosts} entry) or \bfilename{localhost} as it is in the default file.
172
173 \item [bin] is the binary location.
174
175 \item [scripts] is the bacula scripts location (where we could find
176 database creation script, autochanger handler, etc.)
177 \end{bdescription}
178
179 \subsection{Building the Test \bacula{}}
180 \index{Building the Test \bacula{}}
181 \index{\bacula{}!Building the Test}
182
183 Once the above variables are set, you can build the setup by entering:
184
185 \begin{bVerbatim}
186 make setup
187 \end{bVerbatim}
188
189 This will setup the regression testing and you should not need to
190 do this again unless you want to change the database or other regression
191 configuration parameters.
192
193
194 \subsection{Setting up your SQL engine}
195 \index{Setting up your SQL engine}
196 If you are using \sqlite{} or \sqlite{}3, there is nothing more to do; you can
197 simply run the tests as described in the next section.
198
199 If you are using \mysql{} or \postgresql{}, you will need to establish an
200 account with your database engine for the user name \texttt{regress} and
201 you will need to manually create a database named \textbf{regress} that can be
202 used by user name \texttt{regress}, which means you will have to give the user
203 regress sufficient permissions to use the database named \texttt{regress}.
204 There is no password on the \texttt{regress} account.
205
206 You have probably already done this procedure for the user name and
207 database named \texttt{bacula}. If not, the manual describes roughly how to
208 do it, and the scripts in \bdirectoryname{bacula/regress/build/src/cats} named
209 \bfilename{create\_mysql\_database}, \bfilename{create\_postgresql\_database},
210 \bfilename{grant\_mysql\_privileges}, and \bfilename{grant\_postgresql\_privileges}
211 may be of a help to you.
212
213 Generally, to do the above, you will need to run under \texttt{root} to
214 be able to create databases and modify permissions within \mysql{} and
215 \postgresql{}.
216
217 It is possible to configure \mysql{} access for database accounts that
218 require a password to be supplied. This can be done by creating a \bfilename{~/.my.cnf}
219 file which supplies the credentials by default to the \mysql{} commandline
220 utilities.
221
222 \begin{bVerbatim}
223 [client]
224 host = localhost
225 user = regress
226 password = asecret
227 \end{bVerbatim}
228
229 A similar technique can be used \postgresql{} regression testing where the
230 database is configured to require a password. The \bfilename{~/.pgpass} file should
231 contain a line with the database connection properties.
232
233 \begin{bVerbatim}
234 hostname:port:database:username:password
235 \end{bVerbatim}
236
237 \subsection{Running the Disk Only Regression}
238 \index{Regression!Running the Disk Only}
239 \index{Running the Disk Only Regression}
240
241 The simplest way to copy the source code, configure it, compile it, link
242 it, and run the tests is to use a helper script:
243
244 \begin{bVerbatim}
245 ./do_disk
246 \end{bVerbatim}
247
248
249
250
251 This will run the base set of tests using disk Volumes.
252 If you are testing on a
253 non-Linux machine several of the of the tests may not be run. In any case,
254 as we add new tests, the number will vary. It will take about 1 hour
255 and you don't need to be \texttt{root}
256 to run these tests (I run under my regular userid). The result should be
257 something similar to:
258
259 \begin{bVerbatim}
260 Test results
261 ===== auto-label-test OK 12:31:33 =====
262 ===== backup-bacula-test OK 12:32:32 =====
263 ===== bextract-test OK 12:33:27 =====
264 ===== bscan-test OK 12:34:47 =====
265 ===== bsr-opt-test OK 12:35:46 =====
266 ===== compressed-test OK 12:36:52 =====
267 ===== compressed-encrypt-test OK 12:38:18 =====
268 ===== concurrent-jobs-test OK 12:39:49 =====
269 ===== data-encrypt-test OK 12:41:11 =====
270 ===== encrypt-bug-test OK 12:42:00 =====
271 ===== fifo-test OK 12:43:46 =====
272 ===== backup-bacula-fifo OK 12:44:54 =====
273 ===== differential-test OK 12:45:36 =====
274 ===== four-concurrent-jobs-test OK 12:47:39 =====
275 ===== four-jobs-test OK 12:49:22 =====
276 ===== incremental-test OK 12:50:38 =====
277 ===== query-test OK 12:51:37 =====
278 ===== recycle-test OK 12:53:52 =====
279 ===== restore2-by-file-test OK 12:54:53 =====
280 ===== restore-by-file-test OK 12:55:40 =====
281 ===== restore-disk-seek-test OK 12:56:29 =====
282 ===== six-vol-test OK 12:57:44 =====
283 ===== span-vol-test OK 12:58:52 =====
284 ===== sparse-compressed-test OK 13:00:00 =====
285 ===== sparse-test OK 13:01:04 =====
286 ===== two-jobs-test OK 13:02:39 =====
287 ===== two-vol-test OK 13:03:49 =====
288 ===== verify-vol-test OK 13:04:56 =====
289 ===== weird-files2-test OK 13:05:47 =====
290 ===== weird-files-test OK 13:06:33 =====
291 ===== migration-job-test OK 13:08:15 =====
292 ===== migration-jobspan-test OK 13:09:33 =====
293 ===== migration-volume-test OK 13:10:48 =====
294 ===== migration-time-test OK 13:12:59 =====
295 ===== hardlink-test OK 13:13:50 =====
296 ===== two-pool-test OK 13:18:17 =====
297 ===== fast-two-pool-test OK 13:24:02 =====
298 ===== two-volume-test OK 13:25:06 =====
299 ===== incremental-2disk OK 13:25:57 =====
300 ===== 2drive-incremental-2disk OK 13:26:53 =====
301 ===== scratch-pool-test OK 13:28:01 =====
302 Total time = 0:57:55 or 3475 secs
303 \end{bVerbatim}
304
305 and the working tape tests are run with
306
307 \begin{bVerbatim}
308 make full_test
309 \end{bVerbatim}
310
311
312 \begin{bVerbatim}
313 Test results
314
315 ===== Bacula tape test OK =====
316 ===== Small File Size test OK =====
317 ===== restore-by-file-tape test OK =====
318 ===== incremental-tape test OK =====
319 ===== four-concurrent-jobs-tape OK =====
320 ===== four-jobs-tape OK =====
321 \end{bVerbatim}
322
323 Each separate test is self contained in that it initializes to run \bacula{} from
324 scratch (i.e. newly created database). It will also kill any \bacula{} session
325 that is currently running. In addition, it uses ports 8101, 8102, and 8103 so
326 that it does not intefere with a production system.
327
328 Alternatively, you can do the \bfilename{./do\_disk} work by hand with:
329
330 \begin{bVerbatim}
331 make setup
332 \end{bVerbatim}
333
334 The above will then copy the source code within
335 the regression tree (in directory \bdirectoryname{regress/build}), configure it, and build it.
336 There should be no errors. If there are, please correct them before
337 continuing. From this point on, as long as you don't change the \bacula{}
338 source code, you should not need to repeat any of the above steps. If
339 you pull down a new version of the source code, simply run \btool{make setup}
340 again.
341
342
343 Once \bacula{} is built, you can run the basic disk only \texttt{non-root} regression test
344 by entering:
345
346 \begin{bVerbatim}
347 make test
348 \end{bVerbatim}
349
350
351 \subsection{Other Tests}
352 \index{Other Tests}
353 \index{Tests!Other}
354
355 There are a number of other tests that can be run as well. All the tests are a
356 simply shell script keep in the regress directory. For example the \btool{make test}
357 simply executes \btool{./all-non-root-tests}. The other tests, which
358 are invoked by directly running the script are:
359
360 \begin{bdescription}
361
362 \item [all\_non-root-tests]
363 \index{all\_non-root-tests}
364 All non-tape tests not requiring \texttt{root}. This is the standard set of tests,
365 that in general, backup some data, then restore it, and finally compares the
366 restored data with the original data.
367
368 \item [all-root-tests]
369 \index{all-root-tests}
370 All non-tape tests requiring \texttt{root} permission. These are a relatively small
371 number of tests that require running as \texttt{root}. The amount of data backed up
372 can be quite large. For example, one test backs up \bdirectoryname{/usr}, another backs up
373 \bdirectoryname{/etc}. One or more of these tests reports an error -- I'll fix it one day.
374
375 \item [all-non-root-tape-tests]
376 \index{all-non-root-tape-tests}
377 All tape test not requiring root. There are currently three tests, all run
378 without being root, and backup to a tape. The first two tests use one volume,
379 and the third test requires an autochanger, and uses two volumes. If you
380 don't have an autochanger, then this script will probably produce an error.
381
382 \item [all-tape-and-file-tests]
383 \index{all-tape-and-file-tests}
384 All tape and file tests not requiring \texttt{root}. This includes just about
385 everything, and I don't run it very often.
386 \end{bdescription}
387
388 \subsection{If a Test Fails}
389 \index{Fails!If a Test}
390 \index{If a Test Fails}
391
392 If you one or more tests fail, the line output will be similar to:
393
394 \begin{bVerbatim}
395 !!!!! concurrent-jobs-test failed!!! !!!!!
396 \end{bVerbatim}
397
398 If you want to determine why the test failed, you will need to rerun the
399 script with the debug output turned on. You do so by defining the
400 environment variable \textbf{\texttt{REGRESS\_DEBUG}} with commands such as:
401
402 \begin{bVerbatim}
403 REGRESS_DEBUG=1
404 export REGRESS_DEBUG
405 \end{bVerbatim}
406
407 Then from the \bdirectoryname{regress} directory (all regression scripts assume that
408 you have \bdirectoryname{regress} as the current directory), enter:
409
410 \begin{bVerbatim}
411 tests/test-name
412 \end{bVerbatim}
413
414 where test-name should be the name of a test script -- for example:
415 \btool{tests/backup-bacula-test}.
416
417 \section{Testing a Binary Installation}
418 \index{Test!Testing a Binary Installation}
419
420 If you have installed your \bacula{} from a binary release such as (rpms or debs),
421 you can still run regression tests on it.
422 First, make sure that your regression \bfilename{config} file uses the same catalog backend as
423 your installed binaries. Then define the variables \texttt{bin} and \texttt{scripts} variables
424 in your config file.
425
426 Example:
427 \begin{bVerbatim}
428 bin=/opt/bacula/bin
429 scripts=/opt/bacula/scripts
430 \end{bVerbatim}
431
432 The \bfilename{./scripts/prepare-other-loc} will tweak the regress scripts to use
433 your binary location. You will need to run it manually once before you run any
434 regression tests.
435
436 \begin{bVerbatim}
437 $ ./scripts/prepare-other-loc
438 $ ./tests/backup-bacula-test
439 ...
440 \end{bVerbatim}
441
442 All regression scripts must be run by hand or by calling the test scripts.
443 These are principally scripts that begin with \textbf{all\_...} such as \btool{all\_disk\_tests},
444 \btool{./all\_test}, \ldots{}
445 None of the
446 \btool{./do\_disk}, \btool{./do\_all}, \btool{./nightly...} scripts will work.
447
448 If you want to switch back to running the regression scripts from source, first
449 remove the \texttt{bin} and \texttt{scripts} variables from your \bfilename{config} file and
450 rerun the \btool{make setup} step.
451
452 \section{Running a Single Test}
453 \index{Running a Single Test}
454
455 If you wish to run a single test, you can simply:
456
457 \begin{bVerbatim}
458 cd regress
459 tests/<name-of-test>
460 \end{bVerbatim}
461
462 or, if the source code has been updated, you would do:
463
464 \begin{bVerbatim}
465 cd bacula
466 git pull
467 cd regress
468 make setup
469 tests/backup-to-null
470 \end{bVerbatim}
471
472
473 \section{Writing a Regression Test}
474 \index{Test!Writing a Regression}
475 \index{Writing a Regression Test}
476
477 Any developer, who implements a major new feature, should write a regression
478 test that exercises and validates the new feature. Each regression test is a
479 complete test by itself. It terminates any running \bacula{}, initializes the
480 database, starts \bacula{}, then runs the test by using the console program.
481
482 \subsection{Running the Tests by Hand}
483 \index{Hand!Running the Tests by}
484 \index{Running the Tests by Hand}
485
486 You can run any individual test by hand by cd'ing to the \bdirectoryname{regress}
487 directory and entering:
488
489 \begin{bVerbatim}
490 tests/<test-name>
491 \end{bVerbatim}
492
493 \subsection{Directory Structure}
494 \index{Structure!Directory}
495 \index{Directory Structure}
496
497 The directory structure of the regression tests is:
498
499 \begin{bVerbatim}
500 regress - Makefile, scripts to start tests
501 |------ scripts - Scripts and conf files
502 |-------tests - All test scripts are here
503 |
504 |------------------ -- All directories below this point are used
505 | for testing, but are created from the
506 | above directories and are removed with
507 | "make distclean"
508 |
509 |------ bin - This is the install directory for
510 | Bacula to be used testing
511 |------ build - Where the Bacula source build tree is
512 |------ tmp - Most temp files go here
513 |------ working - Bacula working directory
514 |------ weird-files - Weird files used in two of the tests.
515 \end{bVerbatim}
516
517 \subsection{Adding a New Test}
518 \index{Adding a New Test}
519 \index{Test!Adding a New}
520
521 If you want to write a new regression test, it is best to start with one of
522 the existing test scripts, and modify it to do the new test.
523
524 When adding a new test, be extremely careful about adding anything to any of
525 the daemons' configuration files. The reason is that it may change the prompts
526 that are sent to the console. For example, adding a Pool means that the
527 current scripts, which assume that \bacula{} automatically selects a Pool, will
528 now be presented with a new prompt, so the test will fail. If you need to
529 enhance the configuration files, consider making your own versions.
530
531 Once written, if the test can be executed automatically, you can do a separated
532 commit to schedule the new test via CDASH by editing the
533 \texttt{regress/DartTestFile.txt.in}
534
535 \subsection{Adding a Unittest}
536 \index{Adding a Unittest}
537 \index{Test!Adding a New}
538
539 For low level operations, it is important to write unittests. They are usually
540 directly included at the end of the file that has the code to test. It is possible
541 to create a new file in \texttt{regress/src} or \texttt{bacula/src/tools}.
542
543 The C code should be compiled with the appropriate Makefile rule (see \texttt{alist\_test})
544 for example.
545 \begin{bVerbatim}
546 mytest_test: Makefile libbac.la mytest.c unittests.o
547 $(CXX) -DTEST_PROGRAM $(DEFS) $(DEBUG) -c $(CPPFLAGS) -I$(srcdir) -I$(basedir) $(DINCLUDE) $(CFLAGS) mytest.c -o mytest_test.o
548 $(LIBTOOL_LINK) $(CXX) $(LDFLAGS) -L. -o $@ mytest_test.o unittests.o $(DLIB) -lbac -lm $(LIBS) $(OPENSSL_LIBS)
549 $(LIBTOOL_INSTALL) $(INSTALL_PROGRAM) $@ $(DESTDIR)$(sbindir)/
550 \end{bVerbatim}
551
552 The Bacula Unittest framework is a simple set of functions that helps to write
553 a unittest. It has functions to
554 \begin{itemize}
555 \item Test variables
556 \item Log messages
557 \item Configure/Unconfigure Bacula tools such as mempool, lock manager
558 \item Do some high level filesystem functions (mkdir, rmdir, stat)
559 \item Ease some string operations
560 \end{itemize}
561
562 The following code explains how to use the Bacula Unittest framework:
563
564 \begin{bVerbatim}
565 int function_to_test(int parameter1, int parameter2)
566 {
567 /* some interesting code */
568 return 1;
569 }
570
571 /* In this case, the test procedure is at the end of the file,
572 * it can be also in a separated file
573 */
574 #ifdef TEST_PROGRAM
575 #include "lib/unittests.h"
576
577 int main(int argc, char **argv)
578 {
579 Unittests tests("app_test");
580 tests.set_nb_tests(4); /* Plan the number of expected tests */
581 tests.configure(TEST_QUIET|TEST_PRINT_LOCAL); /* Configure */
582
583 log("Test a the function_to_test()");
584 is(function_to_test(1,2), 1, "Test function_to_test(1,2)");
585 ok(function_to_test(1,2) == 1, "Test function_to_test(1,2)");
586 isnt(function_to_test(3,4), 2, "Test function_to_test(3,4)");
587 nok(function_to_test(3,4) == 2, "Test function_to_test(3,4)");
588
589 return report();
590 }
591 #endif /* TEST_PROGRAM */
592 \end{bVerbatim}
593
594 The current options in the unittest library are:
595 \begin{itemize}
596 \item [TEST\_QUIET] Display only test in error
597 \item [TEST\_PRINT\_LOCAL] Display the local variables after an error
598 \end{itemize}
599
600 These options can be set via an environment variable (UNITTEST\_TEST\_QUIET, UNITTEST\_TEST\_PRINT\_LOCAL).
601
602 \subsection{Running a Test Under The Debugger}
603 \index{Debugger}
604 You can run a test under the debugger (actually run a \bacula{} daemon
605 under the debugger) by first setting the environment variable
606 \textbf{\texttt{REGRESS\_WAIT}} with commands such as:
607
608 \begin{bVerbatim}
609 REGRESS_WAIT=1
610 export REGRESS_WAIT
611 \end{bVerbatim}
612
613 Then executing the script. When the script prints the following line:
614
615 \begin{bVerbatim}
616 Start Bacula under debugger and enter anything when ready ...
617 \end{bVerbatim}
618
619 You start the \bacula{} component you want to run under the debugger in a
620 different shell window. For example:
621
622 \begin{bVerbatim}
623 cd .../regress/bin
624 gdb bacula-sd
625 (possibly set breakpoints, ...)
626 run -s -f
627 \end{bVerbatim}
628
629 Then enter any character in the window with the above message.
630 An error message will appear saying that the daemon you are debugging
631 is already running, which is the case. You can simply ignore the
632 error message.