Debian Codegraph @

www/th3.html

raw
1 <!DOCTYPE html>
2 <html><head>
3 <meta name="viewport" content="width=device-width, initial-scale=1.0">
4 <meta http-equiv="content-type" content="text/html; charset=UTF-8">
5 <link href="sqlite.css" rel="stylesheet">
6 <title>TH3</title>
7 <!-- path= -->
8 </head>
9 <body>
10 <div class=nosearch>
11 <a href="index.html">
12 <img class="logo" src="images/sqlite370_banner.svg" alt="SQLite" border="0">
13 </a>
14 <div><!-- IE hack to prevent disappearing logo --></div>
15 <div class="tagline desktoponly">
16 Small. Fast. Reliable.<br>Choose any three.
17 </div>
18 <div class="menu mainmenu">
19 <ul>
20 <li><a href="index.html">Home</a>
21 <li class='mobileonly'><a href="javascript:void(0)" onclick='toggle_div("submenu")'>Menu</a>
22 <li class='wideonly'><a href='about.html'>About</a>
23 <li class='desktoponly'><a href="docs.html">Documentation</a>
24 <li class='desktoponly'><a href="download.html">Download</a>
25 <li class='wideonly'><a href='copyright.html'>License</a>
26 <li class='desktoponly'><a href="support.html">Support</a>
27 <li class='desktoponly'><a href="prosupport.html">Purchase</a>
28 <li class='search' id='search_menubutton'>
29 <a href="javascript:void(0)" onclick='toggle_search()'>Search</a>
30 </ul>
31 </div>
32 <div class="menu submenu" id="submenu">
33 <ul>
34 <li><a href='about.html'>About</a>
35 <li><a href='docs.html'>Documentation</a>
36 <li><a href='download.html'>Download</a>
37 <li><a href='support.html'>Support</a>
38 <li><a href='prosupport.html'>Purchase</a>
39 </ul>
40 </div>
41 <div class="searchmenu" id="searchmenu">
42 <form method="GET" action="search">
43 <select name="s" id="searchtype">
44 <option value="d">Search Documentation</option>
45 <option value="c">Search Changelog</option>
46 </select>
47 <input type="text" name="q" id="searchbox" value="">
48 <input type="submit" value="Go">
49 </form>
50 </div>
51 </div>
52 <script>
53 function toggle_div(nm) {
54 var w = document.getElementById(nm);
55 if( w.style.display=="block" ){
56 w.style.display = "none";
57 }else{
58 w.style.display = "block";
59 }
60 }
61 function toggle_search() {
62 var w = document.getElementById("searchmenu");
63 if( w.style.display=="block" ){
64 w.style.display = "none";
65 } else {
66 w.style.display = "block";
67 setTimeout(function(){
68 document.getElementById("searchbox").focus()
69 }, 30);
70 }
71 }
72 function div_off(nm){document.getElementById(nm).style.display="none";}
73 window.onbeforeunload = function(e){div_off("submenu");}
74 /* Disable the Search feature if we are not operating from CGI, since */
75 /* Search is accomplished using CGI and will not work without it. */
76 if( !location.origin || !location.origin.match || !location.origin.match(/http/) ){
77 document.getElementById("search_menubutton").style.display = "none";
78 }
79 /* Used by the Hide/Show button beside syntax diagrams, to toggle the */
80 function hideorshow(btn,obj){
81 var x = document.getElementById(obj);
82 var b = document.getElementById(btn);
83 if( x.style.display!='none' ){
84 x.style.display = 'none';
85 b.innerHTML='show';
86 }else{
87 x.style.display = '';
88 b.innerHTML='hide';
89 }
90 return false;
91 }
92 var antiRobot = 0;
93 function antiRobotGo(){
94 if( antiRobot!=3 ) return;
95 antiRobot = 7;
96 var j = document.getElementById("mtimelink");
97 if(j && j.hasAttribute("data-href")) j.href=j.getAttribute("data-href");
98 }
99 function antiRobotDefense(){
100 document.body.onmousedown=function(){
101 antiRobot |= 2;
102 antiRobotGo();
103 document.body.onmousedown=null;
104 }
105 document.body.onmousemove=function(){
106 antiRobot |= 2;
107 antiRobotGo();
108 document.body.onmousemove=null;
109 }
110 setTimeout(function(){
111 antiRobot |= 1;
112 antiRobotGo();
113 }, 100)
114 antiRobotGo();
115 }
116 antiRobotDefense();
117 </script>
118 <div class=fancy>
119 <div class=nosearch>
120 <div class="fancy_title">
121 TH3
122 </div>
123 <details class="fancy_toc">
124 <summary>Table Of Contents</summary>
125 <div id="toc_sub"><div class="fancy-toc1"><a href="#overview">1. Overview</a></div>
126 <div class="fancy-toc2"><a href="#history">1.1. History</a></div>
127 <div class="fancy-toc1"><a href="#operation">2. Operation</a></div>
128 <div class="fancy-toc1"><a href="#generating_a_test_program">3. Generating A Test Program</a></div>
129 <div class="fancy-toc2"><a href="#test_automation_scripts">3.1. Test Automation Scripts</a></div>
130 <div class="fancy-toc1"><a href="#test_coverage">4. Test Coverage</a></div>
131 <div class="fancy-toc1"><a href="#mutation_testing">5. Mutation Testing</a></div>
132 <div class="fancy-toc1"><a href="#th3_license">6. TH3 License</a></div>
133 </div>
134 </details>
135 </div>
136
137
138
139
140 <h1 id="overview"><span>1. </span>Overview</h1>
141
142 <p>SQLite Test Harness #3 (hereafter "TH3") is one of
143 the <a href="testing.html#harnesses">test harnesses</a> used for testing SQLite.
144 TH3 meets the following objectives:</p>
145
146 <ul>
147 <li><p> TH3 is able to run on embedded platforms that lack the support
148 infrastructure of workstations.</p></li>
149
150 <li><p> TH3 tests SQLite in an as-deployed configuration using only
151 published and documented interfaces.
152 In other words, TH3 tests the compiled object code, not
153 the source code, thus verifying that no problems were introduced
154 by compiler bugs. "Test what you fly and fly what you test."</p></li>
155
156 <li><p> TH3 checks SQLite's response to out-of-memory errors, disk I/O
157 errors, and power loss during transaction commit. </p></li>
158
159 <li><p> TH3 exercises SQLite in a variety of run-time configurations
160 (UTF8 vs UTF16, different pages sizes, varying journal modes, etc.)
161 </p></li>
162
163 <li><p> TH3 achieves 100% branch test coverage (and 100%
164 <a href="https://en.wikipedia.org/wiki/Modified_condition/decision_coverage">MC/DC</a>)
165 over the SQLite core.
166 (Test coverage of extensions such as FTS and RTREE is less than 100%).
167 </p></li>
168 </ul>
169
170 <p>TH3 was originally written for validation testing only, but has
171 subsequently been used for development testing and debugging
172 as well, and has proven very helpful in those roles. A full-coverage
173 test takes less than five minutes on a workstation and hence
174 serves as a fast regression test during day-to-day maintenance
175 of the SQLite code base.</p>
176
177 <h2 id="history"><span>1.1. </span>History</h2>
178
179 <p>TH3 originated from an effort to test SQLite on
180 <a href="https://en.wikipedia.org/wiki/Symbian">SymbianOS</a>.
181 Prior to TH3, all SQLite tests were run using the
182 <a href="http://www.tcl.tk/">TCL</a> script language, but TCL would not (easily)
183 compile on SymbianOS which made testing difficult. The first attempt
184 to remedy this problem was the "TH1" (Test Harness #1) scripting
185 language - a reimplementation of parts of the TCL language in a
186 more portable form that would compile and run on SymbianOS, and
187 that was sufficient to run the SQLite tests. TH1
188 did not survive as a standard testing tool for SQLite,
189 but it did find continued service as a
190 scripting language used to customize the
191 <a href="http://www.fossil-scm.org/">Fossil</a> version control system.
192 There was also a "Test Harness #2" which was an attempt to
193 create a simple scripting language using operator prefix notation
194 to drive tests. TH3 was the third attempt.
195
196 </p><p>At about that same time, some avionics manufacturers were
197 expressing interest in SQLite, which prompted the SQLite developers
198 to design TH3 to support the rigorous testing standards of
199 <a href="https://en.wikipedia.org/wiki/DO-178B">DO-178B</a>.
200
201 </p><p>The first code for TH3 was laid down on 2008-09-25.
202 An intense effort over the next 10 months resulted in TH3 achieving
203 100% MC/DC on 2009-07-25. The TH3 code continues to be improved and
204 expanded.
205
206 </p><p>As of 2018-05-19, the TH3 source tree consists
207 of well over 500,000 lines of source code in 1709 separate files.
208
209 </p><h1 id="operation"><span>2. </span>Operation</h1>
210
211 <p>TH3 is a test program generator. The output of TH3 is a program
212 implemented in C-code and intended to be
213 linked against the SQLite library under test. The generated test
214 program is compiled and run on the target platform in order to verify
215 correct operation of SQLite on that platform.</p>
216
217 <p>The inputs to TH3 are test modules written in C or SQL and
218 small configuration
219 files that determine how to initialize SQLite. The
220 TH3 package includes 1,444 test
221 modules and more than 47 configurations (as of 2018-05-19).
222 New modules and configurations
223 can be added to customize TH3 for specialized applications.
224 Each time TH3 is run, it reads
225 a subset of the available test modules and configuration files to generate
226 a custom C program that performs all of the specified tests under all
227 specified configurations. A complete test of SQLite normally involves running
228 TH3 multiple times to generate multiple test programs covering different
229 aspects of SQLite's operation, then linking all test programs against
230 a common SQLite library and running them separately on the target platform.
231 </p>
232
233 <p>There are no arbitrary limits in TH3. One could generate a
234 single test program that contained all test modules and all configuration files.
235 However, such a test program might be too large to deploy on embedded
236 platforms. (As of 2018-05-19, a full-up TH3 test is over 850,000 lines and
237 58MB of C code.) TH3 provides the ability to break the library of test
238 modules up into smaller, more easily digested pieces.</p>
239
240 <p>Each individual test module might contain dozens, hundreds, or thousands
241 of separate tests. The test modules can be written in C or as scripts of
242 SQL or a mixture of the two. About two-thirds of the existing test modules are
243 written in pure SQL with the remainder either in pure C or a combination
244 of C and SQL.
245 </p>
246
247 <p>Each test module file contains a header which describes the circumstances
248 under which the test is valid. For a particular configuration, only those
249 modules that are compatible with the configuration are run. </p>
250
251 <h1 id="generating_a_test_program"><span>3. </span>Generating A Test Program</h1>
252
253 <p>The TH3 program generator is a TCL script named "<tt>mkth3.tcl</tt>".
254 To generate a test program, one has merely to run this script and supply
255 the names of files containing test modules and configurations on the
256 command line. Test modules are files that use the "<tt>.test</tt>" suffix
257 and configurations are files that use the "<tt>.cfg</tt>" suffix. A
258 typical invocation of mkth3.tcl might look something like the following:</p>
259
260 <div class="codeblock"><pre>tclsh mkth3.tcl *.test *.cfg &gt;testprog1.c
261 </pre></div>
262
263 <p>The output from the mkth3.tcl script is a C program that contains
264 everything needed to run the tests - everything that is except for
265 the SQLite library itself. The generated test program contains
266 implementations for all of the support interfaces used by the test
267 modules and it contains the <tt>main()</tt> routine that drives the
268 tests. To convert the test program into a working executable, simply
269 compile it against SQLite:</p>
270
271 <div class="codeblock"><pre>cc -o testprog1 testprog1.c sqlite3.c
272 </pre></div>
273
274 <p>The compilation step shown immediately above is merely representative.
275 In a working installation, one would normally want
276 to specify optimization parameters and compile-time switches on the
277 compiler command line.</p>
278
279 <p>For testing on embedded systems, the mkth3.tcl script and the compiler
280 steps shown above are performed on an ordinary workstation using
281 a cross-compiler, then the resulting test program is
282 transferred onto the device to be run.</p>
283
284 <p>Once the test program is generated, it is run with no arguments to
285 perform the tests. Progress information as well as error diagnostics
286 appear on standard output. (Alternative output arrangements can be made
287 using a compile-time option for embedded devices that lack a standard
288 output channel.) The program returns zero if there are no
289 errors and non-zero if any problems were detected.</p>
290
291 <p>Typical output from a single TH3 test program run looks like this:
292
293 </p><div class="codeblock"><pre>With SQLite 3.8.11 2015-05-15 04:13:15 56ef98a04765c34c1c2f3ed7a6f03a732f3b886e
294 -DSQLITE_COVERAGE_TEST
295 -DSQLITE_NO_SYNC
296 -DSQLITE_SYSTEM_MALLOC
297 -DSQLITE_THREADSAFE=1
298 Config-begin c1.
299 Begin c1.pager08
300 End c1.pager08
301 Begin c1.build33
302 End c1.build33
303 Begin c1.orderby01
304 End c1.orderby01
305 <i>... 15014 lines of output omitted ....</i>
306 Begin 64k.syscall01
307 End 64k.syscall01
308 Begin 64k.build01
309 End 64k.build01
310 Begin 64k.auth01
311 End 64k.auth01
312 Config-end 64k. TH3 memory used: 6373738
313 Config-begin wal1.
314 Begin wal1.wal37
315 End wal1.wal37
316 Config-end wal1. TH3 memory used: 100961
317 All 226 VDBE coverage points reached
318 th3: 0 errors out of 1442264 tests in 213.741 seconds. 64-bit little-endian
319 th3: SQLite 3.8.11 2015-05-15 04:13:15 56ef98a04765c34c1c2f3ed7a6f03a732f3b886e
320 </pre></div>
321
322 <p>The output begins with a report of the <a href="c3ref/c_scm_branch.html">SQLITE_SOURCE_ID</a>
323 (cross-checked against <a href="c3ref/libversion.html">sqlite3_sourceid()</a>) for the
324 SQLite under test and the compile-time options used as reported
325 by <a href="c3ref/compileoption_get.html">sqlite3_compileoption_get()</a>. The output concludes with a summary
326 of the test results and a repeat of the <a href="c3ref/c_scm_branch.html">SQLITE_SOURCE_ID</a>. If any
327 errors are detected, additional lines detail the problem. The error
328 reporting lines always begin with a single space character so that they
329 can be quickly extracted from large output files using:
330
331 </p><div class="codeblock"><pre>grep "&#94; "
332 </pre></div>
333
334 <p>The default output shows the beginning and end of each configuration
335 and test module combination. In the example above "c1" and "64k" are
336 configurations and "pager08", "build33", "orderby01", etc. are test modules.
337 Compile-time and run-time options are available to increase or decrease
338 the amount of output.
339 The output can be increased by showing each test case within each
340 test module. The output can be decreased
341 by degrees: omitting test modules starts and stops,
342 omitting configuration starts and stops, and finally by omitting all output.
343
344 </p><h2 id="test_automation_scripts"><span>3.1. </span>Test Automation Scripts</h2>
345
346 <p>TH3 comes with additional TCL scripts that help automate the testing
347 process on workstations. The "th3make" script automatically runs "mkth3.tcl"
348 and "gcc" and then runs the resulting test program and checks the results.
349 Arguments to th3make include all of the "*.test" test modules and
350 "*.cfg" configurations that are to be included in the test. Additional
351 options to th3make can cause the test program to be compiled using different
352 compilers (GCC, Clang, MSVC), to use different output verbosity levels, to
353 run the test program under valgrind, to check the output for coverage using
354 gcov, and so forth. The th3make script also accepts "*.rc" filenames as
355 arguments. These *.rc files are just collections of other arguments that
356 are commonly used together for a single purpose. For example, the "quick.rc"
357 file contains a set of eight arguments to th3make that run a fast (3-minute)
358 full-coverage test. This allows the operator to type "./th3make quick.rc" as
359 a short-cut to typing out all of the required command-line options. The
360 following are a few of the more than 40 available *.rc files:
361
362 </p><ul>
363 <li><b>alignment</b><i>N</i><b>.rc</b> -
364 These files contain -D options to the compiler that are used by
365 various notable downstreams.
366 </li><li><b>cov.rc</b> - Options for measuring test coverage
367 </li><li><b>extensions.rc</b> - Options to enable <a href="fts3.html#fts4">FTS4</a>, <a href="rtree.html">R-Trees</a>,
368 and <a href="fileformat2.html#stat4tab">STAT4</a>.
369 </li><li><b>fast.rc</b> - Run most tests, including those not needed for
370 coverage, skipping only soak tests, using delivery compiler options
371 (ex: -O3)
372 </li><li><b>memdebug.rc</b> - like test.rc but also enable
373 <a href="compile.html#memdebug">-DSQLITE_MEMDEBUG</a>.
374 </li><li><b>min.rc</b> - Run the minimum set of tests needed for 100% coverage.
375 </li><li><b>quick.rc</b> - Run all tests required for 100% coverage tests
376 using -Os and <a href="compile.html#debug">-DSQLITE_DEBUG</a>.
377 </li><li><b>test.rc</b> - Run the same tests as in fast.rc but without
378 compiler optimization and enabling options like
379 <a href="compile.html#debug">-DSQLITE_DEBUG</a> and
380 -DSQLITE_ENABLE_EXPENSIVE_ASSERT.
381 </li><li><b>test-ex.rc</b> - long-running soak tests.
382 </li></ul>
383
384 <p>The TH3 repository also includes the "multitest.tcl" script, another
385 TCL script used to automate TH3 testing on workstations. Multitest.tcl
386 automatically compiles SQLite, then
387 runs ./th3make repeatedly with a variety of alignments, and captures
388 the output in a succinct summary screen. A typical multitest.tcl run
389 generates output that looks like this:
390
391 </p><div class="codeblock"><pre>./multitest.tcl -q --jobs 3
392 start-time: 2018-05-19 03:17:12 UTC
393 file mkdir sqlite3bld
394 cd sqlite3bld
395 exec sh /ramdisk/sqlite/configure
396 file copy -force config.h ../config.h
397 exec make clean sqlite3.c
398 file rename sqlite3.c ../sqlite3.c
399 file rename sqlite3.h ../sqlite3.h
400 exec make clean sqlite3.c OPTS=-DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1
401 file rename sqlite3.c ../sqlite3udl.c
402 exec make clean sqlite3.c OPTS=-DSQLITE_SMALL_STACK=1
403 file rename sqlite3.c ../sqlite3ss.c
404 cd ..
405 *******************************************************************************
406 t01: cov.rc.................................................... Ok (00:03:42)
407 t02: cov.rc ++STAT4 ++DESERIALIZE -D_HAVE_SQLITE_CONFIG_H...... Ok (00:04:45)
408 t03: vfs-cov.rc................................................ Ok (00:03:59)
409 t04: demo.rc................................................... Ok (00:00:05)
410 t07: test.rc ../th3private/*.test.............................. Ok (00:00:21)
411 t08: test.rc ../th3private/*.test ++STAT4...................... Ok (00:01:41)
412 t05: quick.rc.................................................. Ok (00:04:26)
413 t09: quick.rc ~TEST_REALLOC_STRESS -funsigned-char............. Ok (00:05:39)
414 t10: quick.rc ~THREADSAFE=0 -DLONGDOUBLE_TYPE=double........... Ok (00:03:24)
415 t06: quick.rc extensions.rc -D_HAVE_SQLITE_CONFIG_H............ Ok (00:09:03)
416 t11: quick.rc sqlite3ss.c ~MAX_ATTACHED=125.................... Ok (00:04:39)
417 t12: quick.rc ~BYTEORDER=0 ++RTREE............................. Ok (00:07:28)
418 t13: quick.rc ~DISABLE_INTRINSIC ++RTREE....................... Ok (00:07:31)
419 t16: quick.rc ~TRACE_SIZE_LIMIT=15 cov1/main16.test............ Ok (00:00:22)
420 t14: quick.rc ~DIRECT_OVERFLOW_READ -fsigned-char.............. Ok (00:04:35)
421 t15: quick.rc ~UNTESTABLE ~EXTRA_IFNULLROW..................... Ok (00:01:44)
422 t17: quick.rc ~MAX_MMAP_SIZE=0................................. Ok (00:04:46)
423 t18: quick.rc ++NULL_TRIM ++OFFSET_SQL_FUNC.................... Ok (00:04:47)
424 t19: quick.rc ++BATCH_ATOMIC_WRITE ++DESERIALIZE............... Ok (00:05:41)
425 t20: lean1.rc quick.rc......................................... Ok (00:03:09)
426 t22: test.rc alignment2.rc sqlite3udl.c........................ Ok (00:44:22)
427 t21: test.rc alignment1.rc..................................... Ok (01:02:32)
428 t23: memdebug1.rc extensions.rc................................ Ok (01:49:58)
429 t25: valgrind1.rc -O3 extensions.rc............................ Ok (00:56:08)
430 t24: memdebug2.rc extensions.rc................................ Ok (01:43:34)
431 t27: test-ex1.rc............................................... Ok (00:45:00)
432 t26: valgrind2.rc -O3 extensions.rc............................ Ok (01:02:52)
433 t29: test-ex3.rc............................................... Ok (00:31:48)
434 t28: test-ex2.rc............................................... Ok (01:12:03)
435 t30: test-ex4.rc............................................... Ok (01:09:47)
436 t32: test.rc alignment4.rc -m32 CC=clang....................... Ok (00:48:31)
437 t31: test.rc alignment3.rc sqlite3udl.c........................ Ok (01:22:29)
438 t34: test.rc alignment6.rc..................................... Ok (00:35:31)
439 t33: test.rc alignment5.rc extensions.rc....................... Ok (00:59:33)
440 t35: test.rc alignment7.rc..................................... Ok (00:44:10)
441 t40: fast.rc alignment2.rc sqlite3udl.c........................ Ok (00:15:46)
442 t39: fast.rc alignment1.rc extensions.rc -m32.................. Ok (00:33:19)
443 t36: test.rc ~MUTATION_TEST.................................... Ok (00:35:45)
444 t42: fast.rc alignment4.rc..................................... Ok (00:13:03)
445 t43: fast.rc alignment5.rc..................................... Ok (00:13:32)
446 t44: fast.rc alignment6.rc..................................... Ok (00:11:41)
447 t41: fast.rc alignment3.rc sqlite3udl.c........................ Ok (00:26:31)
448 t45: fast.rc alignment7.rc..................................... Ok (00:12:57)
449 t46: fast.rc -fsanitize=undefined.............................. Ok (00:38:18)
450 *******************************************************************************
451 0 failures on 44 th3makes and 198583082 tests in (07:16:01) 3 cores on bella
452 SQLite 3.24.0 2018-05-18 17:58:33 c6071ac99cfa4b6272ac4d739fc61a85acb544f6c1c2a
453 </pre></div>
454
455 <p>As can be seen above, a single run
456 of multitest.tcl invokes th3make dozens of times and takes between 12 and 24
457 CPU hours. The middle section of the output shows the arguments to each
458 individual th3make run and the result and elapse time for that th3make.
459 All build products and output for the separate th3make runs are
460 captured in subdirectories for post-test analysis.
461 The two-line summary at the bottom shows the total number of errors and tests
462 over all th3make runs and the total elapsed time, together with the
463 <a href="c3ref/c_scm_branch.html">SQLITE_SOURCE_ID</a> information for the version of SQLite that was
464 tested. This summary information is recorded in the
465 <a href="https://sqlite.org/checklists">release
466 checklist</a> during final testing.
467
468 </p><p>Abbreviations are applied in the multitest.tcl output so that
469 each th3make invocation will fit on a single 80-column output line.
470 The initial "th3make" verb is omitted.
471 "~" is shorthand for "-DSQLITE_" and "++" is stands for
472 "-DSQLITE_ENABLE". Hence, multitest.tcl output line
473
474 </p><div class="codeblock"><pre>quick.rc ~DISABLE_INTRINSIC ++RTREE
475 </pre></div>
476
477 <p>Really means
478
479 </p><div class="codeblock"><pre>th3make quick.rc -DSQLITE_DISABLE_INTRINSIC -DSQLITE_ENABLE_RTREE
480 </pre></div>
481
482 <h1 id="test_coverage"><span>4. </span>Test Coverage</h1>
483
484 <p>Using one particular subset of the available TH3 test modules (the "cov1"
485 tests) SQLite obtained
486 <a href="testing.html#coverage">100% branch test coverage</a> and 100% <a href="testing.html#mcdc">MC/DC</a> as measured
487 by <a href="http://gcc.gnu.org/onlinedocs/gcc/Gcov.html">gcov</a>
488 on Linux x86 and x86_64 hardware. All releases of SQLite since
489 <a href="releaselog/3_6_17.html">version 3.6.17</a> (2009-08-10) have been tested to this standard.
490 The SQLite developers
491 are committed to maintaining 100% branch coverage and MC/DC for all
492 future releases of SQLite.</p>
493
494 <p>The cov1 test set used to obtain 100% branch test coverage are only a
495 subset of the tests currently implemented using TH3. New test modules are
496 added on a regular basis.</p>
497
498 <a name="muttest"></a>
499
500 <h1 id="mutation_testing"><span>5. </span>Mutation Testing</h1>
501
502 <p>The TH3 source tree contains a script named
503 "mutation-test.tcl" that automates the process of
504 <a href="testing.html#mutationtests">mutation testing</a>.
505
506 </p><p>The mutation-test.tcl script takes care of all of the details for
507 running a mutation test:
508
509 </p><ol>
510 <li> The script compiles the TH3 test harness into machine code ("th3.o") if
511 necessary.
512 </li><li> The script compiles the sqlite3.c source file into assembly language
513 ("sqlite3.s") if necessary.
514 </li><li> The script loops through instructions in the assembly language file
515 to locate branch operations.
516 <ol type="a">
517 <li>The script makes a copy of the original sqlite3.s file.
518 </li><li>The copy is edited to change the branch instruction into either
519 a no-op or an unconditional jump.
520 </li><li>The copy of sqlite3.s is assembled into sqlite3.o then linked
521 against th3.o to generate the "th3" executable.
522 </li><li>The "th3" binary is run and the output checked for errors.
523 </li></ol>
524 </li><li> The script shows progress for each cycle of the previous step then
525 displays a summary of "survivors" at the end. A "survivor" is a
526 mutation that was not detected by TH3.
527 </li></ol>
528
529 <p>Mutation testing can be slow, since each test can take up to 5
530 minutes on a fast workstation, and there are two tests for each
531 branch instruction, and over 20,000 branch instructions. Efforts are
532 made to expedite operation. For example, TH3 is compiled in such a
533 way that it exits as soon as it finds the first error, and as many
534 of the mutations are easily detected, many cycles happen in only
535 a few seconds. Nevertheless, the mutation-test.tcl script includes
536 command-line options to limit the range of code lines tested so that
537 mutation testing only needs to be performed on blocks of code that
538 have recently changed.
539
540 </p><h1 id="th3_license"><span>6. </span>TH3 License</h1>
541
542 <p>SQLite itself is in the <a href="copyright.html">public domain</a> and
543 can be used for any purpose. But TH3 is proprietary and requires a license.
544 </p>
545
546 <p>Even though open-source users do not have direct access to TH3, all
547 users of SQLite benefit from TH3 indirectly since each version of SQLite is
548 validated running TH3 on multiple platforms (Linux, Windows, WinRT, Mac,
549 OpenBSD) prior to release. So anyone using an official release
550 of SQLite can deploy their application with the confidence of knowing that
551 it has been tested using TH3. They simply cannot rerun those tests
552 themselves without purchasing a TH3 license.</p>
553 <p align="center"><small><i>This page was last updated on 2026-04-21 10:28:59Z </small></i></p>
554