| 1 |
<!DOCTYPE html> |
| 2 |
<html><head> |
| 3 |
<meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| 4 |
<meta http-equiv="content-type" content="text/html; charset=UTF-8"> |
| 5 |
<link href="sqlite.css" rel="stylesheet"> |
| 6 |
<title>TH3</title> |
| 7 |
<!-- path= --> |
| 8 |
</head> |
| 9 |
<body> |
| 10 |
<div class=nosearch> |
| 11 |
<a href="index.html"> |
| 12 |
<img class="logo" src="images/sqlite370_banner.svg" alt="SQLite" border="0"> |
| 13 |
</a> |
| 14 |
<div><!-- IE hack to prevent disappearing logo --></div> |
| 15 |
<div class="tagline desktoponly"> |
| 16 |
Small. Fast. Reliable.<br>Choose any three. |
| 17 |
</div> |
| 18 |
<div class="menu mainmenu"> |
| 19 |
<ul> |
| 20 |
<li><a href="index.html">Home</a> |
| 21 |
<li class='mobileonly'><a href="javascript:void(0)" onclick='toggle_div("submenu")'>Menu</a> |
| 22 |
<li class='wideonly'><a href='about.html'>About</a> |
| 23 |
<li class='desktoponly'><a href="docs.html">Documentation</a> |
| 24 |
<li class='desktoponly'><a href="download.html">Download</a> |
| 25 |
<li class='wideonly'><a href='copyright.html'>License</a> |
| 26 |
<li class='desktoponly'><a href="support.html">Support</a> |
| 27 |
<li class='desktoponly'><a href="prosupport.html">Purchase</a> |
| 28 |
<li class='search' id='search_menubutton'> |
| 29 |
<a href="javascript:void(0)" onclick='toggle_search()'>Search</a> |
| 30 |
</ul> |
| 31 |
</div> |
| 32 |
<div class="menu submenu" id="submenu"> |
| 33 |
<ul> |
| 34 |
<li><a href='about.html'>About</a> |
| 35 |
<li><a href='docs.html'>Documentation</a> |
| 36 |
<li><a href='download.html'>Download</a> |
| 37 |
<li><a href='support.html'>Support</a> |
| 38 |
<li><a href='prosupport.html'>Purchase</a> |
| 39 |
</ul> |
| 40 |
</div> |
| 41 |
<div class="searchmenu" id="searchmenu"> |
| 42 |
<form method="GET" action="search"> |
| 43 |
<select name="s" id="searchtype"> |
| 44 |
<option value="d">Search Documentation</option> |
| 45 |
<option value="c">Search Changelog</option> |
| 46 |
</select> |
| 47 |
<input type="text" name="q" id="searchbox" value=""> |
| 48 |
<input type="submit" value="Go"> |
| 49 |
</form> |
| 50 |
</div> |
| 51 |
</div> |
| 52 |
<script> |
| 53 |
function toggle_div(nm) { |
| 54 |
var w = document.getElementById(nm); |
| 55 |
if( w.style.display=="block" ){ |
| 56 |
w.style.display = "none"; |
| 57 |
}else{ |
| 58 |
w.style.display = "block"; |
| 59 |
} |
| 60 |
} |
| 61 |
function toggle_search() { |
| 62 |
var w = document.getElementById("searchmenu"); |
| 63 |
if( w.style.display=="block" ){ |
| 64 |
w.style.display = "none"; |
| 65 |
} else { |
| 66 |
w.style.display = "block"; |
| 67 |
setTimeout(function(){ |
| 68 |
document.getElementById("searchbox").focus() |
| 69 |
}, 30); |
| 70 |
} |
| 71 |
} |
| 72 |
function div_off(nm){document.getElementById(nm).style.display="none";} |
| 73 |
window.onbeforeunload = function(e){div_off("submenu");} |
| 74 |
/* Disable the Search feature if we are not operating from CGI, since */ |
| 75 |
/* Search is accomplished using CGI and will not work without it. */ |
| 76 |
if( !location.origin || !location.origin.match || !location.origin.match(/http/) ){ |
| 77 |
document.getElementById("search_menubutton").style.display = "none"; |
| 78 |
} |
| 79 |
/* Used by the Hide/Show button beside syntax diagrams, to toggle the */ |
| 80 |
function hideorshow(btn,obj){ |
| 81 |
var x = document.getElementById(obj); |
| 82 |
var b = document.getElementById(btn); |
| 83 |
if( x.style.display!='none' ){ |
| 84 |
x.style.display = 'none'; |
| 85 |
b.innerHTML='show'; |
| 86 |
}else{ |
| 87 |
x.style.display = ''; |
| 88 |
b.innerHTML='hide'; |
| 89 |
} |
| 90 |
return false; |
| 91 |
} |
| 92 |
var antiRobot = 0; |
| 93 |
function antiRobotGo(){ |
| 94 |
if( antiRobot!=3 ) return; |
| 95 |
antiRobot = 7; |
| 96 |
var j = document.getElementById("mtimelink"); |
| 97 |
if(j && j.hasAttribute("data-href")) j.href=j.getAttribute("data-href"); |
| 98 |
} |
| 99 |
function antiRobotDefense(){ |
| 100 |
document.body.onmousedown=function(){ |
| 101 |
antiRobot |= 2; |
| 102 |
antiRobotGo(); |
| 103 |
document.body.onmousedown=null; |
| 104 |
} |
| 105 |
document.body.onmousemove=function(){ |
| 106 |
antiRobot |= 2; |
| 107 |
antiRobotGo(); |
| 108 |
document.body.onmousemove=null; |
| 109 |
} |
| 110 |
setTimeout(function(){ |
| 111 |
antiRobot |= 1; |
| 112 |
antiRobotGo(); |
| 113 |
}, 100) |
| 114 |
antiRobotGo(); |
| 115 |
} |
| 116 |
antiRobotDefense(); |
| 117 |
</script> |
| 118 |
<div class=fancy> |
| 119 |
<div class=nosearch> |
| 120 |
<div class="fancy_title"> |
| 121 |
TH3 |
| 122 |
</div> |
| 123 |
<details class="fancy_toc"> |
| 124 |
<summary>Table Of Contents</summary> |
| 125 |
<div id="toc_sub"><div class="fancy-toc1"><a href="#overview">1. Overview</a></div> |
| 126 |
<div class="fancy-toc2"><a href="#history">1.1. History</a></div> |
| 127 |
<div class="fancy-toc1"><a href="#operation">2. Operation</a></div> |
| 128 |
<div class="fancy-toc1"><a href="#generating_a_test_program">3. Generating A Test Program</a></div> |
| 129 |
<div class="fancy-toc2"><a href="#test_automation_scripts">3.1. Test Automation Scripts</a></div> |
| 130 |
<div class="fancy-toc1"><a href="#test_coverage">4. Test Coverage</a></div> |
| 131 |
<div class="fancy-toc1"><a href="#mutation_testing">5. Mutation Testing</a></div> |
| 132 |
<div class="fancy-toc1"><a href="#th3_license">6. TH3 License</a></div> |
| 133 |
</div> |
| 134 |
</details> |
| 135 |
</div> |
| 136 |
|
| 137 |
|
| 138 |
|
| 139 |
|
| 140 |
<h1 id="overview"><span>1. </span>Overview</h1> |
| 141 |
|
| 142 |
<p>SQLite Test Harness #3 (hereafter "TH3") is one of |
| 143 |
the <a href="testing.html#harnesses">test harnesses</a> used for testing SQLite. |
| 144 |
TH3 meets the following objectives:</p> |
| 145 |
|
| 146 |
<ul> |
| 147 |
<li><p> TH3 is able to run on embedded platforms that lack the support |
| 148 |
infrastructure of workstations.</p></li> |
| 149 |
|
| 150 |
<li><p> TH3 tests SQLite in an as-deployed configuration using only |
| 151 |
published and documented interfaces. |
| 152 |
In other words, TH3 tests the compiled object code, not |
| 153 |
the source code, thus verifying that no problems were introduced |
| 154 |
by compiler bugs. "Test what you fly and fly what you test."</p></li> |
| 155 |
|
| 156 |
<li><p> TH3 checks SQLite's response to out-of-memory errors, disk I/O |
| 157 |
errors, and power loss during transaction commit. </p></li> |
| 158 |
|
| 159 |
<li><p> TH3 exercises SQLite in a variety of run-time configurations |
| 160 |
(UTF8 vs UTF16, different pages sizes, varying journal modes, etc.) |
| 161 |
</p></li> |
| 162 |
|
| 163 |
<li><p> TH3 achieves 100% branch test coverage (and 100% |
| 164 |
<a href="https://en.wikipedia.org/wiki/Modified_condition/decision_coverage">MC/DC</a>) |
| 165 |
over the SQLite core. |
| 166 |
(Test coverage of extensions such as FTS and RTREE is less than 100%). |
| 167 |
</p></li> |
| 168 |
</ul> |
| 169 |
|
| 170 |
<p>TH3 was originally written for validation testing only, but has |
| 171 |
subsequently been used for development testing and debugging |
| 172 |
as well, and has proven very helpful in those roles. A full-coverage |
| 173 |
test takes less than five minutes on a workstation and hence |
| 174 |
serves as a fast regression test during day-to-day maintenance |
| 175 |
of the SQLite code base.</p> |
| 176 |
|
| 177 |
<h2 id="history"><span>1.1. </span>History</h2> |
| 178 |
|
| 179 |
<p>TH3 originated from an effort to test SQLite on |
| 180 |
<a href="https://en.wikipedia.org/wiki/Symbian">SymbianOS</a>. |
| 181 |
Prior to TH3, all SQLite tests were run using the |
| 182 |
<a href="http://www.tcl.tk/">TCL</a> script language, but TCL would not (easily) |
| 183 |
compile on SymbianOS which made testing difficult. The first attempt |
| 184 |
to remedy this problem was the "TH1" (Test Harness #1) scripting |
| 185 |
language - a reimplementation of parts of the TCL language in a |
| 186 |
more portable form that would compile and run on SymbianOS, and |
| 187 |
that was sufficient to run the SQLite tests. TH1 |
| 188 |
did not survive as a standard testing tool for SQLite, |
| 189 |
but it did find continued service as a |
| 190 |
scripting language used to customize the |
| 191 |
<a href="http://www.fossil-scm.org/">Fossil</a> version control system. |
| 192 |
There was also a "Test Harness #2" which was an attempt to |
| 193 |
create a simple scripting language using operator prefix notation |
| 194 |
to drive tests. TH3 was the third attempt. |
| 195 |
|
| 196 |
</p><p>At about that same time, some avionics manufacturers were |
| 197 |
expressing interest in SQLite, which prompted the SQLite developers |
| 198 |
to design TH3 to support the rigorous testing standards of |
| 199 |
<a href="https://en.wikipedia.org/wiki/DO-178B">DO-178B</a>. |
| 200 |
|
| 201 |
</p><p>The first code for TH3 was laid down on 2008-09-25. |
| 202 |
An intense effort over the next 10 months resulted in TH3 achieving |
| 203 |
100% MC/DC on 2009-07-25. The TH3 code continues to be improved and |
| 204 |
expanded. |
| 205 |
|
| 206 |
</p><p>As of 2018-05-19, the TH3 source tree consists |
| 207 |
of well over 500,000 lines of source code in 1709 separate files. |
| 208 |
|
| 209 |
</p><h1 id="operation"><span>2. </span>Operation</h1> |
| 210 |
|
| 211 |
<p>TH3 is a test program generator. The output of TH3 is a program |
| 212 |
implemented in C-code and intended to be |
| 213 |
linked against the SQLite library under test. The generated test |
| 214 |
program is compiled and run on the target platform in order to verify |
| 215 |
correct operation of SQLite on that platform.</p> |
| 216 |
|
| 217 |
<p>The inputs to TH3 are test modules written in C or SQL and |
| 218 |
small configuration |
| 219 |
files that determine how to initialize SQLite. The |
| 220 |
TH3 package includes 1,444 test |
| 221 |
modules and more than 47 configurations (as of 2018-05-19). |
| 222 |
New modules and configurations |
| 223 |
can be added to customize TH3 for specialized applications. |
| 224 |
Each time TH3 is run, it reads |
| 225 |
a subset of the available test modules and configuration files to generate |
| 226 |
a custom C program that performs all of the specified tests under all |
| 227 |
specified configurations. A complete test of SQLite normally involves running |
| 228 |
TH3 multiple times to generate multiple test programs covering different |
| 229 |
aspects of SQLite's operation, then linking all test programs against |
| 230 |
a common SQLite library and running them separately on the target platform. |
| 231 |
</p> |
| 232 |
|
| 233 |
<p>There are no arbitrary limits in TH3. One could generate a |
| 234 |
single test program that contained all test modules and all configuration files. |
| 235 |
However, such a test program might be too large to deploy on embedded |
| 236 |
platforms. (As of 2018-05-19, a full-up TH3 test is over 850,000 lines and |
| 237 |
58MB of C code.) TH3 provides the ability to break the library of test |
| 238 |
modules up into smaller, more easily digested pieces.</p> |
| 239 |
|
| 240 |
<p>Each individual test module might contain dozens, hundreds, or thousands |
| 241 |
of separate tests. The test modules can be written in C or as scripts of |
| 242 |
SQL or a mixture of the two. About two-thirds of the existing test modules are |
| 243 |
written in pure SQL with the remainder either in pure C or a combination |
| 244 |
of C and SQL. |
| 245 |
</p> |
| 246 |
|
| 247 |
<p>Each test module file contains a header which describes the circumstances |
| 248 |
under which the test is valid. For a particular configuration, only those |
| 249 |
modules that are compatible with the configuration are run. </p> |
| 250 |
|
| 251 |
<h1 id="generating_a_test_program"><span>3. </span>Generating A Test Program</h1> |
| 252 |
|
| 253 |
<p>The TH3 program generator is a TCL script named "<tt>mkth3.tcl</tt>". |
| 254 |
To generate a test program, one has merely to run this script and supply |
| 255 |
the names of files containing test modules and configurations on the |
| 256 |
command line. Test modules are files that use the "<tt>.test</tt>" suffix |
| 257 |
and configurations are files that use the "<tt>.cfg</tt>" suffix. A |
| 258 |
typical invocation of mkth3.tcl might look something like the following:</p> |
| 259 |
|
| 260 |
<div class="codeblock"><pre>tclsh mkth3.tcl *.test *.cfg >testprog1.c |
| 261 |
</pre></div> |
| 262 |
|
| 263 |
<p>The output from the mkth3.tcl script is a C program that contains |
| 264 |
everything needed to run the tests - everything that is except for |
| 265 |
the SQLite library itself. The generated test program contains |
| 266 |
implementations for all of the support interfaces used by the test |
| 267 |
modules and it contains the <tt>main()</tt> routine that drives the |
| 268 |
tests. To convert the test program into a working executable, simply |
| 269 |
compile it against SQLite:</p> |
| 270 |
|
| 271 |
<div class="codeblock"><pre>cc -o testprog1 testprog1.c sqlite3.c |
| 272 |
</pre></div> |
| 273 |
|
| 274 |
<p>The compilation step shown immediately above is merely representative. |
| 275 |
In a working installation, one would normally want |
| 276 |
to specify optimization parameters and compile-time switches on the |
| 277 |
compiler command line.</p> |
| 278 |
|
| 279 |
<p>For testing on embedded systems, the mkth3.tcl script and the compiler |
| 280 |
steps shown above are performed on an ordinary workstation using |
| 281 |
a cross-compiler, then the resulting test program is |
| 282 |
transferred onto the device to be run.</p> |
| 283 |
|
| 284 |
<p>Once the test program is generated, it is run with no arguments to |
| 285 |
perform the tests. Progress information as well as error diagnostics |
| 286 |
appear on standard output. (Alternative output arrangements can be made |
| 287 |
using a compile-time option for embedded devices that lack a standard |
| 288 |
output channel.) The program returns zero if there are no |
| 289 |
errors and non-zero if any problems were detected.</p> |
| 290 |
|
| 291 |
<p>Typical output from a single TH3 test program run looks like this: |
| 292 |
|
| 293 |
</p><div class="codeblock"><pre>With SQLite 3.8.11 2015-05-15 04:13:15 56ef98a04765c34c1c2f3ed7a6f03a732f3b886e |
| 294 |
-DSQLITE_COVERAGE_TEST |
| 295 |
-DSQLITE_NO_SYNC |
| 296 |
-DSQLITE_SYSTEM_MALLOC |
| 297 |
-DSQLITE_THREADSAFE=1 |
| 298 |
Config-begin c1. |
| 299 |
Begin c1.pager08 |
| 300 |
End c1.pager08 |
| 301 |
Begin c1.build33 |
| 302 |
End c1.build33 |
| 303 |
Begin c1.orderby01 |
| 304 |
End c1.orderby01 |
| 305 |
<i>... 15014 lines of output omitted ....</i> |
| 306 |
Begin 64k.syscall01 |
| 307 |
End 64k.syscall01 |
| 308 |
Begin 64k.build01 |
| 309 |
End 64k.build01 |
| 310 |
Begin 64k.auth01 |
| 311 |
End 64k.auth01 |
| 312 |
Config-end 64k. TH3 memory used: 6373738 |
| 313 |
Config-begin wal1. |
| 314 |
Begin wal1.wal37 |
| 315 |
End wal1.wal37 |
| 316 |
Config-end wal1. TH3 memory used: 100961 |
| 317 |
All 226 VDBE coverage points reached |
| 318 |
th3: 0 errors out of 1442264 tests in 213.741 seconds. 64-bit little-endian |
| 319 |
th3: SQLite 3.8.11 2015-05-15 04:13:15 56ef98a04765c34c1c2f3ed7a6f03a732f3b886e |
| 320 |
</pre></div> |
| 321 |
|
| 322 |
<p>The output begins with a report of the <a href="c3ref/c_scm_branch.html">SQLITE_SOURCE_ID</a> |
| 323 |
(cross-checked against <a href="c3ref/libversion.html">sqlite3_sourceid()</a>) for the |
| 324 |
SQLite under test and the compile-time options used as reported |
| 325 |
by <a href="c3ref/compileoption_get.html">sqlite3_compileoption_get()</a>. The output concludes with a summary |
| 326 |
of the test results and a repeat of the <a href="c3ref/c_scm_branch.html">SQLITE_SOURCE_ID</a>. If any |
| 327 |
errors are detected, additional lines detail the problem. The error |
| 328 |
reporting lines always begin with a single space character so that they |
| 329 |
can be quickly extracted from large output files using: |
| 330 |
|
| 331 |
</p><div class="codeblock"><pre>grep "^ " |
| 332 |
</pre></div> |
| 333 |
|
| 334 |
<p>The default output shows the beginning and end of each configuration |
| 335 |
and test module combination. In the example above "c1" and "64k" are |
| 336 |
configurations and "pager08", "build33", "orderby01", etc. are test modules. |
| 337 |
Compile-time and run-time options are available to increase or decrease |
| 338 |
the amount of output. |
| 339 |
The output can be increased by showing each test case within each |
| 340 |
test module. The output can be decreased |
| 341 |
by degrees: omitting test modules starts and stops, |
| 342 |
omitting configuration starts and stops, and finally by omitting all output. |
| 343 |
|
| 344 |
</p><h2 id="test_automation_scripts"><span>3.1. </span>Test Automation Scripts</h2> |
| 345 |
|
| 346 |
<p>TH3 comes with additional TCL scripts that help automate the testing |
| 347 |
process on workstations. The "th3make" script automatically runs "mkth3.tcl" |
| 348 |
and "gcc" and then runs the resulting test program and checks the results. |
| 349 |
Arguments to th3make include all of the "*.test" test modules and |
| 350 |
"*.cfg" configurations that are to be included in the test. Additional |
| 351 |
options to th3make can cause the test program to be compiled using different |
| 352 |
compilers (GCC, Clang, MSVC), to use different output verbosity levels, to |
| 353 |
run the test program under valgrind, to check the output for coverage using |
| 354 |
gcov, and so forth. The th3make script also accepts "*.rc" filenames as |
| 355 |
arguments. These *.rc files are just collections of other arguments that |
| 356 |
are commonly used together for a single purpose. For example, the "quick.rc" |
| 357 |
file contains a set of eight arguments to th3make that run a fast (3-minute) |
| 358 |
full-coverage test. This allows the operator to type "./th3make quick.rc" as |
| 359 |
a short-cut to typing out all of the required command-line options. The |
| 360 |
following are a few of the more than 40 available *.rc files: |
| 361 |
|
| 362 |
</p><ul> |
| 363 |
<li><b>alignment</b><i>N</i><b>.rc</b> - |
| 364 |
These files contain -D options to the compiler that are used by |
| 365 |
various notable downstreams. |
| 366 |
</li><li><b>cov.rc</b> - Options for measuring test coverage |
| 367 |
</li><li><b>extensions.rc</b> - Options to enable <a href="fts3.html#fts4">FTS4</a>, <a href="rtree.html">R-Trees</a>, |
| 368 |
and <a href="fileformat2.html#stat4tab">STAT4</a>. |
| 369 |
</li><li><b>fast.rc</b> - Run most tests, including those not needed for |
| 370 |
coverage, skipping only soak tests, using delivery compiler options |
| 371 |
(ex: -O3) |
| 372 |
</li><li><b>memdebug.rc</b> - like test.rc but also enable |
| 373 |
<a href="compile.html#memdebug">-DSQLITE_MEMDEBUG</a>. |
| 374 |
</li><li><b>min.rc</b> - Run the minimum set of tests needed for 100% coverage. |
| 375 |
</li><li><b>quick.rc</b> - Run all tests required for 100% coverage tests |
| 376 |
using -Os and <a href="compile.html#debug">-DSQLITE_DEBUG</a>. |
| 377 |
</li><li><b>test.rc</b> - Run the same tests as in fast.rc but without |
| 378 |
compiler optimization and enabling options like |
| 379 |
<a href="compile.html#debug">-DSQLITE_DEBUG</a> and |
| 380 |
-DSQLITE_ENABLE_EXPENSIVE_ASSERT. |
| 381 |
</li><li><b>test-ex.rc</b> - long-running soak tests. |
| 382 |
</li></ul> |
| 383 |
|
| 384 |
<p>The TH3 repository also includes the "multitest.tcl" script, another |
| 385 |
TCL script used to automate TH3 testing on workstations. Multitest.tcl |
| 386 |
automatically compiles SQLite, then |
| 387 |
runs ./th3make repeatedly with a variety of alignments, and captures |
| 388 |
the output in a succinct summary screen. A typical multitest.tcl run |
| 389 |
generates output that looks like this: |
| 390 |
|
| 391 |
</p><div class="codeblock"><pre>./multitest.tcl -q --jobs 3 |
| 392 |
start-time: 2018-05-19 03:17:12 UTC |
| 393 |
file mkdir sqlite3bld |
| 394 |
cd sqlite3bld |
| 395 |
exec sh /ramdisk/sqlite/configure |
| 396 |
file copy -force config.h ../config.h |
| 397 |
exec make clean sqlite3.c |
| 398 |
file rename sqlite3.c ../sqlite3.c |
| 399 |
file rename sqlite3.h ../sqlite3.h |
| 400 |
exec make clean sqlite3.c OPTS=-DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1 |
| 401 |
file rename sqlite3.c ../sqlite3udl.c |
| 402 |
exec make clean sqlite3.c OPTS=-DSQLITE_SMALL_STACK=1 |
| 403 |
file rename sqlite3.c ../sqlite3ss.c |
| 404 |
cd .. |
| 405 |
******************************************************************************* |
| 406 |
t01: cov.rc.................................................... Ok (00:03:42) |
| 407 |
t02: cov.rc ++STAT4 ++DESERIALIZE -D_HAVE_SQLITE_CONFIG_H...... Ok (00:04:45) |
| 408 |
t03: vfs-cov.rc................................................ Ok (00:03:59) |
| 409 |
t04: demo.rc................................................... Ok (00:00:05) |
| 410 |
t07: test.rc ../th3private/*.test.............................. Ok (00:00:21) |
| 411 |
t08: test.rc ../th3private/*.test ++STAT4...................... Ok (00:01:41) |
| 412 |
t05: quick.rc.................................................. Ok (00:04:26) |
| 413 |
t09: quick.rc ~TEST_REALLOC_STRESS -funsigned-char............. Ok (00:05:39) |
| 414 |
t10: quick.rc ~THREADSAFE=0 -DLONGDOUBLE_TYPE=double........... Ok (00:03:24) |
| 415 |
t06: quick.rc extensions.rc -D_HAVE_SQLITE_CONFIG_H............ Ok (00:09:03) |
| 416 |
t11: quick.rc sqlite3ss.c ~MAX_ATTACHED=125.................... Ok (00:04:39) |
| 417 |
t12: quick.rc ~BYTEORDER=0 ++RTREE............................. Ok (00:07:28) |
| 418 |
t13: quick.rc ~DISABLE_INTRINSIC ++RTREE....................... Ok (00:07:31) |
| 419 |
t16: quick.rc ~TRACE_SIZE_LIMIT=15 cov1/main16.test............ Ok (00:00:22) |
| 420 |
t14: quick.rc ~DIRECT_OVERFLOW_READ -fsigned-char.............. Ok (00:04:35) |
| 421 |
t15: quick.rc ~UNTESTABLE ~EXTRA_IFNULLROW..................... Ok (00:01:44) |
| 422 |
t17: quick.rc ~MAX_MMAP_SIZE=0................................. Ok (00:04:46) |
| 423 |
t18: quick.rc ++NULL_TRIM ++OFFSET_SQL_FUNC.................... Ok (00:04:47) |
| 424 |
t19: quick.rc ++BATCH_ATOMIC_WRITE ++DESERIALIZE............... Ok (00:05:41) |
| 425 |
t20: lean1.rc quick.rc......................................... Ok (00:03:09) |
| 426 |
t22: test.rc alignment2.rc sqlite3udl.c........................ Ok (00:44:22) |
| 427 |
t21: test.rc alignment1.rc..................................... Ok (01:02:32) |
| 428 |
t23: memdebug1.rc extensions.rc................................ Ok (01:49:58) |
| 429 |
t25: valgrind1.rc -O3 extensions.rc............................ Ok (00:56:08) |
| 430 |
t24: memdebug2.rc extensions.rc................................ Ok (01:43:34) |
| 431 |
t27: test-ex1.rc............................................... Ok (00:45:00) |
| 432 |
t26: valgrind2.rc -O3 extensions.rc............................ Ok (01:02:52) |
| 433 |
t29: test-ex3.rc............................................... Ok (00:31:48) |
| 434 |
t28: test-ex2.rc............................................... Ok (01:12:03) |
| 435 |
t30: test-ex4.rc............................................... Ok (01:09:47) |
| 436 |
t32: test.rc alignment4.rc -m32 CC=clang....................... Ok (00:48:31) |
| 437 |
t31: test.rc alignment3.rc sqlite3udl.c........................ Ok (01:22:29) |
| 438 |
t34: test.rc alignment6.rc..................................... Ok (00:35:31) |
| 439 |
t33: test.rc alignment5.rc extensions.rc....................... Ok (00:59:33) |
| 440 |
t35: test.rc alignment7.rc..................................... Ok (00:44:10) |
| 441 |
t40: fast.rc alignment2.rc sqlite3udl.c........................ Ok (00:15:46) |
| 442 |
t39: fast.rc alignment1.rc extensions.rc -m32.................. Ok (00:33:19) |
| 443 |
t36: test.rc ~MUTATION_TEST.................................... Ok (00:35:45) |
| 444 |
t42: fast.rc alignment4.rc..................................... Ok (00:13:03) |
| 445 |
t43: fast.rc alignment5.rc..................................... Ok (00:13:32) |
| 446 |
t44: fast.rc alignment6.rc..................................... Ok (00:11:41) |
| 447 |
t41: fast.rc alignment3.rc sqlite3udl.c........................ Ok (00:26:31) |
| 448 |
t45: fast.rc alignment7.rc..................................... Ok (00:12:57) |
| 449 |
t46: fast.rc -fsanitize=undefined.............................. Ok (00:38:18) |
| 450 |
******************************************************************************* |
| 451 |
0 failures on 44 th3makes and 198583082 tests in (07:16:01) 3 cores on bella |
| 452 |
SQLite 3.24.0 2018-05-18 17:58:33 c6071ac99cfa4b6272ac4d739fc61a85acb544f6c1c2a |
| 453 |
</pre></div> |
| 454 |
|
| 455 |
<p>As can be seen above, a single run |
| 456 |
of multitest.tcl invokes th3make dozens of times and takes between 12 and 24 |
| 457 |
CPU hours. The middle section of the output shows the arguments to each |
| 458 |
individual th3make run and the result and elapse time for that th3make. |
| 459 |
All build products and output for the separate th3make runs are |
| 460 |
captured in subdirectories for post-test analysis. |
| 461 |
The two-line summary at the bottom shows the total number of errors and tests |
| 462 |
over all th3make runs and the total elapsed time, together with the |
| 463 |
<a href="c3ref/c_scm_branch.html">SQLITE_SOURCE_ID</a> information for the version of SQLite that was |
| 464 |
tested. This summary information is recorded in the |
| 465 |
<a href="https://sqlite.org/checklists">release |
| 466 |
checklist</a> during final testing. |
| 467 |
|
| 468 |
</p><p>Abbreviations are applied in the multitest.tcl output so that |
| 469 |
each th3make invocation will fit on a single 80-column output line. |
| 470 |
The initial "th3make" verb is omitted. |
| 471 |
"~" is shorthand for "-DSQLITE_" and "++" is stands for |
| 472 |
"-DSQLITE_ENABLE". Hence, multitest.tcl output line |
| 473 |
|
| 474 |
</p><div class="codeblock"><pre>quick.rc ~DISABLE_INTRINSIC ++RTREE |
| 475 |
</pre></div> |
| 476 |
|
| 477 |
<p>Really means |
| 478 |
|
| 479 |
</p><div class="codeblock"><pre>th3make quick.rc -DSQLITE_DISABLE_INTRINSIC -DSQLITE_ENABLE_RTREE |
| 480 |
</pre></div> |
| 481 |
|
| 482 |
<h1 id="test_coverage"><span>4. </span>Test Coverage</h1> |
| 483 |
|
| 484 |
<p>Using one particular subset of the available TH3 test modules (the "cov1" |
| 485 |
tests) SQLite obtained |
| 486 |
<a href="testing.html#coverage">100% branch test coverage</a> and 100% <a href="testing.html#mcdc">MC/DC</a> as measured |
| 487 |
by <a href="http://gcc.gnu.org/onlinedocs/gcc/Gcov.html">gcov</a> |
| 488 |
on Linux x86 and x86_64 hardware. All releases of SQLite since |
| 489 |
<a href="releaselog/3_6_17.html">version 3.6.17</a> (2009-08-10) have been tested to this standard. |
| 490 |
The SQLite developers |
| 491 |
are committed to maintaining 100% branch coverage and MC/DC for all |
| 492 |
future releases of SQLite.</p> |
| 493 |
|
| 494 |
<p>The cov1 test set used to obtain 100% branch test coverage are only a |
| 495 |
subset of the tests currently implemented using TH3. New test modules are |
| 496 |
added on a regular basis.</p> |
| 497 |
|
| 498 |
<a name="muttest"></a> |
| 499 |
|
| 500 |
<h1 id="mutation_testing"><span>5. </span>Mutation Testing</h1> |
| 501 |
|
| 502 |
<p>The TH3 source tree contains a script named |
| 503 |
"mutation-test.tcl" that automates the process of |
| 504 |
<a href="testing.html#mutationtests">mutation testing</a>. |
| 505 |
|
| 506 |
</p><p>The mutation-test.tcl script takes care of all of the details for |
| 507 |
running a mutation test: |
| 508 |
|
| 509 |
</p><ol> |
| 510 |
<li> The script compiles the TH3 test harness into machine code ("th3.o") if |
| 511 |
necessary. |
| 512 |
</li><li> The script compiles the sqlite3.c source file into assembly language |
| 513 |
("sqlite3.s") if necessary. |
| 514 |
</li><li> The script loops through instructions in the assembly language file |
| 515 |
to locate branch operations. |
| 516 |
<ol type="a"> |
| 517 |
<li>The script makes a copy of the original sqlite3.s file. |
| 518 |
</li><li>The copy is edited to change the branch instruction into either |
| 519 |
a no-op or an unconditional jump. |
| 520 |
</li><li>The copy of sqlite3.s is assembled into sqlite3.o then linked |
| 521 |
against th3.o to generate the "th3" executable. |
| 522 |
</li><li>The "th3" binary is run and the output checked for errors. |
| 523 |
</li></ol> |
| 524 |
</li><li> The script shows progress for each cycle of the previous step then |
| 525 |
displays a summary of "survivors" at the end. A "survivor" is a |
| 526 |
mutation that was not detected by TH3. |
| 527 |
</li></ol> |
| 528 |
|
| 529 |
<p>Mutation testing can be slow, since each test can take up to 5 |
| 530 |
minutes on a fast workstation, and there are two tests for each |
| 531 |
branch instruction, and over 20,000 branch instructions. Efforts are |
| 532 |
made to expedite operation. For example, TH3 is compiled in such a |
| 533 |
way that it exits as soon as it finds the first error, and as many |
| 534 |
of the mutations are easily detected, many cycles happen in only |
| 535 |
a few seconds. Nevertheless, the mutation-test.tcl script includes |
| 536 |
command-line options to limit the range of code lines tested so that |
| 537 |
mutation testing only needs to be performed on blocks of code that |
| 538 |
have recently changed. |
| 539 |
|
| 540 |
</p><h1 id="th3_license"><span>6. </span>TH3 License</h1> |
| 541 |
|
| 542 |
<p>SQLite itself is in the <a href="copyright.html">public domain</a> and |
| 543 |
can be used for any purpose. But TH3 is proprietary and requires a license. |
| 544 |
</p> |
| 545 |
|
| 546 |
<p>Even though open-source users do not have direct access to TH3, all |
| 547 |
users of SQLite benefit from TH3 indirectly since each version of SQLite is |
| 548 |
validated running TH3 on multiple platforms (Linux, Windows, WinRT, Mac, |
| 549 |
OpenBSD) prior to release. So anyone using an official release |
| 550 |
of SQLite can deploy their application with the confidence of knowing that |
| 551 |
it has been tested using TH3. They simply cannot rerun those tests |
| 552 |
themselves without purchasing a TH3 license.</p> |
| 553 |
<p align="center"><small><i>This page was last updated on 2026-04-21 10:28:59Z </small></i></p> |
| 554 |
|