| 1 |
<!-- doc/src/sgml/regress.sgml --> |
| 2 |
|
| 3 |
<chapter id="regress"> |
| 4 |
<title>Regression Tests</title> |
| 5 |
|
| 6 |
<indexterm zone="regress"> |
| 7 |
<primary>regression tests</primary> |
| 8 |
</indexterm> |
| 9 |
|
| 10 |
<indexterm zone="regress"> |
| 11 |
<primary>test</primary> |
| 12 |
</indexterm> |
| 13 |
|
| 14 |
<para> |
| 15 |
The regression tests are a comprehensive set of tests for the SQL |
| 16 |
implementation in <productname>PostgreSQL</productname>. They test |
| 17 |
standard SQL operations as well as the extended capabilities of |
| 18 |
<productname>PostgreSQL</productname>. |
| 19 |
</para> |
| 20 |
|
| 21 |
<sect1 id="regress-run"> |
| 22 |
<title>Running the Tests</title> |
| 23 |
|
| 24 |
<para> |
| 25 |
The regression tests can be run against an already installed and |
| 26 |
running server, or using a temporary installation within the build |
| 27 |
tree. Furthermore, there is a <quote>parallel</quote> and a |
| 28 |
<quote>sequential</quote> mode for running the tests. The |
| 29 |
sequential method runs each test script alone, while the |
| 30 |
parallel method starts up multiple server processes to run groups |
| 31 |
of tests in parallel. Parallel testing adds confidence that |
| 32 |
interprocess communication and locking are working correctly. |
| 33 |
Some tests may run sequentially even in the <quote>parallel</quote> |
| 34 |
mode in case this is required by the test. |
| 35 |
</para> |
| 36 |
|
| 37 |
<sect2 id="regress-run-temp-inst"> |
| 38 |
<title>Running the Tests Against a Temporary Installation</title> |
| 39 |
|
| 40 |
<para> |
| 41 |
To run the parallel regression tests after building but before installation, |
| 42 |
type: |
| 43 |
<screen> |
| 44 |
make check |
| 45 |
</screen> |
| 46 |
in the top-level directory. (Or you can change to |
| 47 |
<filename>src/test/regress</filename> and run the command there.) |
| 48 |
Tests which are run in parallel are prefixed with <quote>+</quote>, and |
| 49 |
tests which run sequentially are prefixed with <quote>-</quote>. |
| 50 |
At the end you should see something like: |
| 51 |
<screen> |
| 52 |
<computeroutput> |
| 53 |
# All 213 tests passed. |
| 54 |
</computeroutput> |
| 55 |
</screen> |
| 56 |
or otherwise a note about which tests failed. See <xref |
| 57 |
linkend="regress-evaluation"/> below before assuming that a |
| 58 |
<quote>failure</quote> represents a serious problem. |
| 59 |
</para> |
| 60 |
|
| 61 |
<para> |
| 62 |
Because this test method runs a temporary server, it will not work |
| 63 |
if you did the build as the root user, since the server will not start as |
| 64 |
root. Recommended procedure is not to do the build as root, or else to |
| 65 |
perform testing after completing the installation. |
| 66 |
</para> |
| 67 |
|
| 68 |
<para> |
| 69 |
If you have configured <productname>PostgreSQL</productname> to install |
| 70 |
into a location where an older <productname>PostgreSQL</productname> |
| 71 |
installation already exists, and you perform <literal>make check</literal> |
| 72 |
before installing the new version, you might find that the tests fail |
| 73 |
because the new programs try to use the already-installed shared |
| 74 |
libraries. (Typical symptoms are complaints about undefined symbols.) |
| 75 |
If you wish to run the tests before overwriting the old installation, |
| 76 |
you'll need to build with <literal>configure --disable-rpath</literal>. |
| 77 |
It is not recommended that you use this option for the final installation, |
| 78 |
however. |
| 79 |
</para> |
| 80 |
|
| 81 |
<para> |
| 82 |
The parallel regression test starts quite a few processes under your |
| 83 |
user ID. Presently, the maximum concurrency is twenty parallel test |
| 84 |
scripts, which means forty processes: there's a server process and a |
| 85 |
<application>psql</application> process for each test script. |
| 86 |
So if your system enforces a per-user limit on the number of processes, |
| 87 |
make sure this limit is at least fifty or so, else you might get |
| 88 |
random-seeming failures in the parallel test. If you are not in |
| 89 |
a position to raise the limit, you can cut down the degree of parallelism |
| 90 |
by setting the <literal>MAX_CONNECTIONS</literal> parameter. For example: |
| 91 |
<screen> |
| 92 |
make MAX_CONNECTIONS=10 check |
| 93 |
</screen> |
| 94 |
runs no more than ten tests concurrently. |
| 95 |
</para> |
| 96 |
</sect2> |
| 97 |
|
| 98 |
<sect2 id="regress-run-existing-inst"> |
| 99 |
<title>Running the Tests Against an Existing Installation</title> |
| 100 |
|
| 101 |
<para> |
| 102 |
To run the tests after installation (see <xref linkend="installation"/>), |
| 103 |
initialize a data directory and start the |
| 104 |
server as explained in <xref linkend="runtime"/>, then type: |
| 105 |
<screen> |
| 106 |
make installcheck |
| 107 |
</screen> |
| 108 |
or for a parallel test: |
| 109 |
<screen> |
| 110 |
make installcheck-parallel |
| 111 |
</screen> |
| 112 |
The tests will expect to contact the server at the local host and the |
| 113 |
default port number, unless directed otherwise by <envar>PGHOST</envar> and |
| 114 |
<envar>PGPORT</envar> environment variables. The tests will be run in a |
| 115 |
database named <literal>regression</literal>; any existing database by this name |
| 116 |
will be dropped. |
| 117 |
</para> |
| 118 |
|
| 119 |
<para> |
| 120 |
The tests will also transiently create some cluster-wide objects, such as |
| 121 |
roles, tablespaces, and subscriptions. These objects will have names |
| 122 |
beginning with <literal>regress_</literal>. Beware of |
| 123 |
using <literal>installcheck</literal> mode with an installation that has |
| 124 |
any actual global objects named that way. |
| 125 |
</para> |
| 126 |
</sect2> |
| 127 |
|
| 128 |
<sect2 id="regress-additional"> |
| 129 |
<title>Additional Test Suites</title> |
| 130 |
|
| 131 |
<para> |
| 132 |
The <literal>make check</literal> and <literal>make installcheck</literal> commands |
| 133 |
run only the <quote>core</quote> regression tests, which test built-in |
| 134 |
functionality of the <productname>PostgreSQL</productname> server. The source |
| 135 |
distribution contains many additional test suites, most of them having |
| 136 |
to do with add-on functionality such as optional procedural languages. |
| 137 |
</para> |
| 138 |
|
| 139 |
<para> |
| 140 |
To run all test suites applicable to the modules that have been selected |
| 141 |
to be built, including the core tests, type one of these commands at the |
| 142 |
top of the build tree: |
| 143 |
<screen> |
| 144 |
make check-world |
| 145 |
make installcheck-world |
| 146 |
</screen> |
| 147 |
These commands run the tests using temporary servers or an |
| 148 |
already-installed server, respectively, just as previously explained |
| 149 |
for <literal>make check</literal> and <literal>make installcheck</literal>. Other |
| 150 |
considerations are the same as previously explained for each method. |
| 151 |
Note that <literal>make check-world</literal> builds a separate instance |
| 152 |
(temporary data directory) for each tested module, so it requires more |
| 153 |
time and disk space than <literal>make installcheck-world</literal>. |
| 154 |
</para> |
| 155 |
|
| 156 |
<para> |
| 157 |
On a modern machine with multiple CPU cores and no tight operating-system |
| 158 |
limits, you can make things go substantially faster with parallelism. |
| 159 |
The recipe that most PostgreSQL developers actually use for running all |
| 160 |
tests is something like |
| 161 |
<screen> |
| 162 |
make check-world -j8 >/dev/null |
| 163 |
</screen> |
| 164 |
with a <option>-j</option> limit near to or a bit more than the number |
| 165 |
of available cores. Discarding <systemitem>stdout</systemitem> |
| 166 |
eliminates chatter that's not interesting when you just want to verify |
| 167 |
success. (In case of failure, the <systemitem>stderr</systemitem> |
| 168 |
messages are usually enough to determine where to look closer.) |
| 169 |
</para> |
| 170 |
|
| 171 |
<para> |
| 172 |
Alternatively, you can run individual test suites by typing |
| 173 |
<literal>make check</literal> or <literal>make installcheck</literal> in the appropriate |
| 174 |
subdirectory of the build tree. Keep in mind that <literal>make |
| 175 |
installcheck</literal> assumes you've installed the relevant module(s), not |
| 176 |
only the core server. |
| 177 |
</para> |
| 178 |
|
| 179 |
<para> |
| 180 |
The additional tests that can be invoked this way include: |
| 181 |
</para> |
| 182 |
|
| 183 |
<itemizedlist> |
| 184 |
<listitem> |
| 185 |
<para> |
| 186 |
Regression tests for optional procedural languages. |
| 187 |
These are located under <filename>src/pl</filename>. |
| 188 |
</para> |
| 189 |
</listitem> |
| 190 |
<listitem> |
| 191 |
<para> |
| 192 |
Regression tests for <filename>contrib</filename> modules, |
| 193 |
located under <filename>contrib</filename>. |
| 194 |
Not all <filename>contrib</filename> modules have tests. |
| 195 |
</para> |
| 196 |
</listitem> |
| 197 |
<listitem> |
| 198 |
<para> |
| 199 |
Regression tests for the interface libraries, |
| 200 |
located in <filename>src/interfaces/libpq/test</filename> and |
| 201 |
<filename>src/interfaces/ecpg/test</filename>. |
| 202 |
</para> |
| 203 |
</listitem> |
| 204 |
<listitem> |
| 205 |
<para> |
| 206 |
Tests for core-supported authentication methods, |
| 207 |
located in <filename>src/test/authentication</filename>. |
| 208 |
(See below for additional authentication-related tests.) |
| 209 |
</para> |
| 210 |
</listitem> |
| 211 |
<listitem> |
| 212 |
<para> |
| 213 |
Tests stressing behavior of concurrent sessions, |
| 214 |
located in <filename>src/test/isolation</filename>. |
| 215 |
</para> |
| 216 |
</listitem> |
| 217 |
<listitem> |
| 218 |
<para> |
| 219 |
Tests for crash recovery and physical replication, |
| 220 |
located in <filename>src/test/recovery</filename>. |
| 221 |
</para> |
| 222 |
</listitem> |
| 223 |
<listitem> |
| 224 |
<para> |
| 225 |
Tests for logical replication, |
| 226 |
located in <filename>src/test/subscription</filename>. |
| 227 |
</para> |
| 228 |
</listitem> |
| 229 |
<listitem> |
| 230 |
<para> |
| 231 |
Tests of client programs, located under <filename>src/bin</filename>. |
| 232 |
</para> |
| 233 |
</listitem> |
| 234 |
</itemizedlist> |
| 235 |
|
| 236 |
<para> |
| 237 |
When using <literal>installcheck</literal> mode, these tests will create |
| 238 |
and destroy test databases whose names |
| 239 |
include <literal>regression</literal>, for |
| 240 |
example <literal>pl_regression</literal> |
| 241 |
or <literal>contrib_regression</literal>. Beware of |
| 242 |
using <literal>installcheck</literal> mode with an installation that has |
| 243 |
any non-test databases named that way. |
| 244 |
</para> |
| 245 |
|
| 246 |
<para> |
| 247 |
Some of these auxiliary test suites use the TAP infrastructure explained |
| 248 |
in <xref linkend="regress-tap"/>. |
| 249 |
The TAP-based tests are run only when PostgreSQL was configured with the |
| 250 |
option <option>--enable-tap-tests</option>. This is recommended for |
| 251 |
development, but can be omitted if there is no suitable Perl installation. |
| 252 |
</para> |
| 253 |
|
| 254 |
<para> |
| 255 |
Some test suites are not run by default, either because they are not secure |
| 256 |
to run on a multiuser system, because they require special software or |
| 257 |
because they are resource intensive. You can decide which test suites to |
| 258 |
run additionally by setting the <command>make</command> or environment |
| 259 |
variable <varname>PG_TEST_EXTRA</varname> to a whitespace-separated list, |
| 260 |
for example: |
| 261 |
<programlisting> |
| 262 |
make check-world PG_TEST_EXTRA='kerberos ldap ssl load_balance libpq_encryption' |
| 263 |
</programlisting> |
| 264 |
The following values are currently supported: |
| 265 |
<variablelist> |
| 266 |
<varlistentry> |
| 267 |
<term><literal>kerberos</literal></term> |
| 268 |
<listitem> |
| 269 |
<para> |
| 270 |
Runs the test suite under <filename>src/test/kerberos</filename>. This |
| 271 |
requires an MIT Kerberos installation and opens TCP/IP listen sockets. |
| 272 |
</para> |
| 273 |
</listitem> |
| 274 |
</varlistentry> |
| 275 |
|
| 276 |
<varlistentry> |
| 277 |
<term><literal>ldap</literal></term> |
| 278 |
<listitem> |
| 279 |
<para> |
| 280 |
Runs the test suite under <filename>src/test/ldap</filename>. This |
| 281 |
requires an <productname>OpenLDAP</productname> installation and opens |
| 282 |
TCP/IP listen sockets. |
| 283 |
</para> |
| 284 |
</listitem> |
| 285 |
</varlistentry> |
| 286 |
|
| 287 |
<varlistentry> |
| 288 |
<term><literal>libpq_encryption</literal></term> |
| 289 |
<listitem> |
| 290 |
<para> |
| 291 |
Runs the test <filename>src/interfaces/libpq/t/005_negotiate_encryption.pl</filename>. |
| 292 |
This opens TCP/IP listen sockets. If <varname>PG_TEST_EXTRA</varname> |
| 293 |
also includes <literal>kerberos</literal>, additional tests that require |
| 294 |
an MIT Kerberos installation are enabled. |
| 295 |
</para> |
| 296 |
</listitem> |
| 297 |
</varlistentry> |
| 298 |
|
| 299 |
<varlistentry> |
| 300 |
<term><literal>load_balance</literal></term> |
| 301 |
<listitem> |
| 302 |
<para> |
| 303 |
Runs the test <filename>src/interfaces/libpq/t/004_load_balance_dns.pl</filename>. |
| 304 |
This requires editing the system <filename>hosts</filename> file and |
| 305 |
opens TCP/IP listen sockets. |
| 306 |
</para> |
| 307 |
</listitem> |
| 308 |
</varlistentry> |
| 309 |
|
| 310 |
<varlistentry> |
| 311 |
<term><literal>oauth</literal></term> |
| 312 |
<listitem> |
| 313 |
<para> |
| 314 |
Runs the test suite under <filename>src/test/modules/oauth_validator</filename>. |
| 315 |
This opens TCP/IP listen sockets for a test server running HTTPS. |
| 316 |
</para> |
| 317 |
</listitem> |
| 318 |
</varlistentry> |
| 319 |
|
| 320 |
<varlistentry> |
| 321 |
<term><literal>regress_dump_restore</literal></term> |
| 322 |
<listitem> |
| 323 |
<para> |
| 324 |
Runs an additional test suite in |
| 325 |
<filename>src/bin/pg_upgrade/t/002_pg_upgrade.pl</filename> which |
| 326 |
cycles the regression database through <command>pg_dump</command>/ |
| 327 |
<command>pg_restore</command>. Not enabled by default because it |
| 328 |
is resource intensive. |
| 329 |
</para> |
| 330 |
</listitem> |
| 331 |
</varlistentry> |
| 332 |
|
| 333 |
<varlistentry> |
| 334 |
<term><literal>sepgsql</literal></term> |
| 335 |
<listitem> |
| 336 |
<para> |
| 337 |
Runs the test suite under <filename>contrib/sepgsql</filename>. This |
| 338 |
requires an SELinux environment that is set up in a specific way; see |
| 339 |
<xref linkend="sepgsql-regression"/>. |
| 340 |
</para> |
| 341 |
</listitem> |
| 342 |
</varlistentry> |
| 343 |
|
| 344 |
<varlistentry> |
| 345 |
<term><literal>ssl</literal></term> |
| 346 |
<listitem> |
| 347 |
<para> |
| 348 |
Runs the test suite under <filename>src/test/ssl</filename>. This opens TCP/IP listen sockets. |
| 349 |
</para> |
| 350 |
</listitem> |
| 351 |
</varlistentry> |
| 352 |
|
| 353 |
<varlistentry> |
| 354 |
<term><literal>wal_consistency_checking</literal></term> |
| 355 |
<listitem> |
| 356 |
<para> |
| 357 |
Uses <literal>wal_consistency_checking=all</literal> while running |
| 358 |
certain tests under <filename>src/test/recovery</filename>. Not |
| 359 |
enabled by default because it is resource intensive. |
| 360 |
</para> |
| 361 |
</listitem> |
| 362 |
</varlistentry> |
| 363 |
|
| 364 |
<varlistentry> |
| 365 |
<term><literal>xid_wraparound</literal></term> |
| 366 |
<listitem> |
| 367 |
<para> |
| 368 |
Runs the test suite under <filename>src/test/modules/xid_wraparound</filename>. |
| 369 |
Not enabled by default because it is resource intensive. |
| 370 |
</para> |
| 371 |
</listitem> |
| 372 |
</varlistentry> |
| 373 |
</variablelist> |
| 374 |
|
| 375 |
Tests for features that are not supported by the current build |
| 376 |
configuration are not run even if they are mentioned in |
| 377 |
<varname>PG_TEST_EXTRA</varname>. |
| 378 |
</para> |
| 379 |
|
| 380 |
<para> |
| 381 |
In addition, there are tests in <filename>src/test/modules</filename> |
| 382 |
which will be run by <literal>make check-world</literal> but not |
| 383 |
by <literal>make installcheck-world</literal>. This is because they |
| 384 |
install non-production extensions or have other side-effects that are |
| 385 |
considered undesirable for a production installation. You can |
| 386 |
use <literal>make install</literal> and <literal>make |
| 387 |
installcheck</literal> in one of those subdirectories if you wish, |
| 388 |
but it's not recommended to do so with a non-test server. |
| 389 |
</para> |
| 390 |
</sect2> |
| 391 |
|
| 392 |
<sect2 id="regress-run-locale"> |
| 393 |
<title>Locale and Encoding</title> |
| 394 |
|
| 395 |
<para> |
| 396 |
By default, tests using a temporary installation use the |
| 397 |
locale defined in the current environment and the corresponding |
| 398 |
database encoding as determined by <command>initdb</command>. It |
| 399 |
can be useful to test different locales by setting the appropriate |
| 400 |
environment variables, for example: |
| 401 |
<screen> |
| 402 |
make check LANG=C |
| 403 |
make check LC_COLLATE=en_US.utf8 LC_CTYPE=fr_CA.utf8 |
| 404 |
</screen> |
| 405 |
For implementation reasons, setting <envar>LC_ALL</envar> does not |
| 406 |
work for this purpose; all the other locale-related environment |
| 407 |
variables do work. |
| 408 |
</para> |
| 409 |
|
| 410 |
<para> |
| 411 |
When testing against an existing installation, the locale is |
| 412 |
determined by the existing database cluster and cannot be set |
| 413 |
separately for the test run. |
| 414 |
</para> |
| 415 |
|
| 416 |
<para> |
| 417 |
You can also choose the database encoding explicitly by setting |
| 418 |
the variable <envar>ENCODING</envar>, for example: |
| 419 |
<screen> |
| 420 |
make check LANG=C ENCODING=EUC_JP |
| 421 |
</screen> |
| 422 |
Setting the database encoding this way typically only makes sense |
| 423 |
if the locale is C; otherwise the encoding is chosen automatically |
| 424 |
from the locale, and specifying an encoding that does not match |
| 425 |
the locale will result in an error. |
| 426 |
</para> |
| 427 |
|
| 428 |
<para> |
| 429 |
The database encoding can be set for tests against either a temporary or |
| 430 |
an existing installation, though in the latter case it must be |
| 431 |
compatible with the installation's locale. |
| 432 |
</para> |
| 433 |
</sect2> |
| 434 |
|
| 435 |
<sect2 id="regress-run-custom-settings"> |
| 436 |
<title>Custom Server Settings</title> |
| 437 |
|
| 438 |
<para> |
| 439 |
There are several ways to use custom server settings when running a test |
| 440 |
suite. This can be useful to enable additional logging, adjust resource |
| 441 |
limits, or enable extra run-time checks such as <xref |
| 442 |
linkend="guc-debug-discard-caches"/>. But note that not all tests can be |
| 443 |
expected to pass cleanly with arbitrary settings. |
| 444 |
</para> |
| 445 |
|
| 446 |
<para> |
| 447 |
Extra options can be passed to the various <command>initdb</command> |
| 448 |
commands that are run internally during test setup using the environment |
| 449 |
variable <envar>PG_TEST_INITDB_EXTRA_OPTS</envar>. For example, to run a |
| 450 |
test with checksums enabled and a custom WAL segment size and |
| 451 |
<varname>work_mem</varname> setting, use: |
| 452 |
<screen> |
| 453 |
make check PG_TEST_INITDB_EXTRA_OPTS='-k --wal-segsize=4 -c work_mem=50MB' |
| 454 |
</screen> |
| 455 |
</para> |
| 456 |
|
| 457 |
<para> |
| 458 |
For the core regression test suite and other tests driven by |
| 459 |
<command>pg_regress</command>, custom run-time server settings can also be |
| 460 |
set in the <varname>PGOPTIONS</varname> environment variable (for settings |
| 461 |
that allow this), for example: |
| 462 |
<screen> |
| 463 |
make check PGOPTIONS="-c debug_parallel_query=regress -c work_mem=50MB" |
| 464 |
</screen> |
| 465 |
(This makes use of functionality provided by libpq; see <xref |
| 466 |
linkend="libpq-connect-options"/> for details.) |
| 467 |
</para> |
| 468 |
|
| 469 |
<para> |
| 470 |
When running against a temporary installation, custom settings can also be |
| 471 |
set by supplying a pre-written <filename>postgresql.conf</filename>: |
| 472 |
<screen> |
| 473 |
echo 'log_checkpoints = on' > test_postgresql.conf |
| 474 |
echo 'work_mem = 50MB' >> test_postgresql.conf |
| 475 |
make check EXTRA_REGRESS_OPTS="--temp-config=test_postgresql.conf" |
| 476 |
</screen> |
| 477 |
</para> |
| 478 |
|
| 479 |
</sect2> |
| 480 |
|
| 481 |
<sect2 id="regress-run-extra-tests"> |
| 482 |
<title>Extra Tests</title> |
| 483 |
|
| 484 |
<para> |
| 485 |
The core regression test suite contains a few test files that are not |
| 486 |
run by default, because they might be platform-dependent or take a |
| 487 |
very long time to run. You can run these or other extra test |
| 488 |
files by setting the variable <envar>EXTRA_TESTS</envar>. For |
| 489 |
example, to run the <literal>numeric_big</literal> test: |
| 490 |
<screen> |
| 491 |
make check EXTRA_TESTS=numeric_big |
| 492 |
</screen> |
| 493 |
</para> |
| 494 |
</sect2> |
| 495 |
</sect1> |
| 496 |
|
| 497 |
<sect1 id="regress-evaluation"> |
| 498 |
<title>Test Evaluation</title> |
| 499 |
|
| 500 |
<para> |
| 501 |
Some properly installed and fully functional |
| 502 |
<productname>PostgreSQL</productname> installations can |
| 503 |
<quote>fail</quote> some of these regression tests due to |
| 504 |
platform-specific artifacts such as varying floating-point representation |
| 505 |
and message wording. The tests are currently evaluated using a simple |
| 506 |
<command>diff</command> comparison against the outputs |
| 507 |
generated on a reference system, so the results are sensitive to |
| 508 |
small system differences. When a test is reported as |
| 509 |
<quote>failed</quote>, always examine the differences between |
| 510 |
expected and actual results; you might find that the |
| 511 |
differences are not significant. Nonetheless, we still strive to |
| 512 |
maintain accurate reference files across all supported platforms, |
| 513 |
so it can be expected that all tests pass. |
| 514 |
</para> |
| 515 |
|
| 516 |
<para> |
| 517 |
The actual outputs of the regression tests are in files in the |
| 518 |
<filename>src/test/regress/results</filename> directory. The test |
| 519 |
script uses <command>diff</command> to compare each output |
| 520 |
file against the reference outputs stored in the |
| 521 |
<filename>src/test/regress/expected</filename> directory. Any |
| 522 |
differences are saved for your inspection in |
| 523 |
<filename>src/test/regress/regression.diffs</filename>. |
| 524 |
(When running a test suite other than the core tests, these files |
| 525 |
of course appear in the relevant subdirectory, |
| 526 |
not <filename>src/test/regress</filename>.) |
| 527 |
</para> |
| 528 |
|
| 529 |
<para> |
| 530 |
If you don't |
| 531 |
like the <command>diff</command> options that are used by default, set the |
| 532 |
environment variable <envar>PG_REGRESS_DIFF_OPTS</envar>, for |
| 533 |
instance <literal>PG_REGRESS_DIFF_OPTS='-c'</literal>. (Or you |
| 534 |
can run <command>diff</command> yourself, if you prefer.) |
| 535 |
</para> |
| 536 |
|
| 537 |
<para> |
| 538 |
If for some reason a particular platform generates a <quote>failure</quote> |
| 539 |
for a given test, but inspection of the output convinces you that |
| 540 |
the result is valid, you can add a new comparison file to silence |
| 541 |
the failure report in future test runs. See |
| 542 |
<xref linkend="regress-variant"/> for details. |
| 543 |
</para> |
| 544 |
|
| 545 |
<sect2 id="regress-evaluation-message-differences"> |
| 546 |
<title>Error Message Differences</title> |
| 547 |
|
| 548 |
<para> |
| 549 |
Some of the regression tests involve intentional invalid input |
| 550 |
values. Error messages can come from either the |
| 551 |
<productname>PostgreSQL</productname> code or from the host |
| 552 |
platform system routines. In the latter case, the messages can |
| 553 |
vary between platforms, but should reflect similar |
| 554 |
information. These differences in messages will result in a |
| 555 |
<quote>failed</quote> regression test that can be validated by |
| 556 |
inspection. |
| 557 |
</para> |
| 558 |
</sect2> |
| 559 |
|
| 560 |
<sect2 id="regress-evaluation-locale-differences"> |
| 561 |
<title>Locale Differences</title> |
| 562 |
|
| 563 |
<para> |
| 564 |
If you run the tests against a server that was |
| 565 |
initialized with a collation-order locale other than C, then |
| 566 |
there might be differences due to sort order and subsequent |
| 567 |
failures. The regression test suite is set up to handle this |
| 568 |
problem by providing alternate result files that together are |
| 569 |
known to handle a large number of locales. |
| 570 |
</para> |
| 571 |
|
| 572 |
<para> |
| 573 |
To run the tests in a different locale when using the |
| 574 |
temporary-installation method, pass the appropriate |
| 575 |
locale-related environment variables on |
| 576 |
the <command>make</command> command line, for example: |
| 577 |
<programlisting> |
| 578 |
make check LANG=de_DE.utf8 |
| 579 |
</programlisting> |
| 580 |
(The regression test driver unsets <envar>LC_ALL</envar>, so it |
| 581 |
does not work to choose the locale using that variable.) To use |
| 582 |
no locale, either unset all locale-related environment variables |
| 583 |
(or set them to <literal>C</literal>) or use the following |
| 584 |
special invocation: |
| 585 |
<programlisting> |
| 586 |
make check NO_LOCALE=1 |
| 587 |
</programlisting> |
| 588 |
When running the tests against an existing installation, the |
| 589 |
locale setup is determined by the existing installation. To |
| 590 |
change it, initialize the database cluster with a different |
| 591 |
locale by passing the appropriate options |
| 592 |
to <command>initdb</command>. |
| 593 |
</para> |
| 594 |
|
| 595 |
<para> |
| 596 |
In general, it is advisable to try to run the |
| 597 |
regression tests in the locale setup that is wanted for |
| 598 |
production use, as this will exercise the locale- and |
| 599 |
encoding-related code portions that will actually be used in |
| 600 |
production. Depending on the operating system environment, you |
| 601 |
might get failures, but then you will at least know what |
| 602 |
locale-specific behaviors to expect when running real |
| 603 |
applications. |
| 604 |
</para> |
| 605 |
</sect2> |
| 606 |
|
| 607 |
<sect2 id="regress-evaluation-date-time-differences"> |
| 608 |
<title>Date and Time Differences</title> |
| 609 |
|
| 610 |
<para> |
| 611 |
Most of the date and time results are dependent on the time zone |
| 612 |
environment. The reference files are generated for time zone |
| 613 |
<literal>America/Los_Angeles</literal>, and there will be |
| 614 |
apparent failures if the tests are not run with that time zone setting. |
| 615 |
The regression test driver sets environment variable |
| 616 |
<envar>PGTZ</envar> to <literal>America/Los_Angeles</literal>, |
| 617 |
which normally ensures proper results. |
| 618 |
</para> |
| 619 |
</sect2> |
| 620 |
|
| 621 |
<sect2 id="regress-evaluation-float-differences"> |
| 622 |
<title>Floating-Point Differences</title> |
| 623 |
|
| 624 |
<para> |
| 625 |
Some of the tests involve computing 64-bit floating-point numbers (<type>double |
| 626 |
precision</type>) from table columns. Differences in |
| 627 |
results involving mathematical functions of <type>double |
| 628 |
precision</type> columns have been observed. The <literal>float8</literal> and |
| 629 |
<literal>geometry</literal> tests are particularly prone to small differences |
| 630 |
across platforms, or even with different compiler optimization settings. |
| 631 |
Human eyeball comparison is needed to determine the real |
| 632 |
significance of these differences which are usually 10 places to |
| 633 |
the right of the decimal point. |
| 634 |
</para> |
| 635 |
|
| 636 |
<para> |
| 637 |
Some systems display minus zero as <literal>-0</literal>, while others |
| 638 |
just show <literal>0</literal>. |
| 639 |
</para> |
| 640 |
|
| 641 |
<para> |
| 642 |
Some systems signal errors from <function>pow()</function> and |
| 643 |
<function>exp()</function> differently from the mechanism |
| 644 |
expected by the current <productname>PostgreSQL</productname> |
| 645 |
code. |
| 646 |
</para> |
| 647 |
</sect2> |
| 648 |
|
| 649 |
<sect2 id="regress-evaluation-ordering-differences"> |
| 650 |
<title>Row Ordering Differences</title> |
| 651 |
|
| 652 |
<para> |
| 653 |
You might see differences in which the same rows are output in a |
| 654 |
different order than what appears in the expected file. In most cases |
| 655 |
this is not, strictly speaking, a bug. Most of the regression test |
| 656 |
scripts are not so pedantic as to use an <literal>ORDER BY</literal> for every single |
| 657 |
<literal>SELECT</literal>, and so their result row orderings are not well-defined |
| 658 |
according to the SQL specification. In practice, since we are |
| 659 |
looking at the same queries being executed on the same data by the same |
| 660 |
software, we usually get the same result ordering on all platforms, |
| 661 |
so the lack of <literal>ORDER BY</literal> is not a problem. Some queries do exhibit |
| 662 |
cross-platform ordering differences, however. When testing against an |
| 663 |
already-installed server, ordering differences can also be caused by |
| 664 |
non-C locale settings or non-default parameter settings, such as custom values |
| 665 |
of <varname>work_mem</varname> or the planner cost parameters. |
| 666 |
</para> |
| 667 |
|
| 668 |
<para> |
| 669 |
Therefore, if you see an ordering difference, it's not something to |
| 670 |
worry about, unless the query does have an <literal>ORDER BY</literal> that your |
| 671 |
result is violating. However, please report it anyway, so that we can add an |
| 672 |
<literal>ORDER BY</literal> to that particular query to eliminate the bogus |
| 673 |
<quote>failure</quote> in future releases. |
| 674 |
</para> |
| 675 |
|
| 676 |
<para> |
| 677 |
You might wonder why we don't order all the regression test queries explicitly |
| 678 |
to get rid of this issue once and for all. The reason is that that would |
| 679 |
make the regression tests less useful, not more, since they'd tend |
| 680 |
to exercise query plan types that produce ordered results to the |
| 681 |
exclusion of those that don't. |
| 682 |
</para> |
| 683 |
</sect2> |
| 684 |
|
| 685 |
<sect2 id="regress-evaluation-stack-depth"> |
| 686 |
<title>Insufficient Stack Depth</title> |
| 687 |
|
| 688 |
<para> |
| 689 |
If the <literal>errors</literal> test results in a server crash |
| 690 |
at the <literal>select infinite_recurse()</literal> command, it means that |
| 691 |
the platform's limit on process stack size is smaller than the |
| 692 |
<xref linkend="guc-max-stack-depth"/> parameter indicates. This |
| 693 |
can be fixed by running the server under a higher stack |
| 694 |
size limit (4MB is recommended with the default value of |
| 695 |
<varname>max_stack_depth</varname>). If you are unable to do that, an |
| 696 |
alternative is to reduce the value of <varname>max_stack_depth</varname>. |
| 697 |
</para> |
| 698 |
|
| 699 |
<para> |
| 700 |
On platforms supporting <function>getrlimit()</function>, the server should |
| 701 |
automatically choose a safe value of <varname>max_stack_depth</varname>; |
| 702 |
so unless you've manually overridden this setting, a failure of this |
| 703 |
kind is a reportable bug. |
| 704 |
</para> |
| 705 |
</sect2> |
| 706 |
|
| 707 |
<sect2 id="regress-evaluation-random-test"> |
| 708 |
<title>The <quote>random</quote> Test</title> |
| 709 |
|
| 710 |
<para> |
| 711 |
The <literal>random</literal> test script is intended to produce |
| 712 |
random results. In very rare cases, this causes that regression |
| 713 |
test to fail. Typing: |
| 714 |
<programlisting> |
| 715 |
diff results/random.out expected/random.out |
| 716 |
</programlisting> |
| 717 |
should produce only one or a few lines of differences. You need |
| 718 |
not worry unless the random test fails repeatedly. |
| 719 |
</para> |
| 720 |
</sect2> |
| 721 |
|
| 722 |
<sect2 id="regress-evaluation-config-params"> |
| 723 |
<title>Configuration Parameters</title> |
| 724 |
|
| 725 |
<para> |
| 726 |
When running the tests against an existing installation, some non-default |
| 727 |
parameter settings could cause the tests to fail. For example, changing |
| 728 |
parameters such as <varname>enable_seqscan</varname> or |
| 729 |
<varname>enable_indexscan</varname> could cause plan changes that would |
| 730 |
affect the results of tests that use <command>EXPLAIN</command>. |
| 731 |
</para> |
| 732 |
</sect2> |
| 733 |
</sect1> |
| 734 |
|
| 735 |
<!-- We might want to move the following section into the developer's guide. --> |
| 736 |
<sect1 id="regress-variant"> |
| 737 |
<title>Variant Comparison Files</title> |
| 738 |
|
| 739 |
<para> |
| 740 |
Since some of the tests inherently produce environment-dependent |
| 741 |
results, we have provided ways to specify alternate <quote>expected</quote> |
| 742 |
result files. Each regression test can have several comparison files |
| 743 |
showing possible results on different platforms. There are two |
| 744 |
independent mechanisms for determining which comparison file is used |
| 745 |
for each test. |
| 746 |
</para> |
| 747 |
|
| 748 |
<para> |
| 749 |
The first mechanism allows comparison files to be selected for |
| 750 |
specific platforms. There is a mapping file, |
| 751 |
<filename>src/test/regress/resultmap</filename>, that defines |
| 752 |
which comparison file to use for each platform. |
| 753 |
To eliminate bogus test <quote>failures</quote> for a particular platform, |
| 754 |
you first choose or make a variant result file, and then add a line to the |
| 755 |
<filename>resultmap</filename> file. |
| 756 |
</para> |
| 757 |
|
| 758 |
<para> |
| 759 |
Each line in the mapping file is of the form |
| 760 |
<synopsis> |
| 761 |
testname:output:platformpattern=comparisonfilename |
| 762 |
</synopsis> |
| 763 |
The test name is just the name of the particular regression test |
| 764 |
module. The output value indicates which output file to check. For the |
| 765 |
standard regression tests, this is always <literal>out</literal>. The |
| 766 |
value corresponds to the file extension of the output file. |
| 767 |
The platform pattern is a pattern in the style of the Unix |
| 768 |
tool <command>expr</command> (that is, a regular expression with an implicit |
| 769 |
<literal>^</literal> anchor at the start). It is matched against the |
| 770 |
platform name as printed by <command>config.guess</command>. |
| 771 |
The comparison file name is the base name of the substitute result |
| 772 |
comparison file. |
| 773 |
</para> |
| 774 |
|
| 775 |
<para> |
| 776 |
For example: some systems lack a working <literal>strtof</literal> function, |
| 777 |
for which our workaround causes rounding errors in the |
| 778 |
<filename>float4</filename> regression test. |
| 779 |
Therefore, we provide a variant comparison file, |
| 780 |
<filename>float4-misrounded-input.out</filename>, which includes |
| 781 |
the results to be expected on these systems. To silence the bogus |
| 782 |
<quote>failure</quote> message on <systemitem>Cygwin</systemitem> |
| 783 |
platforms, <filename>resultmap</filename> includes: |
| 784 |
<programlisting> |
| 785 |
float4:out:.*-.*-cygwin.*=float4-misrounded-input.out |
| 786 |
</programlisting> |
| 787 |
which will trigger on any machine where the output of |
| 788 |
<command>config.guess</command> matches <literal>.*-.*-cygwin.*</literal>. |
| 789 |
Other lines in <filename>resultmap</filename> select the variant comparison |
| 790 |
file for other platforms where it's appropriate. |
| 791 |
</para> |
| 792 |
|
| 793 |
<para> |
| 794 |
The second selection mechanism for variant comparison files is |
| 795 |
much more automatic: it simply uses the <quote>best match</quote> among |
| 796 |
several supplied comparison files. The regression test driver |
| 797 |
script considers both the standard comparison file for a test, |
| 798 |
<literal><replaceable>testname</replaceable>.out</literal>, and variant files named |
| 799 |
<literal><replaceable>testname</replaceable>_<replaceable>digit</replaceable>.out</literal> |
| 800 |
(where the <replaceable>digit</replaceable> is any single digit |
| 801 |
<literal>0</literal>-<literal>9</literal>). If any such file is an exact match, |
| 802 |
the test is considered to pass; otherwise, the one that generates |
| 803 |
the shortest diff is used to create the failure report. (If |
| 804 |
<filename>resultmap</filename> includes an entry for the particular |
| 805 |
test, then the base <replaceable>testname</replaceable> is the substitute |
| 806 |
name given in <filename>resultmap</filename>.) |
| 807 |
</para> |
| 808 |
|
| 809 |
<para> |
| 810 |
For example, for the <literal>char</literal> test, the comparison file |
| 811 |
<filename>char.out</filename> contains results that are expected |
| 812 |
in the <literal>C</literal> and <literal>POSIX</literal> locales, while |
| 813 |
the file <filename>char_1.out</filename> contains results sorted as |
| 814 |
they appear in many other locales. |
| 815 |
</para> |
| 816 |
|
| 817 |
<para> |
| 818 |
The best-match mechanism was devised to cope with locale-dependent |
| 819 |
results, but it can be used in any situation where the test results |
| 820 |
cannot be predicted easily from the platform name alone. A limitation of |
| 821 |
this mechanism is that the test driver cannot tell which variant is |
| 822 |
actually <quote>correct</quote> for the current environment; it will just pick |
| 823 |
the variant that seems to work best. Therefore it is safest to use this |
| 824 |
mechanism only for variant results that you are willing to consider |
| 825 |
equally valid in all contexts. |
| 826 |
</para> |
| 827 |
|
| 828 |
</sect1> |
| 829 |
|
| 830 |
<sect1 id="regress-tap"> |
| 831 |
<title>TAP Tests</title> |
| 832 |
|
| 833 |
<para> |
| 834 |
Various tests, particularly the client program tests |
| 835 |
under <filename>src/bin</filename>, use the Perl TAP tools and are run |
| 836 |
using the Perl testing program <command>prove</command>. You can pass |
| 837 |
command-line options to <command>prove</command> by setting |
| 838 |
the <command>make</command> variable <varname>PROVE_FLAGS</varname>, for example: |
| 839 |
<programlisting> |
| 840 |
make -C src/bin check PROVE_FLAGS='--timer' |
| 841 |
</programlisting> |
| 842 |
See the manual page of <command>prove</command> for more information. |
| 843 |
</para> |
| 844 |
|
| 845 |
<para> |
| 846 |
The <command>make</command> variable <varname>PROVE_TESTS</varname> |
| 847 |
can be used to define a whitespace-separated list of paths relative |
| 848 |
to the <filename>Makefile</filename> invoking <command>prove</command> |
| 849 |
to run the specified subset of tests instead of the default |
| 850 |
<filename>t/*.pl</filename>. For example: |
| 851 |
<programlisting> |
| 852 |
make check PROVE_TESTS='t/001_test1.pl t/003_test3.pl' |
| 853 |
</programlisting> |
| 854 |
</para> |
| 855 |
|
| 856 |
<para> |
| 857 |
The TAP tests require the Perl module <literal>IPC::Run</literal>. |
| 858 |
This module is available from |
| 859 |
<ulink url="https://metacpan.org/dist/IPC-Run">CPAN</ulink> |
| 860 |
or an operating system package. |
| 861 |
They also require <productname>PostgreSQL</productname> to be |
| 862 |
configured with the option <option>--enable-tap-tests</option>. |
| 863 |
</para> |
| 864 |
|
| 865 |
<para> |
| 866 |
Generically speaking, the TAP tests will test the executables in a |
| 867 |
previously-installed installation tree if you say <literal>make |
| 868 |
installcheck</literal>, or will build a new local installation tree from |
| 869 |
current sources if you say <literal>make check</literal>. In either |
| 870 |
case they will initialize a local instance (data directory) and |
| 871 |
transiently run a server in it. Some of these tests run more than one |
| 872 |
server. Thus, these tests can be fairly resource-intensive. |
| 873 |
</para> |
| 874 |
|
| 875 |
<para> |
| 876 |
It's important to realize that the TAP tests will start test server(s) |
| 877 |
even when you say <literal>make installcheck</literal>; this is unlike |
| 878 |
the traditional non-TAP testing infrastructure, which expects to use an |
| 879 |
already-running test server in that case. Some PostgreSQL |
| 880 |
subdirectories contain both traditional-style and TAP-style tests, |
| 881 |
meaning that <literal>make installcheck</literal> will produce a mix of |
| 882 |
results from temporary servers and the already-running test server. |
| 883 |
</para> |
| 884 |
|
| 885 |
<sect2 id="regress-tap-vars"> |
| 886 |
<title>Environment Variables</title> |
| 887 |
|
| 888 |
<para> |
| 889 |
Data directories are named according to the test filename, and will be |
| 890 |
retained if a test fails. If the environment variable |
| 891 |
<varname>PG_TEST_NOCLEAN</varname> is set, data directories will be |
| 892 |
retained regardless of test status. For example, retaining the data |
| 893 |
directory regardless of test results when running the |
| 894 |
<application>pg_dump</application> tests: |
| 895 |
<programlisting> |
| 896 |
PG_TEST_NOCLEAN=1 make -C src/bin/pg_dump check |
| 897 |
</programlisting> |
| 898 |
This environment variable also prevents the test's temporary directories |
| 899 |
from being removed. |
| 900 |
</para> |
| 901 |
|
| 902 |
<para> |
| 903 |
Many operations in the test suites use a 180-second timeout, which on slow |
| 904 |
hosts may lead to load-induced timeouts. Setting the environment variable |
| 905 |
<varname>PG_TEST_TIMEOUT_DEFAULT</varname> to a higher number will change |
| 906 |
the default to avoid this. |
| 907 |
</para> |
| 908 |
</sect2> |
| 909 |
|
| 910 |
</sect1> |
| 911 |
|
| 912 |
<sect1 id="regress-coverage"> |
| 913 |
<title>Test Coverage Examination</title> |
| 914 |
|
| 915 |
<para> |
| 916 |
The PostgreSQL source code can be compiled with coverage testing |
| 917 |
instrumentation, so that it becomes possible to examine which |
| 918 |
parts of the code are covered by the regression tests or any other |
| 919 |
test suite that is run with the code. This is currently supported |
| 920 |
when compiling with GCC, and it requires the <literal>gcov</literal> |
| 921 |
and <literal>lcov</literal> packages. |
| 922 |
</para> |
| 923 |
|
| 924 |
<sect2 id="regress-coverage-configure"> |
| 925 |
<title>Coverage with Autoconf and Make</title> |
| 926 |
<para> |
| 927 |
A typical workflow looks like this: |
| 928 |
<screen> |
| 929 |
./configure --enable-coverage ... OTHER OPTIONS ... |
| 930 |
make |
| 931 |
make check # or other test suite |
| 932 |
make coverage-html |
| 933 |
</screen> |
| 934 |
Then point your HTML browser |
| 935 |
to <filename>coverage/index.html</filename>. |
| 936 |
</para> |
| 937 |
|
| 938 |
<para> |
| 939 |
If you don't have <command>lcov</command> or prefer text output over an |
| 940 |
HTML report, you can run |
| 941 |
<screen> |
| 942 |
make coverage |
| 943 |
</screen> |
| 944 |
instead of <literal>make coverage-html</literal>, which will |
| 945 |
produce <filename>.gcov</filename> output files for each source file |
| 946 |
relevant to the test. (<literal>make coverage</literal> and <literal>make |
| 947 |
coverage-html</literal> will overwrite each other's files, so mixing them |
| 948 |
might be confusing.) |
| 949 |
</para> |
| 950 |
|
| 951 |
<para> |
| 952 |
You can run several different tests before making the coverage report; |
| 953 |
the execution counts will accumulate. If you want |
| 954 |
to reset the execution counts between test runs, run: |
| 955 |
<screen> |
| 956 |
make coverage-clean |
| 957 |
</screen> |
| 958 |
</para> |
| 959 |
|
| 960 |
<para> |
| 961 |
You can run the <literal>make coverage-html</literal> or <literal>make |
| 962 |
coverage</literal> command in a subdirectory if you want a coverage |
| 963 |
report for only a portion of the code tree. |
| 964 |
</para> |
| 965 |
|
| 966 |
<para> |
| 967 |
Use <literal>make distclean</literal> to clean up when done. |
| 968 |
</para> |
| 969 |
</sect2> |
| 970 |
|
| 971 |
<sect2 id="regress-coverage-meson"> |
| 972 |
<title>Coverage with Meson</title> |
| 973 |
<para> |
| 974 |
A typical workflow looks like this: |
| 975 |
<screen> |
| 976 |
meson setup -Db_coverage=true ... OTHER OPTIONS ... builddir/ |
| 977 |
meson compile -C builddir/ |
| 978 |
meson test -C builddir/ |
| 979 |
cd builddir/ |
| 980 |
ninja coverage-html |
| 981 |
</screen> |
| 982 |
Then point your HTML browser |
| 983 |
to <filename>./meson-logs/coveragereport/index.html</filename>. |
| 984 |
</para> |
| 985 |
|
| 986 |
<para> |
| 987 |
You can run several different tests before making the coverage report; |
| 988 |
the execution counts will accumulate. |
| 989 |
</para> |
| 990 |
</sect2> |
| 991 |
</sect1> |
| 992 |
|
| 993 |
</chapter> |