| 1 |
.\" ************************************************************************** |
| 2 |
.\" * _ _ ____ _ |
| 3 |
.\" * Project ___| | | | _ \| | |
| 4 |
.\" * / __| | | | |_) | | |
| 5 |
.\" * | (__| |_| | _ <| |___ |
| 6 |
.\" * \___|\___/|_| \_\_____| |
| 7 |
.\" * |
| 8 |
.\" * Copyright (C) 1998 - 2019, Daniel Stenberg, <[email protected]>, et al. |
| 9 |
.\" * |
| 10 |
.\" * This software is licensed as described in the file COPYING, which |
| 11 |
.\" * you should have received as part of this distribution. The terms |
| 12 |
.\" * are also available at https://curl.haxx.se/docs/copyright.html. |
| 13 |
.\" * |
| 14 |
.\" * You may opt to use, copy, modify, merge, publish, distribute and/or sell |
| 15 |
.\" * copies of the Software, and permit persons to whom the Software is |
| 16 |
.\" * furnished to do so, under the terms of the COPYING file. |
| 17 |
.\" * |
| 18 |
.\" * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY |
| 19 |
.\" * KIND, either express or implied. |
| 20 |
.\" * |
| 21 |
.\" ************************************************************************** |
| 22 |
.\" |
| 23 |
.TH runtests.pl 1 "2 Feb 2010" "Curl 7.20.0" "runtests" |
| 24 |
.SH NAME |
| 25 |
runtests.pl \- run one or more test cases |
| 26 |
.SH SYNOPSIS |
| 27 |
.B runtests.pl [options] [test number] [!test number] [key word] [!key word] |
| 28 |
.SH DESCRIPTION |
| 29 |
\fIruntests.pl\fP runs one, several or all the existing test cases in curl's |
| 30 |
test suite. It is often called from the root Makefile of the curl package with |
| 31 |
\&'make test'. |
| 32 |
.SH "TEST NUMBER" |
| 33 |
If no test case number is given, all existing tests that the script can find |
| 34 |
will be considered for running. You can specify single test cases to run, |
| 35 |
space-separated, like "1 3 5 7 11", and you can specify a range like "45 to |
| 36 |
67". You can also specify only the tests you don't want to run by listing |
| 37 |
the numbers with a leading exclamation point, like "!66". |
| 38 |
.P |
| 39 |
It is also possible to specify tests to skip based on a key word describing |
| 40 |
the test. These are specified with a leading exclamation point and the |
| 41 |
key word or phrase, like "!HTTP NTLM auth". Likewise, tests to run can |
| 42 |
be specified simply by specifying the unadorned key words, like "FTPS". |
| 43 |
Remember that the exclamation marks and spaces will need to be quoted somehow |
| 44 |
when entered at many command shells. |
| 45 |
.SH OPTIONS |
| 46 |
.IP "-a" |
| 47 |
Continue running the rest of the test cases even if one test fails. By |
| 48 |
default, the test script stops as soon as an error is detected. |
| 49 |
.IP "-bN" |
| 50 |
Use N as the base TCP/UDP port number on which to start the test servers. |
| 51 |
.IP "-c <curl>" |
| 52 |
Provide a path to a custom curl binary to run the tests with. Default is the |
| 53 |
curl executable in the build tree. |
| 54 |
.IP "-d" |
| 55 |
Enable protocol debug: have the servers display protocol output. |
| 56 |
.IP "-e" |
| 57 |
Run the test event-based (if possible). This will make runtests invoke curl |
| 58 |
with --test-event option. This option only works if both curl and libcurl were |
| 59 |
built debug-enabled. |
| 60 |
.IP "-g" |
| 61 |
Run the given test(s) with gdb. This is best used on a single test case and |
| 62 |
curl built --disable-shared. This then fires up gdb with command line set to |
| 63 |
run the specified test case. Simply (set a break-point and) type 'run' to |
| 64 |
start. |
| 65 |
.IP "-h" |
| 66 |
Displays a help text about this program's command line options. |
| 67 |
.IP "-k" |
| 68 |
Keep output and log files in log/ after a test run, even if no error was |
| 69 |
detected. Useful for debugging. |
| 70 |
.IP "-l" |
| 71 |
Lists all test case names. |
| 72 |
.IP "-n" |
| 73 |
Disable the check for and use of valgrind. |
| 74 |
.IP "-p" |
| 75 |
Prints out all files in "log/" to stdout when a test case fails. Very |
| 76 |
practical when used in the automated and distributed tests since then the |
| 77 |
people checking the failures and the reasons for them might not have physical |
| 78 |
access to the machine and logs. |
| 79 |
.IP "-R" |
| 80 |
Run the tests in a scrambled, or randomized, order instead of sequentially. |
| 81 |
|
| 82 |
The random seed initially set for this is fixed per month and can be set with |
| 83 |
\fI--seed\fP. |
| 84 |
.IP "-r" |
| 85 |
Display run time statistics. (Requires Perl Time::HiRes module) |
| 86 |
.IP "-rf" |
| 87 |
Display full run time statistics. (Requires Perl Time::HiRes module) |
| 88 |
.IP "--repeat=[num]" |
| 89 |
This will repeat the given set of test numbers this many times. If no test |
| 90 |
numbers are given, it will repeat ALL tests this many times. It iteratively |
| 91 |
adds the new sequence at the end of the initially given one. |
| 92 |
|
| 93 |
If \fB-R\fP is also used, the scrambling is done after the repeats have |
| 94 |
extended the test sequence. |
| 95 |
.IP "-s" |
| 96 |
Shorter output. Speaks less than default. |
| 97 |
.IP "--seed=[num]" |
| 98 |
When using \fI--shallow\fP or \fI-R\rP that random certain aspects of the |
| 99 |
behavior, this option can set the initial seed. If not set, the random seed |
| 100 |
will be set based on the currently set local year and month and the first line |
| 101 |
of the "curl -V" output. |
| 102 |
.IP "--shallow=[num]" |
| 103 |
Used together with \fB-t\fP. This limits the number of tests to fail in |
| 104 |
torture mode to no more than 'num' per test case. If this reduces the amount, |
| 105 |
the script will randomly discard entries to fail until the amount is 'num'. |
| 106 |
|
| 107 |
The random seed initially set for this is fixed per month and can be set with |
| 108 |
\fI--seed\fP. |
| 109 |
.IP "-t[num]" |
| 110 |
Selects a \fBtorture\fP test for the given tests. This makes runtests.pl first |
| 111 |
run the tests once and count the number of memory allocations made. It then |
| 112 |
reruns the test that number of times, each time forcing one of the allocations |
| 113 |
to fail until all allocs have been tested. By setting \fInum\fP you can force |
| 114 |
the allocation with that number to be set to fail at once instead of looping |
| 115 |
through everyone, which is very handy when debugging and then often in |
| 116 |
combination with \fI-g\fP. |
| 117 |
.IP "-v" |
| 118 |
Enable verbose output. Speaks more than default. |
| 119 |
.IP "-vc <curl>" |
| 120 |
Provide a path to a custom curl binary to run when verifying that the servers |
| 121 |
running are indeed our test servers. Default is the curl executable in the |
| 122 |
build tree. |
| 123 |
.SH "RUNNING TESTS" |
| 124 |
Many tests have conditions that must be met before the test case can run |
| 125 |
fine. They could depend on built-in features in libcurl or features present in |
| 126 |
the operating system or even in third-party libraries that curl may or may not |
| 127 |
use. |
| 128 |
.P |
| 129 |
The test script checks most of these by itself to determine when it is |
| 130 |
safe to attempt to run each test. Those which cannot be run due to |
| 131 |
failed requirements will simply be skipped and listed at the completion |
| 132 |
of all test cases. In some unusual configurations, the test script |
| 133 |
cannot make the correct determination for all tests. In these cases, |
| 134 |
the problematic tests can be skipped using the "!keyword" skip feature |
| 135 |
documented earlier. |
| 136 |
.SH "WRITING TESTS" |
| 137 |
The simplest way to write test cases is to start with a similar existing test, |
| 138 |
save it with a new number and then adjust it to fit. There's an attempt to |
| 139 |
document the test case file format in the tests/FILEFORMAT. |