| 1 |
_ _ ____ _ |
| 2 |
___| | | | _ \| | |
| 3 |
/ __| | | | |_) | | |
| 4 |
| (__| |_| | _ <| |___ |
| 5 |
\___|\___/|_| \_\_____| |
| 6 |
|
| 7 |
The curl Test Suite |
| 8 |
|
| 9 |
1. Running |
| 10 |
1.1 Requires to run |
| 11 |
1.2 Port numbers used by test servers |
| 12 |
1.3 Test servers |
| 13 |
1.4 Run |
| 14 |
1.5 Shell startup scripts |
| 15 |
1.6 Memory test |
| 16 |
1.7 Debug |
| 17 |
1.8 Logs |
| 18 |
1.9 Test input files |
| 19 |
1.10 Code coverage |
| 20 |
1.11 Remote testing |
| 21 |
|
| 22 |
2. Numbering |
| 23 |
2.1 Test case numbering |
| 24 |
|
| 25 |
3. Write tests |
| 26 |
3.1 test data |
| 27 |
3.2 curl tests |
| 28 |
3.3 libcurl tests |
| 29 |
3.4 unit tests |
| 30 |
|
| 31 |
4. TODO |
| 32 |
4.1 More protocols |
| 33 |
4.2 SOCKS auth |
| 34 |
|
| 35 |
============================================================================== |
| 36 |
|
| 37 |
1. Running |
| 38 |
|
| 39 |
1.1 Requires to run |
| 40 |
|
| 41 |
perl (and a unix-style shell) |
| 42 |
python (and a unix-style shell) |
| 43 |
diff (when a test fails, a diff is shown) |
| 44 |
stunnel (for HTTPS and FTPS tests) |
| 45 |
OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests) |
| 46 |
nghttpx (for HTTP/2 tests) |
| 47 |
nroff (for --manual tests) |
| 48 |
|
| 49 |
1.2 Port numbers used by test servers |
| 50 |
|
| 51 |
- TCP/8990 for HTTP |
| 52 |
- TCP/8991 for HTTPS |
| 53 |
- TCP/8992 for FTP |
| 54 |
- TCP/8993 for FTPS |
| 55 |
- TCP/8994 for HTTP IPv6 |
| 56 |
- TCP/8995 for FTP (2) |
| 57 |
- TCP/8996 for FTP IPv6 |
| 58 |
- UDP/8997 for TFTP |
| 59 |
- UDP/8998 for TFTP IPv6 |
| 60 |
- TCP/8999 for SCP/SFTP |
| 61 |
- TCP/9000 for SOCKS |
| 62 |
- TCP/9001 for POP3 |
| 63 |
- TCP/9002 for POP3 IPv6 |
| 64 |
- TCP/9003 for IMAP |
| 65 |
- TCP/9004 for IMAP IPv6 |
| 66 |
- TCP/9005 for SMTP |
| 67 |
- TCP/9006 for SMTP IPv6 |
| 68 |
- TCP/9007 for RTSP |
| 69 |
- TCP/9008 for RTSP IPv6 |
| 70 |
- TCP/9009 for GOPHER |
| 71 |
- TCP/9010 for GOPHER IPv6 |
| 72 |
- TCP/9011 for HTTPS server with TLS-SRP support |
| 73 |
- TCP/9012 for HTTPS IPv6 server with TLS-SRP support |
| 74 |
- TCP/9013 for HTTP proxy server for CONNECT |
| 75 |
- TCP/9014 for HTTP pipelining server |
| 76 |
- TCP/9015 for HTTP/2 server |
| 77 |
- TCP/9016 for DICT server |
| 78 |
- TCP/9017 for SMB server |
| 79 |
- TCP/9018 for SMBS server (reserved) |
| 80 |
- TCP/9019 for TELNET server with negotiation support |
| 81 |
|
| 82 |
1.3 Test servers |
| 83 |
|
| 84 |
The test suite runs simple FTP, POP3, IMAP, SMTP, HTTP and TFTP stand-alone |
| 85 |
servers on the ports listed above to which it makes requests. For SSL tests, |
| 86 |
it runs stunnel to handle encryption to the regular servers. For SSH, it |
| 87 |
runs a standard OpenSSH server. For SOCKS4/5 tests SSH is used to perform |
| 88 |
the SOCKS functionality and requires a SSH client and server. |
| 89 |
|
| 90 |
The base port number (8990), which all the individual port numbers are |
| 91 |
indexed from, can be set explicitly using runtests.pl' -b option to allow |
| 92 |
running more than one instance of the test suite simultaneously on one |
| 93 |
machine, or just move the servers in case you have local services on any of |
| 94 |
those ports. |
| 95 |
|
| 96 |
The HTTP server supports listening on a Unix domain socket, the default |
| 97 |
location is 'http.sock'. |
| 98 |
|
| 99 |
1.4 Run |
| 100 |
|
| 101 |
'./configure && make && make test'. This builds the test suite support code |
| 102 |
and invokes the 'runtests.pl' perl script to run all the tests. Edit the top |
| 103 |
variables of that script in case you have some specific needs, or run the |
| 104 |
script manually (after the support code has been built). |
| 105 |
|
| 106 |
The script breaks on the first test that doesn't do OK. Use -a to prevent |
| 107 |
the script from aborting on the first error. Run the script with -v for more |
| 108 |
verbose output. Use -d to run the test servers with debug output enabled as |
| 109 |
well. Specifying -k keeps all the log files generated by the test intact. |
| 110 |
|
| 111 |
Use -s for shorter output, or pass test numbers to run specific tests only |
| 112 |
(like "./runtests.pl 3 4" to test 3 and 4 only). It also supports test case |
| 113 |
ranges with 'to', as in "./runtests 3 to 9" which runs the seven tests from |
| 114 |
3 to 9. Any test numbers starting with ! are disabled, as are any test |
| 115 |
numbers found in the files data/DISABLED or data/DISABLED.local (one per |
| 116 |
line). The latter is meant for local temporary disables and will be ignored |
| 117 |
by git. |
| 118 |
|
| 119 |
When -s is not present, each successful test will display on one line the |
| 120 |
test number and description and on the next line a set of flags, the test |
| 121 |
result, current test sequence, total number of tests to be run and an |
| 122 |
estimated amount of time to complete the test run. The flags consist of |
| 123 |
these letters describing what is checked in this test: |
| 124 |
|
| 125 |
s stdout |
| 126 |
d data |
| 127 |
u upload |
| 128 |
p protocol |
| 129 |
o output |
| 130 |
e exit code |
| 131 |
m memory |
| 132 |
v valgrind |
| 133 |
|
| 134 |
1.5 Shell startup scripts |
| 135 |
|
| 136 |
Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly |
| 137 |
influenced by the output of system wide or user specific shell startup |
| 138 |
scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which |
| 139 |
output text messages or escape sequences on user login. When these shell |
| 140 |
startup messages or escape sequences are output they might corrupt the |
| 141 |
expected stream of data which flows to the sftp-server or from the ssh |
| 142 |
client which can result in bad test behaviour or even prevent the test |
| 143 |
server from running. |
| 144 |
|
| 145 |
If the test suite ssh or sftp server fails to start up and logs the message |
| 146 |
'Received message too long' then you are certainly suffering the unwanted |
| 147 |
output of a shell startup script. Locate, cleanup or adjust the shell |
| 148 |
script. |
| 149 |
|
| 150 |
1.6 Memory test |
| 151 |
|
| 152 |
The test script will check that all allocated memory is freed properly IF |
| 153 |
curl has been built with the CURLDEBUG define set. The script will |
| 154 |
automatically detect if that is the case, and it will use the |
| 155 |
'memanalyze.pl' script to analyze the memory debugging output. |
| 156 |
|
| 157 |
Also, if you run tests on a machine where valgrind is found, the script will |
| 158 |
use valgrind to run the test with (unless you use -n) to further verify |
| 159 |
correctness. |
| 160 |
|
| 161 |
runtests.pl's -t option will enable torture testing mode, which runs each |
| 162 |
test many times and makes each different memory allocation fail on each |
| 163 |
successive run. This tests the out of memory error handling code to ensure |
| 164 |
that memory leaks do not occur even in those situations. It can help to |
| 165 |
compile curl with CPPFLAGS=-DMEMDEBUG_LOG_SYNC when using this option, to |
| 166 |
ensure that the memory log file is properly written even if curl crashes. |
| 167 |
|
| 168 |
1.7 Debug |
| 169 |
|
| 170 |
If a test case fails, you can conveniently get the script to invoke the |
| 171 |
debugger (gdb) for you with the server running and the exact same command |
| 172 |
line parameters that failed. Just invoke 'runtests.pl <test number> -g' and |
| 173 |
then just type 'run' in the debugger to perform the command through the |
| 174 |
debugger. |
| 175 |
|
| 176 |
1.8 Logs |
| 177 |
|
| 178 |
All logs are generated in the log/ subdirectory (it is emptied first in the |
| 179 |
runtests.pl script). Use runtests.pl -k to force it to keep the temporary |
| 180 |
files after the test run since successful runs will clean it up otherwise. |
| 181 |
|
| 182 |
1.9 Test input files |
| 183 |
|
| 184 |
All test cases are put in the data/ subdirectory. Each test is stored in the |
| 185 |
file named according to the test number. |
| 186 |
|
| 187 |
See FILEFORMAT for the description of the test case files. |
| 188 |
|
| 189 |
1.10 Code coverage |
| 190 |
|
| 191 |
gcc provides a tool that can determine the code coverage figures for |
| 192 |
the test suite. To use it, configure curl with |
| 193 |
CFLAGS='-fprofile-arcs -ftest-coverage -g -O0'. Make sure you run the normal |
| 194 |
and torture tests to get more full coverage, i.e. do: |
| 195 |
|
| 196 |
make test |
| 197 |
make test-torture |
| 198 |
|
| 199 |
The graphical tool ggcov can be used to browse the source and create |
| 200 |
coverage reports on *NIX hosts: |
| 201 |
|
| 202 |
ggcov -r lib src |
| 203 |
|
| 204 |
The text mode tool gcov may also be used, but it doesn't handle object files |
| 205 |
in more than one directory very well. |
| 206 |
|
| 207 |
1.11 Remote testing |
| 208 |
|
| 209 |
The runtests.pl script provides some hooks to allow curl to be tested on a |
| 210 |
machine where perl can not be run. The test framework in this case runs on |
| 211 |
a workstation where perl is available, while curl itself is run on a remote |
| 212 |
system using ssh or some other remote execution method. See the comments at |
| 213 |
the beginning of runtests.pl for details. |
| 214 |
|
| 215 |
2. Numbering |
| 216 |
|
| 217 |
2.1 Test case numbering |
| 218 |
|
| 219 |
Test cases used to be numbered by category, but the ranges filled |
| 220 |
up. Subsets of tests can now be selected by passing keywords to the |
| 221 |
runtests.pl script via the make TFLAGS variable. |
| 222 |
|
| 223 |
New tests should now be added by finding a free number in |
| 224 |
tests/data/Makefile.inc. |
| 225 |
|
| 226 |
3. Write tests |
| 227 |
|
| 228 |
Here's a quick description on writing test cases. We basically have three |
| 229 |
kinds of tests: the ones that test the curl tool, the ones that build small |
| 230 |
applications and test libcurl directly and the unit tests that test |
| 231 |
individual (possibly internal) functions. |
| 232 |
|
| 233 |
3.1 test data |
| 234 |
|
| 235 |
Each test has a master file that controls all the test data. What to read, |
| 236 |
what the protocol exchange should look like, what exit code to expect and |
| 237 |
what command line arguments to use etc. |
| 238 |
|
| 239 |
These files are tests/data/test[num] where [num] is described in section 2 |
| 240 |
of this document, and the XML-like file format of them is described in the |
| 241 |
separate tests/FILEFORMAT document. |
| 242 |
|
| 243 |
3.2 curl tests |
| 244 |
|
| 245 |
A test case that runs the curl tool and verifies that it gets the correct |
| 246 |
data, it sends the correct data, it uses the correct protocol primitives |
| 247 |
etc. |
| 248 |
|
| 249 |
3.3 libcurl tests |
| 250 |
|
| 251 |
The libcurl tests are identical to the curl ones, except that they use a |
| 252 |
specific and dedicated custom-built program to run instead of "curl". This |
| 253 |
tool is built from source code placed in tests/libtest and if you want to |
| 254 |
make a new libcurl test that is where you add your code. |
| 255 |
|
| 256 |
3.4 unit tests |
| 257 |
|
| 258 |
Unit tests are tests in the 13xx sequence and they are placed in tests/unit. |
| 259 |
There's a tests/unit/README describing the specific set of checks and macros |
| 260 |
that may be used when writing tests that verify behaviors of specific |
| 261 |
individual functions. |
| 262 |
|
| 263 |
The unit tests depend on curl being built with debug enabled. |
| 264 |
|
| 265 |
4. TODO |
| 266 |
|
| 267 |
4.1 More protocols |
| 268 |
|
| 269 |
Add tests for TELNET, LDAP, DICT... |
| 270 |
|
| 271 |
4.2 SOCKS auth |
| 272 |
|
| 273 |
SOCKS4/5 test deficiencies - no proxy authentication tests as SSH (the |
| 274 |
test mechanism) doesn't support them |