Debian Codegraph @

upstream/debian-unstable/man3/TAP::Harness::Beyond.3perl

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man 5.0102 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\" ========================================================================
56 .\"
57 .IX Title "TAP::Harness::Beyond 3perl"
58 .TH TAP::Harness::Beyond 3perl 2025-11-16 "perl v5.40.1" "Perl Programmers Reference Guide"
59 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
60 .\" way too many mistakes in technical documents.
61 .if n .ad l
62 .nh
63 .SH NAME
64 Test::Harness::Beyond \- Beyond make test
65 .SH "Beyond make test"
66 .IX Header "Beyond make test"
67 Test::Harness is responsible for running test scripts, analysing
68 their output and reporting success or failure. When I type
69 \&\fImake test\fR (or \fI./Build test\fR) for a module, Test::Harness is usually
70 used to run the tests (not all modules use Test::Harness but the
71 majority do).
72 .PP
73 To start exploring some of the features of Test::Harness I need to
74 switch from \fImake test\fR to the \fIprove\fR command (which ships with
75 Test::Harness). For the following examples I'll also need a recent
76 version of Test::Harness installed; 3.14 is current as I write.
77 .PP
78 For the examples I'm going to assume that we're working with a
79 \&'normal' Perl module distribution. Specifically I'll assume that
80 typing \fImake\fR or \fI./Build\fR causes the built, ready-to-install module
81 code to be available below ./blib/lib and ./blib/arch and that
82 there's a directory called 't' that contains our tests. Test::Harness
83 isn't hardwired to that configuration but it saves me from explaining
84 which files live where for each example.
85 .PP
86 Back to \fIprove\fR; like \fImake test\fR it runs a test suite \- but it
87 provides far more control over which tests are executed, in what
88 order and how their results are reported. Typically \fImake test\fR
89 runs all the test scripts below the 't' directory. To do the same
90 thing with prove I type:
91 .PP
92 .Vb 1
93 \& prove \-rb t
94 .Ve
95 .PP
96 The switches here are \-r to recurse into any directories below 't'
97 and \-b which adds ./blib/lib and ./blib/arch to Perl's include path
98 so that the tests can find the code they will be testing. If I'm
99 testing a module of which an earlier version is already installed
100 I need to be careful about the include path to make sure I'm not
101 running my tests against the installed version rather than the new
102 one that I'm working on.
103 .PP
104 Unlike \fImake test\fR, typing \fIprove\fR doesn't automatically rebuild
105 my module. If I forget to make before prove I will be testing against
106 older versions of those files \- which inevitably leads to confusion.
107 I either get into the habit of typing
108 .PP
109 .Vb 1
110 \& make && prove \-rb t
111 .Ve
112 .PP
113 or \- if I have no XS code that needs to be built I use the modules
114 below \fIlib\fR instead
115 .PP
116 .Vb 1
117 \& prove \-Ilib \-r t
118 .Ve
119 .PP
120 So far I've shown you nothing that \fImake test\fR doesn't do. Let's
121 fix that.
122 .SS "Saved State"
123 .IX Subsection "Saved State"
124 If I have failing tests in a test suite that consists of more than
125 a handful of scripts and takes more than a few seconds to run it
126 rapidly becomes tedious to run the whole test suite repeatedly as
127 I track down the problems.
128 .PP
129 I can tell prove just to run the tests that are failing like this:
130 .PP
131 .Vb 1
132 \& prove \-b t/this_fails.t t/so_does_this.t
133 .Ve
134 .PP
135 That speeds things up but I have to make a note of which tests are
136 failing and make sure that I run those tests. Instead I can use
137 prove's \-\-state switch and have it keep track of failing tests for
138 me. First I do a complete run of the test suite and tell prove to
139 save the results:
140 .PP
141 .Vb 1
142 \& prove \-rb \-\-state=save t
143 .Ve
144 .PP
145 That stores a machine readable summary of the test run in a file
146 called '.prove' in the current directory. If I have failures I can
147 then run just the failing scripts like this:
148 .PP
149 .Vb 1
150 \& prove \-b \-\-state=failed
151 .Ve
152 .PP
153 I can also tell prove to save the results again so that it updates
154 its idea of which tests failed:
155 .PP
156 .Vb 1
157 \& prove \-b \-\-state=failed,save
158 .Ve
159 .PP
160 As soon as one of my failing tests passes it will be removed from
161 the list of failed tests. Eventually I fix them all and prove can
162 find no failing tests to run:
163 .PP
164 .Vb 2
165 \& Files=0, Tests=0, 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)
166 \& Result: NOTESTS
167 .Ve
168 .PP
169 As I work on a particular part of my module it's most likely that
170 the tests that cover that code will fail. I'd like to run the whole
171 test suite but have it prioritize these 'hot' tests. I can tell
172 prove to do this:
173 .PP
174 .Vb 1
175 \& prove \-rb \-\-state=hot,save t
176 .Ve
177 .PP
178 All the tests will run but those that failed most recently will be
179 run first. If no tests have failed since I started saving state all
180 tests will run in their normal order. This combines full test
181 coverage with early notification of failures.
182 .PP
183 The \-\-state switch supports a number of options; for example to run
184 failed tests first followed by all remaining tests ordered by the
185 timestamps of the test scripts \- and save the results \- I can use
186 .PP
187 .Vb 1
188 \& prove \-rb \-\-state=failed,new,save t
189 .Ve
190 .PP
191 See the prove documentation (type prove \-\-man) for the full list
192 of state options.
193 .PP
194 When I tell prove to save state it writes a file called '.prove'
195 ('_prove' on Windows) in the current directory. It's a YAML document
196 so it's quite easy to write tools of your own that work on the saved
197 test state \- but the format isn't officially documented so it might
198 change without (much) warning in the future.
199 .SS "Parallel Testing"
200 .IX Subsection "Parallel Testing"
201 If my tests take too long to run I may be able to speed them up by
202 running multiple test scripts in parallel. This is particularly
203 effective if the tests are I/O bound or if I have multiple CPU
204 cores. I tell prove to run my tests in parallel like this:
205 .PP
206 .Vb 1
207 \& prove \-rb \-j 9 t
208 .Ve
209 .PP
210 The \-j switch enables parallel testing; the number that follows it
211 is the maximum number of tests to run in parallel. Sometimes tests
212 that pass when run sequentially will fail when run in parallel. For
213 example if two different test scripts use the same temporary file
214 or attempt to listen on the same socket I'll have problems running
215 them in parallel. If I see unexpected failures I need to check my
216 tests to work out which of them are trampling on the same resource
217 and rename temporary files or add locks as appropriate.
218 .PP
219 To get the most performance benefit I want to have the test scripts
220 that take the longest to run start first \- otherwise I'll be waiting
221 for the one test that takes nearly a minute to complete after all
222 the others are done. I can use the \-\-state switch to run the tests
223 in slowest to fastest order:
224 .PP
225 .Vb 1
226 \& prove \-rb \-j 9 \-\-state=slow,save t
227 .Ve
228 .SS "Non-Perl Tests"
229 .IX Subsection "Non-Perl Tests"
230 The Test Anything Protocol (http://testanything.org/) isn't just
231 for Perl. Just about any language can be used to write tests that
232 output TAP. There are TAP based testing libraries for C, C++, PHP,
233 Python and many others. If I can't find a TAP library for my language
234 of choice it's easy to generate valid TAP. It looks like this:
235 .PP
236 .Vb 4
237 \& 1..3
238 \& ok 1 \- init OK
239 \& ok 2 \- opened file
240 \& not ok 3 \- appended to file
241 .Ve
242 .PP
243 The first line is the plan \- it specifies the number of tests I'm
244 going to run so that it's easy to check that the test script didn't
245 exit before running all the expected tests. The following lines are
246 the test results \- 'ok' for pass, 'not ok' for fail. Each test has
247 a number and, optionally, a description. And that's it. Any language
248 that can produce output like that on STDOUT can be used to write
249 tests.
250 .PP
251 Recently I've been rekindling a two-decades-old interest in Forth.
252 Evidently I have a masochistic streak that even Perl can't satisfy.
253 I want to write tests in Forth and run them using prove (you can
254 find my gforth TAP experiments at
255 https://svn.hexten.net/andy/Forth/Testing/). I can use the \-\-exec
256 switch to tell prove to run the tests using gforth like this:
257 .PP
258 .Vb 1
259 \& prove \-r \-\-exec gforth t
260 .Ve
261 .PP
262 Alternately, if the language used to write my tests allows a shebang
263 line I can use that to specify the interpreter. Here's a test written
264 in PHP:
265 .PP
266 .Vb 6
267 \& #!/usr/bin/php
268 \& <?php
269 \& print "1..2\en";
270 \& print "ok 1\en";
271 \& print "not ok 2\en";
272 \& ?>
273 .Ve
274 .PP
275 If I save that as t/phptest.t the shebang line will ensure that it
276 runs correctly along with all my other tests.
277 .SS "Mixing it up"
278 .IX Subsection "Mixing it up"
279 Subtle interdependencies between test programs can mask problems \-
280 for example an earlier test may neglect to remove a temporary file
281 that affects the behaviour of a later test. To find this kind of
282 problem I use the \-\-shuffle and \-\-reverse options to run my tests
283 in random or reversed order.
284 .SS "Rolling My Own"
285 .IX Subsection "Rolling My Own"
286 If I need a feature that prove doesn't provide I can easily write my own.
287 .PP
288 Typically you'll want to change how TAP gets \fIinput\fR into and \fIoutput\fR
289 from the parser. App::Prove supports arbitrary plugins, and TAP::Harness
290 supports custom \fIformatters\fR and \fIsource handlers\fR that you can load using
291 either prove or Module::Build; there are many examples to base mine on.
292 For more details see App::Prove, TAP::Parser::SourceHandler, and
293 TAP::Formatter::Base.
294 .PP
295 If writing a plugin is not enough, you can write your own test harness; one of
296 the motives for the 3.00 rewrite of Test::Harness was to make it easier to
297 subclass and extend.
298 .PP
299 The Test::Harness module is a compatibility wrapper around TAP::Harness.
300 For new applications I should use TAP::Harness directly. As we'll
301 see, prove uses TAP::Harness.
302 .PP
303 When I run prove it processes its arguments, figures out which test
304 scripts to run and then passes control to TAP::Harness to run the
305 tests, parse, analyse and present the results. By subclassing
306 TAP::Harness I can customise many aspects of the test run.
307 .PP
308 I want to log my test results in a database so I can track them
309 over time. To do this I override the summary method in TAP::Harness.
310 I start with a simple prototype that dumps the results as a YAML
311 document:
312 .PP
313 .Vb 1
314 \& package My::TAP::Harness;
315 \&
316 \& use base \*(AqTAP::Harness\*(Aq;
317 \& use YAML;
318 \&
319 \& sub summary {
320 \& my ( $self, $aggregate ) = @_;
321 \& print Dump( $aggregate );
322 \& $self\->SUPER::summary( $aggregate );
323 \& }
324 \&
325 \& 1;
326 .Ve
327 .PP
328 I need to tell prove to use my My::TAP::Harness. If My::TAP::Harness
329 is on Perl's \f(CW@INC\fR include path I can
330 .PP
331 .Vb 1
332 \& prove \-\-harness=My::TAP::Harness \-rb t
333 .Ve
334 .PP
335 If I don't have My::TAP::Harness installed on \f(CW@INC\fR I need to provide
336 the correct path to perl when I run prove:
337 .PP
338 .Vb 1
339 \& perl \-Ilib \`which prove\` \-\-harness=My::TAP::Harness \-rb t
340 .Ve
341 .PP
342 I can incorporate these options into my own version of prove. It's
343 pretty simple. Most of the work of prove is handled by App::Prove.
344 The important code in prove is just:
345 .PP
346 .Vb 1
347 \& use App::Prove;
348 \&
349 \& my $app = App::Prove\->new;
350 \& $app\->process_args(@ARGV);
351 \& exit( $app\->run ? 0 : 1 );
352 .Ve
353 .PP
354 If I write a subclass of App::Prove I can customise any aspect of
355 the test runner while inheriting all of prove's behaviour. Here's
356 myprove:
357 .PP
358 .Vb 2
359 \& #!/usr/bin/env perl use lib qw( lib ); # Add ./lib to @INC
360 \& use App::Prove;
361 \&
362 \& my $app = App::Prove\->new;
363 \&
364 \& # Use custom TAP::Harness subclass
365 \& $app\->harness( \*(AqMy::TAP::Harness\*(Aq );
366 \&
367 \& $app\->process_args( @ARGV ); exit( $app\->run ? 0 : 1 );
368 .Ve
369 .PP
370 Now I can run my tests like this
371 .PP
372 .Vb 1
373 \& ./myprove \-rb t
374 .Ve
375 .SS "Deeper Customisation"
376 .IX Subsection "Deeper Customisation"
377 Now that I know how to subclass and replace TAP::Harness I can
378 replace any other part of the harness. To do that I need to know
379 which classes are responsible for which functionality. Here's a
380 brief guided tour; the default class for each component is shown
381 in parentheses. Normally any replacements I write will be subclasses
382 of these default classes.
383 .PP
384 When I run my tests TAP::Harness creates a scheduler
385 (TAP::Parser::Scheduler) to work out the running order for the
386 tests, an aggregator (TAP::Parser::Aggregator) to collect and analyse
387 the test results and a formatter (TAP::Formatter::Console) to display
388 those results.
389 .PP
390 If I'm running my tests in parallel there may also be a multiplexer
391 (TAP::Parser::Multiplexer) \- the component that allows multiple
392 tests to run simultaneously.
393 .PP
394 Once it has created those helpers TAP::Harness starts running the
395 tests. For each test it creates a new parser (TAP::Parser) which
396 is responsible for running the test script and parsing its output.
397 .PP
398 To replace any of these components I call one of these harness
399 methods with the name of the replacement class:
400 .PP
401 .Vb 5
402 \& aggregator_class
403 \& formatter_class
404 \& multiplexer_class
405 \& parser_class
406 \& scheduler_class
407 .Ve
408 .PP
409 For example, to replace the aggregator I would
410 .PP
411 .Vb 1
412 \& $harness\->aggregator_class( \*(AqMy::Aggregator\*(Aq );
413 .Ve
414 .PP
415 Alternately I can supply the names of my substitute classes to the
416 TAP::Harness constructor:
417 .PP
418 .Vb 3
419 \& my $harness = TAP::Harness\->new(
420 \& { aggregator_class => \*(AqMy::Aggregator\*(Aq }
421 \& );
422 .Ve
423 .PP
424 If I need to reach even deeper into the internals of the harness I
425 can replace the classes that TAP::Parser uses to execute test scripts
426 and tokenise their output. Before running a test script TAP::Parser
427 creates a grammar (TAP::Parser::Grammar) to decode the raw TAP into
428 tokens, a result factory (TAP::Parser::ResultFactory) to turn the
429 decoded TAP results into objects and, depending on whether it's
430 running a test script or reading TAP from a file, scalar or array
431 a source or an iterator (TAP::Parser::IteratorFactory).
432 .PP
433 Each of these objects may be replaced by calling one of these parser
434 methods:
435 .PP
436 .Vb 5
437 \& source_class
438 \& perl_source_class
439 \& grammar_class
440 \& iterator_factory_class
441 \& result_factory_class
442 .Ve
443 .SS Callbacks
444 .IX Subsection "Callbacks"
445 As an alternative to subclassing the components I need to change I
446 can attach callbacks to the default classes. TAP::Harness exposes
447 these callbacks:
448 .PP
449 .Vb 5
450 \& parser_args Tweak the parameters used to create the parser
451 \& made_parser Just made a new parser
452 \& before_runtests About to run tests
453 \& after_runtests Have run all tests
454 \& after_test Have run an individual test script
455 .Ve
456 .PP
457 TAP::Parser also supports callbacks; bailout, comment, plan, test,
458 unknown, version and yaml are called for the corresponding TAP
459 result types, ALL is called for all results, ELSE is called for all
460 results for which a named callback is not installed and EOF is
461 called once at the end of each TAP stream.
462 .PP
463 To install a callback I pass the name of the callback and a subroutine
464 reference to TAP::Harness or TAP::Parser's callback method:
465 .PP
466 .Vb 3
467 \& $harness\->callback( after_test => sub {
468 \& my ( $script, $desc, $parser ) = @_;
469 \& } );
470 .Ve
471 .PP
472 I can also pass callbacks to the constructor:
473 .PP
474 .Vb 8
475 \& my $harness = TAP::Harness\->new({
476 \& callbacks => {
477 \& after_test => sub {
478 \& my ( $script, $desc, $parser ) = @_;
479 \& # Do something interesting here
480 \& }
481 \& }
482 \& });
483 .Ve
484 .PP
485 When it comes to altering the behaviour of the test harness there's
486 more than one way to do it. Which way is best depends on my
487 requirements. In general if I only want to observe test execution
488 without changing the harness' behaviour (for example to log test
489 results to a database) I choose callbacks. If I want to make the
490 harness behave differently subclassing gives me more control.
491 .SS "Parsing TAP"
492 .IX Subsection "Parsing TAP"
493 Perhaps I don't need a complete test harness. If I already have a
494 TAP test log that I need to parse all I need is TAP::Parser and the
495 various classes it depends upon. Here's the code I need to run a
496 test and parse its TAP output
497 .PP
498 .Vb 1
499 \& use TAP::Parser;
500 \&
501 \& my $parser = TAP::Parser\->new( { source => \*(Aqt/simple.t\*(Aq } );
502 \& while ( my $result = $parser\->next ) {
503 \& print $result\->as_string, "\en";
504 \& }
505 .Ve
506 .PP
507 Alternately I can pass an open filehandle as source and have the
508 parser read from that rather than attempting to run a test script:
509 .PP
510 .Vb 6
511 \& open my $tap, \*(Aq<\*(Aq, \*(Aqtests.tap\*(Aq
512 \& or die "Can\*(Aqt read TAP transcript ($!)\en";
513 \& my $parser = TAP::Parser\->new( { source => $tap } );
514 \& while ( my $result = $parser\->next ) {
515 \& print $result\->as_string, "\en";
516 \& }
517 .Ve
518 .PP
519 This approach is useful if I need to convert my TAP based test
520 results into some other representation. See TAP::Convert::TET
521 (http://search.cpan.org/dist/TAP\-Convert\-TET/) for an example of
522 this approach.
523 .SS "Getting Support"
524 .IX Subsection "Getting Support"
525 The Test::Harness developers hang out on the tapx-dev mailing
526 list[1]. For discussion of general, language independent TAP issues
527 there's the tap\-l[2] list. Finally there's a wiki dedicated to the
528 Test Anything Protocol[3]. Contributions to the wiki, patches and
529 suggestions are all welcome.
530 .PP
531 [1] <http://www.hexten.net/mailman/listinfo/tapx\-dev>
532 [2] <http://testanything.org/mailman/listinfo/tap\-l>
533 [3] <http://testanything.org/>