Debian Codegraph @

upstream/mageia-cauldron/man3pm/TAP::Harness::Beyond.3pm

raw
1 .\" -*- mode: troff; coding: utf-8 -*-
2 .\" Automatically generated by Pod::Man v6.0.2 (Pod::Simple 3.45)
3 .\"
4 .\" Standard preamble:
5 .\" ========================================================================
6 .de Sp \" Vertical space (when we can't use .PP)
7 .if t .sp .5v
8 .if n .sp
9 ..
10 .de Vb \" Begin verbatim text
11 .ft CW
12 .nf
13 .ne \\$1
14 ..
15 .de Ve \" End verbatim text
16 .ft R
17 .fi
18 ..
19 .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>.
20 .ie n \{\
21 . ds C` ""
22 . ds C' ""
23 'br\}
24 .el\{\
25 . ds C`
26 . ds C'
27 'br\}
28 .\"
29 .\" Escape single quotes in literal strings from groff's Unicode transform.
30 .ie \n(.g .ds Aq \(aq
31 .el .ds Aq '
32 .\"
33 .\" If the F register is >0, we'll generate index entries on stderr for
34 .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
35 .\" entries marked with X<> in POD. Of course, you'll have to process the
36 .\" output yourself in some meaningful fashion.
37 .\"
38 .\" Avoid warning from groff about undefined register 'F'.
39 .de IX
40 ..
41 .nr rF 0
42 .if \n(.g .if rF .nr rF 1
43 .if (\n(rF:(\n(.g==0)) \{\
44 . if \nF \{\
45 . de IX
46 . tm Index:\\$1\t\\n%\t"\\$2"
47 ..
48 . if !\nF==2 \{\
49 . nr % 0
50 . nr F 2
51 . \}
52 . \}
53 .\}
54 .rr rF
55 .\"
56 .\" Required to disable full justification in groff 1.23.0.
57 .if n .ds AD l
58 .\" ========================================================================
59 .\"
60 .IX Title "TAP::Harness::Beyond 3pm"
61 .TH TAP::Harness::Beyond 3pm 2023-12-22 "perl v5.42.0" "Perl Programmers Reference Guide"
62 .\" For nroff, turn off justification. Always turn off hyphenation; it makes
63 .\" way too many mistakes in technical documents.
64 .if n .ad l
65 .nh
66 .SH NAME
67 Test::Harness::Beyond \- Beyond make test
68 .SH "Beyond make test"
69 .IX Header "Beyond make test"
70 Test::Harness is responsible for running test scripts, analysing
71 their output and reporting success or failure. When I type
72 \&\fImake test\fR (or \fI./Build test\fR) for a module, Test::Harness is usually
73 used to run the tests (not all modules use Test::Harness but the
74 majority do).
75 .PP
76 To start exploring some of the features of Test::Harness I need to
77 switch from \fImake test\fR to the \fIprove\fR command (which ships with
78 Test::Harness). For the following examples I\*(Aqll also need a recent
79 version of Test::Harness installed; 3.14 is current as I write.
80 .PP
81 For the examples I\*(Aqm going to assume that we\*(Aqre working with a
82 \&\*(Aqnormal\*(Aq Perl module distribution. Specifically I\*(Aqll assume that
83 typing \fImake\fR or \fI./Build\fR causes the built, ready\-to\-install module
84 code to be available below ./blib/lib and ./blib/arch and that
85 there\*(Aqs a directory called \*(Aqt\*(Aq that contains our tests. Test::Harness
86 isn\*(Aqt hardwired to that configuration but it saves me from explaining
87 which files live where for each example.
88 .PP
89 Back to \fIprove\fR; like \fImake test\fR it runs a test suite \- but it
90 provides far more control over which tests are executed, in what
91 order and how their results are reported. Typically \fImake test\fR
92 runs all the test scripts below the \*(Aqt\*(Aq directory. To do the same
93 thing with prove I type:
94 .PP
95 .Vb 1
96 \& prove \-rb t
97 .Ve
98 .PP
99 The switches here are \-r to recurse into any directories below \*(Aqt\*(Aq
100 and \-b which adds ./blib/lib and ./blib/arch to Perl\*(Aqs include path
101 so that the tests can find the code they will be testing. If I\*(Aqm
102 testing a module of which an earlier version is already installed
103 I need to be careful about the include path to make sure I\*(Aqm not
104 running my tests against the installed version rather than the new
105 one that I\*(Aqm working on.
106 .PP
107 Unlike \fImake test\fR, typing \fIprove\fR doesn\*(Aqt automatically rebuild
108 my module. If I forget to make before prove I will be testing against
109 older versions of those files \- which inevitably leads to confusion.
110 I either get into the habit of typing
111 .PP
112 .Vb 1
113 \& make && prove \-rb t
114 .Ve
115 .PP
116 or \- if I have no XS code that needs to be built I use the modules
117 below \fIlib\fR instead
118 .PP
119 .Vb 1
120 \& prove \-Ilib \-r t
121 .Ve
122 .PP
123 So far I\*(Aqve shown you nothing that \fImake test\fR doesn\*(Aqt do. Let\*(Aqs
124 fix that.
125 .SS "Saved State"
126 .IX Subsection "Saved State"
127 If I have failing tests in a test suite that consists of more than
128 a handful of scripts and takes more than a few seconds to run it
129 rapidly becomes tedious to run the whole test suite repeatedly as
130 I track down the problems.
131 .PP
132 I can tell prove just to run the tests that are failing like this:
133 .PP
134 .Vb 1
135 \& prove \-b t/this_fails.t t/so_does_this.t
136 .Ve
137 .PP
138 That speeds things up but I have to make a note of which tests are
139 failing and make sure that I run those tests. Instead I can use
140 prove\*(Aqs \-\-state switch and have it keep track of failing tests for
141 me. First I do a complete run of the test suite and tell prove to
142 save the results:
143 .PP
144 .Vb 1
145 \& prove \-rb \-\-state=save t
146 .Ve
147 .PP
148 That stores a machine readable summary of the test run in a file
149 called \*(Aq.prove\*(Aq in the current directory. If I have failures I can
150 then run just the failing scripts like this:
151 .PP
152 .Vb 1
153 \& prove \-b \-\-state=failed
154 .Ve
155 .PP
156 I can also tell prove to save the results again so that it updates
157 its idea of which tests failed:
158 .PP
159 .Vb 1
160 \& prove \-b \-\-state=failed,save
161 .Ve
162 .PP
163 As soon as one of my failing tests passes it will be removed from
164 the list of failed tests. Eventually I fix them all and prove can
165 find no failing tests to run:
166 .PP
167 .Vb 2
168 \& Files=0, Tests=0, 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)
169 \& Result: NOTESTS
170 .Ve
171 .PP
172 As I work on a particular part of my module it\*(Aqs most likely that
173 the tests that cover that code will fail. I\*(Aqd like to run the whole
174 test suite but have it prioritize these \*(Aqhot\*(Aq tests. I can tell
175 prove to do this:
176 .PP
177 .Vb 1
178 \& prove \-rb \-\-state=hot,save t
179 .Ve
180 .PP
181 All the tests will run but those that failed most recently will be
182 run first. If no tests have failed since I started saving state all
183 tests will run in their normal order. This combines full test
184 coverage with early notification of failures.
185 .PP
186 The \-\-state switch supports a number of options; for example to run
187 failed tests first followed by all remaining tests ordered by the
188 timestamps of the test scripts \- and save the results \- I can use
189 .PP
190 .Vb 1
191 \& prove \-rb \-\-state=failed,new,save t
192 .Ve
193 .PP
194 See the prove documentation (type prove \-\-man) for the full list
195 of state options.
196 .PP
197 When I tell prove to save state it writes a file called \*(Aq.prove\*(Aq
198 (\*(Aq_prove\*(Aq on Windows) in the current directory. It\*(Aqs a YAML document
199 so it\*(Aqs quite easy to write tools of your own that work on the saved
200 test state \- but the format isn\*(Aqt officially documented so it might
201 change without (much) warning in the future.
202 .SS "Parallel Testing"
203 .IX Subsection "Parallel Testing"
204 If my tests take too long to run I may be able to speed them up by
205 running multiple test scripts in parallel. This is particularly
206 effective if the tests are I/O bound or if I have multiple CPU
207 cores. I tell prove to run my tests in parallel like this:
208 .PP
209 .Vb 1
210 \& prove \-rb \-j 9 t
211 .Ve
212 .PP
213 The \-j switch enables parallel testing; the number that follows it
214 is the maximum number of tests to run in parallel. Sometimes tests
215 that pass when run sequentially will fail when run in parallel. For
216 example if two different test scripts use the same temporary file
217 or attempt to listen on the same socket I\*(Aqll have problems running
218 them in parallel. If I see unexpected failures I need to check my
219 tests to work out which of them are trampling on the same resource
220 and rename temporary files or add locks as appropriate.
221 .PP
222 To get the most performance benefit I want to have the test scripts
223 that take the longest to run start first \- otherwise I\*(Aqll be waiting
224 for the one test that takes nearly a minute to complete after all
225 the others are done. I can use the \-\-state switch to run the tests
226 in slowest to fastest order:
227 .PP
228 .Vb 1
229 \& prove \-rb \-j 9 \-\-state=slow,save t
230 .Ve
231 .SS "Non\-Perl Tests"
232 .IX Subsection "Non-Perl Tests"
233 The Test Anything Protocol (http://testanything.org/) isn\*(Aqt just
234 for Perl. Just about any language can be used to write tests that
235 output TAP. There are TAP based testing libraries for C, C++, PHP,
236 Python and many others. If I can\*(Aqt find a TAP library for my language
237 of choice it\*(Aqs easy to generate valid TAP. It looks like this:
238 .PP
239 .Vb 4
240 \& 1..3
241 \& ok 1 \- init OK
242 \& ok 2 \- opened file
243 \& not ok 3 \- appended to file
244 .Ve
245 .PP
246 The first line is the plan \- it specifies the number of tests I\*(Aqm
247 going to run so that it\*(Aqs easy to check that the test script didn\*(Aqt
248 exit before running all the expected tests. The following lines are
249 the test results \- \*(Aqok\*(Aq for pass, \*(Aqnot ok\*(Aq for fail. Each test has
250 a number and, optionally, a description. And that\*(Aqs it. Any language
251 that can produce output like that on STDOUT can be used to write
252 tests.
253 .PP
254 Recently I\*(Aqve been rekindling a two\-decades\-old interest in Forth.
255 Evidently I have a masochistic streak that even Perl can\*(Aqt satisfy.
256 I want to write tests in Forth and run them using prove (you can
257 find my gforth TAP experiments at
258 https://svn.hexten.net/andy/Forth/Testing/). I can use the \-\-exec
259 switch to tell prove to run the tests using gforth like this:
260 .PP
261 .Vb 1
262 \& prove \-r \-\-exec gforth t
263 .Ve
264 .PP
265 Alternately, if the language used to write my tests allows a shebang
266 line I can use that to specify the interpreter. Here\*(Aqs a test written
267 in PHP:
268 .PP
269 .Vb 6
270 \& #!/usr/bin/php
271 \& <?php
272 \& print "1..2\en";
273 \& print "ok 1\en";
274 \& print "not ok 2\en";
275 \& ?>
276 .Ve
277 .PP
278 If I save that as t/phptest.t the shebang line will ensure that it
279 runs correctly along with all my other tests.
280 .SS "Mixing it up"
281 .IX Subsection "Mixing it up"
282 Subtle interdependencies between test programs can mask problems \-
283 for example an earlier test may neglect to remove a temporary file
284 that affects the behaviour of a later test. To find this kind of
285 problem I use the \-\-shuffle and \-\-reverse options to run my tests
286 in random or reversed order.
287 .SS "Rolling My Own"
288 .IX Subsection "Rolling My Own"
289 If I need a feature that prove doesn\*(Aqt provide I can easily write my own.
290 .PP
291 Typically you\*(Aqll want to change how TAP gets \fIinput\fR into and \fIoutput\fR
292 from the parser. App::Prove supports arbitrary plugins, and TAP::Harness
293 supports custom \fIformatters\fR and \fIsource handlers\fR that you can load using
294 either prove or Module::Build; there are many examples to base mine on.
295 For more details see App::Prove, TAP::Parser::SourceHandler, and
296 TAP::Formatter::Base.
297 .PP
298 If writing a plugin is not enough, you can write your own test harness; one of
299 the motives for the 3.00 rewrite of Test::Harness was to make it easier to
300 subclass and extend.
301 .PP
302 The Test::Harness module is a compatibility wrapper around TAP::Harness.
303 For new applications I should use TAP::Harness directly. As we\*(Aqll
304 see, prove uses TAP::Harness.
305 .PP
306 When I run prove it processes its arguments, figures out which test
307 scripts to run and then passes control to TAP::Harness to run the
308 tests, parse, analyse and present the results. By subclassing
309 TAP::Harness I can customise many aspects of the test run.
310 .PP
311 I want to log my test results in a database so I can track them
312 over time. To do this I override the summary method in TAP::Harness.
313 I start with a simple prototype that dumps the results as a YAML
314 document:
315 .PP
316 .Vb 1
317 \& package My::TAP::Harness;
318 \&
319 \& use base \*(AqTAP::Harness\*(Aq;
320 \& use YAML;
321 \&
322 \& sub summary {
323 \& my ( $self, $aggregate ) = @_;
324 \& print Dump( $aggregate );
325 \& $self\->SUPER::summary( $aggregate );
326 \& }
327 \&
328 \& 1;
329 .Ve
330 .PP
331 I need to tell prove to use my My::TAP::Harness. If My::TAP::Harness
332 is on Perl\*(Aqs \f(CW@INC\fR include path I can
333 .PP
334 .Vb 1
335 \& prove \-\-harness=My::TAP::Harness \-rb t
336 .Ve
337 .PP
338 If I don\*(Aqt have My::TAP::Harness installed on \f(CW@INC\fR I need to provide
339 the correct path to perl when I run prove:
340 .PP
341 .Vb 1
342 \& perl \-Ilib \`which prove\` \-\-harness=My::TAP::Harness \-rb t
343 .Ve
344 .PP
345 I can incorporate these options into my own version of prove. It\*(Aqs
346 pretty simple. Most of the work of prove is handled by App::Prove.
347 The important code in prove is just:
348 .PP
349 .Vb 1
350 \& use App::Prove;
351 \&
352 \& my $app = App::Prove\->new;
353 \& $app\->process_args(@ARGV);
354 \& exit( $app\->run ? 0 : 1 );
355 .Ve
356 .PP
357 If I write a subclass of App::Prove I can customise any aspect of
358 the test runner while inheriting all of prove\*(Aqs behaviour. Here\*(Aqs
359 myprove:
360 .PP
361 .Vb 2
362 \& #!/usr/bin/env perl use lib qw( lib ); # Add ./lib to @INC
363 \& use App::Prove;
364 \&
365 \& my $app = App::Prove\->new;
366 \&
367 \& # Use custom TAP::Harness subclass
368 \& $app\->harness( \*(AqMy::TAP::Harness\*(Aq );
369 \&
370 \& $app\->process_args( @ARGV ); exit( $app\->run ? 0 : 1 );
371 .Ve
372 .PP
373 Now I can run my tests like this
374 .PP
375 .Vb 1
376 \& ./myprove \-rb t
377 .Ve
378 .SS "Deeper Customisation"
379 .IX Subsection "Deeper Customisation"
380 Now that I know how to subclass and replace TAP::Harness I can
381 replace any other part of the harness. To do that I need to know
382 which classes are responsible for which functionality. Here\*(Aqs a
383 brief guided tour; the default class for each component is shown
384 in parentheses. Normally any replacements I write will be subclasses
385 of these default classes.
386 .PP
387 When I run my tests TAP::Harness creates a scheduler
388 (TAP::Parser::Scheduler) to work out the running order for the
389 tests, an aggregator (TAP::Parser::Aggregator) to collect and analyse
390 the test results and a formatter (TAP::Formatter::Console) to display
391 those results.
392 .PP
393 If I\*(Aqm running my tests in parallel there may also be a multiplexer
394 (TAP::Parser::Multiplexer) \- the component that allows multiple
395 tests to run simultaneously.
396 .PP
397 Once it has created those helpers TAP::Harness starts running the
398 tests. For each test it creates a new parser (TAP::Parser) which
399 is responsible for running the test script and parsing its output.
400 .PP
401 To replace any of these components I call one of these harness
402 methods with the name of the replacement class:
403 .PP
404 .Vb 5
405 \& aggregator_class
406 \& formatter_class
407 \& multiplexer_class
408 \& parser_class
409 \& scheduler_class
410 .Ve
411 .PP
412 For example, to replace the aggregator I would
413 .PP
414 .Vb 1
415 \& $harness\->aggregator_class( \*(AqMy::Aggregator\*(Aq );
416 .Ve
417 .PP
418 Alternately I can supply the names of my substitute classes to the
419 TAP::Harness constructor:
420 .PP
421 .Vb 3
422 \& my $harness = TAP::Harness\->new(
423 \& { aggregator_class => \*(AqMy::Aggregator\*(Aq }
424 \& );
425 .Ve
426 .PP
427 If I need to reach even deeper into the internals of the harness I
428 can replace the classes that TAP::Parser uses to execute test scripts
429 and tokenise their output. Before running a test script TAP::Parser
430 creates a grammar (TAP::Parser::Grammar) to decode the raw TAP into
431 tokens, a result factory (TAP::Parser::ResultFactory) to turn the
432 decoded TAP results into objects and, depending on whether it\*(Aqs
433 running a test script or reading TAP from a file, scalar or array
434 a source or an iterator (TAP::Parser::IteratorFactory).
435 .PP
436 Each of these objects may be replaced by calling one of these parser
437 methods:
438 .PP
439 .Vb 5
440 \& source_class
441 \& perl_source_class
442 \& grammar_class
443 \& iterator_factory_class
444 \& result_factory_class
445 .Ve
446 .SS Callbacks
447 .IX Subsection "Callbacks"
448 As an alternative to subclassing the components I need to change I
449 can attach callbacks to the default classes. TAP::Harness exposes
450 these callbacks:
451 .PP
452 .Vb 5
453 \& parser_args Tweak the parameters used to create the parser
454 \& made_parser Just made a new parser
455 \& before_runtests About to run tests
456 \& after_runtests Have run all tests
457 \& after_test Have run an individual test script
458 .Ve
459 .PP
460 TAP::Parser also supports callbacks; bailout, comment, plan, test,
461 unknown, version and yaml are called for the corresponding TAP
462 result types, ALL is called for all results, ELSE is called for all
463 results for which a named callback is not installed and EOF is
464 called once at the end of each TAP stream.
465 .PP
466 To install a callback I pass the name of the callback and a subroutine
467 reference to TAP::Harness or TAP::Parser\*(Aqs callback method:
468 .PP
469 .Vb 3
470 \& $harness\->callback( after_test => sub {
471 \& my ( $script, $desc, $parser ) = @_;
472 \& } );
473 .Ve
474 .PP
475 I can also pass callbacks to the constructor:
476 .PP
477 .Vb 8
478 \& my $harness = TAP::Harness\->new({
479 \& callbacks => {
480 \& after_test => sub {
481 \& my ( $script, $desc, $parser ) = @_;
482 \& # Do something interesting here
483 \& }
484 \& }
485 \& });
486 .Ve
487 .PP
488 When it comes to altering the behaviour of the test harness there\*(Aqs
489 more than one way to do it. Which way is best depends on my
490 requirements. In general if I only want to observe test execution
491 without changing the harness\*(Aq behaviour (for example to log test
492 results to a database) I choose callbacks. If I want to make the
493 harness behave differently subclassing gives me more control.
494 .SS "Parsing TAP"
495 .IX Subsection "Parsing TAP"
496 Perhaps I don\*(Aqt need a complete test harness. If I already have a
497 TAP test log that I need to parse all I need is TAP::Parser and the
498 various classes it depends upon. Here\*(Aqs the code I need to run a
499 test and parse its TAP output
500 .PP
501 .Vb 1
502 \& use TAP::Parser;
503 \&
504 \& my $parser = TAP::Parser\->new( { source => \*(Aqt/simple.t\*(Aq } );
505 \& while ( my $result = $parser\->next ) {
506 \& print $result\->as_string, "\en";
507 \& }
508 .Ve
509 .PP
510 Alternately I can pass an open filehandle as source and have the
511 parser read from that rather than attempting to run a test script:
512 .PP
513 .Vb 6
514 \& open my $tap, \*(Aq<\*(Aq, \*(Aqtests.tap\*(Aq
515 \& or die "Can\*(Aqt read TAP transcript ($!)\en";
516 \& my $parser = TAP::Parser\->new( { source => $tap } );
517 \& while ( my $result = $parser\->next ) {
518 \& print $result\->as_string, "\en";
519 \& }
520 .Ve
521 .PP
522 This approach is useful if I need to convert my TAP based test
523 results into some other representation. See TAP::Convert::TET
524 (http://search.cpan.org/dist/TAP\-Convert\-TET/) for an example of
525 this approach.
526 .SS "Getting Support"
527 .IX Subsection "Getting Support"
528 The Test::Harness developers hang out on the tapx\-dev mailing
529 list[1]. For discussion of general, language independent TAP issues
530 there\*(Aqs the tap\-l[2] list. Finally there\*(Aqs a wiki dedicated to the
531 Test Anything Protocol[3]. Contributions to the wiki, patches and
532 suggestions are all welcome.
533 .PP
534 [1] <http://www.hexten.net/mailman/listinfo/tapx\-dev>
535 [2] <http://testanything.org/mailman/listinfo/tap\-l>
536 [3] <http://testanything.org/>