| 1 |
# Guide to YAML::Tiny testing |
| 2 |
|
| 3 |
YAML::Tiny tests use several components: |
| 4 |
|
| 5 |
* .t files |
| 6 |
* Test libraries in t/lib |
| 7 |
* YAML data files in t/data |
| 8 |
* TestML data files in t/tml-* |
| 9 |
|
| 10 |
The use of each is described below. |
| 11 |
|
| 12 |
## .t files |
| 13 |
|
| 14 |
The .t files are standard Perl test files. They may use one or more of the |
| 15 |
test libraries in t/lib. They may only use modules available in Perl 5.8.1 or |
| 16 |
later (and not subsequently deprecated), but they may use newer non-XS versions |
| 17 |
of those modules as necessary to avoid known bugs. |
| 18 |
|
| 19 |
Some .t files have complete inputs/outputs for their tests. Others iterate |
| 20 |
over .tml files in the t/tml-* directories. |
| 21 |
|
| 22 |
A .t file should load Test::More and use `done_testing` at the end |
| 23 |
to so that new tests may be added without needing to also update a test plan. |
| 24 |
|
| 25 |
Currently, the convention is to name .t files matching the pattern |
| 26 |
qr/^\d\d_\w+\.t$/ |
| 27 |
|
| 28 |
## Test libraries |
| 29 |
|
| 30 |
There are currently three test libraries in t/lib. A .t file that uses one or |
| 31 |
more of them should put `use lib 't/lib';` at the top of the .t file. Test |
| 32 |
libraries can assume that if they were loaded, that 't/lib' is already in @INC. |
| 33 |
|
| 34 |
The test libraries are: |
| 35 |
|
| 36 |
* SubtestCompat |
| 37 |
* TestML::Tiny |
| 38 |
* TestBridge |
| 39 |
* TestUtils |
| 40 |
|
| 41 |
The "SubtestCompat" library provides a limited emulation of |
| 42 |
Test::More::subtest that is reasonably compatible back to 0.88. If using |
| 43 |
subtests, you must not set a plan in the subtest and you must use |
| 44 |
done_testing in the *.t file. |
| 45 |
|
| 46 |
The TestML::Tiny library contains functions for parsing and executing TestML |
| 47 |
tests with callbacks. TestML is a data-driven testing language; TestML::Tiny |
| 48 |
implements a small subset of its features. See the section on TestML, below, |
| 49 |
for an example. Generally, bugs should be patched upstream on CPAN and then |
| 50 |
a new Test::Tiny CPAN release can be copied here and pod-stripped. |
| 51 |
|
| 52 |
The TestBridge library contains testing functions for use in .t files or to |
| 53 |
be passed to TestML::Tiny functions as callbacks. Test functions should not |
| 54 |
include `done_testing`. They should use `subtest` for any repetitive testing |
| 55 |
that loops over test cases. Callback should check for the expected test |
| 56 |
points (see below) and skip a TML block if those points are not available. |
| 57 |
|
| 58 |
The TestUtils library contains utility functions. Testing functions should |
| 59 |
not be added here (i.e. nothing that uses Test::More). |
| 60 |
|
| 61 |
## YAML data files in t/data |
| 62 |
|
| 63 |
Files in the t/data directory are intended to test how YAML files are loaded |
| 64 |
and decoded and typically need some custom test code to load the file and see |
| 65 |
if the result matches expectations (successful or not). |
| 66 |
|
| 67 |
If a real-world YAML file cannot be loaded due to character set encoding |
| 68 |
issues, it should be placed in this directory for testing. If a real-world |
| 69 |
YAML file is ASCII or UTF-8 encoded, can be decoded successfully, but has |
| 70 |
problems in parsing, it should be reduced to the smallest sample of YAML that |
| 71 |
demonstrates the parsing problem and added to a .tml file for testing. See |
| 72 |
below for more details. |
| 73 |
|
| 74 |
## TestML quick intro |
| 75 |
|
| 76 |
TestML data files are UTF-8 encoded files with a .tml suffix that contain one |
| 77 |
or more test "blocks". Each block has a test label, and one or more 'test |
| 78 |
points', usually representing input and expected output, and possibly |
| 79 |
additional annotations or flags. |
| 80 |
|
| 81 |
Here is an example of a .tml file with a single block: |
| 82 |
|
| 83 |
# This is a TestML block: (this line is a comment) |
| 84 |
=== This is the test label |
| 85 |
Lines until the first point are ignored |
| 86 |
|
| 87 |
# This is a "block" style point. All non-comment lines until next point |
| 88 |
# are the data for the 'yaml' point. The data ends with newline, and |
| 89 |
# trailing blank lines are trimmed. |
| 90 |
--- yaml |
| 91 |
--- |
| 92 |
foo: bar |
| 93 |
# a comment |
| 94 |
\# not a comment |
| 95 |
|
| 96 |
# This is the second point; "inline" style. The data after the colon goes |
| 97 |
# to end of line. Leading/trailing whitespace is trimmed. |
| 98 |
--- perl: [ { foo => 'bar' } ] |
| 99 |
|
| 100 |
# This is a point whose value is the empty string |
| 101 |
--- a_flag |
| 102 |
|
| 103 |
# This is the next block: |
| 104 |
=== Another test case |
| 105 |
|
| 106 |
The test label is provided on a line beginning with qr/^===/. Test "points" |
| 107 |
are provided in sections beginning with qr/^--- +\w+/. All flush-left comment |
| 108 |
lines are stripped. Lines beginning with '\' are escaped. |
| 109 |
|
| 110 |
Different tests expect different test points in a .tml file, based on the |
| 111 |
specific test callback being used. |
| 112 |
|
| 113 |
Many .tml files have the points 'yaml' and 'perl' as in the example above. The |
| 114 |
'yaml' point is a YAML document and the 'perl' point is a Data::Dumper Perl |
| 115 |
data structure. The test checks whether the YAML parses into a data structure |
| 116 |
identical to the Perl one. The 'a_flag' point is an annotation that the |
| 117 |
testing callback can use to affect the run of a given test. |
| 118 |
|
| 119 |
The semantics of points (including annotations) is specific to the callback |
| 120 |
functions used to process test blocks. |
| 121 |
|
| 122 |
# TestML data files in t/tml-* |
| 123 |
|
| 124 |
TestML data files are organized into three directories: |
| 125 |
|
| 126 |
* t/tml-spec — these test files are provided by the YAML spec maintainers and |
| 127 |
should not be modified except to skip testing features that YAML::Tiny does not |
| 128 |
support |
| 129 |
|
| 130 |
* t/tml-local — these test files are YAML::Tiny's own unit tests; generally new |
| 131 |
test cases for coverage or correctness should be added here; these are |
| 132 |
broken into subdirectories, described later |
| 133 |
|
| 134 |
* t/tml-world — these test files represent "real world" YAML and their |
| 135 |
corresponding expected Perl output |
| 136 |
|
| 137 |
Generally, if a "real world" problem can be isolated to a particular snippet of |
| 138 |
YAML, it's best to add it to a t/tml-local file (or create a new one). If the |
| 139 |
problem can only be seen in the context of the entire YAML document, include it |
| 140 |
in t/tml-world. If the problem relates to encoding, it should be put into |
| 141 |
t/data instead. |
| 142 |
|
| 143 |
# t/tml-local subdirectories |
| 144 |
|
| 145 |
The subdirectories in t/tml-local define four types of tests: |
| 146 |
|
| 147 |
* perl-to-yaml: test that perl data dump to an expected YAML string |
| 148 |
|
| 149 |
* yaml-roundtrip: test that a YAML string loads to an expected perl data |
| 150 |
structure; also tests that the perl data can be dumped and loaded back; |
| 151 |
|
| 152 |
* dump-error: test that certain perl data trigger expected errors |
| 153 |
|
| 154 |
* load-error: test that certain YAML strings trigger expected errors |
| 155 |
|
| 156 |
All .tml files in a t/tml-local directory must have the TestML |
| 157 |
test points required by the corresponding test functions defined |
| 158 |
in the TestBridge library. |
| 159 |
|
| 160 |
Generally, files should be grouped by data type or feature so that |
| 161 |
related tests are kept together. |