| 1 |
This document describes how to write and run tests for the Checker |
| 2 |
Framework. Writing and running tests is useful for bug submitters, |
| 3 |
checker writers, and Checker Framework maintainers. Users of the Checker |
| 4 |
Framework and of the checkers packaged with it should read the manual |
| 5 |
instead; see file ../../docs/manual/manual.html . |
| 6 |
|
| 7 |
|
| 8 |
How to run all the tests for the Checker Framework |
| 9 |
================================================== |
| 10 |
|
| 11 |
# To run from the checker-framework directory: |
| 12 |
./gradlew allTests |
| 13 |
|
| 14 |
Other Gradle tasks also exist to run a subset of the tests; for example, |
| 15 |
# Run from the checker-framework directory: |
| 16 |
./gradlew :checker:NullnessFbcTest |
| 17 |
# Run from the checker directory: |
| 18 |
../gradlew NullnessFbcTest |
| 19 |
# To see all tasks |
| 20 |
./gradlew tasks |
| 21 |
|
| 22 |
|
| 23 |
How to run just one test for the Checker Framework |
| 24 |
================================================== |
| 25 |
|
| 26 |
To run an individual test (check one source code file), |
| 27 |
do something like the following, assuming that the source code |
| 28 |
file to be checked is AssertNonNullTest.java in directory |
| 29 |
$CHECKERFRAMEWORK/checker/tests/nullness/ and the checker is |
| 30 |
org.checkerframework.checker.nullness.NullnessChecker. |
| 31 |
|
| 32 |
cd $CHECKERFRAMEWORK/checker/tests/nullness |
| 33 |
(cd $CHECKERFRAMEWORK && ./gradlew compileJava) && $CHECKERFRAMEWORK/checker/bin-devel/javac -processor org.checkerframework.checker.nullness.NullnessChecker -implicit:class AssertNonNullTest.java |
| 34 |
|
| 35 |
or (to build distribution files) |
| 36 |
|
| 37 |
cd $CHECKERFRAMEWORK/checker/tests/nullness |
| 38 |
(cd $CHECKERFRAMEWORK && ./gradlew assemble) && $CHECKERFRAMEWORK/checker/bin/javac -processor org.checkerframework.checker.nullness.NullnessChecker -implicit:class AssertNonNullTest.java |
| 39 |
|
| 40 |
or, less commonly (to rebuild the JDK): |
| 41 |
|
| 42 |
cd $CHECKERFRAMEWORK/checker/tests/framework |
| 43 |
(cd $CHECKERFRAMEWORK && ./gradlew buildJdk assemble -PuseLocalJdk) && $CHECKERFRAMEWORK/checker/bin/javac -processor TestChecker -implicit:class GenericTest10.java |
| 44 |
|
| 45 |
where the specific checker and command-line arguments are often clear from |
| 46 |
the directory name but can also be determined from a file such as |
| 47 |
checker-framework/checker/tests/src/tests/MyTypeSystemTest.java |
| 48 |
which is the source code for the test itself. |
| 49 |
|
| 50 |
|
| 51 |
Writing new tests for an existing checker |
| 52 |
========================================= |
| 53 |
|
| 54 |
To create a new test case, just place a Java file in the test directory, |
| 55 |
whose name usually corresponds to the checker name, such as |
| 56 |
checker-framework/checker/tests/nullness/ . Unless the README file in |
| 57 |
the test directory specifies otherwise, then the Java file must |
| 58 |
1. Not issue any javac errors. |
| 59 |
2. Not declare a class with the same (fully qualified) name as any other class in |
| 60 |
the test directory. |
| 61 |
3. Not declare a class with the same (simple) name as any commonly used Java |
| 62 |
library class such as List. |
| 63 |
|
| 64 |
The testing framework for the Checker Framework is built on top of JUnit. |
| 65 |
However, its tests are more like end-to-end system tests than unit tests. |
| 66 |
|
| 67 |
A checker test case has two parts: |
| 68 |
1. the Java class to be compiled, and |
| 69 |
2. a set of expected errors. |
| 70 |
Both parts can be expressed in one Java file (see below). |
| 71 |
|
| 72 |
Classes in checker-framework/framework/tests/src/tests/lib can be referenced by |
| 73 |
tests to check behavior related to unchecked bytecode. |
| 74 |
|
| 75 |
By convention, when a test is about an issue in the issue tracker, we write |
| 76 |
a comment at the top of the file, in this format: |
| 77 |
|
| 78 |
// Test case for issue 266: |
| 79 |
// https://github.com/typetools/checker-framework/issues/266 |
| 80 |
|
| 81 |
// @skip-test until the issue is fixed |
| 82 |
|
| 83 |
|
| 84 |
Specifying expected warnings and errors |
| 85 |
======================================= |
| 86 |
|
| 87 |
A test case is a Java file that uses stylized comments to indicate expected |
| 88 |
error messages. |
| 89 |
|
| 90 |
Suppose that you want to test the Nullness Checker's behavior when |
| 91 |
type-checking the following Java class: |
| 92 |
|
| 93 |
class MyNullnessTest { |
| 94 |
void method() { |
| 95 |
Object nullable = null; |
| 96 |
nullable.toString(); // should emit error |
| 97 |
} |
| 98 |
} |
| 99 |
|
| 100 |
The Nullness Checker should report an error for the dereference in line 4. |
| 101 |
The non-localized message key for such an error is |
| 102 |
'dereference.of.nullable'. You could learn that by reading the Javadoc (or |
| 103 |
the source code) for org.checkerframework.checker.nullness.NullnessVisitor, |
| 104 |
or by creating the test and observing the failure. |
| 105 |
|
| 106 |
To indicate the expected failure, insert the line |
| 107 |
// :: error: (<error-message-key>) |
| 108 |
directly preceding the expected error line. |
| 109 |
If a warning rather than an error is expected, insert the line |
| 110 |
// :: warning: (<warning-message-key>) |
| 111 |
If a stub parser warning is expected, insert the line |
| 112 |
//warning: StubParser: <stub parser warning> |
| 113 |
If multiple errors are expected on a single line, duplicate everthing |
| 114 |
except the "//" comment characters, as in |
| 115 |
// :: error: (<error-message-key1>) :: error: (<error-message-key2>) |
| 116 |
If the expected failures line would be very long, you may break it across |
| 117 |
multiple comment lines. |
| 118 |
It is permitted to write a "// ::" comment after "{" that was at the end of |
| 119 |
a line, to indicate a warning on the line immediately after the "{". |
| 120 |
|
| 121 |
So the final test case would be: |
| 122 |
|
| 123 |
class MyNullnessTest { |
| 124 |
void method() { |
| 125 |
Object nullable = null; |
| 126 |
// :: error: (dereference.of.nullable) |
| 127 |
nullable.toString(); |
| 128 |
} |
| 129 |
} |
| 130 |
|
| 131 |
The file may appear anywhere in or under |
| 132 |
checker-framework/checker/tests/nullness/. (You may find it useful to use |
| 133 |
separate subfolders, such as nullness/tests/nullness/dereference/.) Each |
| 134 |
checker should have its own folder under checker-framework/checker/tests/, |
| 135 |
such as checker-framework/checker/tests/interning/, |
| 136 |
checker-framework/checker/tests/regex/, etc. |
| 137 |
|
| 138 |
You can indicate an expected warning (as opposed to error) by using |
| 139 |
"warning:" instead of "error:", as in |
| 140 |
|
| 141 |
// :: warning: (known.nonnull) |
| 142 |
assert val != null; |
| 143 |
|
| 144 |
Multiple expected messages can be given on the same line using the |
| 145 |
"// :: A :: B :: C" syntax. This example expects both an error and |
| 146 |
a warning from the same line of code: |
| 147 |
|
| 148 |
@Regex String s1 = null; |
| 149 |
// :: error: (assignment.type.incompatible) :: warning: (cast.unsafe) |
| 150 |
@Regex(3) String s2 = (@Regex(2) String) s; |
| 151 |
|
| 152 |
|
| 153 |
As an alternative to writing expected errors in the source file using "// ::" |
| 154 |
syntax, expected errors can be specified in a separate file using the .out |
| 155 |
file extension. These files contain lines of the following format: |
| 156 |
|
| 157 |
:19: error: (dereference.of.nullable) |
| 158 |
|
| 159 |
The number between the colons is the line number of the expected error |
| 160 |
message. This format is harder to maintain, and we suggest using the |
| 161 |
in-line comment format. |
| 162 |
|
| 163 |
|
| 164 |
Writing tests for a new checker or with different command-line arguments |
| 165 |
======================================================================== |
| 166 |
|
| 167 |
To create tests for a new checker, mimic some existing checker's tests: |
| 168 |
* create a top-level test directory, such as |
| 169 |
checker-framework/checker/tests/regex for the test cases |
| 170 |
* create a top-level JUnit test in checker-framework/checker/src/test/tests, |
| 171 |
such as: RegexTest.java |
| 172 |
It is a subclass of CheckerFrameworkPerDirectoryTest, and its list of checker |
| 173 |
options must include "-Anomsgtext". (See the API documentation for |
| 174 |
CheckerFrameworkPerDirectoryTest for more detailed information.) |
| 175 |
* include "all-systems" as a test directly by adding it to the array created |
| 176 |
in getTestDirs in the new test class. See |
| 177 |
checker-framework/framework/tests/all-systems/README for more details about |
| 178 |
the all-systems tests. |
| 179 |
|
| 180 |
Different test cases may need to pass different command-line arguments |
| 181 |
(flags) to the checker -- for instance, to check an optional command-line |
| 182 |
argument that should not be enabled for every test. Follow the same |
| 183 |
instructions as for writing tests for a new checker. |
| 184 |
|
| 185 |
A Gradle task is created for each Junit test class. The task is named the same |
| 186 |
as the Junit test classname, for example, ./gradlew RegexTest runs the Regex |
| 187 |
Checker tests. |
| 188 |
|
| 189 |
|
| 190 |
Disabling a test case |
| 191 |
===================== |
| 192 |
|
| 193 |
Write @skip-test anywhere in a test file to disable that test. |
| 194 |
|
| 195 |
Write @below-java9-jdk-skip-test anywhere in a test file to disable that |
| 196 |
test if the executing JDK version is lower than 9. |
| 197 |
This is useful when the test depends on Java 9 language features that |
| 198 |
need runtime support or depends on Java 9 APIs. |
| 199 |
Tests that only exercise the static type annotation API do not need |
| 200 |
this marker. |
| 201 |
|
| 202 |
To disable all tests for a given type system, annotated the Junit test class |
| 203 |
with @Ignore. |
| 204 |
|
| 205 |
|
| 206 |
Annotated JDK |
| 207 |
============= |
| 208 |
|
| 209 |
The tests run with the annotated JDK. Keep this in mind when writing tests. |
| 210 |
|
| 211 |
|
| 212 |
Seeing the javac commands that are run |
| 213 |
====================================== |
| 214 |
|
| 215 |
To see the exact javac commands that the Checker Framework test framework |
| 216 |
runs, use |
| 217 |
-Pemit.test.debug=true |
| 218 |
For example: |
| 219 |
./gradlew NullnessStubfileTest -Pemit.test.debug=true |
| 220 |
This may be helpful during debugging. |