Debian Codegraph @

checker/tests/README

raw
1 This document describes how to write and run tests for the Checker
2 Framework. Writing and running tests is useful for bug submitters,
3 checker writers, and Checker Framework maintainers. Users of the Checker
4 Framework and of the checkers packaged with it should read the manual
5 instead; see file ../../docs/manual/manual.html .
6
7
8 How to run all the tests for the Checker Framework
9 ==================================================
10
11 # To run from the checker-framework directory:
12 ./gradlew allTests
13
14 Other Gradle tasks also exist to run a subset of the tests; for example,
15 # Run from the checker-framework directory:
16 ./gradlew :checker:NullnessFbcTest
17 # Run from the checker directory:
18 ../gradlew NullnessFbcTest
19 # To see all tasks
20 ./gradlew tasks
21
22
23 How to run just one test for the Checker Framework
24 ==================================================
25
26 To run an individual test (check one source code file),
27 do something like the following, assuming that the source code
28 file to be checked is AssertNonNullTest.java in directory
29 $CHECKERFRAMEWORK/checker/tests/nullness/ and the checker is
30 org.checkerframework.checker.nullness.NullnessChecker.
31
32 cd $CHECKERFRAMEWORK/checker/tests/nullness
33 (cd $CHECKERFRAMEWORK && ./gradlew compileJava) && $CHECKERFRAMEWORK/checker/bin-devel/javac -processor org.checkerframework.checker.nullness.NullnessChecker -implicit:class AssertNonNullTest.java
34
35 or (to build distribution files)
36
37 cd $CHECKERFRAMEWORK/checker/tests/nullness
38 (cd $CHECKERFRAMEWORK && ./gradlew assemble) && $CHECKERFRAMEWORK/checker/bin/javac -processor org.checkerframework.checker.nullness.NullnessChecker -implicit:class AssertNonNullTest.java
39
40 or, less commonly (to rebuild the JDK):
41
42 cd $CHECKERFRAMEWORK/checker/tests/framework
43 (cd $CHECKERFRAMEWORK && ./gradlew buildJdk assemble -PuseLocalJdk) && $CHECKERFRAMEWORK/checker/bin/javac -processor TestChecker -implicit:class GenericTest10.java
44
45 where the specific checker and command-line arguments are often clear from
46 the directory name but can also be determined from a file such as
47 checker-framework/checker/tests/src/tests/MyTypeSystemTest.java
48 which is the source code for the test itself.
49
50
51 Writing new tests for an existing checker
52 =========================================
53
54 To create a new test case, just place a Java file in the test directory,
55 whose name usually corresponds to the checker name, such as
56 checker-framework/checker/tests/nullness/ . Unless the README file in
57 the test directory specifies otherwise, then the Java file must
58 1. Not issue any javac errors.
59 2. Not declare a class with the same (fully qualified) name as any other class in
60 the test directory.
61 3. Not declare a class with the same (simple) name as any commonly used Java
62 library class such as List.
63
64 The testing framework for the Checker Framework is built on top of JUnit.
65 However, its tests are more like end-to-end system tests than unit tests.
66
67 A checker test case has two parts:
68 1. the Java class to be compiled, and
69 2. a set of expected errors.
70 Both parts can be expressed in one Java file (see below).
71
72 Classes in checker-framework/framework/tests/src/tests/lib can be referenced by
73 tests to check behavior related to unchecked bytecode.
74
75 By convention, when a test is about an issue in the issue tracker, we write
76 a comment at the top of the file, in this format:
77
78 // Test case for issue 266:
79 // https://github.com/typetools/checker-framework/issues/266
80
81 // @skip-test until the issue is fixed
82
83
84 Specifying expected warnings and errors
85 =======================================
86
87 A test case is a Java file that uses stylized comments to indicate expected
88 error messages.
89
90 Suppose that you want to test the Nullness Checker's behavior when
91 type-checking the following Java class:
92
93 class MyNullnessTest {
94 void method() {
95 Object nullable = null;
96 nullable.toString(); // should emit error
97 }
98 }
99
100 The Nullness Checker should report an error for the dereference in line 4.
101 The non-localized message key for such an error is
102 'dereference.of.nullable'. You could learn that by reading the Javadoc (or
103 the source code) for org.checkerframework.checker.nullness.NullnessVisitor,
104 or by creating the test and observing the failure.
105
106 To indicate the expected failure, insert the line
107 // :: error: (<error-message-key>)
108 directly preceding the expected error line.
109 If a warning rather than an error is expected, insert the line
110 // :: warning: (<warning-message-key>)
111 If a stub parser warning is expected, insert the line
112 //warning: StubParser: <stub parser warning>
113 If multiple errors are expected on a single line, duplicate everthing
114 except the "//" comment characters, as in
115 // :: error: (<error-message-key1>) :: error: (<error-message-key2>)
116 If the expected failures line would be very long, you may break it across
117 multiple comment lines.
118 It is permitted to write a "// ::" comment after "{" that was at the end of
119 a line, to indicate a warning on the line immediately after the "{".
120
121 So the final test case would be:
122
123 class MyNullnessTest {
124 void method() {
125 Object nullable = null;
126 // :: error: (dereference.of.nullable)
127 nullable.toString();
128 }
129 }
130
131 The file may appear anywhere in or under
132 checker-framework/checker/tests/nullness/. (You may find it useful to use
133 separate subfolders, such as nullness/tests/nullness/dereference/.) Each
134 checker should have its own folder under checker-framework/checker/tests/,
135 such as checker-framework/checker/tests/interning/,
136 checker-framework/checker/tests/regex/, etc.
137
138 You can indicate an expected warning (as opposed to error) by using
139 "warning:" instead of "error:", as in
140
141 // :: warning: (known.nonnull)
142 assert val != null;
143
144 Multiple expected messages can be given on the same line using the
145 "// :: A :: B :: C" syntax. This example expects both an error and
146 a warning from the same line of code:
147
148 @Regex String s1 = null;
149 // :: error: (assignment.type.incompatible) :: warning: (cast.unsafe)
150 @Regex(3) String s2 = (@Regex(2) String) s;
151
152
153 As an alternative to writing expected errors in the source file using "// ::"
154 syntax, expected errors can be specified in a separate file using the .out
155 file extension. These files contain lines of the following format:
156
157 :19: error: (dereference.of.nullable)
158
159 The number between the colons is the line number of the expected error
160 message. This format is harder to maintain, and we suggest using the
161 in-line comment format.
162
163
164 Writing tests for a new checker or with different command-line arguments
165 ========================================================================
166
167 To create tests for a new checker, mimic some existing checker's tests:
168 * create a top-level test directory, such as
169 checker-framework/checker/tests/regex for the test cases
170 * create a top-level JUnit test in checker-framework/checker/src/test/tests,
171 such as: RegexTest.java
172 It is a subclass of CheckerFrameworkPerDirectoryTest, and its list of checker
173 options must include "-Anomsgtext". (See the API documentation for
174 CheckerFrameworkPerDirectoryTest for more detailed information.)
175 * include "all-systems" as a test directly by adding it to the array created
176 in getTestDirs in the new test class. See
177 checker-framework/framework/tests/all-systems/README for more details about
178 the all-systems tests.
179
180 Different test cases may need to pass different command-line arguments
181 (flags) to the checker -- for instance, to check an optional command-line
182 argument that should not be enabled for every test. Follow the same
183 instructions as for writing tests for a new checker.
184
185 A Gradle task is created for each Junit test class. The task is named the same
186 as the Junit test classname, for example, ./gradlew RegexTest runs the Regex
187 Checker tests.
188
189
190 Disabling a test case
191 =====================
192
193 Write @skip-test anywhere in a test file to disable that test.
194
195 Write @below-java9-jdk-skip-test anywhere in a test file to disable that
196 test if the executing JDK version is lower than 9.
197 This is useful when the test depends on Java 9 language features that
198 need runtime support or depends on Java 9 APIs.
199 Tests that only exercise the static type annotation API do not need
200 this marker.
201
202 To disable all tests for a given type system, annotated the Junit test class
203 with @Ignore.
204
205
206 Annotated JDK
207 =============
208
209 The tests run with the annotated JDK. Keep this in mind when writing tests.
210
211
212 Seeing the javac commands that are run
213 ======================================
214
215 To see the exact javac commands that the Checker Framework test framework
216 runs, use
217 -Pemit.test.debug=true
218 For example:
219 ./gradlew NullnessStubfileTest -Pemit.test.debug=true
220 This may be helpful during debugging.