Debian Codegraph @

doc/antlr-project-testing.md

raw
1 # ANTLR project unit tests
2
3 ## Introduction
4
5 Because ANTLR supports multiple target languages, the unit tests are broken into two groups: the unit tests that test the tool itself (in `tool-testsuite`) and the unit tests that test the parser runtimes (in `antlr4/runtime-testsuite`). The tool tests are straightforward because they are Java code testing Java code; see the section at the bottom of this file.
6
7 The runtime tests must be specified in a generic fashion to work across language targets. Furthermore, we must test the various targets from Java. This usually means Java launching processes to compile, say, C++ and run parsers.
8
9 As of 4.6, we use [a Java descriptor object](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) to describe each runtime test. Unit tests are grouped together into categories such as [ParserExecDescriptors](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/ParserExecDescriptors.java), which has multiple nested descriptor objects, one per test. For example, here is the start of that file:
10
11 ```java
12 public class ParserExecDescriptors {
13 public static class APlus extends BaseParserTestDescriptor {
14 public String input = "a b c";
15 public String output = "abc\n";
16 public String errors = "";
17 public String startRule = "a";
18 public String grammarName = "T";
19
20 /**
21 grammar T;
22 a : ID+ {
23 <writeln("$text")>
24 };
25 ID : 'a'..'z'+;
26 WS : (' '|'\n') -> skip;
27 */
28 @CommentHasStringValue
29 public String grammar;
30 }
31 ```
32
33 The mysterious `@CommentHasStringValue` annotation is a bit of a hack that allows multi-line strings in Java. This kung fu is required so that we can use Java classes rather than StringTemplate group files to specify runtime tests (the legacy system used those and it was hard to get them right). Here are all the [Runtime test descriptors](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors) organized into groups.
34
35 The grammars are strings representing StringTemplates (`ST` objects) so `<writeln("$text")>` will get replace when the unit test file is generated (`Test.java`, `Test.cs`, ...). The `writeln` template must be defined per target. Here are all of the
36 [Target templates for runtime tests](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates).
37
38 ## Requirements
39
40 In order to perform the tests on all target languages, you need to have the following languages installed:
41
42 * `mono` (e.g., `brew install mono`) on non-Windows boxes (on Windows it uses the Microsoft .net stack). Also must [`xbuild` the runtime](https://github.com/antlr/antlr4/blob/master/doc/releasing-antlr.md) before tests will run; see below
43 * `nodejs`
44 * Python 2.7
45 * Python 3.6
46 * Go
47 * Swift 4 (via XCode 10.x) tested currently only osx
48 * clang (for C++ target)
49 *
50 To **install into local repository** `~/.m2/repository/org/antlr`, do this:
51
52 ```bash
53 $ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
54 $ mvn install -DskipTests=true # make sure all artifacts are visible on this machine
55 ```
56
57 Now, make sure C# runtime is built and installed locally.
58
59 ```bash
60 cd ~/antlr/code/antlr4/runtime/CSharp/src
61 rm -rf `find . -name '{obj,bin}'`
62 dotnet build -c Release runtime/CSharp/src/Antlr4.csproj
63 ```
64
65 C++ test rig automatically builds C++ runtime during tests. Others don't need a prebuilt lib.
66
67
68 ## Running the runtime tests
69
70 A single test rig is sufficient to test all targets against all descriptors using the [junit parameterized tests](https://github.com/junit-team/junit4/wiki/parameterized-tests) mechanism. But, that is inconvenient because we often want to test just a single target or perhaps even just a single test within a single group of a single target. I have automatically generated a bunch of
71 [Target runtime test rigs](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/test/org/antlr/v4/test/runtime) that allow developers such flexibility. For example, here are the Python3 test rigs in intellij:
72
73 <img src=images/testrigs.png width=300>
74
75 And the result of testing the entire subdirectory:
76
77 <img src=images/python3-tests.png width=400>
78
79 From `mvn`, on the commandline, you will see:
80
81 ```bash
82 $ cd antlr4
83 $ mvn test
84 ...
85 -------------------------------------------------------
86 T E S T S
87 -------------------------------------------------------
88 Running org.antlr.v4.test.runtime.csharp.TestCompositeLexers
89 dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451
90 Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068612451/Antlr4.Test.mono.csproj
91 dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081
92 Starting build /usr/bin/xbuild /p:Configuration=Release /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeLexers-1446068615081/Antlr4.Test.mono.csproj
93 Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.451 sec
94 Running org.antlr.v4.test.runtime.csharp.TestCompositeParsers
95 dir /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864
96 antlr reports warnings from [-visitor, -Dlanguage=CSharp, -o, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -lib, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864, -encoding, UTF-8, /var/folders/s1/h3qgww1x0ks3pb30l8t1wgd80000gn/T/TestCompositeParsers-1446068615864/M.g4]
97 ...
98 [INFO] ------------------------------------------------------------------------
99 [INFO] Reactor Summary:
100 [INFO]
101 [INFO] ANTLR 4 ............................................ SUCCESS [ 0.445 s]
102 [INFO] ANTLR 4 Runtime .................................... SUCCESS [ 3.392 s]
103 [INFO] ANTLR 4 Tool ....................................... SUCCESS [ 1.373 s]
104 [INFO] ANTLR 4 Maven plugin ............................... SUCCESS [ 1.519 s]
105 [INFO] ANTLR 4 Runtime Test Annotations ................... SUCCESS [ 0.086 s]
106 [INFO] ANTLR 4 Runtime Test Processors .................... SUCCESS [ 0.014 s]
107 [INFO] ANTLR 4 Runtime Tests (2nd generation) ............. SUCCESS [06:39 min]
108 [INFO] ANTLR 4 Tool Tests ................................. SUCCESS [ 6.922 s]
109 [INFO] ------------------------------------------------------------------------
110 [INFO] BUILD SUCCESS
111 [INFO] ------------------------------------------------------------------------
112 [INFO] Total time: 06:53 min
113 [INFO] Finished at: 2016-11-16T15:36:56-08:00
114 [INFO] Final Memory: 44M/458M
115 [INFO] ------------------------------------------------------------------------
116 ```
117
118 Note: That is actually result of running the much faster:
119
120 ```bash
121 mvn -Dparallel=methods -DthreadCount=4 install
122 ```
123
124 ## Running test subsets
125
126 *From the `runtime-testsuite` dir*
127
128 ### Run one test group across targets
129
130 ```bash
131 $ cd runtime-testsuite
132 $ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux
133 $ mvn -Dtest=TestParserExec test
134 -------------------------------------------------------
135 T E S T S
136 -------------------------------------------------------
137 Running org.antlr.v4.test.runtime.cpp.TestParserExec
138 ...
139 Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 114.283 sec
140 Running org.antlr.v4.test.runtime.csharp.TestParserExec
141 ...
142 ```
143
144 Or run all lexer related tests:
145
146 ```
147 $ cd runtime-testsuite
148 $ mvn -Dtest=Test*Lexer* test
149 -------------------------------------------------------
150 T E S T S
151 -------------------------------------------------------
152 Running org.antlr.v4.test.runtime.cpp.TestCompositeLexers
153 ...
154 ```
155
156 ### Run all tests for a single target
157
158 ```bash
159 $ cd runtime-testsuite
160 $ mvn -Dtest=java.* test
161 ...
162 ```
163
164 Or run all lexer related tests in Java target only:
165
166 ```bash
167 $ cd runtime-testsuite
168 $ mvn -Dtest=java.*Lexer* test
169 ...
170 -------------------------------------------------------
171 T E S T S
172 -------------------------------------------------------
173 Running org.antlr.v4.test.runtime.java.TestCompositeLexers
174 Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec
175 Running org.antlr.v4.test.runtime.java.TestLexerErrors
176 Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.376 sec
177 Running org.antlr.v4.test.runtime.java.TestLexerExec
178 Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.07 sec
179 Running org.antlr.v4.test.runtime.java.TestSemPredEvalLexer
180 Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.255 sec
181
182 Results :
183
184 Tests run: 59, Failures: 0, Errors: 0, Skipped: 0
185 ```
186
187 ## Testing in parallel
188
189 Use this to run tests in parallel:
190
191 ```bash
192 $ export MAVEN_OPTS="-Xmx1G"
193 $ mvn -Dparallel=methods -DthreadCount=4 test
194 ...
195 -------------------------------------------------------
196 T E S T S
197 -------------------------------------------------------
198 Concurrency config is parallel='methods', perCoreThreadCount=true, threadCount=4, useUnlimitedThreads=false
199 ...
200 ```
201
202 This can be combined with other `-D` above.
203
204 ## Adding a runtime test
205
206 To add a new runtime test, first determine which [group of tests](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors) it belongs to. Then, add a new [RuntimeTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) implementation by subclassing one of:
207
208 * [BaseParserTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseParserTestDescriptor.java); see example [APlus](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/ParserExecDescriptors.java#L7).
209 * [BaseDiagnosticParserTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseDiagnosticParserTestDescriptor) if you want to test parser diagnostic output; see [example output](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/FullContextParsingDescriptors.java#L16).
210 * [BaseCompositeParserTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseCompositeParserTestDescriptor.java); see example [BringInLiteralsFromDelegate](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/CompositeParsersDescriptors.java#L11)
211 * [BaseLexerTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseLexerTestDescriptor.java); see example [ActionPlacement](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/LexerExecDescriptors.java#L12).
212 * [BaseCompositeLexerTestDescriptor](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/BaseCompositeLexerTestDescriptor.java); see example [LexerDelegatorInvokesDelegateRule](https://github.com/antlr/antlr4/blob/master/runtime-testsuite/test/org/antlr/v4/test/runtime/descriptors/CompositeLexersDescriptors.java#L11)
213
214
215 Each descriptor object describes the following mandatory elements for the test:
216
217 * the test type
218 * the grammar
219 * the start rule
220 * the input text to parse or lex
221 * the expected output
222 * the expected errors
223
224 Your best bet is to find a similar test in the appropriate group and then copy and paste the descriptor object, creating a new nested class within the test group class. Modify the field definitions to suit your new problem.
225
226 If you need to create a whole new group of tests, it requires a new descriptor class; call it `XDescriptors`. Then, in each [target subdirectory](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/test/org/antlr/v4/test/runtime), you need to create a new test rig `TestX.java` file:
227
228 ```java
229 package org.antlr.v4.test.runtime.java;
230
231 import org.antlr.v4.test.runtime.BaseRuntimeTest;
232 import org.antlr.v4.test.runtime.RuntimeTestDescriptor;
233 import org.antlr.v4.test.runtime.descriptors.ListenersDescriptors;
234 import org.junit.runner.RunWith;
235 import org.junit.runners.Parameterized;
236
237 @RunWith(Parameterized.class)
238 public class TestX extends BaseRuntimeTest {
239 public TestX(RuntimeTestDescriptor descriptor) {
240 super(descriptor,new Base<TARGET>Test());
241 }
242
243 @Parameterized.Parameters(name="{0}")
244 public static RuntimeTestDescriptor[] getAllTestDescriptors() {
245 return BaseRuntimeTest.getRuntimeTestDescriptors(XDescriptors.class, "<TARGET>");
246 }
247 }
248 ```
249
250 where `<TARGET>` is replaced with Java, Cpp, CSharp, Python2, ... in the various subdirectories.
251
252 ### Ignoring tests
253
254 In order to turn off a test for a particular target, we need to use the `ignore` method. Given a target name, a descriptor object can decide whether to ignore the test. This is not always convenient but it is fully general and works well for the one case we have now where we have to ignore `Visitor` tests in all targets except JavaScript.
255
256 ### Target API/library testing
257
258 Some parts of the runtime API need to be tested with code written specifically in the target language. For example, you can see all of the Java runtime API tests here:
259
260 [https://github.com/antlr/antlr4/tree/master/runtime-testsuite/test/org/antlr/v4/test/runtime/java/api](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/test/org/antlr/v4/test/runtime/java/api)
261
262 Notice that it is under an `api` dir. The directory above is where all of the `Test*` files go.
263
264 ### Cross-language actions embedded within grammars
265
266 To get:
267
268 ```
269 System.out.println($set.stop);
270 ```
271
272 Use instead the language-neutral:
273
274 ```
275 <writeln("$set.stop")>
276 ```
277
278 Template file [runtime-testsuite/resources/org/antlr/v4/test/runtime/templates/Java.test.stg](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime/templates/Java.test.stg) has templates like:
279
280 ```
281 writeln(s) ::= <<System.out.println(<s>);>>
282 ```
283
284 that translate generic operations to target-specific language statements or expressions.
285
286 ## Adding an ANTLR tool unit test
287
288 Just go into the appropriate Java test class in dir [antlr4/tool-testsuite/test/org/antlr/v4/test/tool](https://github.com/antlr/antlr4/tree/master/tool-testsuite/test/org/antlr/v4/test/tool) and add your unit test.
289
290