Debian Codegraph @

features/formatter.rerun.feature

raw
1 @sequential
2 Feature: Rerun Formatter
3
4 To simplify to run the scenarios that have failed during the last test run
5 As a tester
6 I want that behave generates the necessary information for me.
7
8 . SOLUTION:
9 .
10 . Put RerunFormatter into "behave.ini" configuration file:
11 .
12 . # -- file:behave.ini
13 . [behave]
14 . format = rerun
15 . outfiles = rerun.txt
16 .
17 . Then a "rerun.txt" file is generated during each test run
18 . that contains the file locations of the failing scenarios, like:
19 .
20 . # -- file:rerun.txt
21 . # RERUN: Failing scenarios during last test run.
22 . features/alice.feature:10
23 . features/alice.feature:42
24 .
25 . To rerun the failing scenarios of the last test run, use:
26 .
27 . behave @rerun.txt
28
29
30 @setup
31 Scenario: Feature Setup
32 Given a new working directory
33 And a file named "features/steps/steps.py" with:
34 """
35 from behave import step
36
37 @step('a step passes')
38 def step_passes(context):
39 pass
40
41 @step('a step fails')
42 def step_fails(context):
43 assert False, "XFAIL-STEP"
44 """
45 And a file named "features/alice.feature" with:
46 """
47 Feature: Alice
48
49 Scenario: Passing
50 Given a step passes
51 When a step passes
52 Then a step passes
53
54 @xfail
55 Scenario: When-step fails
56 Given a step passes
57 When a step fails
58 Then a step passes
59
60 @xfail
61 Scenario: Then-step fails
62 Given a step passes
63 When a step passes
64 Then a step fails
65 """
66 And a file named "features/bob.feature" with:
67 """
68 Feature: Bob
69
70 @xfail
71 Scenario: Failing early
72 Given a step fails
73 When a step passes
74 Then a step fails
75
76 Scenario: Passing
77 Given a step passes
78
79 @xfail
80 Scenario: Failing
81 When a step passes
82 Then a step fails
83 """
84
85 @usecase.step1
86 Scenario: Rerun file is generated when failures occur
87 When I run "behave -f rerun -o rerun.txt -f plain -T features/"
88 Then it should fail with:
89 """
90 0 features passed, 2 failed, 0 skipped
91 2 scenarios passed, 4 failed, 0 skipped
92 """
93 And the command output should contain:
94 """
95 Failing scenarios:
96 features/alice.feature:9 When-step fails
97 features/alice.feature:15 Then-step fails
98 features/bob.feature:4 Failing early
99 features/bob.feature:13 Failing
100 """
101 And the file "rerun.txt" should contain:
102 """
103 # -- RERUN: 4 failing scenarios during last test run.
104 features/alice.feature:9
105 features/alice.feature:15
106 features/bob.feature:4
107 features/bob.feature:13
108 """
109
110 @usecase.step2
111 Scenario: Use rerun file during next test run to test only failing scenarios
112 Given a file named "rerun.txt" with:
113 """
114 # -- RERUN: 4 failing scenarios during last test run.
115 features/alice.feature:9
116 features/alice.feature:15
117 features/bob.feature:4
118 features/bob.feature:13
119 """
120 When I run "behave -f plain -T --no-skipped @rerun.txt"
121 Then it should fail with:
122 """
123 0 features passed, 2 failed, 0 skipped
124 0 scenarios passed, 4 failed, 2 skipped
125 """
126 And the command output should contain:
127 """
128 Feature: Alice
129
130 Scenario: When-step fails
131 Given a step passes ... passed
132 When a step fails ... failed
133 Assertion Failed: XFAIL-STEP
134
135 Scenario: Then-step fails
136 Given a step passes ... passed
137 When a step passes ... passed
138 Then a step fails ... failed
139 Assertion Failed: XFAIL-STEP
140
141 Feature: Bob
142
143 Scenario: Failing early
144 Given a step fails ... failed
145 Assertion Failed: XFAIL-STEP
146
147 Scenario: Failing
148 When a step passes ... passed
149 Then a step fails ... failed
150 Assertion Failed: XFAIL-STEP
151 """
152 And the command output should contain:
153 """
154 Failing scenarios:
155 features/alice.feature:9 When-step fails
156 features/alice.feature:15 Then-step fails
157 features/bob.feature:4 Failing early
158 features/bob.feature:13 Failing
159 """
160
161
162 Scenario: Rerun file is deleted when no failures occur
163
164 The rerun output file should be deleted
165 if the test run was successful and no failures occurred.
166
167 Given an empty file named "rerun.txt"
168 When I run "behave -f rerun -o rerun.txt -f plain -T --no-skipped --tags=~@xfail features/"
169 Then it should pass with:
170 """
171 2 features passed, 0 failed, 0 skipped
172 2 scenarios passed, 0 failed, 4 skipped
173 """
174 And a file named "rerun.txt" should not exist
175
176
177 Scenario: Use RerunFormatter with output=stdout
178 When I run "behave -f rerun -f plain -T --no-skipped features/"
179 Then it should fail with:
180 """
181 0 features passed, 2 failed, 0 skipped
182 2 scenarios passed, 4 failed, 0 skipped
183 """
184 And the command output should contain:
185 """
186 # -- RERUN: 4 failing scenarios during last test run.
187 features/alice.feature:9
188 features/alice.feature:15
189 features/bob.feature:4
190 features/bob.feature:13
191 """
192
193
194 @with.behave_configfile
195 Scenario: Use rerun file with RerunFormatter in behave configuration file
196 Given a file named "behave.ini" with:
197 """
198 [behave]
199 format = rerun
200 outfiles = rerun.txt
201 show_timings = false
202 show_skipped = false
203 """
204 When I run "behave -f plain features/"
205 Then it should fail with:
206 """
207 0 features passed, 2 failed, 0 skipped
208 2 scenarios passed, 4 failed, 0 skipped
209 """
210 And the file "rerun.txt" should contain:
211 """
212 # -- RERUN: 4 failing scenarios during last test run.
213 features/alice.feature:9
214 features/alice.feature:15
215 features/bob.feature:4
216 features/bob.feature:13
217 """
218 When I run "behave -f plain @rerun.txt"
219 Then it should fail with:
220 """
221 0 features passed, 2 failed, 0 skipped
222 0 scenarios passed, 4 failed, 2 skipped
223 """
224 And the command output should contain:
225 """
226 Feature: Alice
227
228 Scenario: When-step fails
229 Given a step passes ... passed
230 When a step fails ... failed
231 Assertion Failed: XFAIL-STEP
232
233 Scenario: Then-step fails
234 Given a step passes ... passed
235 When a step passes ... passed
236 Then a step fails ... failed
237 Assertion Failed: XFAIL-STEP
238
239 Feature: Bob
240
241 Scenario: Failing early
242 Given a step fails ... failed
243 Assertion Failed: XFAIL-STEP
244
245 Scenario: Failing
246 When a step passes ... passed
247 Then a step fails ... failed
248 Assertion Failed: XFAIL-STEP
249 """
250 And the file "rerun.txt" should contain:
251 """
252 # -- RERUN: 4 failing scenarios during last test run.
253 features/alice.feature:9
254 features/alice.feature:15
255 features/bob.feature:4
256 features/bob.feature:13
257 """
258
259 @sad.case
260 @with.behave_configfile
261 Scenario: Two RerunFormatter use same output file
262 Given a file named "behave.ini" with:
263 """
264 [behave]
265 format = rerun
266 outfiles = rerun.txt
267 """
268 When I run "behave -f rerun -o rerun.txt -f plain features/"
269 Then it should fail with:
270 """
271 0 features passed, 2 failed, 0 skipped
272 2 scenarios passed, 4 failed, 0 skipped
273 """
274 And the file "rerun.txt" should contain:
275 """
276 # -- RERUN: 4 failing scenarios during last test run.
277 features/alice.feature:9
278 features/alice.feature:15
279 features/bob.feature:4
280 features/bob.feature:13
281 """
282 But the file "rerun.txt" should not contain:
283 """
284 # -- RERUN: 4 failing scenarios during last test run.
285 features/alice.feature:9
286 features/alice.feature:15
287 features/bob.feature:4
288 features/bob.feature:13
289
290 # -- RERUN: 4 failing scenarios during last test run.
291 features/alice.feature:9
292 features/alice.feature:15
293 features/bob.feature:4
294 features/bob.feature:13
295 """
296 And note that "the second RerunFormatter overwrites the output of the first one"