Debian Codegraph @

features/runner.hook_errors.feature

raw
1 Feature: Hooks processing in case of errors (exceptions)
2
3 . SPECIFICATION:
4 . * Hook errors in before_all/after_all hook aborts the test run (and fail).
5 . * Hook errors in before_feature/after_feature causes the feature to fail.
6 . * Hook errors in before_feature causes feature scenarios to be skipped (untested).
7 . * Hook errors in before_scenario/after_scenario causes the scenario to fail.
8 . * Hook errors in before_scenario causes scenario steps to be skipped (untested).
9 . * Hook errors in before_step/after_step causes step to fail.
10 . * Hook errors in before_tag/after_tag of a feature causes feature to fail.
11 . * Hook errors in before_tag/after_tag of a scenario causes scenario to fail.
12 . * If a hook error occurs in a "before_xxx" hook,
13 . then the "after_xxx" hook will also be called (to cleanup some stuff).
14 .
15 . NOTE:
16 . The --verbose flag can be used to show the exception traceback.
17
18 @setup
19 Scenario: Feature Setup
20 Given a new working directory
21 And a file named "features/steps/steps.py" with:
22 """
23 from behave import step
24
25 @step(u'{word:w} step passes')
26 def step(context, word):
27 pass
28 """
29 And a file named "features/passing.feature" with:
30 """
31 @foo
32 Feature: Alice
33 @soo
34 Scenario: A1
35 Given a step passes
36 """
37 And a file named "features/environment.py" with:
38 """
39 from __future__ import print_function
40
41 def cause_hook_to_fail():
42 raise RuntimeError("FAIL")
43
44 def before_all(context):
45 userdata = context.config.userdata
46 if userdata.get("HOOK_ERROR_LOC") == "before_all":
47 cause_hook_to_fail()
48
49 def after_all(context):
50 userdata = context.config.userdata
51 if userdata.get("HOOK_ERROR_LOC") == "before_all":
52 print("called_hook:after_all")
53 if userdata.get("HOOK_ERROR_LOC") == "after_all":
54 cause_hook_to_fail()
55
56 def before_feature(context, feature):
57 userdata = context.config.userdata
58 if userdata.get("HOOK_ERROR_LOC") == "before_feature":
59 cause_hook_to_fail()
60
61 def after_feature(context, feature):
62 userdata = context.config.userdata
63 if userdata.get("HOOK_ERROR_LOC") == "before_feature":
64 print("called_hook:after_feature")
65 if userdata.get("HOOK_ERROR_LOC") == "after_feature":
66 cause_hook_to_fail()
67
68 def before_scenario(context, scenario):
69 userdata = context.config.userdata
70 if userdata.get("HOOK_ERROR_LOC") == "before_scenario":
71 cause_hook_to_fail()
72
73 def after_scenario(context, scenario):
74 userdata = context.config.userdata
75 if userdata.get("HOOK_ERROR_LOC") == "before_scenario":
76 print("called_hook:after_scenario")
77 if userdata.get("HOOK_ERROR_LOC") == "after_scenario":
78 cause_hook_to_fail()
79
80 def before_step(context, step):
81 userdata = context.config.userdata
82 if userdata.get("HOOK_ERROR_LOC") == "before_step":
83 cause_hook_to_fail()
84
85 def after_step(context, step):
86 userdata = context.config.userdata
87 if userdata.get("HOOK_ERROR_LOC") == "before_step":
88 print("called_hook:after_step")
89 if userdata.get("HOOK_ERROR_LOC") == "after_step":
90 cause_hook_to_fail()
91
92 def before_tag(context, tag):
93 userdata = context.config.userdata
94 if userdata.get("HOOK_ERROR_LOC") == "before_tag":
95 error_tag = userdata.get("HOOK_ERROR_TAG")
96 if not error_tag or tag == error_tag:
97 cause_hook_to_fail()
98
99 def after_tag(context, tag):
100 userdata = context.config.userdata
101 if userdata.get("HOOK_ERROR_LOC") == "before_tag":
102 error_tag = userdata.get("HOOK_ERROR_TAG")
103 if tag == error_tag:
104 print("called_hook:after_tag: tag=%s" % tag)
105 if userdata.get("HOOK_ERROR_LOC") == "after_tag":
106 error_tag = userdata.get("HOOK_ERROR_TAG")
107 if not error_tag or tag == error_tag:
108 cause_hook_to_fail()
109 """
110 And a file named "behave.ini" with:
111 """
112 [behave]
113 show_skipped = false
114 show_timings = false
115 """
116
117
118 @hook.before_all
119 Scenario: Hook error in before_all
120 When I run "behave -f plain -D HOOK_ERROR_LOC=before_all features/passing.feature"
121 Then it should fail with
122 """
123 HOOK-ERROR in before_all: RuntimeError: FAIL
124 called_hook:after_all
125
126 ABORTED: By user.
127 0 features passed, 0 failed, 0 skipped, 1 untested
128 0 scenarios passed, 0 failed, 0 skipped, 1 untested
129 0 steps passed, 0 failed, 0 skipped, 0 undefined, 1 untested
130 """
131 But note that "the after_all hook is called, too"
132
133
134 @hook.after_all
135 Scenario: Hook error in after_all
136 When I run "behave -f plain -D HOOK_ERROR_LOC=after_all features/passing.feature"
137 Then it should fail with:
138 """
139 Feature: Alice
140
141 Scenario: A1
142 Given a step passes ... passed
143
144 HOOK-ERROR in after_all: RuntimeError: FAIL
145 ABORTED: By user.
146 1 feature passed, 0 failed, 0 skipped
147 1 scenario passed, 0 failed, 0 skipped
148 1 step passed, 0 failed, 0 skipped, 0 undefined
149 """
150
151
152 @hook.before_feature
153 Scenario: Hook error in before_feature
154 When I run "behave -f plain -D HOOK_ERROR_LOC=before_feature features/passing.feature"
155 Then it should fail with:
156 """
157 HOOK-ERROR in before_feature: RuntimeError: FAIL
158 Feature: Alice
159 called_hook:after_feature
160
161 0 features passed, 1 failed, 0 skipped
162 0 scenarios passed, 0 failed, 0 skipped, 1 untested
163 0 steps passed, 0 failed, 0 skipped, 0 undefined, 1 untested
164 """
165 But note that "the after_feature hook is called, too."
166
167
168 @hook.after_feature
169 Scenario: Hook error in after_feature
170 When I run "behave -f plain -D HOOK_ERROR_LOC=after_feature features/passing.feature"
171 Then it should fail with:
172 """
173 Feature: Alice
174 Scenario: A1
175 Given a step passes ... passed
176 HOOK-ERROR in after_feature: RuntimeError: FAIL
177
178 0 features passed, 1 failed, 0 skipped
179 1 scenario passed, 0 failed, 0 skipped
180 1 step passed, 0 failed, 0 skipped, 0 undefined
181 """
182
183
184 @hook.before_scenario
185 Scenario: Hook error in before_scenario
186 When I run "behave -f plain -D HOOK_ERROR_LOC=before_scenario features/passing.feature"
187 Then it should fail with:
188 """
189 Feature: Alice
190
191 HOOK-ERROR in before_scenario: RuntimeError: FAIL
192 Scenario: A1
193 called_hook:after_scenario
194
195 Failing scenarios:
196 features/passing.feature:4 A1
197
198 0 features passed, 1 failed, 0 skipped
199 0 scenarios passed, 1 failed, 0 skipped
200 0 steps passed, 0 failed, 0 skipped, 0 undefined, 1 untested
201 """
202 But note that "the after_scenario hook is called, too."
203
204
205 @hook.after_scenario
206 Scenario: Hook error in after_scenario
207 When I run "behave -f plain -D HOOK_ERROR_LOC=after_scenario features/passing.feature"
208 Then it should fail with:
209 """
210 Feature: Alice
211
212 Scenario: A1
213 Given a step passes ... passed
214 HOOK-ERROR in after_scenario: RuntimeError: FAIL
215
216
217 Failing scenarios:
218 features/passing.feature:4 A1
219
220 0 features passed, 1 failed, 0 skipped
221 0 scenarios passed, 1 failed, 0 skipped
222 1 step passed, 0 failed, 0 skipped, 0 undefined
223 """
224
225 @hook.before_step
226 Scenario: Hook error in before_step
227 When I run "behave -f plain -D HOOK_ERROR_LOC=before_step features/passing.feature"
228 Then it should fail with:
229 """
230 Feature: Alice
231
232 Scenario: A1
233 Given a step passes ... failed
234
235 Captured stdout:
236 HOOK-ERROR in before_step: RuntimeError: FAIL
237 called_hook:after_step
238
239 Failing scenarios:
240 features/passing.feature:4 A1
241
242 0 features passed, 1 failed, 0 skipped
243 0 scenarios passed, 1 failed, 0 skipped
244 0 steps passed, 1 failed, 0 skipped, 0 undefined
245 """
246 But note that "the after_step hook is called, too."
247
248
249 @hook.after_step
250 Scenario: Hook error in after_step
251 When I run "behave -f plain -D HOOK_ERROR_LOC=after_step features/passing.feature"
252 Then it should fail with:
253 """
254 Feature: Alice
255
256 Scenario: A1
257 Given a step passes ... failed
258
259 Captured stdout:
260 HOOK-ERROR in after_step: RuntimeError: FAIL
261
262 Failing scenarios:
263 features/passing.feature:4 A1
264
265 0 features passed, 1 failed, 0 skipped
266 0 scenarios passed, 1 failed, 0 skipped
267 0 steps passed, 1 failed, 0 skipped, 0 undefined
268 """
269
270
271 @hook.before_tag
272 Scenario: Hook error in before_tag for feature
273 When I run "behave -f plain -D HOOK_ERROR_LOC=before_tag -D HOOK_ERROR_TAG=foo features/passing.feature"
274 Then it should fail with:
275 """
276 HOOK-ERROR in before_tag(tag=foo): RuntimeError: FAIL
277 Feature: Alice
278 called_hook:after_tag: tag=foo
279
280 0 features passed, 1 failed, 0 skipped
281 0 scenarios passed, 0 failed, 0 skipped, 1 untested
282 0 steps passed, 0 failed, 0 skipped, 0 undefined, 1 untested
283 """
284 And the command output should contain:
285 """
286 HOOK-ERROR in before_tag(tag=foo): RuntimeError: FAIL
287 Feature: Alice
288 """
289 But note that "the hook-error in before_tag of the feature causes it to fail and be skipped (untested)"
290 And note that "the after_tag hook is still called"
291
292
293 @hook.after_tag
294 Scenario: Hook error in after_tag for feature
295 When I run "behave -f plain -D HOOK_ERROR_LOC=after_tag -D HOOK_ERROR_TAG=foo features/passing.feature"
296 Then it should fail with:
297 """
298 Feature: Alice
299
300 Scenario: A1
301 Given a step passes ... passed
302 HOOK-ERROR in after_tag(tag=foo): RuntimeError: FAIL
303
304 0 features passed, 1 failed, 0 skipped
305 1 scenario passed, 0 failed, 0 skipped
306 1 step passed, 0 failed, 0 skipped, 0 undefined
307 """
308 But note that "the hook-error in after_tag of the scenario causes it to fail"
309
310
311 @hook.before_tag
312 Scenario: Hook error in before_tag for scenario
313 When I run "behave -f plain -D HOOK_ERROR_LOC=before_tag -D HOOK_ERROR_TAG=soo features/passing.feature"
314 Then it should fail with:
315 """
316 Feature: Alice
317 HOOK-ERROR in before_tag(tag=soo): RuntimeError: FAIL
318 Scenario: A1
319 called_hook:after_tag: tag=soo
320
321 Failing scenarios:
322 features/passing.feature:4 A1
323
324 0 features passed, 1 failed, 0 skipped
325 0 scenarios passed, 1 failed, 0 skipped
326 0 steps passed, 0 failed, 0 skipped, 0 undefined, 1 untested
327 """
328 But note that "the hook-error in before_tag of the scenario causes it to fail and be skipped (untested)"
329 And note that "the after_tag hook is still called"
330
331
332 @hook.after_tag
333 Scenario: Hook error in after_tag for scenario
334 When I run "behave -f plain -D HOOK_ERROR_LOC=after_tag -D HOOK_ERROR_TAG=soo features/passing.feature"
335 Then it should fail with:
336 """
337 Feature: Alice
338
339 Scenario: A1
340 Given a step passes ... passed
341 HOOK-ERROR in after_tag(tag=soo): RuntimeError: FAIL
342
343 Failing scenarios:
344 features/passing.feature:4 A1
345
346 0 features passed, 1 failed, 0 skipped
347 0 scenarios passed, 1 failed, 0 skipped
348 1 step passed, 0 failed, 0 skipped, 0 undefined
349 """
350 But note that "the hook-error in after_tag of the scenario causes it to fail"
351
352
353 @skipped.hook.after_feature
354 Scenario: Skipped feature with potential hook error (hooks are not run)
355
356 This goes unnoticed because hooks are not run for a skipped feature/scenario.
357 NOTE: Except if before_feature(), before_scenario() hook skips the feature/scenario.
358
359 When I run "behave -f plain -D HOOK_ERROR_LOC=after_feature -t ~@foo features/passing.feature"
360 Then it should pass with:
361 """
362 0 features passed, 0 failed, 1 skipped
363 0 scenarios passed, 0 failed, 1 skipped
364 0 steps passed, 0 failed, 1 skipped, 0 undefined
365 """
366 But note that "hooks are not executed for skipped features/scenarios"
367
368
369 Scenario: Show hook error details (traceback)
370 When I run "behave -f plain -D HOOK_ERROR_LOC=before_feature --verbose features/passing.feature"
371 Then it should fail with:
372 """
373 HOOK-ERROR in before_feature: RuntimeError: FAIL
374 """
375 And the command output should contain:
376 """
377 self.hooks[name](context, *args)
378 File "features/environment.py", line 21, in before_feature
379 cause_hook_to_fail()
380 File "features/environment.py", line 4, in cause_hook_to_fail
381 raise RuntimeError("FAIL")
382 """
383 But note that "the traceback caused by the hook-error is shown"