Debian Codegraph @

features/runner.scenario_autoretry.feature

raw
1 Feature: Auto-retry failed scenarios (a number of times)
2
3 As a tester in an unreliable environment
4 I want that failed scenarios are automatically retried a number of times
5 So that these scenarios are passing
6
7 NOTE:
8 This problem should occur rather seldom, but may occur when the server or
9 network infrastructure, etc. is unreliable and causes random failures.
10
11
12 @setup
13 Scenario: Feature Setup
14 Given a new working directory
15 And a file named "features/steps/unreliable_steps.py" with:
16 """
17 from behave import given, step
18
19 unreliable_step_passed_calls = 0
20
21 @step(u'{word:w} unreliable step fails sometimes')
22 def step_unreliable_step_fails_sometimes(context, word):
23 global unreliable_step_passed_calls
24 userdata = context.config.userdata
25 fault_pos = userdata.getint("UNRELIABLE_FAULTPOS", 0)
26 if fault_pos <= 0:
27 return
28
29 # -- NORMAL CASE: Unreliable step should every N calls.
30 unreliable_step_passed_calls += 1
31 if unreliable_step_passed_calls >= fault_pos:
32 unreliable_step_passed_calls = 0
33 assert False, "UNRELIABLE-STEP FAILURE"
34 """
35 And a file named "features/steps/reuse_steps.py" with:
36 """
37 import behave4cmd0.passing_steps
38 """
39 And a file named "features/environment.py" with:
40 """
41 from behave.contrib.scenario_autoretry import patch_scenario_with_autoretry
42
43 def before_feature(context, feature):
44 for scenario in feature.scenarios:
45 if "autoretry" in scenario.effective_tags:
46 patch_scenario_with_autoretry(scenario, max_attempts=2)
47 """
48 And a file named "features/unreliable.feature" with:
49 """
50 @autoretry
51 Feature: Alice
52 Scenario: A1
53 Given a step passes
54 When an unreliable step fails sometimes
55 Then another unreliable step fails sometimes
56
57 Scenario: A2
58 Given an unreliable step fails sometimes
59 Then another step passes
60 """
61 And a file named "behave.ini" with:
62 """
63 [behave]
64 show_timings = false
65 """
66
67
68 Scenario: Retry failed scenarios
69 When I run "behave -f plain -D UNRELIABLE_FAULTPOS=3 features/unreliable.feature"
70 Then it should pass with:
71 """
72 2 scenarios passed, 0 failed, 0 skipped
73 5 steps passed, 0 failed, 0 skipped, 0 undefined
74 """
75 And the command output should contain:
76 """
77 AUTO-RETRY SCENARIO (attempt 1)
78 """
79 But the command output should not contain:
80 """
81 AUTO-RETRY SCENARIO (attempt 2)
82 """
83 And the command output should contain:
84 """
85 Scenario: A1
86 Given a step passes ... passed
87 When an unreliable step fails sometimes ... passed
88 Then another unreliable step fails sometimes ... passed
89
90 Scenario: A2
91 Given an unreliable step fails sometimes ... failed
92 Assertion Failed: UNRELIABLE-STEP FAILURE
93 AUTO-RETRY SCENARIO (attempt 1)
94
95 Scenario: A2
96 Given an unreliable step fails sometimes ... passed
97 Then another step passes ... passed
98 """
99
100 Scenario: Scenarios failures are accepted after N=2 attempts
101 When I run "behave -f plain -D UNRELIABLE_FAULTPOS=2 features/unreliable.feature"
102 Then it should fail with:
103 """
104 1 scenario passed, 1 failed, 0 skipped
105 4 steps passed, 1 failed, 0 skipped, 0 undefined
106 """
107 And the command output should not contain:
108 """
109 AUTO-RETRY SCENARIO (attempt 2)
110 SCENARIO-FAILURE ACCEPTED (after 2 attempts)
111 """
112 And the command output should contain:
113 """
114 Scenario: A1
115 Given a step passes ... passed
116 When an unreliable step fails sometimes ... passed
117 Then another unreliable step fails sometimes ... failed
118 Assertion Failed: UNRELIABLE-STEP FAILURE
119 AUTO-RETRY SCENARIO (attempt 1)
120
121 Scenario: A1
122 Given a step passes ... passed
123 When an unreliable step fails sometimes ... passed
124 Then another unreliable step fails sometimes ... failed
125 Assertion Failed: UNRELIABLE-STEP FAILURE
126 AUTO-RETRY SCENARIO FAILED (after 2 attempts)
127
128 Scenario: A2
129 Given an unreliable step fails sometimes ... passed
130 Then another step passes ... passed
131 """