Debian Codegraph @

features/scenario.exclude_from_run.feature

raw
1 Feature: Exclude Scenario from Test Run
2
3 As a test writer
4 I want sometimes to decide at runtime
5 that a scenario is excluded from a test run
6 So that the command-line configuration becomes simpler
7 (and auto-configuration is supported).
8
9 . MECHANISM:
10 . The "before_scenario()" hook can decide just before a scenario should run
11 . that the scenario should be excluded from the test-run.
12 . NOTE: Hooks are not called in dry-run mode.
13 .
14 . RATIONALE:
15 . There are certain situations where it is better to skip a scenario
16 . than to run and fail the scenario.
17 .
18 . Reasons for these cases are of often test environment related:
19 . * test environment does not fulfill the desired criteria
20 . * used testbed does not fulfill test requirements
21 .
22 . Instead of providing the exclude-scenario selection on the command-line,
23 . the test (environment) and configuration logic should determine
24 . if a test should be excluded (as auto-configuration functionality).
25 .
26 . EXAMPLE:
27 . Certain scenarios should not run on Windows (or Linux, ...).
28 .
29 . EVALUATION ORDER:
30 . Before the user can exclude a scenario from a test-run,
31 . additional mechanisms decide, if the scenario is part of the selected run-set.
32 . These are:
33 . * tags
34 . * ...
35 .
36 . RELATED:
37 . * features/feature.exclude_from_run.feature
38
39
40 @setup
41 Scenario:
42 Given a new working directory
43 And a file named "features/example.feature" with:
44 """
45 Feature:
46
47 Scenario: Alice
48 Given a step passes
49 When another step passes
50 Then some step passes
51
52 Scenario: Bob and Alice
53 Given some step passes
54
55 Scenario: Bob
56 Given another step passes
57 """
58 And a file named "features/steps/steps.py" with:
59 """
60 from behave import step
61
62 @step('{word:w} step passes')
63 def step_passes(context, word):
64 pass
65
66 @step('{word:w} step fails')
67 def step_fails(context, word):
68 assert False, "XFAIL-STEP"
69 """
70
71 @use_hook.before_scenario
72 Scenario: Exclude a scenario from the test run (using: before_scenario() hook)
73 Given a file named "features/environment.py" with:
74 """
75 import sys
76
77 def should_exclude_scenario(context, scenario):
78 if scenario.name.startswith("Alice"):
79 return True
80 return False
81
82 def before_scenario(context, scenario):
83 if should_exclude_scenario(context, scenario):
84 sys.stdout.write("EXCLUDED-BY-USER: Scenario %s\n" % scenario.name)
85 scenario.skip()
86 """
87 When I run "behave -f plain -T features/example.feature"
88 Then it should pass with:
89 """
90 2 scenarios passed, 0 failed, 1 skipped
91 2 steps passed, 0 failed, 3 skipped, 0 undefined
92 """
93 And the command output should contain:
94 """
95 EXCLUDED-BY-USER: Scenario Alice
96 """
97
98
99 @use_hook.before_feature
100 Scenario: Exclude a scenario from the test run (using: before_feature() hook)
101 Given a file named "features/environment.py" with:
102 """
103 import sys
104
105 def should_exclude_scenario(scenario):
106 if "Alice" in scenario.name: # MATCHES: Alice, Bob and Alice
107 return True
108 return False
109
110 def before_feature(context, feature):
111 # -- NOTE: walk_scenarios() flattens ScenarioOutline.scenarios
112 for scenario in feature.walk_scenarios():
113 if should_exclude_scenario(scenario):
114 sys.stdout.write("EXCLUDED-BEFORE-FEATURE: Scenario %s\n" % scenario.name)
115 scenario.skip()
116 """
117 When I run "behave -f plain -T features/example.feature"
118 Then it should pass with:
119 """
120 1 scenario passed, 0 failed, 2 skipped
121 1 step passed, 0 failed, 4 skipped, 0 undefined
122 """
123 And the command output should contain:
124 """
125 EXCLUDED-BEFORE-FEATURE: Scenario Alice
126 EXCLUDED-BEFORE-FEATURE: Scenario Bob and Alice
127 """
128
129
130 Scenario: Skip scenario in a step
131
132 Expect that remaining steps of the scenario are skipped.
133 The skipping step is also marked as skipped
134 to better detect scenarios that are partly executed and then skipped
135 (otherwise a passed step would hide that the remaining steps are skipped).
136
137 Given a file named "features/skip_scenario.feature" with:
138 """
139 Feature:
140
141 Scenario: Alice2
142 Given a step passes
143 And the assumption "location:Wonderland" is not met
144 When another step passes
145 Then some step passes
146
147 Scenario: Bob and Alice2
148 Given some step passes
149 When I skip the remaining scenario
150 Then another step passes
151 """
152 And a file named "features/steps/skip_scenario_steps.py" with:
153 """
154 from behave import given, step
155
156 @given('the assumption "{name}" is not met')
157 def step_assumption_not_met(context, name):
158 context.scenario.skip("Assumption %s not met" % name)
159
160 @step('I skip the remaining scenario')
161 def step_skip_scenario(context):
162 context.scenario.skip()
163 """
164 And a file named "features/environment.py" with:
165 """
166 # -- OVERRIDE WITH EMPTY-ENVIRONMENT.
167 """
168 When I run "behave -f plain -T features/skip_scenario.feature"
169 Then it should pass with:
170 """
171 0 scenarios passed, 0 failed, 2 skipped
172 2 steps passed, 0 failed, 5 skipped, 0 undefined
173 """
174 And the command output should contain:
175 """
176 Scenario: Alice2
177 Given a step passes ... passed
178 And the assumption "location:Wonderland" is not met ... skipped
179
180 Scenario: Bob and Alice2
181 Given some step passes ... passed
182 When I skip the remaining scenario ... skipped
183 """
184 But note that "the step that skipped the scenario is also marked as skipped"
185
186
187 Scenario: Skip scenario in after_scenario hook
188
189 Expect that scenario is not marked as skipped
190 because it was already executed (with status: passed, failed, ...).
191
192 Given a file named "features/pass_and_fail.feature" with:
193 """
194 Feature:
195
196 Scenario: Passing
197 Given a step passes
198
199 Scenario: Failing
200 Given some step passes
201 When a step fails
202 """
203 And a file named "features/environment.py" with:
204 """
205 def before_all(context):
206 context.config.setup_logging()
207
208 def after_scenario(context, scenario):
209 scenario.skip("AFTER-SCENARIO")
210 """
211 When I run "behave -f plain -T features/pass_and_fail.feature"
212 Then it should fail with:
213 """
214 1 scenario passed, 1 failed, 0 skipped
215 2 steps passed, 1 failed, 0 skipped, 0 undefined
216 """
217 But note that "the scenarios are not marked as skipped (SKIP-TOO-LATE)"