Some context…
One of the biggest challenges about functional automated tests is to look for strategies which help us to save time on automated tests execution. Although you can implement mapping strategies that help you save some milliseconds looking for screen elements or taking care of implicit an explicit waits. When you execute all tests cases in unattended way, if at least one of those test cases fails, you must re run your entire suite losing all the time you spent, and making your test execution costs double or more, although the application does’n have new changes. To solve this problem, I’ll show you how to use the rerun plugin of Cucumber to enable you to rerun only the failed test on your last execution.
If you make a manual execution of automated tests, you have entire control on which runner you’d like to use but, when a test automated execution is orchestrated by an automated process, you need to have a way to ensure you just rerun tests that makes sense to run if your application doesn’t have source changes, to enable the auto-healing process, you could implement the rerun plugin of Cucumber to run only failed tests.
Let’s illustrate the problem using the following situation:
We have an automated process which has a stage to run automated tests that may take 2 hours to end one execution, if one (or in this case 2) tests fail, the pipeline must run all tests indifferent if the application has changes or if the application has none. You will waste a lot of time.
The solution…
Our automated tests are created using Serenity BDD and Cucumber and you can take advantage of it to solve the problem and I’ll give you a step-by-step process.
Step 1: Configure rerun plugin on your test runner
Step 2: Let’s create a specific runner to rerun all failed scenarios stored in ./target/rerunFailed folder, or specific path
Optional step: You can apply the rerun plugin on the runner you created to execute failed tests, this will allow you to rerun failed scenarios on a retest execution.
Step 3: When you need to rerun failed scenarios you can run the specific runner created to rerun failed scenarios as we configured our example, we will run RerunFailedScenarios
gradle --tests "test.app.runners.RerunFailedScenarios"How does it work?
The rerun plugin will create a file named as you configure on each runner to specify the feature path and the line when the failed scenario starts.
plugin = {"rerun:path/to/failed/tests.txt"}To keep in mid our example, the plugin must create the path:
The file will be filled with the feature path and the line when the failed scenario starts, as you can see, every scenario is separated with character “:”
If you configure all your tests runners with the same path on the rerun plugin configuration, each runner will override all information on your exit file in order to execute your last runner execution. If you want to preserve all runners failed scenarios information you need to use different paths, I suggest you to create a file with the name of your feature or runner in order to maintain as much failed scenarios as posible to rerun.
The runner created on step 3 will run every scenario stored on the path created to maintain the information of failed scenarios, in our example it will be:
features = "@target/rerunFailed"Optional: You can specify what file you will run in your RerunFailedScenarios runner
features = "@target/rerunFailed/FailedScenarios.txt"
Special thanks to Alexi Ruben Hernandez Ochoa who guided me in the search for the possibilities that Cucumber offers us with its plugins.