Answered

Display real number of failures out of total test cases in the set when there are failed-reruns

Hasib Rostaiee

September 09, 2024 21:26

When we have failed tests and rerun them, the number of total tests also increases (it counts number of runs rather than number of tests in the set) and this will confuse users about the actual number of failures and failure percentage.

For example, I have a suite which has 40 test caes which intitially passed with rate of 22/40. When I rerun the test cases, the result became 37/62 which incorrectly interpreted as 37 tests failed out of 62 total test cases. Inside the set result only few test cases failed actually. So we want to see the Result as "37/40" in order to correctly represent how many test in the set passed and have more clear % of failure.

3 comments

Official comment

Jonathan Hotz September 10, 2024 13:56

Hi Hasib,

We are actively working on a big improvement to the executions results, and we heard similar concerns/questions from other users about the execution results information when we conducted user interviews and user testing with our prototype designs. In the new design, we will have a "pass rate" value that shows "test cases that passed / total test cases in the execution set", which will make it easier to understand the final result, along with additional details so you can also see how many passed on the first attempt and how many passed on subsequent attempts.

We're really excited for this, and hope to have it out in the next couple months. Thank you again for posting this!

Joseph Hamdan September 10, 2024 08:27

Hi Hasib,

Thanks for creating a feature request.

While I understand that this is creating some confusion, this field is meant to be a result of execution (how many passed out of the entire number of executions).

We will however take a look at this field and potentially improve it to resolve that confusion.

Regards,
Subject7 Team

Jonathan Hotz December 18, 2024 16:41

In addition to the revised Pass Rate value that I mentioned above, there will also be a graphical widget to help you understand what passed on an initial run, what passed with re-runs, and the final Pass Rate.

Please sign in to leave a comment.