Answered

Display real number of failures out of total test cases in the set when there are failed-reruns

Hasib Rostaiee

September 09, 2024 21:26

When we have failed tests and rerun them, the number of total tests also increases (it counts number of runs rather than number of tests in the set) and this will confuse users about the actual number of failures and failure percentage.

For example, I have a suite which has 40 test caes which intitially passed with rate of 22/40. When I rerun the test cases, the result became 37/62 which incorrectly interpreted as 37 tests failed out of 62 total test cases. Inside the set result only few test cases failed actually. So we want to see the Result as "37/40" in order to correctly represent how many test in the set passed and have more clear % of failure.

4 comments

Official comment

Jonathan Hotz September 10, 2024 13:56

Hi Hasib,

We are actively working on a big improvement to the executions results, and we heard similar concerns/questions from other users about the execution results information when we conducted user interviews and user testing with our prototype designs. In the new design, we will have a "pass rate" value that shows "test cases that passed / total test cases in the execution set", which will make it easier to understand the final result, along with additional details so you can also see how many passed on the first attempt and how many passed on subsequent attempts.

We're really excited for this, and hope to have it out in the next couple months. Thank you again for posting this!

Joseph Hamdan September 10, 2024 08:27

Hi Hasib,

Thanks for creating a feature request.

While I understand that this is creating some confusion, this field is meant to be a result of execution (how many passed out of the entire number of executions).

We will however take a look at this field and potentially improve it to resolve that confusion.

Regards,
Subject7 Team

Jonathan Hotz December 18, 2024 16:41

In addition to the revised Pass Rate value that I mentioned above, there will also be a graphical widget to help you understand what passed on an initial run, what passed with re-runs, and the final Pass Rate.

Hasib Rostaiee April 29, 2025 15:00

Hi Jonathan,
I see recently, widget added for the pass rate and it has Intial, Re-run, and Final. It's nice to have feature, however we are looking for a pass rate of (number of passed test/out of all tests in the execution). Which still not considered.

Here is an example of howe current widget works and we are expecting for the Final.

Total Number of Tests in Execution: 40
Total Number of Tests Passed Initally = 34 ++++> Resulted in 34/40 = 85% Pass
We had 3 test rerun of the failures
The final Results is counted as 37/43 instead of 37/40, which shows 86% of Pass rate in stead of the actual 37/40 which is 92% Rate.

Would you be able to consider this change? We should not add reruns to total number of tests, only consider their passed one. Basically we want to know how many tests passed out of execution , not howmany runs passed.

Thanks

Please sign in to leave a comment.