Answered

The "Result" column in execution Results, doesn't display correct results if there are re-run of failed tests

Hasib Rostaiee

April 22, 2020 19:12

When there are test cases fail, the "Result" column of execution set show number of passed out of all tests, however when the failed ones rerun, the numbers in the results doesn't display correct numbers (especially when any of the re-run tests fails).

For example, an execution set had the result of 24/33 (9 test cases failed), after re-run the failure, all tests passed except one, the result now displayed as 32/42. This give an interpretation that 10 test fails, however only 1 test is failed.

We use this calculation in the metrics and it's hard to to know how many test cases exactly failed.

2 comments

Rex Feizi April 23, 2020 16:40

Hassib,

Logically, in that set, there were 10 failures (9 on the first run and one on the second run). Showing 41/42 would be incorrect and could confuse other users. We try to reflect exactly what happened and it's up to the users to interpret them as they see fit to their specific projects.

That being said, I see the benefit of what you are suggesting. We could potentially introduce a new column of "Overall Status" similar to the one in test case execution. Let me discuss this with the team, meanwhile, I would encourage you to submit an enhancement request, and demonstrate your use-case to further make your case.

--Rex

Hasib Rostaiee April 23, 2020 17:16

Thank you Rex,

I understand that design is to get any number of failed and passed inside the execution regardless of how many times tests run.The regular execution runs when there are no re-run on them is easier to understand. However, when re-run involves, the numbers interpretation may be confusing and the graphs we add in our dashboard pages would also display that 10 tests are failed (which is after re-run only 1 test is failing).

Having an overall Result which counts the re-run status (and not their older failed status), would be giving good numbers and is easier to interpret what's going on. I'll post this in the "Feature Requests".

-Hasib

Please sign in to leave a comment.