Incorrect total "Execution Duration" for the execution set

Hasib Rostaiee

January 29, 2020 21:37

How does "Execution Duration" is calculated for an execution set? Apparently, right now it's the cumulative of "Execution Duration" of all individual tests in the set, however since the tests are running in parallel, we need this time to represent the time from start to end of an execution.

Below execution shows "15286" seconds of "Execution Duration" which is 4+ hours, however to total time execution completed was around 40 mins (from 15:45 to 16:25):

Note: "Execution Duration" for individual tests are correct.

Thanks,
Hasib

5 comments

Joseph Hamdan January 31, 2020 17:32

Hasib,

Thanks for your post. We are currently evaluating it and will provide a response as soon as possible.

Regards,

Subject7 Team

Hasib Rostaiee March 16, 2020 19:29

Hi Joseph,

Is there any update on this issue. An accurate execution duration time is helpful for us to used it for some metrics we need.

Joseph Hamdan March 17, 2020 16:19

Hi Hasib,

Sorry for the delay, this is what we are planning to do in 8.11.0 which is due in less than two weeks:

1. For an execution, we will be adding how long the entire execution took from start to finish. For example, suppose you have an execution set with 10 test cases where every test case takes 5 minutes. With 1 VM, this should take 50 minutes, with 10 VMs, this should take 5 minutes. Deepening on how many VMs a pool will have, we will be reporting the effective execution time.
2. For a test case execution, we will be adding endTime-startTime

3. For an execution, we will be adding endTime-startTime

4. For legacy reasons, we will be keeping the sum of total the steps.

Regards,

Subject7 Team

Hasib Rostaiee April 23, 2020 17:18

Hi Joseph,

Are these changes going to be available on 8.11 release?

Jonathan Hotz April 23, 2020 20:11

Hi Hasib.

Test case execution already has actual duration that is calculated as (end date - start date).

Unfortunately, same actual duration for execution set execution is not planned for 8.11/8.12. Probably we will add it in 8.13.0. We collect ideas for better calculation in special cases, for example, like re-runs.

Please sign in to leave a comment.