2013 R1: huge delay when running test lists

Progress® Telerik® Test Studio® Feedback Portal

Create an account Log In

Back to Feed

Request a Feature Report a Bug

Completed

Last Updated: 10 Sep 2013 08:45 by ADMIN

ADMIN

Ivaylo

Created on: 26 Jul 2013 14:24

Type: Bug Report

2013 R1: huge delay when running test lists

When executing test lists in a larger project there is huge delay before the test list start executing. There is a long wait between each test in the list as well.

Steps to reproduce provided in the internal description.

22 comments

ADMIN

Shtilianov

Posted on: 10 Sep 2013 08:45

Hi Ewin, I did try to reproduce your problem but I had no luck reproducing it. It will help if you create a new feedback item where you give more details like - what type of execution are you using (remote/local), are you using dynamic/static or performance test lists, do you use recycle browser or any other specific test list setting.

Ewin

Posted on: 05 Sep 2013 22:35

This is a repost.  Could you guys check about clicking stop after starting the test list?

Something I just noticed when trying to run lists.  When I had selected multiple test lists and click play,  I had to stop them for something.  When click to stop it, the test lists were still executing.  I stopped them by using the restful api http://url:8009 with http delete verb.

ADMIN

Shtilianov

Posted on: 04 Sep 2013 14:11

Alexander I have created a new feedback item about this problem http://feedback.telerik.com/Project/117/Feedback/Details/57612-logon-dialog-is-not-handled-in-a-timely-fashion-if-dialog-monitor-is-enabled-for
Seems that there is a more general problem with dialog monitor and test lists, I'll update the new feedback item when we have a fix ready (which will be soon). Meanwhile the only workaround is to add Manager.DialogMonitor.Start(); back as it was and disable dialog monitor from test list settings. I'm sorry for the incovinience.

Ewin

Posted on: 04 Sep 2013 13:39

Here is my last piece of suggestion for you Alexander since I was only hoping that it would be a easy thing is:

Right before you add the dialog to the dialogmonitor, use dialog.InitializationTime = 500;

I imagine that the logondialog should appear after your navigateto action. So try adding: 

System.Threading.Thread.Sleep(250);

before and after the waituntilhandled to allow the test to catch up.  Try using waituntilhandled with some length of time as the client timeout.  I usually use 20000.  

Hope this help

algot

Posted on: 04 Sep 2013 13:35

I deleted string
Manager.DialogMonitor.Start();
but tests still works slow and login takes up to 25-30 seconds near the end of test list.
My test list contains 35 tests and login is slower till the end of the test list.

>>As for the second problem with pop-ups I advise you to write a seperate ticket so we can track the problem and find a solution.
This problem reproduces from time to time so it will take time to gather info about it.

ADMIN

Shtilianov

Posted on: 04 Sep 2013 12:01

Hi Alexander, the problem is that by running the code you say "Manager.DialogMonitor.Start();" which is not needed and when you execute it in a test list where "dialog monitoring" is not disabled tow instances of DialogHandler are created which leads to the problem. This was introduced in 2013R1 and we'll investigate it. Anyway in order to solve your problem you should either open test list settings and check "Disable Dialog monitoring" or preferably you should just remove Manager.DialogMonitor.Start(); from your code. As for the second problem with pop-ups I advise you to write a seperate ticket so we can track the problem and find a solution.

algot

Posted on: 04 Sep 2013 09:23

Some clarification about my tests.
I use following code to login
LogonDialog dialog = LogonDialog.CreateLogonDialog(ActiveBrowser, domain + "\\" + userName, password, DialogButton.OK);
Manager.DialogMonitor.AddDialog(dialog);
Manager.DialogMonitor.Start();
Manager.ActiveBrowser.NavigateTo(Utility.Context.BaseUrl);
Manager.ActiveBrowser.Window.Maximize();
dialog.WaitUntilHandled();
Manager.DialogMonitor.RemoveDialog(dialog);

and new version of Test Studio could login to my app up to 15 second and previous version of test studio spends about 1-2 second on login.

Also sometimes generic "Connect to pop-up" step can't connect to pop-up which was loaded successfully. It looks like pop-up is opened quickly and Test Studio is still waiting for it.

Ewin

Posted on: 03 Sep 2013 21:55

Konstantin, I do this because there were plenty of times where the test lists is STUCK on the dialog and does not fail to go to the next test or do anything.  I have been doing this ever since the new dialog() was deprecated but is no longer.  

Compared to a recorded dialog, the initialization of the dialog and the actual waituntilhandled() are so far apart.  It is easier for manage a dialog waituntilhandled is next to the initialization of the dialog.

David

Posted on: 03 Sep 2013 19:07

We had noticed the performance issues prior to the internal build - went away after the internal build, but only for Static Lists.  Dynamic lists don't appear to run on an upgraded project when run from the IDE (they run with the command line runner).  Support ticket on its way.

ADMIN

Konstantin Petkov

Posted on: 03 Sep 2013 18:05

Alexander, do you get delays because of timeouts in handling dialogs? Also do you get these failures in test list execution only or also if you run the test in quick execution mode?

Ewin, do you do that to speed up the test execution in general? Or is that specific for the internal build only?

Thanks guys!

Ewin

Posted on: 03 Sep 2013 16:33

When I run my test lists, I had the disable the dialog monitor on the test list.  Also on the individual tests, I would manually turn the dialog monitoring on and off only when it needed to be on.  All the dialog boxes I have are only from the coded steps where it captured the specific button or action to cause the dialog box to come up. 

like below:

Manager.DialogMonitor.Start();
ConfirmDialog confirm = new ConfirmDialog(ActiveBrowser, DialogButton.OK);
confirm.initialization = 250;
Manager.DialogMonitor.AddDialog(confirm);
Action.Click();
system.threading.thread.sleep(250);
confirm.waituntilhandled(20000);
system.threading.thread.sleep(250);
Manager.DialogMonitor.remove(confirm);
Manager.DialogMonitor.Stop();

I am not working from my work computer...this is what I remember it to look like.

algot

Posted on: 03 Sep 2013 15:37

I have updated to last build but performance is still "bad".
Most of pop-ups can't be handled during test execution and ~40% tests are failed because of it.

Ewin

Posted on: 30 Aug 2013 20:32

Something I just noticed when trying to run lists.  When I had selected multiple test lists and click play,  I had to stop them for something.  

When click to stop it, the test lists were still executing.  

I stopped them by using the restful api http://url:8009 with http delete verb.

ADMIN

Konstantin Petkov

Posted on: 30 Aug 2013 20:03

Hello Everyone,

Indeed, the issue reported here was one of the main goals of this internal build. The QA department report test list execution times similar to those with the 2012 releases despite the additional scenarios Test Studio 2013.1 Release adds.

Having said that we recommend the tests distribution feature as well. You can get as many times faster execution as many execution machines host your tests.

Thanks a lot for your feedback and please accept our apologies for the temporary inconvenience!

Regards!

Ewin

Posted on: 30 Aug 2013 19:28

The internal build 827 is working nicely for executing test lists by pushing play.  FYI for those following this item.

Imported User

Posted on: 16 Aug 2013 19:52

I am glad to see this is a known issue and is being addressed. I am going to roll-back my "home" installation and not update my "work" installation until this is fixed. Thanks.

Todd

Posted on: 09 Aug 2013 11:20

Bloody awful! I am going to roll back to the old Test Studio software version until this issue is addressed! It takes WAY TOO LONG to execute my automated tests in our Agile Environment!!

Tekiyu

Posted on: 06 Aug 2013 19:10

This is not acceptable.

ADMIN

Jared Twing

Posted on: 06 Aug 2013 18:53

All, 

Thanks for the comments. I would recommend running distributed across machines to speed up execution for now if that is an option for your environment. It allows tests to run in parallel, cutting down the times dramatically. 

That being said, we are working very hard on the first service pack for this release and the speed of running test lists is very high on the priority list.

Thanks,
Jared

Tekiyu

Posted on: 06 Aug 2013 14:26

I just updated to the non-Beta release and I still see this issue.

Adrian

Posted on: 06 Aug 2013 14:12

I also have the same isssue, it is taking anything up to 20 minutes to run a test list that previously took about 5 minutes to run in the current release.

Tekiyu

Posted on: 05 Aug 2013 17:40

Any update on this? Will it be addressed with the release on 8/6?