This might be the same issue as http://feedback.telerik.com/Project/161/Feedback/Details/138950
After installing the web application in IIS from:
https://www.dropbox.com/s/7qqlym3f31r1mmm/TestStudioDemo.zip?dl=0

1) Open the attached test
2) Using FF, run to here on step 3

Expected: The RadWindow to stay open
Actual: It closes as soon as the recorder attaches

Further, if you manually open the RadWindow and try to Run Selected Steps for step 3, it results in:
'10/3/2014 3:27:03 PM' - 'Fail' : 3. Enter text 'tester' in 'TxtTestText'
------------------------------------------------------------
Failure Information: 
~~~~~~~~~~~~~~~
Unable to find the target host (Browser/SilverlightApp) to locate an element. Failure: ArtOfTest.WebAii.Exceptions.ExecuteCommandException: ExecuteCommand failed!
InError set by the client. Client Error:
mozCommandProcessor: GetCurrentDocumentMarkup(): Exception thrown while trying to serialize the Dom. Error: TypeError: this.TargetDocument is null
BrowserCommand (Type:'Information',Info:'DocumentMarkup',Action:'NotSet',Target:'null',Data:'',ClientId:'a96a0605-d19e-4aa1-b1e0-fd0d18743cd6',HasFrames:'False',FramesInfo:'',TargetFrameIndex:'5',InError:'True',Response:'mozCommandProcessor: GetCurrentDocumentMarkup(): Exception thrown while trying to serialize the Dom. Error: TypeError: this.TargetDocument is null')
InnerException: none.

   at ArtOfTest.WebAii.Core.Browser.ExecuteCommandInternal(BrowserCommand request)
   at ArtOfTest.WebAii.Core.Browser.ExecuteCommand(BrowserCommand request, Boolean performDomRefresh, Boolean waitUntilReady)
   at ArtOfTest.WebAii.Core.Browser.RefreshDomTree()
   at ArtOfTest.WebAii.Core.FramesCollection.EnsureDomLoaded(Browser browser)
   at ArtOfTest.WebAii.Core.FramesCollection.get_Item(FrameInfo frameInfo)
   at ArtOfTest.WebAii.Design.Execution.ExecutionUtils.GetFrameBrowserInstance(Browser topBrowser, FrameInfo frame, Int32 waitTimeout, String& stringError)
InnerException:
System.Exception: Unable to find the target host (Browser/SilverlightApp) to locate an element. Failure: ArtOfTest.WebAii.Exceptions.ExecuteCommandException: ExecuteCommand failed!
InError set by the client. Client Error:
mozCommandProcessor: GetCurrentDocumentMarkup(): Exception thrown while trying to serialize the Dom. Error: TypeError: this.TargetDocument is null
BrowserCommand (Type:'Information',Info:'DocumentMarkup',Action:'NotSet',Target:'null',Data:'',ClientId:'a96a0605-d19e-4aa1-b1e0-fd0d18743cd6',HasFrames:'False',FramesInfo:'',TargetFrameIndex:'5',InError:'True',Response:'mozCommandProcessor: GetCurrentDocumentMarkup(): Exception thrown while trying to serialize the Dom. Error: TypeError: this.TargetDocument is null')
InnerException: none.

   at ArtOfTest.WebAii.Core.Browser.ExecuteCommandInternal(BrowserCommand request)
   at ArtOfTest.WebAii.Core.Browser.ExecuteCommand(BrowserCommand request, Boolean performDomRefresh, Boolean waitUntilReady)
   at ArtOfTest.WebAii.Core.Browser.RefreshDomTree()
   at ArtOfTest.WebAii.Core.FramesCollection.EnsureDomLoaded(Browser browser)
   at ArtOfTest.WebAii.Core.FramesCollection.get_Item(FrameInfo frameInfo)
   at ArtOfTest.WebAii.Design.Execution.ExecutionUtils.GetFrameBrowserInstance(Browser topBrowser, FrameInfo frame, Int32 waitTimeout, String& stringError)
   at ArtOfTest.WebAii.Design.Execution.ExecutionUtils.WaitForAllElements(IAutomationHost host, AutomationDescriptor descriptor, Int32 timeout)
   at ArtOfTest.WebAii.Design.Execution.ExecutionEngine.ExecuteStep(Int32 order)

The same problem happens in Safari. It works as expected in IE and Chrome.

The same symptom happens in ServerTransferTest. Run to Here in FF on step 2 or 3 causes the browser to return to the starting page when the recorder attaches. It should remain on the second page of the app.