This is reproducible when you disable the automation peers by setting the AutomationManager.AutomationMode static property to Disabled.
It happens when you open a new RadWindow dialog (using the ShowDialog method), and the close it. After this, when you interact with the main window (click a button, hover a GridView, etc.) the exception occurs.
The exception is thrown only in the Output pane of Visual Studio and it doesn't cause any harm to the application. You can safely ignore it.
The exception message is "Invalid window handle".
To work this around you can enable the "Enable Just My Code" option in Visual Studio.