RadSpell uses a RadWindow for its dialog which, in turn, uses RadFormDecorator. Thus, RadSpell should expose its RenderMode property in order to allow the developer to make the popups consistent on the page (i.e. avoid mixing modes which is not supported) and to avoid styling issues with the dialog.
Currently the following web.config setting can be used to change the rendering mode of all RadWindows in the application, including the ones used by RadSpell:
<add key="Telerik.Web.UI.Window.RenderMode" value="Lightweight" />
and this one for the RadFormDecorators:
 <add key="Telerik.Web.UI.FormDecorator.RenderMode" value="lightweight" />