When a textbox is focused pressing enter will have the browser click the first available button in the form, which is, in this case the first command button from the RadWindow.
If the only available command button is Close the RadWindow will close. 

Pressing enter with a textbox focused fires the next button on the form (you can see this with a simple textbox and button without anything special at all) and the page will be posted):
<asp:TextBox ID="Textbox2" runat="server" />
<button onclick="alert(2);" id="testButton">test button</button>
Since the RadWindow is the first thing rendered in a form the first button from its titlebar gets fired when the textbox is inside its ContentTemplate and enter is pressed.


One of the most suitable workarounds is to add a Pin button and cancel the Pin command:
		function OnClientCommand(sender, args)
		{
			if (args.get_commandName() == "Pin")
			{
				args.set_cancel(true);
				$telerik.$(".rwPinButton", sender.get_popupElement()).blur();
			}
		}
where this is attached to the OnClientCommand event of the RadWindow:
	<telerik:RadWindow ID="RadWindow1" runat="server" OpenerElementID="Button1" RenderMode="Lightweight"
		Behaviors="Pin, Move, Close"  OnClientCommand="OnClientCommand">
		<ContentTemplate>
			<asp:TextBox ID="Textbox1" runat="server" />
		</ContentTemplate>
	</telerik:RadWindow>
	<asp:Button ID="Button1" Text="open the RadWindow" runat="server" />