The "Link to original" property should insert an image link pointing to the original image and the "Open in a new window" property should add the target attribute to the link with value "_blank". 
None of these modifications happen if the properties are set to true.

Possible workaround is to add the attached ImageManager dialog as an external dialog using the ExternalDialogsPath property of the RadEditor control. If you need step-by-step example for such approach you could follow this live example: http://demos.telerik.com/aspnet-ajax/editor/examples/externaldialogspath/defaultcs.aspx