The problematic tool window has a user control that is docked to fill and it is built entirely with Telerik controls. The dock control, tab strips, windows and even the controls inside the user control are correctly scaled. The issue comes when you start undocking and then docking the tool window. The size of the user control increases with each operation. The controls have anchors and although they have correct scaling they get stretched. This results in messed layout. 

The DocumentTabStrip and ToolTabStrip are created dynamically and when they initialize they call their Scale method. The tool window, the user control, and also all other controls added to it are created only once. When you undock the window a new tab strip is created and the existing controls are added to it. When we load the tab strip we call PerformRadAutoScale which calls the stip's Scale method. Calling Scale propagates to the child controls causing additional scaling to the Microsoft controls. RadControls don't have issues because we have logic to check the scaling of the window and we don't allow double scaling.

As a workaround, we can create custom tab strips and override the ScaleChildren property to return false.

this.radDock1.DockTabStripNeeded += this.RadDock1_DockTabStripNeeded;
private void RadDock1_DockTabStripNeeded(object sender, DockTabStripNeededEventArgs e)
{
    if (e.DockType == DockType.ToolWindow)
    {
        e.Strip = new MyToolTabStrip();
    }
    else
    {
        e.Strip = new MyDocumentTabStrip();
    }
}

public class MyToolTabStrip : ToolTabStrip
{
    protected override bool ScaleChildren => false;
}

public class MyDocumentTabStrip : DocumentTabStrip
{
    protected override bool ScaleChildren => false;
}