Most controls that show a drop down capture the mouse to cancel the scrolling of any parent ScrollViewers while the drop down is opened. The ToolBar overflow items drop down doesn't prevent the scrolling in parent ScrollViewers (a ScrollViewer that wraps the content where the ToolBar is defined). In that case if the user scrolls the ScrollViewer while the overflow items are opened, the ToolBar control moves according to the new position, but the popup stays open at its original position.

The common behavior here is to prevent scrolling while the drop down is opened. 

To work this around, you can subscribe to the MouseWheel event of RadToolBar and handle it in case the IsOverflowOpen property is True.

private void ToolBar_MouseWheel(object sender, MouseWheelEventArgs e)
        {
            var toolBar = (RadToolBar)sender;
            if (toolBar.IsOverflowOpen)
            {
                e.Handled = true;
            }
        }