To avoid this, handle the scrolling of the ScrollViewer when the dropdown content is opened. This behavior is already implemented in RadComboBox by overriding its OnMouseWheel method and handling the event arguments.
To work this around subscribe to the MouseWheel event of RadDropDownButton and set the Handled property of the event arguments to True.
private void RadDropDownButton_MouseWheel(object sender, MouseWheelEventArgs e)
{
var btn = (RadDropDownButton)sender;
if (btn.IsOpen)
{
e.Handled = true;
}
}