This functionality is not supported for RadGridView.GroupRenderMode="Nested". We recommend using RadGridView.GroupRenderMode="Flat".

Directly using the Style that targets the GridViewRow element in the Resources of the RadGridView instance does not cause the undesired behavior. For example:

<telerik:RadGridView.Resources>
                <Style  TargetType="{x:Type telerik:GridViewRow}" BasedOn="{StaticResource {x:Type telerik:GridViewRow}}">
                    <Setter Property="Background" 
                Value="{Binding Background, 
                                RelativeSource={RelativeSource FindAncestor,AncestorType=Window}, Converter={StaticResource myConv}}"/>
                </Style>
            </telerik:RadGridView.Resources>