Setting the LayoutMode to Simplified doesn't do anything in the RadRibbonTabs that are assigned to a RadRibbonContextualGroup.
To work this around, extract the "ContextualTabsTemplate" ControlTemplate and the following elements in the "RibbonScrollViewer".
<telerikRibbonViewPrimitives:RibbonScrollViewer x:Name="TabItemsScrollViewer">
<Grid>
<ItemsPresenter x:Name="PART_DefaultItemsPresenter" HorizontalAlignment="Left" />
<ItemsControl x:Name="PART_SimplifiedItemsControl" Visibility="Collapsed"
ItemsSource="{Binding SimplifiedItems, RelativeSource={RelativeSource TemplatedParent}}">
<ItemsControl.ItemsPanel>
<ItemsPanelTemplate>
<telerikRibbonViewPrimitives:RibbonGroupsPanel />
</ItemsPanelTemplate>
</ItemsControl.ItemsPanel>
</ItemsControl>
</Grid>
</telerikRibbonViewPrimitives:RibbonScrollViewer>
Then define the following DataTrigger in the ControlTemplate.Triggers collection:
<DataTrigger Binding="{Binding RelativeSource={RelativeSource AncestorType=telerikRibbonView:RadRibbonView}, Path=LayoutMode}" Value="Simplified">
<Setter TargetName="PART_SimplifiedItemsControl" Property="Visibility" Value="Visible"/>
<Setter TargetName="PART_DefaultItemsPresenter" Property="Visibility" Value="Collapsed"/>
</DataTrigger>
Then, assign the custom ControlTemplate via the Template property of RadRibbonTab.
<telerik:RadRibbonTab Template="{StaticResource CustomContextualRibbonTabTemplate}" />