By default the drop down buttons (like RadRibbonDropDownButton and RadRibbonSplitButton are opening their drop down content on mouse enter, when hosted in ApplicationMenu. When a drop down of a button is opened and you hover another button, the currently opened drop down should get closed. But this doesn't happen when the ApplicationMenu is populated through its ItemsSource and the buttons are defined in the ItemTemplate.
To work this around avoid using ItemsSource and ItemTemplate. Instead add the buttons directly in the Items collection of ApplicationMenu.