The gray background that is shown when you hover an item is not hidden when the pointer leaves the item. That happens when you scroll on a touch device. 

To work this around, you can hide the pointer over background completely. To do this, you can edit the ControlTemplate of RadListView item and remove the Setter from the PointerOver Visual State.