If you have already selected item(s) and then click on a new one to select it without holding Shift or Ctrl, the previous selection is cleared. This action should raise SelectionChanged with items in the RemovedItems collection of the event arguments. You can see this behavior in the native ListBox control. Also, this is how the RadDataGrid control is behaving in Single SelectionMode.
However, currently, if the SelectionMode is set to Extended, the RemovedItems collection doesn't contain any items.