RecentColors functionality doesn't work when you set RecentColorsItemsSource. The recent colors are not shown at all but they should be. When using this property the user should be able to customize the recent colors functionality on his own. Showing the recent colors is a must when setting non-empty colletion via RecentColorsItemSource.
You can't customize ButtonPart ot TogglePart of SplitButton in ColorPicker as SplitButtonStyle doesn't get applied with this property.