You can't customize ButtonPart ot TogglePart of SplitButton in ColorPicker as SplitButtonStyle doesn't get applied with this property.
The fix will be included in the upcoming official Q2 2014 release.