Can you please add a property CustomFormat to RadDateTimePicker? Possibly we should have 3 options for DisplayMode: Short, Long and Custom. If Custom is selected, user can specify format into CustomFormat field. Ex. CustomFormat = "MMM dd, yyyy". Also, make this field bindable, so that it can be specified from the ViewModel.
Please check the following SDK example which demonstrates how to achieve the required approach: https://github.com/telerik/xaml-sdk/tree/master/DateTimePicker/StringFormatMVVM