The format should be set like this instead of using the custom format:
radDateTimePicker1.Culture = new System.Globalization.CultureInfo("bg-BG");
var behaviour = radDateTimePicker1.DateTimePickerElement.GetCurrentBehavior() as RadDateTimePickerCalendar;
behaviour.TimePicker.Culture = new System.Globalization.CultureInfo("bg-BG");