Setting the custom format is not applied to the cell. To reproduce:
Dim customColumn As New GridViewDateTimeColumn()
customColumn.Width = 200
customColumn.Name = "custom"
customColumn.EditorType = GridViewDateTimeEditorType.TimePicker
customColumn.FormatString = "{0:HH:mm}"
customColumn.Format = DateTimePickerFormat.Custom
customColumn.CustomFormat = "HH:mm"
RadGridView1.MasterTemplate.Columns.Add(customColumn)
RadGridView1.Rows.Add(DateTime.Now)