Generally, GridComboBoxColumn is quite performance sensitive, especially in scenarios, where it has a large ItemsSource. In display mode (cell element), GridViewComboBoxColumn utilizes a LookUpElement to calculate the relation between the selected key and the value that should be displayed. Internally, it hosts an instance of RadComboBox that manages these calculations and then passes the display value to the LookUpElement. While this solution is optimal in terms of memory, since it does not require additional copies of the ItemsSource data, it lacks in performance. 
Improving this behavior will require updating the whole GridViewComboBoxColumn. Currently, this is not considered a bug, but rather a limitation. A possible workaround is to work wit Dictionary instead thus improving the speed of matching the two sources. Attached is a sample project illustrating the approach.