This happened multiple times. In the property builder, I tried to delete the column and Visual Studio crashed. I was able to delete other columns fine, but this particular column caused a problem. The only difference that I can see between the other columns and the one that caused the problem was the conditionalformattingobject. Once I deleted this conditionalformattingobject, I was able to delete the column without any problem.
The issue occurs when the deleted column is part of an expression which is set to an ExpressionFormattingObject of another column. The same thing happens if you try to set an expression which contains an invalid column name to any column. We were able to prevent VS from crashing by showing an error message dialog, however, due to the nature of the issue it might happen that the error dialog appears more times than needed.