Here are two workaround options:

1. Cancel the client-side Command event when it is fired by a non-groupable column with GroupByColumn command.
   Wire the OnCommand event listener in the markup:
   

   <ClientSettings>
       <ClientEvents OnCommand="onCommand" />
   </ClientSettings>

   JavaScript:
   

   function onCommand(sender, args) {
       if (args.get_commandName() == "GroupByColumn") {
           var tableView = args.get_tableView();
           var columnName = args.get_commandArgument();
           var col = tableView.getColumnByUniqueName(columnName);
           var isColumnGroupable = col._data.Groupable;
   
           if (!isColumnGroupable) {
               args.set_cancel(true);
           }
       }
   }

   
   
2. Hide the GroupBy option on the menu.
   Use the client page load event to wire 'onclick' event listener to the options button (three dots button) and hide the GroupBy option when the column is not groupable:
   JavaScript
   

   function pageLoad(app, args) {
                   $('.rgActionButton.rgOptions').on('click', function (e) {
                       var headerCell = $(this).parents()[0];
                       var grid = $find('<%= RadGrid1.ClientID%>');
                       var masterTableView = grid.get_masterTableView();
                       var col = masterTableView.get_columns()[headerCell.cellIndex];
   
                       setTimeout(function () {
                           if (!col._data.Groupable) {
                               $('.rgMobileColumnForm').find('.rgButtonGroup').first().hide();
                           }
                           else {
                               $('.rgMobileColumnForm').find('.rgButtonGroup').first().show();
                           }
                       })
                   });
               }

Kind regards,
Doncho
Progress Telerik