When you have a checkbox column in the grid and group by a certain column, you should have the possibility to check-all rows inside a group without the necessity of expanding the group and checking each separate row.
The group row should have a checkbox as well. When you toggle it, all data rows or nested groups should also be toggled. The parent group should also be adjusted correctly.
Workaround: The following knowledge base article demonstrates a sample approach how to extend the default group row and add a checkbox. However, it would be necessary to modify and extend it in a way to cover the cases with nested groups: