To reproduce:

Create a RadRibbonForm and set its ApplicationMenuStyle to BackstageView. Start the application open the backstage, click the maximize button and then again to set the Form back to normal. You will notice that the Form cannot be set back to normal.

Workaround:

Manually change the WindowState of the Form.

this.RibbonBar.BackstageControl.BackstageElement.TitleBarElement.MaximizeButton.MouseDown +=MaximizeButton_Click;

void MaximizeButton_Click(object sender, EventArgs e)
        {
            if (this.WindowState == FormWindowState.Maximized)
            {
                this.WindowState = FormWindowState.Normal;
            }
            else if (this.WindowState == FormWindowState.Normal)
            {
                this.WindowState = FormWindowState.Maximized;
            }
        }