To reproduce:

Add a RadRichTextBox, write some text and set the ScaleFactor property to 0.5f. You will see that the document itself is smaller than the control and is not centered.

Workaround:

Set the font size of the document. This will only scale the text, but will keep the document centered:

void RadTextBox_TextChanged(object sender, EventArgs e)
{

    if (textBox.Text == "")
    {
        return;
    }
    rtb.Document.Selection.SelectAll();
    rtb.DocumentView.ChangeFontSize(float.Parse(this.Get<RadTextBox>().Text));
    rtb.Document.Selection.Clear();
}

Resolution: 
This issue is addressed in the new version of the control - RadRichTextEditor. Please use the new control instead the RadRichTextBox.