I noticed that in the recurring scheduled test the settings do not persist if we select " Only on error..." .

It defaults back to "Always".  I would expect it to remember the original setting.