When we try to record a test step to handle RadUpload dialog, Test Studio records two steps:

RadUpload(''): input file 0 select  and Handle 'FileUpload' dialog.

When you examine the properties for the  Handle 'FileUpload' dialog step, you will see that the step was recorded with HandleButton Close. For that reason the dialog is handled by closing and not by selecting the right HandleButton Open and the test fails.

Steps to reproduce and a sample test project are attached in the internal description.