Repository navigation
Error: Failed to validate user configuration and data #2568
Description
System Information (please complete the following information):
- Model Builder Version (available in Manage Extensions dialog): 17.14.4.2313404
- Visual Studio Version: Microsoft Visual Studio Enterprise 2022 (64-bit) - Current Version 17.5.3
Describe the bug
- On which step of the process did you run into an issue:
Object detection on Azure, Training step - Clear description of the problem:
Initially, there was no prompt from the model builder. In Azure ML Studio, the job failed with the error message, "Error: Failed to validate user configuration and data. 1. Service failed to retrieve the data. Ensure data correctness and availability.", Eventually, the model builder encountered "A network issue has been detected. ", or "Didn't find a child run."
To Reproduce
Steps to reproduce the behavior:
- Go to 'Object Detection(Azure)'
- Setup Environment
- Select json file
- Click "Train"
- Go to Azure ML Studio, see job failed,
Expected behavior
A clear and concise description of what you expected to happen.
The job in Azure ML Studio should have succeeded, and the model builder should then consume the model.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
The older version model builder seemed fine, so I rolled back to 16.13.9.2235601, and the issue disappeared.
Activity
LittleLittleCloud commentedon Mar 29, 2023
@zewditu Can you take a look at this?
LittleLittleCloud commentedon Mar 29, 2023
Hitting this in the current main as well. Marking it as release blocker
LittleLittleCloud commentedon Mar 29, 2023
Verifying azure training over the previous release and it seems to be not working as well. Perhaps there's a break change on azure automl service side
zewditu commentedon Mar 30, 2023
@LittleLittleCloud , @johnseed I get successful run today, are you able to try it again?
johnseed commentedon Apr 1, 2023
It's still not working, the same error persists.
zewditu commentedon Apr 4, 2023
@johnseed it will be resolved in our next release.
johnseed commentedon Apr 7, 2023
I can confirm that the old version 16.13.9.2235601 does not have this issue, but the old version 16 is unable to select V100 or T4 Compute. Due to the large amount of data, the training times out. These accumulated issues have completely hindered my progress, causing not only financial loss from failed training but also impacting my project, which relies on this. I am currently under tremendous pressure.
LittleLittleCloud commentedon Apr 7, 2023
@johnseed Really sorry to hear that. We are going to release early next week which will include the fix for this issue.
LittleLittleCloud commentedon Apr 7, 2023
Also @zewditu, is it the case that @johnseed can overcome this issue by creating a training compute from another region
zewditu commentedon Apr 7, 2023
@johnseed probably your region is not westus2? you can use resources created in westus2 region to unblock you till our next release
johnseed commentedon Apr 7, 2023
I don't know what it has to do with westus2, as all my compute resources are in eastus. Regardless, I managed to use the T4 GPU, and I hope it works.
LittleLittleCloud commentedon Apr 18, 2023
@johnseed We just release a newer version with the fix for this issue, please let us know if it fixes this problem
johnseed commentedon Apr 19, 2023
Thank you for the update! I'll try the new version. I'll let you know if it resolves the issue.
johnseed commentedon Apr 29, 2023
Can't say it's perfect, but it's working now, thank you very much!
zewditu commentedon Apr 29, 2023
@johnseed what makes you "Can't say it's perfect, but it's working now" ? let us know
jpcintegral commentedon May 9, 2023
Hi, I am having the same problem today
johnseed commentedon May 9, 2023
LittleLittleCloud commentedon May 9, 2023
@jpcintegral can you share the model builder version you are using? And is the azure training fails with the same error information on azure portal as well.