Repository navigation
Ability to limit AutoML resource using (amount of parallel threads) #6061
Description
The current AutoML is not good to use when you have trained a lot of models at the same time in the cloud.
If you have some amount of pods in your Kubernetes cluster it doesn't matter how many AutoML experiments you execute at the same time. 4 experiments or 1 experiment at the same time use 100% of CPU (It brokes health checks and so on)
Low-level API of trainers (like FastForestBinaryTrainer) has options like NumberOfThreads and some other trainer-specific options you can use to handle the workload.
Is it able to add something like this to AutoML API?
But the most brilliant solution is some sort of smart property like ResourceUsingRatio which can be from 0.0 to 1.0
ResourceUsingRatio = 1.0 means the experiment use the maximum of potential resources (mainly CPU) it needs or the machine has.
Activity
michaelgsharp commentedon Feb 1, 2022
@80LevelElf thanks for your suggestion, this is something that would be good to address. We are currently in the process of updating our AutoML implementation, so this is something that we can look at during this time.
@JakeRadMSFT @LittleLittleCloud thoughts?
LittleLittleCloud commentedon Feb 2, 2022
In model builder and mlnet cli, users can limit on the number of thread one experiment can use via
env:MLNET_MAX_THREAD, we could probably do the same via AutoML experiment settingmichaelgsharp commentedon Feb 2, 2022
@LittleLittleCloud will that environment variable work with AutoML just as is? As a potential workaround for now:?
LittleLittleCloud commentedon Feb 2, 2022
Nope,, it's only available in modelbuilder/cli
michaelgsharp commentedon Feb 2, 2022
Yeah, I just checked and the main ML.NET framework itself doesn't know anything about that..
With the re-work going on with AutoML, is this something that could be included pretty easily with it?
LittleLittleCloud commentedon Feb 2, 2022
Yup, just need to expose it through AutoML experiment setting and override the default value when creating trainer