Skip to content

Pause and Resume AutoML experiments #5736

New issue
New issue
Closed
Closed
Pause and Resume AutoML experiments#5736
Milestone
ML.NET Future

Description

@stoyandimov
stoyandimov
opened on Apr 2, 2021

I wonder if there are any plans to provide a way to pause/resume ML.NET AutoML experiments (with exiting the ML.NET process).

One can use AutoML to run experiments and set duration limit in seconds. However, this requires the process running for the entire duration of the experiment.

Has anyone tried running an experiment and implementing some sort of pause/resume functionality?

Use Case: Pause an experiment and resume after reboot

  • Developer starts an AutoML Regression experiment that should run for 20 hours
  • 8 hours into the experiment developer pauses the experiment
  • ML.NET persists the current state and results of the experiment
  • Developer shutdowns or reboots the OS
  • Developer resumes the experiment
  • ML.NET picks up where it left of and continues the experiment until it has run the whole 20 hours

The real use case for me is running long experiments at night. I'm not using my computer between 02:00 and 07:00. How can I run experiment for those 5 hours, pause the experiment and continue the next night?

Activity

  1. michaelgsharp
    added
    AutoML.NET
    P2
    on Apr 8, 2021

  2. briacht commented on Jul 9, 2021

    @briacht
    briacht
    on Jul 9, 2021
    Contributor

    This is something we are working on in tooling right now as we continue to improve our AutoML experience!

    @LittleLittleCloud can provide more details if you have additional questions.

  3. andrasfuchs commented on Sep 12, 2022

    @andrasfuchs
    andrasfuchs
    on Sep 12, 2022
    Contributor

    I would like to add that it would be great to have "skip current trial" and "stop training" functionality alongside the pause/resume mentioned above, ideally accessible through the experiment's IMonitor interface.

    Sometimes there are certain trials in an experiment that run significant longer than the previous ones. In those cases I would like to implement a logic that skips that particular trial, because that probably doesn't worth the time spending on it, hoping that the next trial would run normally within the time limits I expect.

    I'll include my recent logs to illustrate my point:

    14:38:16.0 info: BBD.BodyMonitor.API[0] Bio Balance Detector Body Monitor v0.9.0.0
14:38:16.0 info: BBD.BodyMonitor.API[0] (The current UTC time is 2022-09-11 12:38:16)
14:38:16.6 info: Microsoft.Hosting.Lifetime[14] Now listening on: https://localhost:7061
14:38:16.6 info: Microsoft.Hosting.Lifetime[14] Now listening on: http://localhost:5061
14:38:16.6 info: Microsoft.Hosting.Lifetime[0] Application started. Press Ctrl+C to shut down.
14:38:16.6 info: Microsoft.Hosting.Lifetime[0] Hosting environment: Development
14:38:16.6 info: Microsoft.Hosting.Lifetime[0] Content root path: C:\Work\BioBalanceDetector\Software\Source\BBDProto08\BBD.BodyMonitor\BBD.BodyMonitor.API\
14:38:29.7 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Starting model training for 1800 seconds, using 'BBD_20220829_TrainingData_MLP09_5p0Hz-150000Hz_IsSubject_AndrasFuchs_3500rows.csv' as data source with the 'MLP09' profile.
14:39:05.9 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #   0 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:   -0,03 - Duration:     36 seconds
14:39:05.9 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0]  New Best Trial #   0 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:   -0,03
14:39:54.1 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #   1 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:   -0,71 - Duration:     48 seconds
14:40:41.7 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #   2 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:    0,92 - Duration:     48 seconds
14:40:41.7 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0]  New Best Trial #   2 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:    0,92
14:41:17.5 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #   3 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:    0,66 - Duration:     36 seconds
14:42:25.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #   4 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:    0,91 - Duration:     68 seconds
14:43:16.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #   5 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:    0,81 - Duration:     51 seconds
14:44:08.7 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #   6 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:    0,95 - Duration:     52 seconds
14:44:08.8 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0]  New Best Trial #   6 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:    0,95
14:45:04.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #   7 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:    0,90 - Duration:     56 seconds
14:46:17.7 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #   8 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:   -0,89 - Duration:     73 seconds
14:47:05.3 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #   9 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:    0,93 - Duration:     48 seconds
14:48:08.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #  10 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:   -0,95 - Duration:     63 seconds
14:49:18.9 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #  11 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:   -0,55 - Duration:     70 seconds
14:50:11.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #  12 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:    0,85 - Duration:     52 seconds
14:51:03.6 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #  13 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:    0,92 - Duration:     52 seconds
14:52:11.3 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #  14 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:    0,97 - Duration:     68 seconds
14:52:11.3 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0]  New Best Trial #  14 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:    0,97
15:01:54.2 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #  15 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:    0,89 - Duration:     583 seconds
15:03:03.9 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #  16 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:   -0,97 - Duration:     70 seconds
15:08:05.1 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] Completed Trial #  17 - Pipeline: ReplaceMissingValues=>Concatenate=>FastTreeRegression - Metric:   -0,98 - Duration:     301 seconds
15:08:29.9 info: BBD.BodyMonitor.API.Controllers.MachineLearningController[0] AutoML result: 0,9660835989610341. Saving model as 'o:\Work\BBD.BodyMonitor\BBD_20220829_TrainingData_MLP09_5p0Hz-150000Hz_IsSubject_AndrasFuchs_3500rows.zip'

    You can see here that the 15th and 17th trials ran much longer than the others.

  4. LittleLittleCloud commented on Sep 12, 2022

    @LittleLittleCloud
    LittleLittleCloud
    on Sep 12, 2022
    Member

    We are exploring the option of continue training in AutoML now. And for excessive long trials, it most likely happens in tree-base trainers when the NumberOfTree or NumberOfIteration is large. I'll push a PR to mitigate this.

  5. michaelgsharp
    added this to the ML.NET Future milestone on Oct 10, 2022

  6. rosebyte commented on Sep 7, 2026

    @rosebyte
    rosebyte
    on Sep 7, 2026
    Member

    closing as fixed by 6335

  7. rosebyte
    closed this as completedon Sep 7, 2026
  8. github-actions
    locked and limited conversation to collaborators on Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions