Skip to content

[ML.net cli] Resume a cli training after a crash  #6286

Open

Description

@wil70

Hello, is there a way to resume an ML.net cli training to where it was before a crash?
I have a lot of data in the folder C:\Users\wwww\AppData\Local\Temp\AutoML-NNI\Experiment-9K67B4 but I do not know how to make mlnet start from there.

Detail:
I used the cli, ie "mlnet classicfiaction...."
I trained for a few days, but I made a mistake which used a lot of memory on my computer, which stopped the mlnet process. I would like to start mlnet to where it left so it can continue from there

Thanks
w

2022-08-10 15:03:24.3091 DEBUG System.InvalidOperationException: Event we were waiting on was subject to an exception
---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Array.Resize[T](T[]& array, Int32 newSize)
at Microsoft.ML.Internal.Utilities.ArrayUtils.EnsureSize[T](T[]& array, Int32 min, Int32 max, Boolean keepOld, Boolean& resized)
at Microsoft.ML.Data.CacheDataView.ColumnCache.ImplOne`1.CacheCurrent()
at Microsoft.ML.Data.CacheDataView.Filler(DataViewRowCursor cursor, ColumnCache[] caches, OrderedWaiter waiter)

Activity

  1. ghost added
    untriagedNew issue has not been triaged
    on Aug 17, 2022
  2. changed the title [-]Resume a cli training[/-] [+]ML.net cli -- Resume a cli training [/+] on Aug 17, 2022
  3. changed the title [-]ML.net cli -- Resume a cli training [/-] [+]ML.net cli -- Resume a cli training after a crash [/+] on Aug 17, 2022
  4. changed the title [-]ML.net cli -- Resume a cli training after a crash [/-] [+][ML.net cli] Resume a cli training after a crash [/+] on Aug 18, 2022
  5. dakersnar commented on Aug 25, 2022

    @dakersnar
    Contributor

    I don't believe this functionality is currently supported.

    cc @luisquintanilla for confirmation. Is this something that we are tracking support for in the future?

  6. added
    area-AutoMLAutomated machine learning, Model Builder, and ML.NET CLI
    on Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-AutoMLAutomated machine learning, Model Builder, and ML.NET CLIenhancementNew feature or requestyellowmaintenance only

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions