Skip to content

[Issue, ML.net CLI] 330GB csv file of data cause a OutOfMemoryException (1/2) #6288

New issue
New issue
Closed
Closed
[Issue, ML.net CLI] 330GB csv file of data cause a OutOfMemoryException (1/2)#6288

Description

@wil70
wil70
opened on Aug 18, 2022

System Information (please complete the following information):

  • OS & Version: Win8, latest version as of this bug entry
  • ML.NET Version: 16.13.9
  • .NET Version:6.0.303

Describe the bug
When I start ML.net from CLI, I get a OutOfMemoryException
I have 64GB Ram, I have a 330GB csv file of data.

I tried with
To Reproduce
Steps to reproduce the behavior:

  1. Generate a 330GB file with 4209 columns with random data
  2. open prompt
  3. type in command line:
    mlnet classification --train-time 75600 --name SampleClassification --log-file-path c:\Log_data.txt --has-header true --label-col 4209 --ignore-cols 0,1,4206,4207,4208 --dataset "c:\data.csv" --test-dataset "c:\test_data.csv"
  4. See error log at the end of this message with the OutOfMemoryException

Expected behavior
I expect ml.net to continue and feed the data as it stream it, so there should be no OutOfMemoryException
When I monitor the mknet.exe prices with task manager, the mlnet.exe process doesn't go high at all, like less than ~14GB. So something is not right as I have 64GB and also it shouldn't matter isn't it as .

Screenshots, Code, Sample Projects
Additional context
Here is the log
Start Training
start nni training
Experiment output folder: C:\Users\W\AppData\Local\Temp\AutoML-NNI\Experiment-GET3JS
System.FormatException: Parsing failed with an exception: Stream reading encountered exception
---> System.FormatException: Stream reading encountered exception
---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Text.StringBuilder.ToString()
at System.IO.StreamReader.ReadLine()
at Microsoft.ML.Data.TextLoader.Cursor.LineReader.ThreadProc()
--- End of inner exception stack trace ---
at Microsoft.ML.Data.TextLoader.Cursor.LineReader.GetBatch()
at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.Parse(Int32 tid)
at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.ThreadProc(Object obj)
--- End of inner exception stack trace ---
at Microsoft.ML.Data.TextLoader.Cursor.ParseParallel(ParallelState state)+MoveNext()
at Microsoft.ML.Data.TextLoader.Cursor.MoveNextCore()
at Microsoft.ML.Data.RootCursorBase.MoveNext()
at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.Controller.CountRows(IDataView data, Int64 maxRows) in //src/Microsoft.ML.ModelBuilder.AutoMLService/Proposer/Controller.cs:line 174
at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.Controller.Initialize() in //src/Microsoft.ML.ModelBuilder.AutoMLService/Proposer/Controller.cs:line 111
at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.LocalAutoMLExperiment.ExecuteAsync(IDataView trainData, IDataView validateData, ColumnInformation columnInformation, CancellationToken cancellationToken, CancellationToken timeout) in //src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/LocalAutoMLExperiment.cs:line 138
at Microsoft.ML.ModelBuilder.AutoMLEngine.StartTrainingAsync(TrainingConfiguration config, PathConfiguration pathConfig, CancellationToken userCancellationToken) in //src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 160
at Microsoft.ML.CLI.Runners.AutoMLRunner.ExecuteAsync() in //src/mlnet/Runners/AutoMLRunner.cs:line 88
at Microsoft.ML.CLI.Program.TrainAsync(TrainingConfiguration trainingConfiguration, PathConfiguration pathConfig, AutoMLServiceLogLevel logLevel) in //src/mlnet/Program.cs:line 348
at Microsoft.ML.CLI.Program.AutoMLCommandRunner(AutoMLCommand command, Boolean skipGenerateConsoleApp) in //src/mlnet/Program.cs:line 329
at Microsoft.ML.CLI.Program.<>c.<b__4_0>d.MoveNext() in //src/mlnet/Program.cs:line 89
--- End of stack trace from previous location ---
at System.CommandLine.Invocation.CommandHandler.GetExitCodeAsync(Object value, InvocationContext context)
at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass23_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.ML.CLI.Program.<>c__DisplayClass4_0.<b__9>d.MoveNext() in /_/src/mlnet/Program.cs:line 290
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<b__24_0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass11_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<b__10_0>d.MoveNext()
--- End of stack trace from previous location ---
at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass14_0.<b__0>d.MoveNext()
Check out log file for more information: c:\Log_data.txt
Exiting ...

C:\Users\W>'

Activity

  1. ghost added
    untriagedNew issue has not been triaged
    on Aug 18, 2022
  2. wil70
    changed the title [-]330GB csv file cause a OutOfMemoryException [/-] [+]330GB csv file of data cause a OutOfMemoryException [/+] on Aug 18, 2022
  3. wil70
    changed the title [-]330GB csv file of data cause a OutOfMemoryException [/-] [+][Issue] 330GB csv file of data cause a OutOfMemoryException [/+] on Aug 18, 2022

  4. LittleLittleCloud commented on Aug 18, 2022

    @LittleLittleCloud
    LittleLittleCloud
    on Aug 18, 2022
    Member

    Looks like the OOM happens even before training start
    @michaelgsharp is CountRows using stream data or it load IDataview in memory

  5. wil70
    changed the title [-][Issue] 330GB csv file of data cause a OutOfMemoryException [/-] [+][Issue] 330GB csv file of data cause a OutOfMemoryException with ML/NET CLI[/+] on Aug 19, 2022
  6. wil70
    changed the title [-][Issue] 330GB csv file of data cause a OutOfMemoryException with ML/NET CLI[/-] [+][Issue] 330GB csv file of data cause a OutOfMemoryException [/+] on Aug 19, 2022
  7. wil70
    changed the title [-][Issue] 330GB csv file of data cause a OutOfMemoryException [/-] [+][Issue, ML.net CLI] 330GB csv file of data cause a OutOfMemoryException [/+] on Aug 19, 2022
  8. wil70
    changed the title [-][Issue, ML.net CLI] 330GB csv file of data cause a OutOfMemoryException [/-] [+][Issue, ML.net CLI] 330GB csv file of data cause a OutOfMemoryException (1/2)[/+] on Aug 19, 2022

  9. wil70 commented on Aug 20, 2022

    @wil70
    wil70
    on Aug 20, 2022
    Author

    I also tried with MLnet --version: 16.13.9+3066041f1761935763bdd267a9bcbb64cb3ec543
    and I got the same Exception as above

    Sometimes, I get a different OutOfMemoryException stack trace:

    
Start Training
start nni training
Experiment output folder: C:\Users\W\AppData\Local\Temp\AutoML-NNI\Experiment-GNJIOK
|     Trainer                              MicroAccuracy  MacroAccuracy  Duration #Iteration                     |
System.InvalidOperationException: Event we were waiting on was subject to an exception
 ---> System.FormatException: Parsing failed with an exception: Stream reading encountered exception
 ---> System.FormatException: Stream reading encountered exception
 ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
   at System.IO.StreamReader.ReadLine()
   at Microsoft.ML.Data.TextLoader.Cursor.LineReader.ThreadProc()
   --- End of inner exception stack trace ---
   at Microsoft.ML.Data.TextLoader.Cursor.LineReader.GetBatch()
   at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.Parse(Int32 tid)
   at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.ThreadProc(Object obj)
   --- End of inner exception stack trace ---
   at Microsoft.ML.Data.TextLoader.Cursor.ParseParallel(ParallelState state)+MoveNext()
   at Microsoft.ML.Data.TextLoader.Cursor.MoveNextCore()
   at Microsoft.ML.Data.RootCursorBase.MoveNext()
   at Microsoft.ML.Data.CacheDataView.Filler(DataViewRowCursor cursor, ColumnCache[] caches, OrderedWaiter waiter)
   --- End of inner exception stack trace ---
   at Microsoft.ML.Internal.Utilities.OrderedWaiter.Wait(Int64 position, CancellationToken token)
   at Microsoft.ML.Data.CacheDataView.GetPermutationOrNull(Random rand)
   at Microsoft.ML.Data.CacheDataView.GetRowCursorWaiterCore[TWaiter](TWaiter waiter, Func`2 predicate, Random rand)
   at Microsoft.ML.Data.CacheDataView.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Transforms.RowShufflingTransformer.GetRowCursorCore(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Data.TransformBase.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Transforms.GenerateNumberTransform.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
   at Microsoft.ML.Transforms.RangeFilter.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
   at Microsoft.ML.Data.DataViewUtils.TryCreateConsolidatingCursor(DataViewRowCursor& curs, IDataView view, IEnumerable`1 columnsNeeded, IHost host, Random rand)
   at Microsoft.ML.Data.TransformBase.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Transforms.ColumnSelectingTransformer.SelectColumnsDataTransform.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Data.IO.BinarySaver.RowsPerBlockHeuristic(IDataView data, ColumnCodec[] actives)
   at Microsoft.ML.Data.IO.BinarySaver.SaveData(Stream stream, IDataView data, Int32[] colIndices)
   at Microsoft.ML.Data.DataSaverUtils.SaveDataView(IChannel ch, IDataSaver saver, IDataView view, Stream stream, Boolean keepHidden)
   at Microsoft.ML.BinaryLoaderSaverCatalog.SaveAsBinary(DataOperationsCatalog catalog, IDataView data, Stream stream, Boolean keepHidden)
   at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.ResampleStrategyProposer.Initialize(String outputFolder)
   at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.ResampleStrategyProposer.Propose(TrialParameter trialParameter)
   at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.Controller.Propose()
   at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.LocalAutoMLExperiment.ExecuteAsync(IDataView trainData, IDataView validateData, ColumnInformation columnInformation, CancellationToken cancellationToken, CancellationToken timeout)
   at Microsoft.ML.ModelBuilder.AutoMLEngine.StartTrainingAsync(TrainingConfiguration config, PathConfiguration pathConfig, CancellationToken userCancellationToken)
   at Microsoft.ML.CLI.Runners.AutoMLRunner.ExecuteAsync() in /_/src/mlnet/Runners/AutoMLRunner.cs:line 88
   at Microsoft.ML.CLI.Program.TrainAsync(TrainingConfiguration trainingConfiguration, PathConfiguration pathConfig, AutoMLServiceLogLevel logLevel) in /_/src/mlnet/Program.cs:line 348
   at Microsoft.ML.CLI.Program.AutoMLCommandRunner(AutoMLCommand command, Boolean skipGenerateConsoleApp) in /_/src/mlnet/Program.cs:line 329
   at Microsoft.ML.CLI.Program.<>c.<<CreateRootCommandLineBuilder>b__4_0>d.MoveNext() in /_/src/mlnet/Program.cs:line 89
--- End of stack trace from previous location ---
   at System.CommandLine.Invocation.CommandHandler.GetExitCodeAsync(Object value, InvocationContext context)
   at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass23_0.<<UseParseErrorReporting>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.ML.CLI.Program.<>c__DisplayClass4_0.<<CreateRootCommandLineBuilder>b__9>d.MoveNext() in /_/src/mlnet/Program.cs:line 290
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__24_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseParseDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass11_0.<<UseDebugDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__10_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass14_0.<<UseExceptionHandler>b__0>d.MoveNext()
Check out log file for more information: G:\log_data.txt
Exiting ...

  10. wil70 commented on Aug 26, 2022

    @wil70
    wil70
    on Aug 26, 2022
    Author

    Related issue on c# instead of CLI
    #6297

  11. wil70 commented on Aug 26, 2022

    @wil70
    wil70
    on Aug 26, 2022
    Author

    Any ETA for a fix? TY!

  12. tarekgh commented on Aug 26, 2022

    @tarekgh
    tarekgh
    on Aug 26, 2022
    Member

    @wil70 what kind of data do you have inside the csv file? does it contain strings with length greater than 2G? In .NET, the string length is limited to 2G (int.MaxValue).

  13. wil70 commented on Aug 27, 2022

    @wil70
    wil70
    on Aug 27, 2022
    Author

    Hi @tarekgh ,
    All the fields are "double" except 1 of the fields is a "string" which has 3 characters maximum
    ie.
    DOO,-0.85748191341578595,-1.0763921379331161,-3,-1,...
    UUU,-1.153315959380012,-1.2316112856397838,0,1,...
    ...
    I hope this help
    TY!

  14. tarekgh commented on Aug 27, 2022

    @tarekgh
    tarekgh
    on Aug 27, 2022
    Member

    Thanks @wil70. In this case it looks to me you are running into legitimate OutOfMemoryException https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/performance#issue-an-out-of-memory-exception-is-thrown. I suggest you try to check the GC state when getting this exception. either through a profiler or a debugger or using the GC APIs https://docs.microsoft.com/en-us/dotnet/api/system.gcmemoryinfo?view=net-6.0.

  15. wil70 commented on Aug 28, 2022

    @wil70
    wil70
    on Aug 28, 2022
    Author

    Thanks @tarekgh
    I would expect ML.NET CLI to stream data via a IDataView
    Each of my row is only ~4000 floats + 3 letter string, I wouldn't expect the memory to blow up with such small data.

    Note: I have 64GB on this machine

    What is the biggest dataset that has been handled with ML.NET CLI? and ML.NET via c# api?

    Thanks
    Wil

  16. 5 remaining items

  17. tarekgh commented on Sep 2, 2022

    @tarekgh
    tarekgh
    on Sep 2, 2022
    Member

    @wil70 thanks for the info. I have tried it myself and I can repro it with 100GB data file on my system.

    I am seeing OOO exception stacks in different places like:

     ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.GC.AllocateNewArray(IntPtr typeHandle, Int32 length, GC_ALLOC_FLAGS flags)
   at System.GC.<AllocateUninitializedArray>g__AllocateNewUninitializedArray|66_0[T](Int32 length, Boolean pinned)
   at System.Text.StringBuilder.ExpandByABlock(Int32 minBlockCharCount)
   at System.Text.StringBuilder.Append(Char* value, Int32 valueCount)
   at System.Text.StringBuilder.Append(Char[] value, Int32 startIndex, Int32 charCount)
   at System.IO.StreamReader.ReadLine()
   at Microsoft.ML.Data.TextLoader.Cursor.LineReader.ThreadProc()

    and

     ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
   at System.IO.StreamReader.ReadLine()
   at Microsoft.ML.Data.TextLoader.Cursor.LineReader.ThreadProc()

    The issue here is not ML.NET (either CLI or C#). The issue is the system limitation coming from the OS (Windows in this case). As the training uses exceptionally large data, that will need allocating a lot of memory. On Windows the memory allocation will depend on how big the pagefile maximum size setting is. Note, the page files are global for the entire system which means all running applications will be contributing to such file too. If any application is trying to allocate memory which can exceed the maximum size of the page file, it will fail the allocation and in managed code will start experiencing the OutOfMemoryException.

    What you may try to do is first ensure you have enough space on the disk. second try to adjust the system settings to increase the page file maximum size to allow big memory allocations. Here is the image of the settings dialog from Windows 11 machine. You can reach this setting from System Settings->Advanced->Performance Settings->Advanced

    image

  18. wil70 commented on Sep 5, 2022

    @wil70
    wil70
    on Sep 5, 2022
    Author

    Thanks @tarekgh !
    How did you figure this out?

    I have 64GB RAM on the machine and 34GB on Virtual memory, I will increase it and try - TY!
    It seems my max was set to 1,144,296MB as Custom Size on the F drive, but maybe it went above which seems a lot of somehow it didn't increase to the max

    image

    If I set "System managed size" instead of "custom size", will windows be able to increase as ML.net ask for more?

    Note: I'm only running ML.net, I'm not running anything else in parallel to keep the ressources for ML.net

    Interresting - Thank you @tarekgh

  19. tarekgh commented on Sep 6, 2022

    @tarekgh
    tarekgh
    on Sep 6, 2022
    Member

    If I set "System managed size" instead of "custom size", will windows be able to increase as ML.net ask for more?

    For the 330GB data, I don't think the system will allow that much size automatically. Try to use custom size and specify a substantial number allow 330GB size swapping. Thanks!

  20. wil70 commented on Sep 6, 2022

    @wil70
    wil70
    on Sep 6, 2022
    Author

    Thanks @tarekgh
    I tried over the weekend with "System managed" and it failed quickly. It would be great if microsoft os could handle this.
    I tried with Custom size, like 50 to 70GB on c drive and 500 to 1000GB on f drive, it did run longer.
    image

    The input file is 330gb, so I'm wondering as AutoML.net try model after model if it doesn't clear well the memory?

    Thanks
    Wil
    Start Training start nni training [##### ] Time left: 34452s [##### ] Time left: 34447s [##### ] Time left: 34445s [##### ] Time left: 34445s [##### ] Time left: 34424s [##### ] Time left: 34424s [##### ] Time left: 34035s [####### ] Time left: 34034s [####### ] Time left: 33964s [####### ] Time left: 33964s [####### ] Time left: 33952s [########## ] Time left: 32896s [########### ] Time left: 32784s [########### ] Time left: 32763s [########### ] Time left: 32648s [########### ] Time left: 32637s [########### ] Time left: 32637s [########### ] Time left: 32629s [########### ] Time left: 32621s [########### ] Time left: 32601s [########### ] Time left: 32599s [############### ] Time left: 31383s [############### ] Time left: 31304s [############### ] Time left: 31304s [############### ] Time left: 31299s [############### ] Time left: 31299s [############### ] Time left: 31299s [############### ] Time left: 31299s [############### ] Time left: 31299s [############### ] Time left: 31283s [############### ] Time left: 31283s [############### ] Time left: 31279s [############### ] Time left: 31279s System.InvalidOperationException: Event we were waiting on was subject to an exception ---> System.FormatException: Parsing failed with an exception: Stream reading encountered exception ---> System.FormatException: Stream reading encountered exception ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at System.Text.StringBuilder.ToString() at System.IO.StreamReader.ReadLine() at Microsoft.ML.Data.TextLoader.Cursor.LineReader.ThreadProc() --- End of inner exception stack trace --- at Microsoft.ML.Data.TextLoader.Cursor.LineReader.GetBatch() at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.Parse(Int32 tid) at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.ThreadProc(Object obj) --- End of inner exception stack trace --- at Microsoft.ML.Data.TextLoader.Cursor.ParseParallel(ParallelState state)+MoveNext() at Microsoft.ML.Data.TextLoader.Cursor.MoveNextCore() at Microsoft.ML.Data.RootCursorBase.MoveNext() at Microsoft.ML.Data.CacheDataView.Filler(DataViewRowCursor cursor, ColumnCache[] caches, OrderedWaiter waiter) --- End of inner exception stack trace --- at Microsoft.ML.Internal.Utilities.OrderedWaiter.Wait(Int64 position, CancellationToken token) at Microsoft.ML.Data.CacheDataView.GetPermutationOrNull(Random rand) at Microsoft.ML.Data.CacheDataView.GetRowCursorWaiterCore[TWaiter](TWaiter waiter, Func2 predicate, Random rand)
    at Microsoft.ML.Data.CacheDataView.GetRowCursor(IEnumerable1 columnsNeeded, Random rand) at Microsoft.ML.Transforms.RowShufflingTransformer.GetRowCursorCore(IEnumerable1 columnsNeeded, Random rand)
    at Microsoft.ML.Data.TransformBase.GetRowCursor(IEnumerable1 columnsNeeded, Random rand) at Microsoft.ML.Transforms.GenerateNumberTransform.GetRowCursorSet(IEnumerable1 columnsNeeded, Int32 n, Random rand)
    at Microsoft.ML.Transforms.RangeFilter.GetRowCursorSet(IEnumerable1 columnsNeeded, Int32 n, Random rand) at Microsoft.ML.Data.DataViewUtils.TryCreateConsolidatingCursor(DataViewRowCursor& curs, IDataView view, IEnumerable1 columnsNeeded, IHost host, Random rand)
    at Microsoft.ML.Data.TransformBase.GetRowCursor(IEnumerable1 columnsNeeded, Random rand) at Microsoft.ML.Transforms.ColumnSelectingTransformer.SelectColumnsDataTransform.GetRowCursor(IEnumerable1 columnsNeeded, Random rand)
    at Microsoft.ML.Data.IO.BinarySaver.RowsPerBlockHeuristic(IDataView data, ColumnCodec[] actives)
    at Microsoft.ML.Data.IO.BinarySaver.SaveData(Stream stream, IDataView data, Int32[] colIndices)
    at Microsoft.ML.Data.DataSaverUtils.SaveDataView(IChannel ch, IDataSaver saver, IDataView view, Stream stream, Boolean keepHidden)
    at Microsoft.ML.BinaryLoaderSaverCatalog.SaveAsBinary(DataOperationsCatalog catalog, IDataView data, Stream stream, Boolean keepHidden)
    at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.ResampleStrategyProposer.Initialize(String outputFolder) in //src/Microsoft.ML.ModelBuilder.AutoMLService/Proposer/ResampleStrategyProposer.cs:line 102
    at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.ResampleStrategyProposer.Propose(TrialParameter trialParameter) in /    /src/Microsoft.ML.ModelBuilder.AutoMLService/Proposer/ResampleStrategyProposer.cs:line 55
    at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.Controller.Propose() in //src/Microsoft.ML.ModelBuilder.AutoMLService/Proposer/Controller.cs:line 65
    at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.LocalAutoMLExperiment.ExecuteAsync(IDataView trainData, IDataView validateData, ColumnInformation columnInformation, CancellationToken cancellationToken, CancellationToken timeout) in /    /src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/LocalAutoMLExperiment.cs:line 163
    at Microsoft.ML.ModelBuilder.AutoMLEngine.StartTrainingAsync(TrainingConfiguration config, PathConfiguration pathConfig, CancellationToken userCancellationToken) in //src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 160
    at Microsoft.ML.CLI.Runners.AutoMLRunner.ExecuteAsync() in /    /src/mlnet/Runners/AutoMLRunner.cs:line 88
    at Microsoft.ML.CLI.Program.TrainAsync(TrainingConfiguration trainingConfiguration, PathConfiguration pathConfig, AutoMLServiceLogLevel logLevel) in //src/mlnet/Program.cs:line 348
    at Microsoft.ML.CLI.Program.AutoMLCommandRunner(AutoMLCommand command, Boolean skipGenerateConsoleApp) in /    /src/mlnet/Program.cs:line 329
    at Microsoft.ML.CLI.Program.<>c.<b__4_0>d.MoveNext() in //src/mlnet/Program.cs:line 89
    --- End of stack trace from previous location ---
    at System.CommandLine.Invocation.CommandHandler.GetExitCodeAsync(Object value, InvocationContext context)
    at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
    at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<b__0>d.MoveNext()
    --- End of stack trace from previous location ---
    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass23_0.<b__0>d.MoveNext()
    --- End of stack trace from previous location ---
    at Microsoft.ML.CLI.Program.<>c__DisplayClass4_0.<b__9>d.MoveNext() in /    /src/mlnet/Program.cs:line 290
    --- End of stack trace from previous location ---
    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<b__24_0>d.MoveNext()
    --- End of stack trace from previous location ---
    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<b__0>d.MoveNext()
    --- End of stack trace from previous location ---
    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass11_0.<b__0>d.MoveNext()
    --- End of stack trace from previous location ---
    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<b__10_0>d.MoveNext()
    --- End of stack trace from previous location ---
    at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass14_0.<b__0>d.MoveNext()
    Check out log file for more information: S:\CATS\files\data_analysis\output\Log_ALL_INST_smallAttributes.txt
    Exiting ...
    `

  21. tarekgh commented on Sep 7, 2022

    @tarekgh
    tarekgh
    on Sep 7, 2022
    Member

    The input file is 330gb, so I'm wondering as AutoML.net try model after model if it doesn't clear well the memory?

    I expect this is not the case. But it could be the model loading and caching the whole data into the memory. looking at the stack and seeing Microsoft.ML.Data.CacheDataView which suggests what I am saying. Considering you are using an excessively big data set, caching loaded data will cause out of memory for sure.

    CC @luisquintanilla @LittleLittleCloud in case they know more about that.

  22. wil70 commented on Sep 7, 2022

    @wil70
    wil70
    on Sep 7, 2022
    Author

    Thanks @tarekgh

    Arg, I was hoping being able to use a much bigger input file later

    Right now I'm not using the "-cache" which means it default to "Automatic" by default.
    I could try with "-cache Off" if you think that would help?

    --cache <Auto|Off|On> Specify [On|Off|Auto] for cache to be turned on, off, or auto-determined (default).

    Thanks

    Wil

  23. tarekgh commented on Sep 7, 2022

    @tarekgh
    tarekgh
    on Sep 7, 2022
    Member

    @wil70 yes, it is worth trying to turn off the cache option. Thank you for your patience and trials.

  24. wil70 commented on Sep 8, 2022

    @wil70
    wil70
    on Sep 8, 2022
    Author

    @terlochan, Here you go with --cache Off
    I hope this help
    TY!
    Wil

    2022-09-07 22:36:05.7438 INFO Check out log file for more information: c:\output\log.txt (Microsoft.ML.CLI.Program+<>c__DisplayClass4_0.<CreateRootCommandLineBuilder>b__7)
2022-09-07 22:36:05.7438 INFO Exiting ... (Microsoft.ML.CLI.Program+<>c__DisplayClass4_0.<CreateRootCommandLineBuilder>b__7)
2022-09-08 08:00:38.1674 DEBUG Set log file path to c:\output\log.txt (Microsoft.ML.CLI.Commands.MLCommand.set_LogFilePath)
2022-09-08 08:01:15.7140 DEBUG Set log file path to c:\output\log.txt (Microsoft.ML.CLI.Commands.MLCommand.set_LogFilePath)
2022-09-08 08:01:56.0343 INFO Start Training (Microsoft.ML.CLI.Runners.AutoMLRunner+<ExecuteAsync>d__8.MoveNext)
2022-09-08 08:01:56.2187 INFO start nni training (Microsoft.ML.CLI.Utilities.PBarConsolePrinter.Print)
2022-09-08 08:01:56.2588 INFO Experiment output folder: C:\Users\Wilhelm\AppData\Local\Temp\AutoML-NNI\Experiment-M5PI59 (Microsoft.ML.CLI.Utilities.PBarConsolePrinter.Print)
2022-09-08 08:01:56.2588 DEBUG row count is unknown, count it explicitly (Microsoft.ML.CLI.Runners.AutoMLRunner.Instance_DiagnosticDataReceived)
2022-09-08 08:16:43.1825 DEBUG count elapse 886919ms (Microsoft.ML.CLI.Runners.AutoMLRunner.Instance_DiagnosticDataReceived)
2022-09-08 08:16:43.1825 INFO |     Trainer                              MicroAccuracy  MacroAccuracy  Duration #Iteration                     | (Microsoft.ML.CLI.Utilities.PBarConsolePrinter.Print)
2022-09-08 08:23:19.3495 ERROR System.InvalidOperationException: Event we were waiting on was subject to an exception
 ---> System.FormatException: Parsing failed with an exception: Stream reading encountered exception
 ---> System.FormatException: Stream reading encountered exception
 ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
   at System.IO.StreamReader.ReadLine()
   at Microsoft.ML.Data.TextLoader.Cursor.LineReader.ThreadProc()
   --- End of inner exception stack trace ---
   at Microsoft.ML.Data.TextLoader.Cursor.LineReader.GetBatch()
   at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.Parse(Int32 tid)
   at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.ThreadProc(Object obj)
   --- End of inner exception stack trace ---
   at Microsoft.ML.Data.TextLoader.Cursor.ParseParallel(ParallelState state)+MoveNext()
   at Microsoft.ML.Data.TextLoader.Cursor.MoveNextCore()
   at Microsoft.ML.Data.RootCursorBase.MoveNext()
   at Microsoft.ML.Data.CacheDataView.Filler(DataViewRowCursor cursor, ColumnCache[] caches, OrderedWaiter waiter)
   --- End of inner exception stack trace ---
   at Microsoft.ML.Internal.Utilities.OrderedWaiter.Wait(Int64 position, CancellationToken token)
   at Microsoft.ML.Data.CacheDataView.GetPermutationOrNull(Random rand)
   at Microsoft.ML.Data.CacheDataView.GetRowCursorWaiterCore[TWaiter](TWaiter waiter, Func`2 predicate, Random rand)
   at Microsoft.ML.Data.CacheDataView.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Transforms.RowShufflingTransformer.GetRowCursorCore(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Data.TransformBase.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Transforms.GenerateNumberTransform.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
   at Microsoft.ML.Transforms.RangeFilter.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
   at Microsoft.ML.Data.DataViewUtils.TryCreateConsolidatingCursor(DataViewRowCursor& curs, IDataView view, IEnumerable`1 columnsNeeded, IHost host, Random rand)
   at Microsoft.ML.Data.TransformBase.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Transforms.ColumnSelectingTransformer.SelectColumnsDataTransform.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Data.IO.BinarySaver.RowsPerBlockHeuristic(IDataView data, ColumnCodec[] actives)
   at Microsoft.ML.Data.IO.BinarySaver.SaveData(Stream stream, IDataView data, Int32[] colIndices)
   at Microsoft.ML.Data.DataSaverUtils.SaveDataView(IChannel ch, IDataSaver saver, IDataView view, Stream stream, Boolean keepHidden)
   at Microsoft.ML.BinaryLoaderSaverCatalog.SaveAsBinary(DataOperationsCatalog catalog, IDataView data, Stream stream, Boolean keepHidden)
   at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.ResampleStrategyProposer.Initialize(String outputFolder) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Proposer/ResampleStrategyProposer.cs:line 102
   at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.ResampleStrategyProposer.Propose(TrialParameter trialParameter) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Proposer/ResampleStrategyProposer.cs:line 55
   at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.Controller.Propose() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Proposer/Controller.cs:line 65
   at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.LocalAutoMLExperiment.ExecuteAsync(IDataView trainData, IDataView validateData, ColumnInformation columnInformation, CancellationToken cancellationToken, CancellationToken timeout) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/LocalAutoMLExperiment.cs:line 163
   at Microsoft.ML.ModelBuilder.AutoMLEngine.StartTrainingAsync(TrainingConfiguration config, PathConfiguration pathConfig, CancellationToken userCancellationToken) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 160
   at Microsoft.ML.CLI.Runners.AutoMLRunner.ExecuteAsync() in /_/src/mlnet/Runners/AutoMLRunner.cs:line 88
   at Microsoft.ML.CLI.Program.TrainAsync(TrainingConfiguration trainingConfiguration, PathConfiguration pathConfig, AutoMLServiceLogLevel logLevel) in /_/src/mlnet/Program.cs:line 348
   at Microsoft.ML.CLI.Program.AutoMLCommandRunner(AutoMLCommand command, Boolean skipGenerateConsoleApp) in /_/src/mlnet/Program.cs:line 329
   at Microsoft.ML.CLI.Program.<>c.<<CreateRootCommandLineBuilder>b__4_0>d.MoveNext() in /_/src/mlnet/Program.cs:line 89
--- End of stack trace from previous location ---
   at System.CommandLine.Invocation.CommandHandler.GetExitCodeAsync(Object value, InvocationContext context)
   at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass23_0.<<UseParseErrorReporting>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.ML.CLI.Program.<>c__DisplayClass4_0.<<CreateRootCommandLineBuilder>b__9>d.MoveNext() in /_/src/mlnet/Program.cs:line 290
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__24_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseParseDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass11_0.<<UseDebugDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__10_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass14_0.<<UseExceptionHandler>b__0>d.MoveNext() (Microsoft.ML.CLI.Program+<>c__DisplayClass4_0.<CreateRootCommandLineBuilder>b__7)
2022-09-08 08:23:27.4475 DEBUG System.InvalidOperationException: Event we were waiting on was subject to an exception
 ---> System.FormatException: Parsing failed with an exception: Stream reading encountered exception
 ---> System.FormatException: Stream reading encountered exception
 ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ToString()
   at System.IO.StreamReader.ReadLine()
   at Microsoft.ML.Data.TextLoader.Cursor.LineReader.ThreadProc()
   --- End of inner exception stack trace ---
   at Microsoft.ML.Data.TextLoader.Cursor.LineReader.GetBatch()
   at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.Parse(Int32 tid)
   at Microsoft.ML.Data.TextLoader.Cursor.ParallelState.ThreadProc(Object obj)
   --- End of inner exception stack trace ---
   at Microsoft.ML.Data.TextLoader.Cursor.ParseParallel(ParallelState state)+MoveNext()
   at Microsoft.ML.Data.TextLoader.Cursor.MoveNextCore()
   at Microsoft.ML.Data.RootCursorBase.MoveNext()
   at Microsoft.ML.Data.CacheDataView.Filler(DataViewRowCursor cursor, ColumnCache[] caches, OrderedWaiter waiter)
   --- End of inner exception stack trace ---
   at Microsoft.ML.Internal.Utilities.OrderedWaiter.Wait(Int64 position, CancellationToken token)
   at Microsoft.ML.Data.CacheDataView.GetPermutationOrNull(Random rand)
   at Microsoft.ML.Data.CacheDataView.GetRowCursorWaiterCore[TWaiter](TWaiter waiter, Func`2 predicate, Random rand)
   at Microsoft.ML.Data.CacheDataView.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Transforms.RowShufflingTransformer.GetRowCursorCore(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Data.TransformBase.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Transforms.GenerateNumberTransform.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
   at Microsoft.ML.Transforms.RangeFilter.GetRowCursorSet(IEnumerable`1 columnsNeeded, Int32 n, Random rand)
   at Microsoft.ML.Data.DataViewUtils.TryCreateConsolidatingCursor(DataViewRowCursor& curs, IDataView view, IEnumerable`1 columnsNeeded, IHost host, Random rand)
   at Microsoft.ML.Data.TransformBase.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Transforms.ColumnSelectingTransformer.SelectColumnsDataTransform.GetRowCursor(IEnumerable`1 columnsNeeded, Random rand)
   at Microsoft.ML.Data.IO.BinarySaver.RowsPerBlockHeuristic(IDataView data, ColumnCodec[] actives)
   at Microsoft.ML.Data.IO.BinarySaver.SaveData(Stream stream, IDataView data, Int32[] colIndices)
   at Microsoft.ML.Data.DataSaverUtils.SaveDataView(IChannel ch, IDataSaver saver, IDataView view, Stream stream, Boolean keepHidden)
   at Microsoft.ML.BinaryLoaderSaverCatalog.SaveAsBinary(DataOperationsCatalog catalog, IDataView data, Stream stream, Boolean keepHidden)
   at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.ResampleStrategyProposer.Initialize(String outputFolder) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Proposer/ResampleStrategyProposer.cs:line 102
   at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.ResampleStrategyProposer.Propose(TrialParameter trialParameter) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Proposer/ResampleStrategyProposer.cs:line 55
   at Microsoft.ML.ModelBuilder.AutoMLService.Proposer.Controller.Propose() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Proposer/Controller.cs:line 65
   at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.LocalAutoMLExperiment.ExecuteAsync(IDataView trainData, IDataView validateData, ColumnInformation columnInformation, CancellationToken cancellationToken, CancellationToken timeout) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/LocalAutoMLExperiment.cs:line 163
   at Microsoft.ML.ModelBuilder.AutoMLEngine.StartTrainingAsync(TrainingConfiguration config, PathConfiguration pathConfig, CancellationToken userCancellationToken) in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 160
   at Microsoft.ML.CLI.Runners.AutoMLRunner.ExecuteAsync() in /_/src/mlnet/Runners/AutoMLRunner.cs:line 88
   at Microsoft.ML.CLI.Program.TrainAsync(TrainingConfiguration trainingConfiguration, PathConfiguration pathConfig, AutoMLServiceLogLevel logLevel) in /_/src/mlnet/Program.cs:line 348
   at Microsoft.ML.CLI.Program.AutoMLCommandRunner(AutoMLCommand command, Boolean skipGenerateConsoleApp) in /_/src/mlnet/Program.cs:line 329
   at Microsoft.ML.CLI.Program.<>c.<<CreateRootCommandLineBuilder>b__4_0>d.MoveNext() in /_/src/mlnet/Program.cs:line 89
--- End of stack trace from previous location ---
   at System.CommandLine.Invocation.CommandHandler.GetExitCodeAsync(Object value, InvocationContext context)
   at System.CommandLine.Invocation.ModelBindingCommandHandler.InvokeAsync(InvocationContext context)
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass23_0.<<UseParseErrorReporting>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.ML.CLI.Program.<>c__DisplayClass4_0.<<CreateRootCommandLineBuilder>b__9>d.MoveNext() in /_/src/mlnet/Program.cs:line 290
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__24_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseParseDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass11_0.<<UseDebugDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__10_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass14_0.<<UseExceptionHandler>b__0>d.MoveNext() (Microsoft.ML.CLI.Program+<>c__DisplayClass4_0.<CreateRootCommandLineBuilder>b__7)
2022-09-08 08:23:27.4475 INFO Check out log file for more information: c:\output\log.txt (Microsoft.ML.CLI.Program+<>c__DisplayClass4_0.<CreateRootCommandLineBuilder>b__7)
2022-09-08 08:23:27.4475 INFO Exiting ... (Microsoft.ML.CLI.Program+<>c__DisplayClass4_0.<CreateRootCommandLineBuilder>b__7)

  25. wil70 commented on Sep 8, 2022

    @wil70
    wil70
    on Sep 8, 2022
    Author

    Note: After this work, I will try with 2TB+ of input file. TY!

  26. tarekgh commented on Sep 8, 2022

    @tarekgh
    tarekgh
    on Sep 8, 2022
    Member

    @LittleLittleCloud with --cache off parameter, I am seeing CacheDataView still used in the stack. Any thoughts on how we can avoid caching the data in general?

    CC @luisquintanilla

  27. wil70 commented on Sep 9, 2022

    @wil70
    wil70
    on Sep 9, 2022
    Author

    and next may be treat it the cache as an intermediate buffer between the disk and the ram so you do not wear off the drive too fast and still gain some speed. IF this is loaded linearly then that won't be useful, but if some algo train huge dataset by batches then that might help...not sure.

  28. michaelgsharp commented on Nov 28, 2022

    @michaelgsharp
    michaelgsharp
    on Nov 28, 2022
    Contributor

    Closing this as it seems like the issue has been solved.

  29. michaelgsharp
    closed this as completedon Nov 28, 2022
  30. ghost removed
    untriagedNew issue has not been triaged
    on Nov 28, 2022
  31. ghost locked as resolved and limited conversation to collaborators on Dec 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions