Repository navigation
e06d7363 with big datasets: "The process was terminated due to an unhandled exception." #6679
Description
System Information (please complete the following information):
- OS & Version: Windows Server 2019 & Ubuntu 22.04
- ML.NET Version: Daily / nightly build
- .NET Version: 7.0
Question
For FastTree.DiskTranspose: how much free disk space is needed? I have plenty, but perhaps I did not have for full copy of the dataset if it is needed.
Describe the bug
I am training a big dataset, 450 GB as binary IDV file, maybe about 2TB as uncompressed CSV. Smaller datasets (~10-20GB) work fine.
LightGBM and FastTree with DiskTranspose are able to run wihtout exception, but at some point "unhandled exception" occurs. I have try...catch around contents of main() to print the error. Nothing shows up in console. Many other trainers give OOM.
Ubuntu 22.04, 128 GB RAM, 650 GB swap
Windows 2019, 220 GB RAM, total of 2.8 TB paging files
From Windows Logs / Application I can see these 2 errors
Application: Kwork.MLTrainer2023.exe
CoreCLR Version: 7.0.523.17405
.NET Version: 7.0.5
Description: The process was terminated due to an unhandled exception.
Exception Info: exception code e06d7363, exception address 00007FFB2A0049B9
Application: Kwork.MLTrainer2023.exe CoreCLR Version: 7.0.523.17405 .NET
Version: 7.0.5 Description: The process was terminated due to an unhandled
exception. Exception Info: exception code e06d7363, exception address
00007FFB2A0049B9
Faulting application name: Kwork.MLTrainer2023.exe, version: 0.2100.23.26401, time stamp: 0x641de100
Faulting module name: ucrtbase.dll, version: 10.0.17763.1490, time stamp: 0x48ac8393
Exception code: 0xc0000409
Fault offset: 0x000000000006e77e
Faulting process id: 0x15c0
Faulting application start time: 0x01d98649658be43b
Faulting application path: C:\Users\antti\Desktop\net7.0\Kwork.MLTrainer2023.exe
Faulting module path: C:\Windows\System32\ucrtbase.dll
Report Id: 316601db-768e-4398-b14a-5c958b66723d
Faulting package full name:
Faulting package-relative application ID:
Kwork.MLTrainer2023.exe
0.2100.23.26401
641de100
ucrtbase.dll
10.0.17763.1490
48ac8393
c0000409
000000000006e77e
15c0
01d98649658be43b
C:\Users\antti\Desktop\net7.0\Kwork.MLTrainer2023.exe
C:\Windows\System32\ucrtbase.dll
316601db-768e-4398-b14a-5c958b66723d
Activity
torronen commentedon May 17, 2023
Temporary workaround: use smaller dataset. Something like this:
TrainTestData dataSplit = mlContext.Data.TrainTestSplit(data, testFraction: 0.2);For some reason, I was unable to run FilterByCustomPredicate. Here is the draft as it will allow more flexibility for sampling but it probably has some bug:
Smaller dataset is not the same, but at least it should be representative of the full dataset. I have not checked the research but my intuition says huge datasets (1+ TB) might not have too much benefit for non-NN algorithms, such as in Microsoft.ML library. At least the if the feature count is in the thousands or less. This is just my intuitive feeling.
[-]"The process was terminated due to an unhandled exception."[/-][+]e06d7363 with big datasets: "The process was terminated due to an unhandled exception."[/+]michaelgsharp commentedon Jan 24, 2024
Yeah, worst case scenario you will need enough disk space to store the full file. Not sure though if it stores it as the compressed binary version or plain text.
I'll close this issue, but I have added your duplication issue to our backlog.