Repository navigation
End-to-end perf scenarios #7
Description
Performance Test
test\Microsoft.ML.Benchmarks\StochasticDualCoordinateAscentClassifierBench.cs
Results
.NET Core App 2.0 (Job=Default):
BenchmarkDotNet=v0.10.14, OS=Windows 10.0.17134
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.1.200
[Host] : .NET Core 2.0.7 (CoreCLR 4.6.26328.01, CoreFX 4.6.26403.03), 64bit RyuJIT
Toolchain=InProcessToolchain| Method | Mean | Error | StdDev | AccuracyMacro | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|
| TrainIris | 340,686,491.15 ns | 3,571,557.1056 ns | 2,982,411.6534 ns | 0.98 | 8625.0000 | 250.0000 | 62.5000 | 12596368 B |
| PredictIris | 1,292,667.88 ns | 25,756.8687 ns | 64,618.9476 ns | 0.98 | 25.3906 | 9.7656 | 1.9531 | 87571 B |
| PredictIrisBatchOf1 | 14.80 ns | 0.2822 ns | 0.2357 ns | 0.98 | 0.0171 | - | - | 72 B |
| PredictIrisBatchOf2 | 15.14 ns | 0.3011 ns | 0.3468 ns | 0.98 | 0.0171 | - | - | 72 B |
| PredictIrisBatchOf5 | 16.40 ns | 0.3957 ns | 0.5145 ns | 0.98 | 0.0171 | - | - | 72 B |
// * Legends *
Mean : Arithmetic mean of all measurements
Error : Half of 99.9% confidence interval
StdDev : Standard deviation of all measurements
AccuracyMacro : Macro-average accuracy of the model
Gen 0 : GC Generation 0 collects per 1k Operations
Gen 1 : GC Generation 1 collects per 1k Operations
Gen 2 : GC Generation 2 collects per 1k Operations
Allocated : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)
1 ns : 1 Nanosecond (0.000000001 sec).NET Core App 3.0 (Job=Default):
BenchmarkDotNet=v0.10.14, OS=Windows 10.0.17134
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.0.100-alpha1-20180720-2
[Host] : .NET Core 3.0.0-preview1-26710-03 (CoreCLR 4.6.26710.05, CoreFX 4.6.26708.04), 64bit RyuJIT
Toolchain=InProcessToolchain| Method | Mean | Error | StdDev | AccuracyMacro | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|
| TrainIris | 340,069,667.23 ns | 4,074,874.9274 ns | 3,612,270.1309 ns | 0.98 | 10187.5000 | 312.5000 | 62.5000 | 14416239 B |
| PredictIris | 1,163,328.54 ns | 21,833.5386 ns | 21,443.4523 ns | 0.98 | 27.3438 | 13.6719 | 1.9531 | 89840 B |
| PredictIrisBatchOf1 | 13.58 ns | 0.1441 ns | 0.1277 ns | 0.98 | 0.0171 | - | - | 72 B |
| PredictIrisBatchOf2 | 13.58 ns | 0.1992 ns | 0.1863 ns | 0.98 | 0.0171 | - | - | 72 B |
| PredictIrisBatchOf5 | 13.63 ns | 0.2046 ns | 0.1598 ns | 0.98 | 0.0171 | - | - | 72 B |
// * Legends *
Mean : Arithmetic mean of all measurements
Error : Half of 99.9% confidence interval
StdDev : Standard deviation of all measurements
AccuracyMacro : Macro-average accuracy of the model
Gen 0 : GC Generation 0 collects per 1k Operations
Gen 1 : GC Generation 1 collects per 1k Operations
Gen 2 : GC Generation 2 collects per 1k Operations
Allocated : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)
1 ns : 1 Nanosecond (0.000000001 sec)
Activity
helloguo commentedon Aug 22, 2018
@briancylui , any idea which CpuMath APIs are used in this example? And what's the percentage of the execution time spending on these CpuMath APIs vs overall execution time ?
briancylui commentedon Aug 22, 2018
@helloguo:
(1) While I don't have this specific info,
SdcaL1UpdatedUandSdcaL1UpdateSUcould probably be used sinceSdcais short forStochasticDualCoordinateAscent. Meanwhile, the code here (https://github.com/dotnet/machinelearning/blob/master/test/Microsoft.ML.Benchmarks/StochasticDualCoordinateAscentClassifierBench.cs) could provide some context.(2) Unfortunately I don't have this specific info, but it will be useful to know about it.
eerhardt commentedon Aug 23, 2018
Thanks for the results, @briancylui. It's good to see the time drop, while keeping the same accuracy. However, one concern I have is that on .NET Core 3.0, it looks like we are allocating MORE memory for both
TrainIrisandPredictIris.For one more comparison, can you run the same perf test against .NET Core 2.1? .NET Core 2.0 is a bit old by now, and a bunch of perf changes came in 2.1.