Progress on porting ML.NET native SIMD algorithms to managed code #1

Goals

1. Gain an understanding of SIMD operations and use cases
2. Port ML.NET C++ SIMD algorithms to C#
3. Increase ML.NET performance by using AVX operations when supported and where beneficial
4. Ensure C# Hardware Intrinsics feature meets the needs of ML.NET
5. Unit test all functions and get performance benchmark numbers for before and after changes
6. (Stretch) provide software fallback implementations to support more architectures
7. (Stretch) Implement ARM64 SIMD algorithms

Progress

Week 1: Familiarize with .NET Development

• Get familiar with C# (Tutorial and Quick starts)
• Make a console app, use the debugger in Visual Studio
• Learn about ML.NET SIMD operations from the Intel Intrinsics Guide and mapping from C to C# functions
• Adopt the team's GitHub workflow and fork a local repo for work
• Implement SSE support and software fallbacks in managed code for DotU on a new .NET Core 2.1 console app with a NuGet package reference
• Write a unit test for the managed code of DotU using XUnit

Week 2: Learn SIMD operations and use them in .NET outside of ML.NET

• Complete first connect with recruiter
• Implement SSE support and software fallbacks in managed code for all key intrinsics
• Comply with coding style standard
• Implement working unit tests for all key intrinsics
• Implement working performance tests for all key intrinsics using BenchmarkDotNet (slides and recording)
• Present performance results in a table (SsePerf-report-github.pdf)

BenchmarkDotNet=v0.10.14, OS=Windows 10.0.15063.1155 (1703/CreatorsUpdate/Redstone2)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
Frequency=3515623 Hz, Resolution=284.4446 ns, Timer=TSC
.NET Core SDK=2.1.300
  [Host]     : .NET Core 2.1.0 (CoreCLR 4.6.26515.07, CoreFX 4.6.26515.06), 64bit RyuJIT
  DefaultJob : .NET Core 2.1.0 (CoreCLR 4.6.26515.07, CoreFX 4.6.26515.06), 64bit RyuJIT

Week 3-5: Port algo to C#, write unit tests and performance tests, check in code

• Think about why managed codes for "sparse" intrinsics are slower than native codes
• Apply real data to test implemented managed code using BenchmarkDotNet
• Schedule a meeting for midpoint review with Dan, Eric, Santi, Tanner, and Ivan on Skype at the end of Week 5 on July 20
• Get familiarized with the entire pipeline of ML.NET by creating a ML project
• Integrate local code into ML.NET repo to prepare for checking in code, including:

• C# implementations of intrinsics
• Unit tests
• Performance tests

• Implement additional unit tests to test the complete code paths for two different target frameworks
• Enable multi-targeting
• Make the switch to turn on or off implemented code at will with the UseIntrinsics build attribute
• Check in code with PR Port C# key hardware intrinsics APIs for SSE from SIMD native algorithms dotnet/machinelearning#562

Week 6

• Participate in Microsoft Hackathon
• Attend IEEE conference

Week 7

• Respond to PR comments and Intel partners
• Fix build issues in multi-targeting and disabling netcoreapp3.0 test projects
• Hard-code unit tests
• Introduced a custom random seed in perf tests based on environmental variables for better testing
• Major style changes to best utilize existing libraries and ensure aggressive inlining wherever needed
• Document follow-up action items for performance enhancement in an issue page (Suggestions on CpuMath enhancement #2)
• Fix perf issues of some SSE intrinsics in compliance with C# 7.3 updates
• Fix merge conflicts and obtain green builds for PR
• PR on SSE key intrinsics, as well as their unit tests and perf tests, with multi-targeting, is approved

Week 8-9

• Scale up implementation, unit tests, and performance tests to cover all SSE intrinsics
• Write AVX implementations
• Performance test before and after. We should see some perf gains here.
• Check in code to ML.NET (submitted PR)

Perf test results for all active SSE hardware intrinsics:

BenchmarkDotNet=v0.10.14, OS=Windows 10.0.17134
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.0.100-alpha1-20180720-2
  [Host] : .NET Core 3.0.0-preview1-26710-03 (CoreCLR 4.6.26710.05, CoreFX 4.6.26708.04), 64bit RyuJIT

Toolchain=InProcessToolchain

Perf tests results for all managed intrinsics with AVX enhancement:

BenchmarkDotNet=v0.10.14, OS=Windows 10.0.17134
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.0.100-alpha1-20180720-2
  [Host] : .NET Core 3.0.0-preview1-26710-03 (CoreCLR 4.6.26710.05, CoreFX 4.6.26708.04), 64bit RyuJIT

Toolchain=InProcessToolchain

In one summary:

BenchmarkDotNet=v0.10.14, OS=Windows 10.0.17134
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.0.100-alpha1-20180720-2
  [Host] : .NET Core 3.0.0-preview1-26710-03 (CoreCLR 4.6.26710.05, CoreFX 4.6.26708.04), 64bit RyuJIT

Toolchain=InProcessToolchain

Week 10-11 (Stretch)

• Provide software fallback implementations (stretch goals)
• Respond to PR feedback for AVX intrinsics
• Streamlined perf test layout
• Report improvement in running time of intrinsics: averaged 17.78%
• Report improvement in running time of end-to-end real-life user scenarios: 13.88%
• Get ML.NET to run on Raspberry Pi
• Present on August 31 (11am-12nn 25/3365, also on Skype)

Week 12

• Improve perf by optimizing loops and alignment issues (Suggestions on CpuMath enhancement #2) at the assembly/instruction level
• Clean up, presentation, close out remaining issues
• Write blog post on how ML.NET is taking advantage of .NET Core hardware intrinsics, and AVX vs SSE comparisons (both implementation and runtime perf)

Latest perf results:

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.0.100-alpha1-20180720-2
  [Host] : .NET Core 3.0.0-preview1-26710-03 (CoreCLR 4.6.26710.05, CoreFX 4.6.26708.04), 64bit RyuJIT

Toolchain=InProcessToolchain