Repository navigation
GPU groundwork: standard-ops forecast graph and exact-order device linear kernels - #1
Draft
mudler-agent wants to merge 8 commits into
Draft
mudler-agent wants to merge 8 commits into
mudler-agent wants to merge 8 commits into
Conversation
The default graph still uses the ggml_custom_4d callbacks that match the reference rounding bit for bit. A new switch builds the same graph from built-in ggml operations only, so a GPU backend can schedule every node. The switch is ts::standard_ops(). It is off by default. The CMake option TIMESTRAL_STANDARD_OPS=ON turns it on by default, and the environment variable TIMESTRAL_STANDARD_OPS=0|1 overrides it for one process. The flag is read when a graph is built. No custom-op code changed. Standard ops used: - linear layers and the head: ggml_mul_mat plus a bias add - RMS norm: ggml_rms_norm times the scale - SiLU and sigmoid: ggml_silu and ggml_sigmoid - attention: mul_mat for the scores, ggml_soft_max_ext with the additive time or variate mask, mul_mat for the value product - RoPE already uses mul/neg/concat/add on the stored tables and is shared Scaler, refine, covariates and the float64 quantile steps stay on the host. This path is held to the NORM gate, not to bit-exact gates. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
test_standard_ops runs the patch, layer (with RoPE), encoder, head, predict_block and forecast components on the standard-op graph against the stored reference. It uses the NORM gate (err <= max(2 * floor, 16u)) and prints one GATE line per comparison with error, tolerance and ratio. It does not stop at the first failure and no tolerance is changed. The executable is not registered with ctest. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
The reference F32 GEMM reduces K in panels of 192: one accumulator chain per panel, panel sums added in order. A single ggml_mul_mat over all of K rounds along a different path. In layers with outlier activations that moves the largest output by several units in the last place, which is more than the NORM gate allows. standard_linear now issues one mul_mat per panel of 192 and adds the panels in order, then the bias. Dots shorter than 192 stay one mul_mat. Gates and tolerances are unchanged. On the 209M F32 suite the failed NORM checks go from 102 to 56 of 5571. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The reference sums x*x in F32 with a fixed cascade: 32 lanes, power of two merge blocks, a fixed lane fold, then divides by D and takes 1/sqrt(mean + eps). ggml_rms_norm sums in double. Both are accurate, but they round differently, and the last bit of 1/sqrt(mean + eps) shows in the output of any row with large channels. standard_rms_norm now rebuilds the cascade from element-wise nodes (sqr, view, add, div, sqrt, mul), which are exact IEEE F32 on every backend. Rows with D not a multiple of 32 fall back to ggml_rms_norm. The scalars D and 1.0 come from a new graph_constant() that the graph runner writes when it allocates the graph. The encoder graph metadata budget goes from 512 to 1024 nodes per layer, because panels and the cascade add nodes. Default-path graphs are unchanged. On the 209M F32 suite the failed NORM checks go from 56 to 29 of 5571, and from 20 to 4 of 2328 on the extended cases. docs/standard-ops.md now records the arithmetic choices and the remaining failures. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
One thread per output with the same ascending fmaf chain, 192-wide panels and bias-last order as the CPU reference. CUDA and HIP share one source. Not compiled for a device yet; a CPU emulation build matches the reference on all 880 cases. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
One MSL thread per output, same operation order as the CUDA kernel, explicit fma(), fast math off. The host test compares bitwise with the CPU reference using a platform-independent splitmix64 generator. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The CUDA test drew inputs from std::mt19937 and std::normal_distribution. The normal distribution is not specified bit for bit, so the data differed between standard libraries and from the Metal test, which used splitmix64. Move the splitmix64 generator (seed 11) into src/gpu/shared_inputs.hpp and use it in both tests, so every platform tests identical data. Both tests now print a checksum of the first 64 weights of the first case. Shapes, case order, checks and the kernels are unchanged. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Work toward GPU support, in three parts. Nothing here changes the default CPU path, which stays bit-identical to the existing baseline.
TIMESTRAL_STANDARD_OPS, CMakeTIMESTRAL_STANDARD_OPS). It uses built-in ggml operations for linear layers, RMS norm, SiLU, sigmoid and attention, and keeps the host-side scaler, refine, covariate and float64 quantile steps as they are. Seedocs/standard-ops.md.src/gpu/) and a Metal source (src/gpu/metal/), each with a bitwise test against the CPU reference.src/gpu/shared_inputs.hpp), so the same data is checked on every device.Status
mul_matwhere the reference accumulates in double) and from outlier channels. This is open; seedocs/standard-ops.md.tests/test_standard_ops.cpp) is built but not registered with ctest.Review focus
src/gpu/linear_exact_gpu.hpp: the per-output order must matchlinear_exact_ref.hpp.src/layers.cpp: the standard-operations switch is read at graph build time, and each custom-op site returns early only when the switch is on.🤖 Generated with Claude Code