test(parity): Wolf-pattern attention stress test (zerfoo plan T2.4) - #141
Merged
Conversation
Adds TestAttentionTraining_WolfPattern_GPU / _StressCI: the Wolf gr-12 hazard schedule on an attention-shaped synthetic graph (Q@K^T -> scale -> softmax -> @v plus residual + RMS-norm), built on the graph/node APIs with SaveForBackward (ADR 006) for the multi-consumer saved intermediates (Q/K/V/attention weights, pre-norm activation, normalization inverse). Schedule: per-sample Forward+Backward with an arena reset BETWEEN forward and backward (saved lifetimes load-bearing) plus the per-sample ResetPool, in-place gradient accumulation into persistent graph.Parameters, in-place SGD per batch, 2 epochs x 2 batches x 3 samples, all under ZTENSOR_ARENA_POISON. Asserts params/grads finite, GPU within f32 tolerance of the byte-identical CPU run, persistent storage never re-homed, and zero leaked pins. Red-proof: removing the SaveForBackward calls makes the CI variant fail with poison NaN at the first gradient check. scripts/parity/run.sh gains the corresponding GB10 PASS gate (exit 7).
…tensor-parity-853a7fa1)
This was referenced Jun 12, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements T2.4 of zerfoo
docs/plan-gpu-training-hardening.md(verifies UC-GH-4, UC-GH-7): the Wolf-pattern integration stress test on an attention-shaped synthetic graph, building on the linear-net Wolf-pattern training loop from #140.Graph shape (the increment over #140)
attentionNode: single-head self-attentionQ=x@Wq, K=x@Wk, V=x@Wv; A=softmax(Q@K^T/sqrt(d)); out=A@Vbuilt as agraph.Nodewith trainablegraph.Parameters. Forward registers Q, K, V, and the attention weights A viaSaveForBackward(ADR 006). A is multi-consumer in backward: dV, dA, and the softmax denominator-style row reduction.residualNormNode: residual + RMS normalization, saving the pre-norm activation and the normalization inverse (the Wolf QK-norm cached-inverse shape).xfeeds both nodes, so backward exercises the graph's converging-gradient accumulation path.Schedule (the hazard that bit Wolf, sharpened)
Per sample:
Graph.Forward→ host-materialized loss gradient → arena reset BETWEEN forward and backward (every saved lifetime is load-bearing) →Graph.Backwardwith in-place gradient accumulation into persistentgraph.Parameters → per-sample ResetPool. In-place SGD step per batch; 2 epochs × 2 batches × 3 samples; all underZTENSOR_ARENA_POISONwith a deliberately small arena.Assertions
Red-proof
Removing the
SaveForBackwardcalls makesTestAttentionTraining_WolfPattern_StressCIfail deterministically withWq.Gradient[0] = NaN: non-finite under poison (stale arena read)— the saved lifetimes are load-bearing, the test has teeth.CI / GB10
TestAttentionTraining_WolfPattern_StressCIruns in plain CI (host-backed-arena StressEngine, poison on).TestAttentionTraining_WolfPattern_GPUskips without CUDA;scripts/parity/run.shgains a hard PASS gate (exit 7) so the Spark GB10 pod fails unless it passes on real hardware. GB10 evidence will be added to the devlog in this PR before merge.Quality gates:
go build ./..., CI-scopedgo vet,gofmtclean,go test -race ./...full suite green locally.