Repository navigation
tests: fix CI flake in test_rope by bounding its tolerance by n_ctx - #31874
Merged
Merged
Conversation
The rotation angle of a position p is rounded in f32 to about p*FLT_EPSILON radians, and the CPU reference and the GPU kernels reach it through different rounding sequences (cumulative product versus pow). With positions up to 8192 this puts the NMSE floor between backends near 8e-8, so the default 1e-7 threshold fails at random on Metal. Derive the tolerance from n_ctx instead: it stays at 1e-7 for n_ctx=512 and becomes 9.5e-7 for the YaRN cases at n_ctx=8192.
ggerganov
approved these changes
Nov 8, 2026
CISC
approved these changes
Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Fix CI: test-backend-ops ROPE(type=f16, n_ctx=8192, ...) occasionally exceeds the default NMSE threshold of 1e-7 on gpu-metal (seen at 1.09e-7 on the M4 runner).
The n_ctx=8192 YaRN cases added in #30206 draw positions up to 8191, where the f32 rotation angle is only resolved to about 5e-4 rad. The CPU reference builds the angle by cumulative product and the GPU kernels by pos * pow(), so they disagree at that resolution whatever the trig accuracy.
The tolerance now follows that bound, max(1e-7, (n_ctx * FLT_EPSILON)^2): unchanged for n_ctx=512, 9.5e-7 for n_ctx=8192. This follows test_rms_norm_mul_rope, which already relaxes its tolerance for large positions (#28175).
Additional information
The floor is not Metal specific: over 400 runs of the same 20 cases, Metal on an M5 Max reaches 8.7e-8, CUDA and Vulkan on an RTX PRO 6000 reach 8.4e-8 and 9.7e-8. Metal simply has the thickest tail, and the M4 runner is the one that crosses it.
#30802 enables the same cases on Vulkan, WebGPU, SYCL and OpenVINO.
Requirements