Skip to content

test(ci): give ASAN recursion-constraint test a 20m timeout - #24318

Closed
AztecBot wants to merge 2 commits into
merge-train/spartan-v5from
cb/asan-recursion-timeout-spartan-v5
Closed

test(ci): give ASAN recursion-constraint test a 20m timeout#24318
AztecBot wants to merge 2 commits into
merge-train/spartan-v5from
cb/asan-recursion-timeout-spartan-v5

Conversation

@AztecBot

Copy link
Copy Markdown
Collaborator

Why merge-train/spartan-v5 (PR #24272) keeps getting dequeued

The train PR's merge-queue CI (merge-queue-heavy, 10 grind runs) fails because a single ASAN unit test times out, not because of anything in the train's commits.

The failing test, on every grind run:

FAILED: barretenberg/cpp/build-asan-fast/bin/dsl_tests \
  --gtest_filter=HonkRecursionConstraintTestWithoutPredicate/2.Tampering (601s) (code: 124)

code: 124 is a timeout kill, not an assertion failure. The dashboard history for this test on v5-next shows it creeping up over a few days — ~270s (06-24) → ~400s (06-25) → >600s (06-26), and now all 10 grind runs hit the 600s default and are killed, dequeuing the train on each Auto-merge/dequeue cycle.

Root cause

This ASAN test proves and verifies a full UltraHonk circuit. Its cost has grown with the verifier circuit on the v5 line. In barretenberg/cpp/bootstrap.sh, test_cmds_asan emits every ASAN test with CPUS=4:MEM=8g, so they inherit the 600s default timeout (ci3/source_test_params: TIMEOUT=${TIMEOUT:-600s}). The full suite runs through non-strict parallelize (CPU-based concurrency + GNU-parallel --memsuspend backpressure), so under the parallel grind the test slows well past 600s and is killed.

Fix

Give just this heavy ASAN recursion test a 20m timeout — the same budget already used for the other heavy recursion proving test (bbapi_tests ChonkPinnedIvcInputsTest.AllPinnedFlows, TIMEOUT=20m). One-line behavioral change: the emitted command becomes …:CPUS=4:MEM=8g:TIMEOUT=20m ….

Local verification

  • Built build-asan-fast dsl_tests and reproduced the original grumpkin g1 data not found only because the warm CRS was the pre-v2 format; after barretenberg/scripts/download_bb_crs.sh (fetches grumpkin_g1_v2.flat.dat), the test runs for real.
  • Ran the exact command: [ OK ] HonkRecursionConstraintTestWithoutPredicate/2.Tampering (178752 ms) — passes in ~3min in isolation (peak ~10.9 GB). So this is a wall-clock-budget problem under grind contention, not a logic failure or hang; a longer timeout is the correct, minimal fix.
  • Confirmed source_test_params parses the multi-segment prefix and exec_test applies timeout -v 20m.

Full ./bootstrap.sh ci (merge-queue-heavy, EC2-orchestrated) can't run on this host, so ci-full is set here to exercise the ASAN path through PR CI.

This is shared CI infra; landing it via the spartan-v5 train carries it into v5-next, where the regression is observed.


Created by claudebox · group: slackbot

The build-asan-fast dsl_tests HonkRecursionConstraintTestWithoutPredicate/2.Tampering
proves and verifies a full UltraHonk circuit. In isolation it runs ~3min, but its cost
has grown with the verifier circuit and under the parallel full-test grind it now exceeds
the 600s default timeout and is killed (code 124), dequeuing merge-train/spartan-v5 on
every merge-queue attempt. Give it the same 20m budget as the other heavy recursion test.
@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-full Run all master checks. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Jun 26, 2026
@AztecBot

AztecBot commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator Author

Automatically closing this stale claudebox draft PR (no updates for 5+ days). Re-open if still needed.

@AztecBot AztecBot closed this Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-full Run all master checks. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant