Repository navigation
ci: Rust Integration Tests job is failing across branches — validators_and_full_nodes_reach_height_3 killed by SIGKILL #298
Description
Summary
The Rust Integration Tests job is currently red on unrelated branches, including Circle's own sync/v0_8_0. In each case the same test is killed by the OOM killer:
SIGKILL [9.729s] (11/13) arc-test-integration::basic validators_and_full_nodes_reach_height_3
(test aborted with signal 9: SIGKILL)
Summary [65.769s] 11/13 tests run: 10 passed, 1 failed, 0 skipped
warning: 2/13 tests were not run due to signal
error: test run failed
This does not look branch-specific, so I wanted to flag it rather than assume it is known.
Affected runs
| Branch | Run | Result |
|---|---|---|
sync/v0_8_0 (internal release sync) |
33160093215 | SIGKILL, same test |
fix/executor-cumulative-gas-overflow |
33372562134 | Rust Integration Tests failed |
fix/key-file-permissions (mine, #288) |
33388046314 | SIGKILL, same test |
The sync/v0_8_0 failure is dated 2026-08-28 and has the identical 10 passed, 1 failed split, so this predates the PRs above.
Notes on the failing test
crates/test/integration/tests/basic.rs:65. From the run logs it brings up seven nodes (node-0 through node-6) in-process. The job runs on ubuntu-latest and already passes --test-threads 1 (ci.yml:133), so test-level parallelism is not the cause; a single test appears to exceed the runner's memory.
For what it is worth, I saw this same binary reach roughly 7.3 GB RSS when running cargo test --workspace on a 7 GB VM, where the kernel OOM killer reported:
Out of memory: Killed process (basic-5d309b0f0) total-vm:21601360kB, anon-rss:7349572kB
I have not been able to determine whether the failure is deterministic or marginal, so I cannot say whether it is a genuine memory regression or a test that has simply grown past the runner budget.
Possibly related: the nextest ci profile is never used
.config/nextest.toml defines a [profile.ci] (line 8) with fail-fast = false (line 11) and retries (line 15):
[profile.ci]
fail-fast = false
retries = { backoff = "exponential", count = 2, delay = "1s", max-delay = "10s" }None of the three CI invocations pass --profile ci:
ci.yml:91—cargo nextest run --locked --workspace --exclude arc-test-integrationci.yml:133—cargo nextest run --locked -p arc-test-integration --test-threads 1ci.yml:139—cargo nextest run --locked --workspace --exclude arc-test-integration --features integration
So CI runs under [profile.default], which sets fail-fast = true and no retries. That matches the observed 2/13 tests were not run due to signal: the run aborts rather than completing, and the failing test is never retried.
To be clear about what this is and is not: wiring up --profile ci would give fuller failure reporting and would retry a flaky test, but if the OOM is deterministic it will not fix the underlying problem. I am flagging it as a likely-unintended gap, not as a fix.
Happy to open a PR for the --profile ci wiring if that is useful. The memory issue itself looks like it needs someone with more context on the integration harness.
🤖 Generated with Claude Code
Activity
osr21 commentedon Aug 31, 2026
Good report — I verified the repo-side claims and can add two things: an answer to the deterministic-vs-marginal question, and a concrete v0.8.0 suspect for why this test's memory grew.
Claims verified
validators_and_full_nodes_reach_height(5, 5, 2, 3)= 5 validators to height 5 plus 2 full nodes to height 3 — 7 in-process nodes, the largest node count of any test inbasic.rs(the others run 5). Consistent with it being the one that dies.ci.ymlnextest invocations run under[profile.default]; nothing setsNEXTEST_PROFILEeither, so the[profile.ci]block is indeed dead config.fail-fast = truematches the observed2/13 tests were not run due to signal.It is marginal, not deterministic
Job-level results for every completed
Public CIrun that reached theRust Integration Testsjob since the v0.8.0 sync landed:main@66ad2d5(the v0.8.0 merge itself, Aug 28)sync/v0_8_0@42cf057fix/executor-cumulative-gas-overflow@2a31a4ffix/key-file-permissions@f8334faThe trees of
42cf057and66ad2d5are not byte-identical overall, but the test harness (crates/test/framework/src/lib.rs) is identical between them, and the two failing PR branches are small deltas on top of66ad2d5that don't touch the harness or node internals. So effectively the same code has gone 1 pass / 3 fails: the test sits right at the runner's memory boundary and the outcome is a coin flip weighted toward failure.One calibration note on the numbers:
ubuntu-latestfor public repos is 4 vCPU / 16 GB RAM (per GitHub's hosted-runner docs), so what CI is exceeding is ~16 GB — your 7.3 GB RSS kill on a 7 GB VM is just where the smaller box gave out, not the peak the binary reaches when given room.A concrete v0.8.0 suspect: the harness now keeps every node alive until the whole test finishes
git diff v0.7.3..v0.8.0 -- crates/test/framework/src/lib.rsshows a node-handle lifetime refactor:run_nodereturned only aNodeOutcome, and the node's handle was dropped when its own step sequence completed — dropping the handle shuts the node down. Nodes were torn down incrementally: in this test, the 2 full nodes died at height 3 and each validator died as it reached height 5, freeing memory as the test progressed.run_nodereturns(NodeOutcome, Option<Handle>), andcheck_resultscollects the handles intolive_handles, dropping them only after every task has joined. The new doc comment is explicit: "handles are released only once all tasks complete."Net effect on this test: all 7 nodes now stay resident — still producing blocks, gossiping, and growing their caches — until the slowest node finishes. Peak memory therefore scales with how long the slowest node takes, and consensus timing varies run to run (missed rounds, rebroadcast timers). That gives you both halves of the observed behavior in one mechanism: higher peak than v0.7.3, and nondeterministic OOM.
The keep-alive change is well-motivated for the crash/restart scenarios (a restarted node needs live peers to sync from), but in this scenario it buys nothing: full nodes never provide sync support (they're not in the validator registry), and validator availability for the full nodes was already guaranteed by the
val_height > full_heightassertion in the scenario builder. A targeted fix could drop handles for nodes no remaining peer depends on, rather than holding all of them to the end.Cheap A/B for anyone with a big enough machine to confirm:
cargo nextest run --locked -p arc-test-integration -E 'test(validators_and_full_nodes_reach_height_3)'at v0.7.3 vs v0.8.0 while recording peak RSS (/usr/bin/time -v, or/sys/fs/cgroup/memory.peakin a scoped cgroup).On wiring up
--profile ciAgreed it looks like an unintended gap, with one caveat given the failure is marginal rather than deterministic:
retries = 2would likely turn most of these red runs green, which unblocks PRs but also silently hides the memory regression. If it gets wired up, it's worth pairing with a visibility step (dump peak memory after the nextest invocation) so the drift stays measurable instead of being retried away.fail-fast = falsealone would at least let the remaining 2 tests run and give full reports in the meantime.fix(ci): reduce integration test memory usage (circlefin#310)