chore: merge next into merge-train/fairies (resolve conflicts, unblock #24951) - #24965
Merged
Conversation
Updates every README in the e2e test section to describe the suite as it will look once the nine open round-2 consolidation PRs land. This is docs-only — no code, config, or `bootstrap.sh` changes. ## What changed per README - **`end-to-end/README.md`** (root): repointed the `LOG_LEVEL` example off the deleted `proving/empty_blocks.test.ts` to `proving/default_node.test.ts`; linked the new p2p README from the category table (the p2p row now reads "yes" and points at `src/p2p/README.md`); and renamed the `setupBlockProducer` submission-window literal `1024` to the named constant `NO_REORG_SUBMISSION_EPOCHS`. - **`end-to-end/src/single-node/README.md`**: rewrote the `cross-chain/` row for the merged layout (`CrossChainMessagingTest` + `message_test_helpers.ts`, and the `l1_to_l2` / `l1_to_l2_inbox_drift` / `l2_to_l1` / `token_bridge` files); folded `public_payments`/`sponsored_payments` into `private_payments.parallel` and fixed the stale `failures.parallel` name to `failures`; replaced `slasher_config`/`sequencer_config` with the merged `runtime_config`; replaced `world_state_pruning`/`empty_blocks` with the merged `default_node`; added the previously-missing `snapshot_sync` to the `sync/` row; and applied the `NO_REORG_SUBMISSION_EPOCHS` rename. - **`end-to-end/src/multi-node/README.md`**: named `equivocation_offenses` in the slashing row (in place of the separate `duplicate_proposal`/`duplicate_attestation` files) and applied the `NO_REORG_SUBMISSION_EPOCHS` rename in the `MOCK_GOSSIP_MULTI_VALIDATOR_OPTS` preset. - **`end-to-end/src/p2p/README.md`** (new): created with the content added by #24500. - **`end-to-end/src/automine/README.md`** and **`end-to-end/src/infra/README.md`**: untouched — both are behavior-level (no file names) or cover a directory none of the nine PRs change, so they remain accurate as-landed. --------- Co-authored-by: AztecBot <tech@aztec-labs.com> (cherry picked from commit 1cc5f91)
## What / why
Round-4 e2e speedup, PR 3 of the effort. The `single-node/bot` suite's
file-level `beforeAll` costs
~178s on CI with 97% untagged, and the residual proven-chain waits in
the `private_payments`/`failures`
fees suites (~32s/shard) were also invisible. This PR instruments both
so the cost is attributable, and
diagnoses the bot residual.
Spans are pure passthrough when `TEST_TIMING_FILE` is unset (see
`fixtures/timing.ts`), so this is zero
behavior change for normal runs and CI.
## Commit
- `test(e2e): instrument bot suite setup and fees proven-chain
residuals`
- Bot suite (`single-node/bot/bot.test.ts`): `setup:wallet` around
`EmbeddedWallet.create`,
`wallet:create` around `createSchnorrInitializerlessAccount`, and
`setup:bot` around the four nested
`describe` beforeAll hooks (`Bot.create` / `AmmBot.create` /
`CrossChainBot.create` / `BotStore`).
- Fees harness (`single-node/fees`): `wait:proven-checkpoint` around
`catchUpProvenChain`'s poll loop,
and `warp:proven-checkpoint-epoch` around `advanceToNextEpoch` via a new
instrumented
`FeesTest.advanceToNextEpoch()` wrapper; `waitForEpochProven` and the
direct call sites in
`failures`/`private_payments` now route through it.
- Reuses existing tags where the concept matches (`wallet:create` is
already what
`createFundedInitializerlessAccounts` fires;
`warp:proven-checkpoint-epoch` is already what
`advanceToNextEpoch` fires in `block-production/setup.ts`) rather than
minting near-duplicates.
## Diagnosis
The timing environment (`shared/timing_env.mjs`) folds *every*
`beforeAll` in a file — the file-level
one plus each nested `describe`'s — into a single suite-scoped
`beforeHooksMs`, and every span fired
during any beforeAll lands on that same suite line. The shortlist read
the whole number as the
file-level hook and, seeing only `setup:env:*` tagged, attributed the
rest to the one visible untagged
file-level call (`createSchnorrInitializerlessAccount`). It is actually
the nested hooks.
Local read (`TEST_TIMING_SPANS=1`, `PIPELINING_SETUP_OPTS` = 12s L2
slots, production sequencer, fake
prover), suite `beforeHooksMs = 153,733 ms`:
- `setup:env:*` (file-level `setup()`): ~4.6s — already tagged
- `setup:wallet` (`EmbeddedWallet.create`, ephemeral): **137 ms** — the
PXE has `autoSync:false`, no
genesis walk
- `wallet:create` (`createSchnorrInitializerlessAccount`): **69 ms** —
initializerless, no deploy tx, as
designed
- `setup:bot` (4 nested hooks): **149,582 ms = 97.3%** of the file's
beforeAll
- transaction-bot `Bot.create`: 32.2s (token deploy class+instance, then
private+public mint)
- bridge-resume `new BotStore(...)`: 0.9 ms
- amm-bot `AmmBot.create`: 73.9s (deploys 4 contracts + set_minter +
mint + add_liquidity)
- cross-chain-bot `CrossChainBot.create`: 43.5s (TestContract deploy +
seed 2 L1→L2 msgs + wait ready)
**Root cause: structural, not a bug.** The bot suite runs the production
Sequencer to exercise
CHECKPOINTED/PROPOSED follow modes and L1 fee-juice bridging, so each
`Bot.create` deploys real
contracts and mints serially at slot cadence (`bot/src/factory.ts`). The
two calls the shortlist
suspected are ~0.1s each.
No fix is shipped here because there is no safe one-liner: speeding this
up means batching/parallelizing
the `bot/src/factory.ts` deploy/mint paths (production code,
PR-6-shaped), genesis-seeding (PR 1/2), or
cutting the CI slot time (PR 5) — none belong in the instrumentation PR.
Tracked as a round-5 item; the
new `setup:bot` span makes that win measurable. Full write-up in
`tmp/SPEEDUP_ROUND4_FINDINGS.md`.
## Expected effect
No wall-clock change (passthrough spans). The deliverable is visibility:
the previously-invisible
`setup:bot` / `setup:wallet` / `wallet:create` decomposition of the bot
beforeAll, and the
`wait:proven-checkpoint` / `warp:proven-checkpoint-epoch` residual in
the fees suites.
Measured numbers from a full green CI run will be appended below once
available.
## Measured impact
Both runs are full CI runs. PR run: `ci/x-fast` on head `1f7898cd2b` (CI
1783345614459429, 1,799 test
lines). Baseline: `ci/x-full-no-test-cache` on the exact merge-base
`a8cfa93df5` (CI 1783341726388943,
1,863 test lines). No wall-clock change claimed (spans are passthrough;
suite-level noise floor is
~11s median / ~52s p90) — the deliverable is attribution, measured as
span `busyMs`.
Bot suite (`suite` line, beforeAll = 181.5s on the PR run vs 182.1s
baseline):
- Baseline: 7.4s tagged (`setup:env:*` only) → **174.8s / 96%
invisible**.
- PR run: `setup:bot` **174,203 ms** (4 occurrences, max 86,961 ms = the
amm-bot hook),
`setup:wallet` **250 ms**, `wallet:create` **144 ms**; residual now
**−1.5s ≈ 0** →
**100% of the beforeAll is attributed**.
- This confirms the diagnosis: `EmbeddedWallet.create` +
`createSchnorrInitializerlessAccount` are
~0.4s combined; the "invisible 178s" is the four nested describes'
bot-factory setup
(deploys + mints at production 12s-slot cadence), folded into the file's
suite line by the
hook accounting.
Fees suites (`private_payments.parallel` × 8 shards + `failures`):
- Baseline: avg **32.3s/shard untagged residual** (zero
`wait:proven-checkpoint` /
`warp:proven-checkpoint-epoch` lines on these suites).
- PR run: `wait:proven-checkpoint` fires on all 9 suite hooks, **293,888
ms total**
(avg 32.7s/shard — matches the residual exactly), plus one in-test
occurrence (32.1s);
`warp:proven-checkpoint-epoch` 19 occurrences, 361 ms total (the warp
RPC itself is
near-free — the cost is entirely the proven-chain poll that follows).
Per-shard residual is now
≈ −1s, i.e. fully attributed.
Net: **~469s/run of previously invisible setup/wait cost is now tagged
and attributable**
(174.2s `setup:bot` + 0.4s wallet spans + 294.3s fees proven-chain
waits), with no behavior or
wall-clock change.
Fixes A-1407
(cherry picked from commit 1476179)
…24566) PR 4 of the round-4 e2e speedup effort: extend the merged `warpWithSequencersPaused` primitive (from #24475, proven on `pipeline_prune`) to the remaining production-sequencer-bound dead waits. One commit per converted site so a CI-hostile one reverts in isolation. On the fresh `merge-train/spartan-v5` base, most of the sites the plan targeted have already been warp-optimized (they use `warpToBuildWindowForSlot` / `warpToEpochStart` / `waitUntilNextEpochStarts`, and `pipeline_prune` already uses `warpWithSequencersPaused`). The residual waits in the remaining sites ride real-time consensus/production that a warp cannot safely collapse — warping would skip the checkpoints/attestations/offenses the tests are verifying. After reading each file on the fresh base, only one genuinely-dead wall-clock wait was left to convert; the rest are honest skips (per the round-3 skip rule: an honest skip beats a flaky convert). ## Converted (one commit) - **`multi-node/recovery/proposal_failure_recovery` — missed-L1-publish test.** Between "proposed chain reached slotTwo" and the prune, the test waited in wall-clock (`waitUntilL1Timestamp`) for the L1 clock to roll past slotOne so the archiver prunes the uncheckpointed slotOne/slotTwo blocks. The pipelined slotTwo broadcast has already reached every node and slotThree does not build until slotTwo, so nothing must be produced in that gap — a genuine dead wait, the direct `pipeline_prune` analog. Replaced with `warpWithSequencersPaused(..., { restart: false })`: pause across the warp (warping under a running sequencer would interrupt in-flight builds), keep the sequencers stopped until the prune is confirmed so no proposer builds on the unpruned tip, then restart them for recovery. Expected saving: up to ~1 L2 slot (aztecSlot=36s on this suite's cadence) per run of that test. ## Skipped, with reasons - **`proposal_failure_recovery` — orphan-prune test (second `it`).** The prune wait overlaps P2's concurrent S2 checkpoint rebuild inside the same slot (pipelining builds S2 during S1 while the orphan is pruned). Pausing the sequencers to warp would stop the S2 rebuild the test then asserts on. No pausable dead gap. - **`multi-node/recovery/equivocation_recovery`.** The dominant waits are the heal phase (`waitUntilCheckpointNumber` / `waitForAllNodesToReachCheckpoint` for baseline+2) which requires B/C to actually produce two checkpoints after A is stopped, plus `waitForOffenseOnNodes` which requires real slashing-round detection. Warping skips the very production being verified. - **`multi-node/slashing/slash_veto_demo`.** `waitForSubmittableRound` accrues inactivity offenses to quorum over proven blocks, and the veto/expiry logic rides real slashing-round consensus. This is proof-submission/consensus real time (the prover must actually run) — warp does not help, same as the fees `catchUpProvenChain` finding from round 3. - **`multi-node/slashing/attested_invalid_proposal`** (now under `slashing/`, not `invalid-attestations/` — base drift). Epoch advances already use cheatcode warps (`advanceToEpoch` / `advanceToNextEpoch`); the remaining waits are real-time committee/attestation/offense-detection rounds (bad proposer must build and broadcast, lazy validator must attest, honest node must detect the offense). - **`multi-node/block-production/multi_validator_node`.** Cost is real-time attestation + checkpoint building via an actual deploy tx; the only clock jump (`advanceToEpoch` past the validator-set lag) is already a cheatcode warp, and this test was just flake-fixed (#24543) so it is left untouched. - **`single-node/proving/optimistic.parallel` (`wait:epoch` tails, pool 2).** Already warp-optimized on this base: every epoch-boundary skip uses `warpToEpochStart` / `waitUntilNextEpochStarts` / `warpToBuildWindowForSlot`. The residual `wait:epoch` is the deliberate ~2-slot real-time tail (the epoch's final checkpoint must land and the sequencer must settle) — pausing across it would suppress that checkpoint. The `waitUntilCheckpointNumber(midCheckpoint)` waits require the sequencer to produce two mid-epoch checkpoints, and `waitUntilProvenCheckpointNumber(..., 240)` rides real prover latency. Nothing left to safely convert. - **`wait:tx-mined` sites (pool 3).** These are inclusion waits for in-flight builds, not waits for a future slot, so they do not qualify (as anticipated in the plan). Not converted. ## Local verification - `yarn build` (full TS project): passes. - `yarn format end-to-end` / `yarn lint end-to-end`: clean. - The converted test is a 4-validator mock-gossip pipelining suite, too heavy to run on the dev machine, so it was not executed locally. CI (`ci-no-fail-fast`) is the verifier; per-commit isolation contains any flake. ## Measured impact The conversion landed as scoped: `warp:sequencers-paused` goes from 1 occurrence run-wide (pipeline_prune, pre-existing from #24475) to 2 — the new one on the converted `proposal_failure_recovery` test. - Converted test ("prune and recover when proposer fails to publish to L1"): total 130.8s → 122.5s (−8.3s), body 114.6s → 109.2s (−5.4s). The added `warp:sequencers-paused` span is 36.1s. - The net saving is modest: the pause+warp+drain+deferred-restart cycle itself costs ~36s, only ~8s less than the real-time prune-boundary wait it replaced — not a full slot. The clean attribution is the span count (1→2); the ~8s test-total delta sits near the multi-node noise floor. - Untouched sites are flat within noise: the sibling orphan-prune test 93.7s → 91.5s, and pipeline_prune 260.4s → 258.7s. This is consistent with the PR's own finding that the warp lever is largely exhausted on this base — one convertible site, a single-test gain. Baseline CI run 1783374468714213 (merge-base e1711d3, full). PR CI run 1783387686257121 (x-fast). ## Notes for the rest of round 4 - No new shared helper was added — the converted site reuses the existing `warpWithSequencersPaused` on `SingleNodeTestContext` (inherited by `MultiNodeTestContext`). PRs 1a/1b/6/5/7 have nothing new to reuse from here. - Finding relevant to the round's premise: on this drifted base the multi-node recovery/slashing suites' large `wait:checkpoint`/`wait:epoch` pools are dominated by real production (checkpoint building, slashing rounds, prover latency), not dead clock time. The warp lever is largely exhausted there; PR 5 (shrink slot times) is the remaining cadence lever for those suites. Fixes A-1183 (cherry picked from commit cfb6c7f)
## What
PR 5 of the round-4 e2e speedup effort: shrink e2e slot times, the
highest-leverage remaining lever
now that the warp lever is exhausted (PR 4) and the fees setup costs are
pure inclusion latency at
CI cadence (PR 6). Every cadence-bound cost — setup txs, inclusion
waits, epoch walks — scales
linearly with what these commits cut.
Config-only. Three commits, in **ascending risk order** so the
CI-survey/fix phase can revert from
the top. A fourth candidate (WIDE_SLOT 72s → 48s) was investigated and
**dropped** — see below.
All guard math is recomputed from `stdlib/src/timetable/budgets.ts` +
`stdlib/src/timetable/proposer_timetable.ts`:
maxBlocksPerCheckpoint = floor((S - init - D - 2P - prepCp) / D) must be
>= 1
where `S` = aztecSlotDuration, `init` = checkpointProposalInitTime (1s,
never clamped), `D` =
blockDurationMs/1000, `P` = p2pPropagationTime, `prepCp` =
checkpointProposalPrepareTime. Fast-profile
clamping (`P -> 0.5`, `prepCp -> 0.5`, `minBlock -> 1`) applies only
when `ethereumSlotDuration < 8`
(`FAST_PROFILE_ETHEREUM_SLOT_DURATION`). At or above 8s the production
budgets apply verbatim.
## Commit 1 (lowest risk) — `perf(e2e): cut default CI L1 block time to
8s`
`DEFAULT_L1_BLOCK_TIME`: `process.env.CI ? 12 : 8` → `8`. Local dev
already ran at 8, so this only
removes a CI-vs-local cadence asymmetry. Default single-node L2 slot
goes 24s → 16s.
Guard (default single-node: D = DEFAULT_BLOCK_DURATION_MS/1000 = 3,
production budgets P=2,
prepCp=1, init=1):
- before (eth 12): S = 2x12 = 24 → floor((24 - 1 - 3 - 4 - 1)/3) =
floor(15/3) = **5** blocks/cp
- after (eth 8): S = 2x8 = 16 → floor((16 - 1 - 3 - 4 - 1)/3) =
floor(7/3) = **2** blocks/cp
Both >= 1. eth=8 is AT the fast-profile boundary, so budgets stay at
production values (no clamping).
**Blast radius is much smaller than the round-4 plan assumed.** The plan
expected this to cut the
fees / cross-chain / bot families, but the base has drifted: those
suites now run on
`PIPELINING_SETUP_OPTS` / `AUTOMINE_E2E_OPTS` (both set
`ethereumSlotDuration: 4` explicitly) and no
longer read `DEFAULT_L1_BLOCK_TIME`. The genuine default-cadence
consumers (eth from
`SingleNodeTestContext.getSlotDurations`, no explicit eth) are four
single-node proving/recovery
suites: `proving/default_node`, `proving/cross_chain_public_message`,
`recovery/sync_after_reorg`,
`partial-proofs/multi_root`. The only other `process.env.CI` timing
coupling
(`multi-node/slashing/inactivity_setup.ts`, `process.env.CI ? 8 : 4`)
sets its own eth slot and does
not read `DEFAULT_L1_BLOCK_TIME`, so it is untouched.
## Commit 2 (medium risk) — `perf(e2e): cut reorg cadence to 24s slots`
`REORG_TIMING_BASE`: aztecSlotDuration 36 → 24, blockDurationMs 8000 →
5000 (epoch stays 4 slots).
Shared by both reorg profiles, both below the fast-profile boundary:
- `FAST_REORG_TIMING` (eth 4): proving/optimistic reorg describes,
l1-reorgs/
- `MULTI_VALIDATOR_REORG_TIMING` (eth 6, attestationPropagationTime
0.5): recovery/proposal_failure_recovery,
recovery/equivocation_recovery, high-availability/ha_sync,
high-availability/ha_checkpoint_handoff
Both run fast-profile: P = 0.5, prepCp = 0.5, init = 1.
Guard, FAST_REORG_TIMING and MULTI_VALIDATOR_REORG_TIMING (identical
budgets):
- before: S = 36, D = 8 → floor((36 - 1 - 8 - 1 - 0.5)/8) =
floor(25.5/8) = **3** blocks/cp
- after: S = 24, D = 5 → floor((24 - 1 - 5 - 1 - 0.5)/5) = floor(16.5/5)
= **3** blocks/cp
blockDurationMs drops 8000 → 5000 specifically to preserve 3
blocks/checkpoint. Leaving it at 8000
would give floor(13.5/8) = **1**, which would break the l1-reorgs
suites' `assertMultipleBlocksPerSlot(2)`
(they push TX_COUNT=8 with maxTxsPerBlock=1 to force multi-block
checkpoints). All other timing scales
off `test.constants.slotDuration` (waits, warps).
equivocation_recovery's manual fit calc still holds
(3 x 5s = 15 + 0.5 init + 5 final block + 2.5 finalization = 23s <=
24s); its comment is updated.
## Commit 3 (higher risk) — `perf(e2e): cut multi-validator
block-production cadence to 24s slots`
`MULTI_VALIDATOR_BLOCK_PRODUCTION_TIMING`: aztecSlotDurationInL1Slots 3
→ 2 (36s → 24s at eth 12),
blockDurationMs 6000 → 4000. eth stays 12 (production budgets).
Consumers: block-production/simple,
first_slot, proof_boundary.
Guard (eth 12, production budgets prepCp=1, init=1; P = per-test
attestationPropagationTime), S=24, D=4:
- simple (P=default 2): floor((24 - 1 - 4 - 4 - 1)/4) = floor(14/4) =
**3** (was 4 at 36/6)
- proof_boundary (P=default 2): floor((24 - 1 - 4 - 4 - 1)/4) = **3**
(was 4)
- first_slot (P=0.5): floor((24 - 1 - 4 - 1 - 1)/4) = floor(17/4) =
**4** (was 4)
All >= 1. simple asserts no block-count (only no-sequencer-failures);
proof_boundary asserts
proof-vs-boundary timing that scales off `test.constants`; first_slot
keeps 4 blocks/checkpoint
(its comment stays accurate). So the drop 4 → 3 is harmless for those
three.
`high_tps` is **pinned to the old 36s/6s cadence at its call site**
(`aztecSlotDurationInL1Slots: 3`,
`blockDurationMs: 6000` in its setupOpts, overriding the shared
profile). Its checkpoint packing is
2 txs x 2.5s = 5s per block, which needs a 6s block sub-slot; a 4s block
cannot hold it, and the suite
hard-asserts `max-checkpoint-length == 4` and `max-txs-per-block == 2`.
Pinning keeps high_tps at its
tuned cadence rather than rewriting its whole timing model. The other
three suites still take the cut.
## Dropped candidate — WIDE_SLOT 72s → 48s
Investigated per the plan's stretch item and **dropped**.
`blob_promotion` hard-asserts a checkpoint
with >= `PIPELINE_EXPECTED_BLOCKS_PER_CHECKPOINT = 8` blocks. At the
profile's D = 5.5s:
- current 72/12: floor((72 - 1 - 5.5 - 4 - 1)/5.5) = floor(60.5/5.5) =
**11** blocks/cp (fits 8)
- 48/12: floor((48 - 1 - 5.5 - 4 - 1)/5.5) = floor(36.5/5.5) = **6**
blocks/cp (cannot fit 8)
Forcing 8 blocks would require shrinking D to ~4.5s, which lands the
guard at exactly 8 with zero
margin — under the deliberate `mockGossipSubNetworkLatency: 500ms` the
suite runs with, and on top of
the profile's documented A-914 constraint (pipelined
multiple-blocks-per-slot starves non-proposer
nodes with `CheckpointNumberNotSequentialError` when the L2 slot is
tightened, worsened by a shorter
slot). This is the "provably can't fit" case, and it cannot be verified
locally (heavy multi-node
pipelining), so it is not shipped. proposed_chain / cross_chain_messages
(only need 2 blocks/cp) would
survive the guard but share the A-914 risk; not worth the exposure for
one profile.
## Local verification
- **Commit 1: verified.** Ran `single-node/proving/default_node.test.ts
-t 'returns initial block data'`
locally (eth=8 = the new CI cadence, which local already uses). The node
came up at
`ethereumSlotDuration: 8, aztecSlotDuration: 16, blockDurationMs: 3000`
and logged
`Sequencer timetable initialized with 2 blocks per slot
{"maxNumberOfBlocks":2}` — exactly the guard
value above. No "Invalid timing configuration", no missed-slot warnings,
test passed. Budgets stayed
at production values (`attestationPropagationTime: 2, minBlockDuration:
2`), confirming eth=8 does not
trip fast-profile clamping.
- **Commits 2 and 3: guard math + CI only.** The reorg and
multi-validator block-production suites are
too heavy for the dev machine (multi-node + real-time waits + proving).
The guard math above is the
verification; CI validates end-to-end. The formula was confirmed exact
against commit 1's live run.
## Revert protocol for the fix phase
Commits are ordered by ascending risk; revert individual commits from
the top on flake rather than
fighting them:
1. **Any block-production suite flake** (simple / first_slot /
proof_boundary) → revert commit 3 first.
proof_boundary is the most real-time-sensitive (it warps to N-3 then
runs the boundary in real time
at the tighter 24s cadence); watch it first.
2. **Any reorg / prune / HA flake** (l1-reorgs, proving/optimistic reorg
cases,
recovery/proposal_failure_recovery, recovery/equivocation_recovery,
high-availability/*) → revert
commit 2. Watch the l1-reorgs `assertMultipleBlocksPerSlot(2)`
assertions and equivocation_recovery's
slashing-round timing.
3. **Any of the four default-cadence single-node proving/recovery
suites** → revert commit 1 (lowest
probability; the cadence is what local already runs).
## Notes for PR 7 (bot suite cadence)
The bot suite (`single-node/bot/bot.test.ts`) uses `setup(0, {
...PIPELINING_SETUP_OPTS, ... })`, which
sets `ethereumSlotDuration: 4` explicitly. It does not read
`DEFAULT_L1_BLOCK_TIME`, so **commit 1 does
not affect the bot suite** — its cadence lever remains its own
PIPELINING preset, unchanged here.
## Measured impact
The cadence cuts land a run-wide −6.4%: −1,600s summed across shards
over the 135 suites present in
both runs (−2,046s of improvements against +446s of regressions). The
wall-clock benefit is smaller,
since shards run in parallel across workers.
Controls confirm the deltas are real cadence effects, not run-type
artifacts. `high_tps` (deliberately
pinned at the old cadence) is unchanged (+0.1s), and the WIDE_SLOT /
explicit-preset proving suites are
flat — pipeline_prune −1.2s, blob_promotion −0.1s, long_proving_time
−1.1s, prune_when_cannot_build
−0.1s. Only the default-cadence and reorg/block-production timings
moved.
Largest per-suite wall cuts (default-cadence proving/recovery from
commit 1; reorg from commit 2):
- optimistic.parallel −399.9s (−23.5%, ×8 shards)
- proof_boundary.parallel −346.0s (−25.6%, ×5 shards)
- full −202.7s; blocks.parallel −196.3s (−28.8%, ×5 shards)
- invalidate_block.parallel −118.5s (×11 shards); sync_after_reorg
−108.3s
- data_withholding_slash −72.0s; proposal_failure_recovery.parallel
−60.1s;
cross_chain_public_message −48.2s; equivocation_recovery −46.9s;
late_prover_tx_collection −43.3s
Regressions, all on high-variance multi-node suites that are not
cadence-cut here (run-to-run noise,
not attributable to this change): multi_root +74.1s, inactivity_slash
+64.4s,
attested_invalid_proposal.parallel +32.1s, proposed_chain.parallel
+26.1s, bot +22.3s.
The metric is whole-suite wall time (sum of test durations + hooks); a
cadence change carries no new
span to attribute against, so these are run-vs-run wall deltas — the
flat pinned/preset controls above
are what make the cadence attribution defensible.
Baseline CI run 1783417125211175 (merge-base 13a53f1, full). PR CI
run 1783426164237985
(x-fast).
Fixes A-1184
(cherry picked from commit eedf4eb)
…24564) ## What / why PR 2 of the round-4 e2e speedup effort. The fees-family suites pay a `setup:bridge` span (~48s ≈ 4 production slots per process) in `FeesTest.applyFPCSetup`, bridging fee juice from L1 to the BananaFPC via the gas portal (`bridgeFromL1ToL2` = prepare-on-L1 + advance 2 blocks + a claim tx). That funding is pure setup plumbing — the fees tests snapshot the FPC's gas balance and assert deltas, they don't assert on the bridge itself. Fee-juice balances are just public-data leaves at `computeFeePayerBalanceLeafSlot(address)`, which `getGenesisValues` (`world-state/src/testing.ts`) already prefills for genesis-funded accounts — the same mechanism that funds the initial accounts and (already, on this base) the sponsored FPC. This PR seeds the BananaFPC's fee juice at genesis instead of bridging it during setup. Note: the plan's premise that `fundSponsoredFPC: true` bridges was stale — the sponsored FPC has been genesis-funded since #19532 (setup pushes its deterministic address into the genesis-funded list). The remaining `setup:bridge` cost in the fees family is the BananaFPC bridge, which this PR removes. ## Mechanism - The BananaFPC address depends on the BananaCoin address and the FPC admin (the first setup account), and the accounts are generated *inside* `setup()` with random secrets — so the FPC address can't be precomputed by the test before genesis. Fixed deploy salts are added for BananaCoin and BananaFPC so both addresses become deterministic once the admin is known, and a `computeExtraGenesisFundedAddresses(defaultAccounts)` hook on `SetupOptions` runs after the accounts are generated and before genesis values are computed. `FeesTest` uses it to derive the BananaFPC address from the first account and add it to the genesis-funded list. - The returned address flows through the existing initial-accounts path (`addressesToFund` → `getGenesisValues`), so it is funded with the same fee juice as an initial account (`10^22`) and automatically included in the L1 `FeeJuicePortal` `feeJuicePortalInitialBalance` (`fundingNeeded`) accounting — the portal's locked L1 balance stays consistent with total L2 fee-juice supply, exactly as for every other genesis-funded balance. - `applyFPCSetup` deploys the FPC with the fixed salt and asserts the deployed address equals the seeded one, so any drift in the deterministic deploy params surfaces as a clear error instead of a downstream "insufficient fee payer balance". The `bridgeFromL1ToL2` call is removed. - The `setup:bridge` span wrapper is kept in place for the bridging that remains, so its disappearance from the fees setup hooks is the measured proof. ## Scope decisions (site by site) - **`FeesTest.applyFPCSetup` BananaFPC funding** → moved to genesis. Used by `account_init`, `private_payments.parallel`, `failures`, `gas_estimation.parallel`. This is the only funding that moved. - **`FeesTest.mintAndBridgeFeeJuice`** (bridges to an arbitrary recipient) → kept. It fires in `account_init` test bodies that test the bridge/claim flow ("pays natively in the Fee Juice after Alice bridges funds") — real behavior under test. - **`account_init` `FeeJuicePaymentMethodWithClaim` / `prepareTokensOnL1`** → kept. Tests the atomic claim-in-deploy flow. - **Sponsored FPC** → already genesis-funded via `fundSponsoredFPC: true` on this base; no change. - **`cross_chain_messaging_test`** (`fundSponsoredFPC: true`) → already genesis-funded; its own cross-chain bridging harness is the behavior under test, kept. - **`bench/client_flows_benchmark`** bridges to its BananaFPC → out of scope (benchmark harness, not in the fees e2e timing family), kept. - Production/sandbox funding paths (cli, `aztec sandbox`) → untouched. The new hook defaults to unset; only the e2e fixtures opt in. ## Expected effect ~48s × the number of fees processes that ran the FPC bridge in setup (account_init, private_payments, failures, gas_estimation). No C++, no new config, no change to production genesis roots. ## Local verification - `single-node/fees/account_init -t 'pays privately through an FPC'` passes; the span leaderboard for the run shows **zero `setup:bridge`** occurrences (was ~48s), while `deploy:fpc`/`deploy:token` remain. The FPC-paying test asserts the FPC's gas balance decreases by the fee, so the genesis funding is exercised end to end, and the deployed-address assertion held. - `single-node/fees/account_init -t 'after Alice bridges funds'` passes with `setup:bridge` present in the test body, confirming the retained bridge/claim harness still works. ## Measured impact The `setup:bridge` span is eliminated across the run: 17 occurrences totalling 646.4s of bridge setup time in the baseline drop to zero in the PR run. The 13 fees-hook bridges (~48s each) are the bulk of it. - Fees beforeHooks per shard: `private_payments.parallel` −51.1s (×8 shards), `gas_estimation.parallel` −48.1s (×3), `failures` −47.8s, `account_init` −47.8s — each matching the ~48s (4-slot) bridge cost. - `setup:bridge` busyMs removed per suite: account_init 48.1s, failures 48.2s, gas_estimation ×3 = 144.6s, private_payments ×8 = 384.7s (13 fees-hook bridges ≈ 625.5s), plus the ~5s sponsored-FPC bridges on `amm` and `storage_proof`. - Total beforeHooks reduction over the suites present in both runs: −907.4s summed across shards (wall-clock benefit is smaller, since the fees shards run in parallel). - One suite moved the wrong way within noise: `fee_juice_payments` +11.0s. It keeps its real bridging/claim txs (no `setup:bridge` span removed), so this is run-to-run variance, not a regression from this change. Baseline CI run 1783374468714213 (merge-base e1711d3, full). PR CI run 1783375878950264 (x-fast — the comparison is per-suite over the suites present in both runs). Fixes A-1405 (cherry picked from commit 1321e98)
Round-4 e2e speedup, PR 1b (adoption half of the AuthRegistry prize). **Stacked on #24567** (`spl/genesis-prefilled-nullifiers`, the genesis-nullifier mechanism). This PR targets that branch; retarget to `merge-train/spartan-v5` once #24567 merges. ## What Make e2e environments start with the standard contracts (AuthRegistry, PublicChecks, HandshakeRegistry) already "published", so `ensureAuthRegistryPublished` and its two siblings skip their two publish txs each. At production cadence `setup:auth-registry` alone was ~16 min/run (2 sequential txs per process, paid per-process across the suite). This collapses that span to sub-second. ## Mechanism (three parts, all default-off outside e2e) - **Genesis nullifiers.** Per standard contract, seed `siloNullifier(ContractClassRegistry, classId)` and `siloNullifier(ContractInstanceRegistry, instanceAddress)` (the real derived address — standard contracts are not magic-address protocol contracts) into the fixtures' `getGenesisValues` call, via PR 1a's new 5th param. These are exactly the nullifiers the publish txs would emit, so the AVM's deployment-nullifier check passes when the contracts are called publicly. A single helper (`getStandardContractGenesisNullifiers`) feeds all four e2e genesis builders (`fixtures/setup.ts`, `e2e_prover_test.ts`, `p2p/p2p_network.ts`, `multi-node/governance/add_rollup.test.ts`) — the latter three recompute a genesis that must reproduce the L1-deployed archive root, so they must seed the identical set. Non-e2e callers (cli, sandbox/local-network) are untouched. - **Archiver preload.** `registerStandardContracts` mirrors `registerProtocolContracts`: it seeds each contract's class (with recomputed `publicBytecodeCommitment`), instance, and public-function signatures into the contract store at block 0, reading the bundled `@aztec/standard-contracts` artifacts. It is idempotent (skips already-registered classes on restart) and gated behind a new archiver config flag `testPreloadStandardContracts` (env `TEST_PRELOAD_STANDARD_CONTRACTS`, default **false**). The flag is set from the e2e node config so every spawned node (validators, prover nodes) picks it up through the normal config path. This adds `@aztec/standard-contracts` as an archiver dependency. - **Guard short-circuit.** The `ensure*Published` helpers are unchanged. Their guards read `wallet.getContractClassMetadata(id).isContractClassPubliclyRegistered` (to `aztecNode.getContractClass`) and `wallet.getContractMetadata(addr).isContractPublished` (to `aztecNode.getContract`) — both are archiver-store reads, not nullifier-tree reads. The store preload alone makes both short-circuit; the genesis nullifiers are what make the contract actually callable in the AVM afterwards. Keeping the helpers doubles as a regression check: if seeding ever breaks, the tx path re-engages and tests still pass, just slow. ## Why the flag is test-only (A-1257 rationale) Preloading unconditionally in production would recreate the A-1257 collision #24254 fixed: a real on-chain publish of a preloaded class would collide with the block-0 preload, because production genesis will **not** carry the matching nullifiers. The flag is only set by the e2e fixtures, which also seed the nullifiers, keeping the store and the nullifier tree consistent. A single source of truth (`getPublishableStandardContracts`) drives both the preloaded set and the seeded-nullifier set so they cannot drift. ## State-write verification Inspected the two publish txs on this base: - `ContractClassRegistry.publish` pushes one nullifier (`classId`, siloed to the class-registry address) and broadcasts a `ContractClassPublished` contract-class log carrying the packed bytecode. No public-data writes. - `ContractInstanceRegistry.publish_for_public_execution` asserts the class-registration nullifier exists, then pushes one nullifier (`address`, siloed to the instance-registry address) and broadcasts a `ContractInstancePublished` private log. No public-data writes. So the full state-write set is two nullifiers + two broadcast logs. The nullifiers are replicated via genesis `prefilledNullifiers`; the archiver normally learns the class/instance from those two logs (`data_store_updater.ts`), and the block-0 preload replaces that path exactly. No `genesisPublicData` is required. PXE-side registration (`wallet.registerContract`) is unaffected and still runs in the helpers. ## Local verification - `automine/accounts/authwit.test.ts` (public-authwit path against the standard AuthRegistry), timing on: passes; `setup:auth-registry` span = **10 ms** (was ~33 s at production cadence). No on-chain AuthRegistry publish (the only `ContractClassRegistry.publish` executions are the test's own AuthWitTest/GenericProxy deploys); the public AuthRegistry contract executes fine against the seeded nullifier. `testPreloadStandardContracts: true` confirmed in the node config dump. - `single-node/fees/account_init.test.ts` (production sequencer + simulated prover node), timing on: 5/5 pass; `setup:auth-registry` span = **10 ms**. Prover node syncs against the seeded genesis (no root divergence), and the flag propagates to it. No duplicate-nullifier errors. - New unit test in `archiver/src/modules/data_store_updater.test.ts`: `registerStandardContracts` preloads each publishable standard contract's class + instance into the store and is idempotent on a second call. ## Expected impact ~14–16 min/run from auth-registry alone, plus more from the two untagged siblings (PublicChecks / HandshakeRegistry) wherever used. ## Measured impact `setup:auth-registry` collapses from 702.2s of setup time run-wide (39 occurrences, 5–35s each) to 482ms total (max 87ms per occurrence) — sub-second run-wide, every shard clean, no seeding or flag-propagation slow path. - The measured beforeHooks reduction is larger than the auth-registry span alone, because the two untagged sibling publishes (PublicChecks, HandshakeRegistry) are removed too. Example: account_init beforeHooks −49.6s vs a 30.9s tagged auth-registry span; the extra ~19s is the two siblings. - Per-suite beforeHooks deltas: account_init −49.6s, fee_settings −49.5s, failures −48.7s, private_payments.parallel −41.8s/shard (×8), gas_estimation.parallel −41.5s/shard (×3), l1_to_l2 −38.1s, l2_to_l1 −37.5s, l1_to_l2_inbox_drift −38.2s, token_bridge −37.7s, fee_juice_payments −37.4s, bot −34.3s; automine/parallel suites shed 5–15s each (their fast-cadence publishes). - Total beforeHooks reduction over the suites present in both runs: −1,649.7s summed across shards (wall-clock benefit is smaller, since shards run in parallel across workers). Baseline CI run 1783389062016888 (#24567 `ci/x-fast`; mechanism-only, so timing-neutral vs merge-base). PR CI run 1783391194852197 (#24568 `ci/x-fast`). ## Notes for downstream - **PR 6 (fees harness):** `applyEnsureAuthRegistryPublished` now short-circuits; the fees setup chain loses the ~32 s/shard auth-registry step, so rebase the overlap/batch shape onto what remains (token deploy, FPC, mints). - **Fix phase / timing capture:** expected span assertion is `setup:auth-registry` around sub-second across the run (10 ms observed locally per process). Any shard still showing tens of seconds means the guards took the slow path (a seeding or flag-propagation bug), not noise. Fixes A-1404 (cherry picked from commit f320049)
Adds `yarn-project/.claude/skills/writing-e2e-tests/SKILL.md`: a skill
that gives an agent everything it needs to place, structure, and write a
robust e2e test with little guidance. Written against the v5-next e2e
layout (category directories with per-category READMEs and context
classes).
## What it covers
- **Step 0 decision ladder**: unit test → new expectation in an existing
test → new `it` in an existing suite → new file on an existing
context/harness → new standalone test.
- **Where to place the test**: recap of the category directories
(`automine/` via `AutomineTestContext`, `single-node/` via
`setupWithProver`/`setupBlockProducer`, `multi-node/` via
`MultiNodeTestContext` on the mock-gossip bus, `p2p/` via
`P2PNetworkTest` on real libp2p, plus `composed/`, `infra/`, `spartan/`,
`bench/`), the multi-node-vs-p2p decision rule, pointers to the
per-category READMEs as the authoritative reference, file naming
(`describe` matches path, header comment), automatic CI registration
through `bootstrap.sh` globs, `.parallel.test.ts` semantics, and
jest/bash timeout sync.
- **Setup reuse**: the three layers (category contexts → domain
harnesses → root `SetupOptions`), the standard suite file shape,
one-environment-per-file, preset spread order, guarded teardown.
- **Readability**: intent-only test bodies, the named-waiter surface
(`fixtures/wait_helpers.ts`, context waiters, `ChainMonitor`), shared
helpers and co-located `*_test_helpers.ts`, simulators with `afterEach`
checks, shared error constants.
- **Speed**: themes distilled from the recent
`test(e2e)`/`perf(e2e)`/`chore(e2e)` speedup PRs and their tracking
issues — avoid unneeded setup, genesis seeding over setup txs,
`BatchCall` batching over `Promise.all` overlap (with the
PXE-serialization caveat), warping dead waits with
`markProvenAndWarp`/`warpWithSequencersPaused` ("an honest wait beats a
flaky warp"), named timing profiles over ad-hoc cadences, and measuring
with the `testSpan` instrumentation + `track-e2e-times` skill before
optimizing.
- **Flakiness**: twelve golden rules synthesized from six months of
deflake PRs and the accumulated flaky-test gotchas (poll-don't-sleep via
named waiters, receipt-anchored assertions, mined ≠ checkpointed ≠
proven, `syncChainTip` tag awareness, pipelining timing margins +
`findSlotsWithProposers`, invariant-not-exact-value assertions, L1
account/nonce and port hygiene, freeze-L1-across-restarts, gossip-mesh
readiness via `runGossipScenario`, fee padding, determinism, honest
timeouts), plus `deflaker.sh` validation and `.test_patterns.yml` as a
last resort.
- Final pre-ship checklist.
All helper/API names cited were verified to exist on
`merge-train/spartan-v5` (`AutomineTestContext.setup`,
`setupWithProver`/`setupBlockProducer`,
`MOCK_GOSSIP_MULTI_VALIDATOR_OPTS`, `wait_helpers.ts` waiters,
`warpWithSequencersPaused`, `markProvenAndWarp`,
`testSpan`/`TEST_TIMING_FILE`, `runGossipScenario`,
`waitForP2PMeshConnectivity`, `findSlotsWithProposers`,
`getPaddedMaxFeesPerGas`, etc.). Genesis-prefill techniques that only
exist in still-open speedup PRs are described as themes to look for
rather than as concrete APIs.
(cherry picked from commit 4c2b6d8)
(cherry picked from commit 23e8e1e)
## What Fixes a low-frequency (~1 in 140) startup-race flake in the `proof_boundary` multi-node e2e suite (CI hash `c06563e7d2b7e067`, group `e2e-p2p-epoch-flakes`). All five scenarios time out in the shared `computeBoundarySlot()` helper at `waitUntilCheckpointNumber(1)` — the 3-validator committee never produces its first checkpoint. ## Why `setupTest` created the three validator nodes sequentially (`asyncMap`) and started each sequencer at node construction. A node subscribes to the gossip topics when its P2P client starts (unconditionally, at node creation), but the in-memory mock gossip bus has no message replay. So when the first-created validator happened to draw the first slot's proposer, it broadcast the inaugural block proposal before the other two committee members had been created and subscribed — the proposal was lost, the 3-of-3 quorum was missed, and the chain forked onto genesis and never produced checkpoint 1. It's randao-dependent (fatal only when the first-created node is the first proposer), hence the low frequency. ## Fix Create the validator nodes with `dontStartSequencer: true`, then `startSequencers(nodes)` once all three exist. All peers subscribe to gossip at creation, so no proposal is broadcast before the whole committee is on the bus. This matches the pattern already used by the non-flaky sibling tests `first_slot.test.ts` and `block-production/setup.ts`. ## Scope Test-only change. Related to A-1419, which tracks the underlying product-level recovery race (competing block-1 re-executions surfacing as `BlockNumberNotSequentialError`). This PR implements that issue's "keep the test startup barrier" acceptance criterion but does **not** resolve A-1419 — the product-side race is unchanged. (cherry picked from commit 54edf7a)
…state (#24578) v5 version of #24558 (same change, based on `merge-train/spartan-v5`). ## Problem When the prover broker cancelled a proving job it settled the job as a cached `{ status: 'rejected', reason: 'Aborted' }` result — in memory and in the database. Because a job's id is a deterministic hash of its inputs, any later enqueue of the same job returned that cached abort (`Cached proving job … Not enqueuing again`) and the facade failed the job with `Aborted`. So once a proof was aborted it stayed aborted, and a retry — a fresh epoch attempt after a reorg/failure, or a re-orchestration after a restart — got the cached abort instead of re-proving. A planned mid-epoch redeploy hit this and turned into a testnet epoch prune. ## Fix Cancellation is now its own first-class, **revivable** state: - A new `aborted` proving-job status (`ProvingJobStatus` / `ProvingJobSettledResult`). `cancelProvingJob` records it, notifies the current waiter, and **persists** it (`ProvingBrokerDatabase.setProvingJobAborted`). - `enqueueProvingJob` **revives** an aborted job: when the producer re-requests it, the broker clears the aborted state — in memory and, via the new `deleteProvingJobResult`, in the database — and re-enqueues it as if new. So a restart mid-revival keeps the job pending rather than resurrecting the abort. Genuine agent errors and timeouts still settle as terminal `rejected` results. Net effect: cancelling frees the job cleanly and records why, but re-requesting the same proof always brings it back to life — in the same process or after a restart — so an abort can never permanently block an epoch from proving. The stacked PR (prover-node clean-shutdown, v5) stops a clean prover-node shutdown from cancelling its in-flight jobs in the first place. ## Tests `proving_broker.test.ts` (both DB variants): cancel leaves a job `aborted`; re-request revives it and it completes; the aborted state persists across a restart; and a revived job survives a restart mid-revival as pending (not aborted). Note: not run locally in this session — the prover-client suite needs a full noir/wasm bootstrap that wasn't available here — so relied on CI. --- *Created by [claudebox](https://claudebox.work/v2/sessions/6fa5e242ed9ceae7) · group: `slackbot`* --------- Co-authored-by: Phil Windle <philip.windle@gmail.com> (cherry picked from commit a4e88ca)
…poch (#24436) Resolves A-1290. ## Problem A `CheckpointProver`'s sub-tree work forks world-state per block. When an L1 reorg prunes a base block, those fork reads fault and the prover permanently rejects its one-shot `blockProofs` promise. The store keeps the prover alive across a prune (`markPruned`) so a re-add of identical content can reuse its in-flight work. But a prune that removes the checkpoint also breaks the fork reads, so that reuse only ever hands back a poisoned prover: a re-add or a full `EpochSession` recreate re-references the same rejected `blockProofs` and fails immediately. The affected node then silently abandons the epoch until a process restart (a healthy prover elsewhere still proves it, so it's not a network stall). ## Fix **Drop the reuse machinery — a pruned prover cannot survive, so rebuild instead of reuse.** - `CheckpointStore.cancelAndRemoveAboveBlock` cancels and **removes** orphaned provers (replacing `markPrunedAboveBlock`); a re-add builds a fresh prover with a fresh `blockProofs`. - Removed the `pruned` flag / `markPruned` / `markCanonical` from `CheckpointProver`, and the `SlotWatcher` that only reaped lingering pruned provers. - `listCanonical` / `addOrUpdate` simplified — every prover in the store is now canonical. **Recover the epoch by rebuilding on re-add, rather than trying to prevent the terminal state.** A prune unwinds world-state as soon as it's detected — before the prover-node cancels the affected session — so a checkpoint prover mid-fork faults while still live and its session goes terminal `failed`. That race is inherent (the world-state rejection and the prover being cancelled are unordered), so instead of trying to reclassify it as a cancellation, we make the terminal state harmless: - `SessionManager.openFullSessionIfReady` no longer treats a terminal session as "already open" — it drops it and rebuilds. So re-adding an epoch's checkpoints reopens a fresh live session over fresh provers, via the checkpoint/prune triggers (ungated by the tick high-water mark). This reopen depends on the store no longer holding the poisoned prover: with the reuse machinery gone the rebuilt session gets a fresh prover with a healthy `blockProofs`, so it can actually prove rather than re-inheriting the rejected one. - `runSession` skips the failure-upload when the session's checkpoints no longer match canonical content — a prune-invalidated failure isn't a genuine proving failure, so it no longer emits a spurious post-mortem upload / alert. The only non-recovered case is "content pruned and never re-added" (chain stops publishing that epoch's checkpoints) — there's nothing to prove on this node then, so abandoning the epoch is correct. **Cancel the pruned prover's in-flight work promptly.** `CheckpointProver` now threads its abort signal into `PublicProcessor.process`, so a prune-driven cancel stops the current block's public execution immediately instead of running it to completion before the next `signal.aborted` check. This is independent of the recovery logic above — it just avoids wasting CPU re-executing a block whose checkpoint is being discarded. The in-flight block has not yet enqueued a broker proof, so aborting it loses no reusable work. ### Note: broker-level proof reuse is unaffected Removing the `CheckpointProver`-level reuse does not waste proving work. The expensive part — the SNARK proofs — is cached in the proving broker, content-addressed by job id, independent of world-state. `cancelJobsOnStop` defaults to `false`, so cancelling a prover does not abort its broker jobs; on an identical re-add the fresh prover regenerates identical inputs, and the broker dedups against the cached jobs/results (bounded by the broker's per-epoch cleanup). The reuse we deleted was witness-generation reuse, whose world-state forks cannot survive a prune anyway. ## Testing - `session-manager.test.ts`: pins the retry/recovery invariant — the periodic tick does not re-attempt a failed epoch (gated by `lastTickEpoch`), but a checkpoint re-add recovers it (ungated); a `failed` session's epoch is reopened once its checkpoints are re-added; a genuine proving failure still uploads a post-mortem; a failure coinciding with a canonical content change (prune) skips the upload. (Red/green verified for both the gate and the upload suppression.) - `checkpoint-store.test.ts` / `prover-node.test.ts`: pruned provers are cancelled and removed (not flagged); a re-add builds a fresh prover. - `checkpoint-prover.test.ts`: cancelling a prover mid-block aborts the signal its public execution is running under (red/green verified). - Full prover-node suite passes (166 tests); package typechecks clean; lint OK. - Updated `optimistic.parallel.test.ts` (the e2e that spawned this issue) to the cancelled+removed semantics. ## Notes - No operator/contract-dev-facing config or API changed, so no changelog entry. - The terminal `failed` state is not prevented, only recovered from: the world-state unwind and the prover cancellation are unordered, so the failure can't be reliably reclassified from the session side. Recovering on re-add is deterministic and needs no such race to be won. (cherry picked from commit c1e6229)
Upstreaming the utilities build for `aztec-kit` so external projects can benefit from our e2e scaffolding. Essentially the same thing we already use, but pure ts rather than the unpublishable .sh script. Verified with `ci-full-no-test-cache` (cherry picked from commit 7e09008)
…l validation (#24694) ## Problem On a live mainnet node, checkpoint-proposal validation could throw an uncaught error out of the gossipsub message handler whenever it ran while the world-state synchronizer trailed the archiver (block source) by a block: ``` ERROR world-state:database Call CREATE_FORK failed: Error: Unable to initialize from future block: 117860 unfinalizedBlockHeight: 117859. Tree name: NullifierTree ERROR p2p:libp2p_service Error handling gossipsub message: Error: Unable to initialize from future block: 117860 unfinalizedBlockHeight: 117859. Tree name: NullifierTree ``` The archiver already had block N (so the block lookups inside checkpoint validation succeeded), but world state's `unfinalizedBlockHeight` was still N-1. When `validateCheckpointProposal` forked world state at the parent block before it had been applied, the raw tree error escaped as an uncaught ERROR-level gossipsub message. The node lost that checkpoint validation/attestation for the round and spammed error logs; it self-healed the next tick, but it should not happen. ## Root cause The two world-state fork sites were asymmetric. The block re-execution path (`reexecuteTransactions`) syncs world state *before* forking. The checkpoint validation path (`validateCheckpointProposal`) forked via `checkpointsBuilder.getFork` *without* a preceding sync. The checkpoint path does sync the block source (archiver) earlier, but that does not advance the world state, so the gap remained. ## Fix - Move the sync-before-fork invariant into `FullNodeCheckpointsBuilder.getFork`: it now calls `worldState.syncImmediate(blockNumber, blockHash)` before `worldState.fork(blockNumber)`, so the single fork helper owns the invariant rather than each caller. `syncImmediate` blocks until world state reaches the block, or throws a typed error if it genuinely cannot, instead of leaking the raw tree error. - In `validateCheckpointProposal`, look up the parent block's hash from the block source and pass it through so the sync re-syncs on a world-state reorg. Wrap the fork in a try/catch that maps any failure to a clean, non-slashable `world_state_not_synced` result, so a sync/fork failure never surfaces as an uncaught ERROR-level gossipsub message. - As a second, independent guard, check that the forked world state's archive root matches the proposal's `lastArchiveRoot` before rebuilding the checkpoint. A mismatch means world state forked from a different chain than the proposal was built on, so we fail fast with a clean, non-slashable `initial_archive_mismatch` result — mirroring the block-proposal re-execution check — instead of a confusing downstream header/archive mismatch. Both new reasons record an `unvalidated` outcome and are non-slashable, matching the other "we couldn't validate right now" / local-state-divergence reasons. Other checkpoint-builder fork sites were reviewed and need no change: the proposer's `checkpoint_proposal_job` forks at its own already-synced, coherence-checked world-state tip, and the block-proposal re-execution path already syncs before forking and performs the equivalent archive-root check. ## Tests - `checkpoint_builder.test.ts`: `getFork` syncs world state to the block before forking, and propagates a sync failure without forking. - `proposal_handler.test.ts`: forks at the parent block number passing its block hash; a fork failure returns `world_state_not_synced`; and a fork whose archive root diverges from the proposal returns `initial_archive_mismatch`. ## Notes The same bug exists on the v4 line — the v4 site is the same function, forking via `this.worldState.fork(parentBlockNumber)` directly without a preceding `syncImmediate`. This fix should be ported there as well. Fixes A-1421 (cherry picked from commit b7a7d1b)
The sqlite3mc we vendor clobbers config params used to find the sqlite wasm file to load. As a result, the bundler-visible default wasm URL was silently discarded on the first call in a given env and wasmBinary was never honored at all. We introduce an index.ts wrapper that takes care of ensuring options survive the first call and every call after. It honors `locateFile`, `wasmBinary`, and `instantiateWasm`, and it defaults to a static `SQLITE3_WASM_URL` that bundlers rewrite to their emitted asset. (We need to follow this up with some simplification of aztec-kit) Closes TRIAGE-200 Fixes #24895
(cherry picked from commit 1d280af)
(cherry picked from commit da9ac1c) CONFLICTS LEFT INTENTIONALLY for follow-up resolution.
) Adapt the upstream NAPI forwarding to the current IPC world-state backend. (cherry picked from commit a4e3a44)
Required for the ported fee-payer phase assertion to exercise the supplied deferred payment method. (cherry picked from commit 019608c)
Use the direct merged-payload simulation path on next, where BatchCall.simulate enters the phase before the first app call.
Forward-ports the **end-to-end / testing** slice of the v5-next → next backlog (work merged to `v5-next` after the ~2026-07-08 cut that reshaped `next`). ## Applied (clean cherry-picks, chronological) - #24518 docs(e2e): update READMEs for the consolidated suite layout - #24534 test(e2e): instrument and diagnose bot suite setup cost - #24566 perf(e2e): warp dead waits in multi-node recovery and proving tests - #24570 perf(e2e): shrink e2e slot times - #24564 perf(e2e): seed BananaFPC fee juice at genesis instead of bridging - #24568 perf(e2e): seed standard contracts at genesis - #24597 chore: add writing-e2e-tests skill - #24590 feat(e2e): interactive handshake e2e - #24671 test: fix proof_boundary startup race ##⚠️ Needs owner conflict-resolution (conflict against reshaped `next`; not included here) Cherry-pick onto this branch and resolve: - [ ] #24569 `git cherry-pick -x 1d280af` — perf(e2e): overlap and batch setup txs in e2e harnesses - [ ] #24479 `git cherry-pick -x da9ac1c` — feat: assert non revertible phase when setting fee payer Part of the manual v5-next backlog sweep. Draft until owners resolve the conflicts above and CI is green.
…24937) (#24944) ## Summary Port of #24937 to `next`. The cherry-pick conflicted in `yarn-project/sqlite3mc-wasm/src/index.ts` because `next` still had the simple sqlite3mc re-export while the source PR replaced it with an init wrapper. I resolved the conflict by using the wrapper from the source PR so `locateFile`, `wasmBinary`, and `instantiateWasm` survive sqlite initialization, while retaining the package's existing exported sqlite types. `origin/port-to-next-staging` was not present after repeated fetches, so this PR targets `next` directly instead of a missing staging base. Fixes #24895 ## Tests - `yarn install --immutable` - `yarn workspace @aztec/foundation build` - `yarn workspace @aztec/sqlite3mc-wasm test src/index.test.ts` - `yarn workspace @aztec/sqlite3mc-wasm build` Full `yarn build` was attempted, but this fresh checkout lacked generated repo-wide artifacts such as `@aztec/l1-artifacts`; the narrower sqlite workspace build passed after restoring the required ignored local artifacts for testing. Closes #24895 --- *Created by [claudebox](https://claudebox.work/v2/sessions/597504b9b3d7d66e/jobs/1) · group: `slackbot` · [Slack thread](https://aztecprotocol.slack.com/archives/C0AGN2WT3CP/p1784817938944179?thread_ts=1784817938.944179&cid=C0AGN2WT3CP)*
…down (#24579) v5 version of #24562 (based on `merge-train/spartan-v5`). Stacked on #24578 — review/merge that first. Note: this is a fresh implementation, not a port — the v5 prover-node architecture differs from `next` (it uses `EpochSession` / `TopTreeJob` rather than `EpochProvingJob` / `ProvingOrchestrator`), which is the structure actually running on testnet. ## Problem On a clean shutdown, `SessionManager.stop()` cancels every live session with reason `'prover-node stopping'`. That flows `EpochSession.cancel` → `TopTreeJob.cancel()` → `topTree.cancel({ abortJobs: true })` — the `abortJobs: true` was **hardcoded**, so a deploy/restart aborted the in-flight top-tree broker jobs. Aborting on a clean restart is wasteful: the proofs are still valid, agents are mid-flight, and a restarted node re-orchestrating the same epoch produces the same deterministically-hashed job ids, so it re-proves from scratch. ## Fix Thread an `abortJobs` decision from the cancel reason down to the broker: - `SessionManager.stop()` cancels with `{ abortJobs: false }` — a clean shutdown preserves the jobs. - `EpochSession.cancel(reason, { abortJobs })` forwards it to `TopTreeJob.cancel(abortJobs)` → `topTree.cancel({ abortJobs })`. - All other cancels (reorg / supersede / deadline) keep the default `abortJobs: true`, since their inputs are stale. Composes with #24578: a clean redeploy mid-epoch neither cancels the in-flight jobs nor (if one ever were aborted) poisons them, so the epoch keeps proving across the restart. ## Tests - `session-manager.test.ts`: `stop()` cancels every session with `abortJobs: false`. - `epoch-session.test.ts`: a normal cancel forwards `abortJobs: true` to the top-tree orchestrator; a clean-shutdown cancel forwards `abortJobs: false`. Note: not run locally in this session — the prover-node suite needs a full noir/wasm bootstrap that wasn't available here — so relied on CI. --- *Created by [claudebox](https://claudebox.work/v2/sessions/6fa5e242ed9ceae7) · group: `slackbot`* --------- Co-authored-by: Phil Windle <philip.windle@gmail.com> (cherry picked from commit c44e74e)
This pr adds an optional prover-node config, `PROVER_NODE_PROOF_SUBMISSION_TARGET_ADDRESS`, that lets a prover send the `submitEpochRootProof` tx to a contract other than the rollup. Everything else is unchanged: all rollup state is still read from the real rollup, only the destination of the submit tx is redirected. When unset, it defaults to the rollup address, so existing provers are unaffected. Use case: A prover that wants to claim prover tips from oxide-styled withdrawal txs must register itself in the Pool contract as the first prover to submit a proof of a given length L. To do that, it deploys a wrapper contract that calls the Pool before (check proven length = L-T) and after (check proven length = L) calling `submitEpochRootProof` on the rollup, then points this config at the wrapper. Closes GK-752 (cherry picked from commit 1039d38)
…ults (#24678) Follow-up to #24436 (A-1290). That PR fixed the immediate prune-induced failure but left a class of race conditions around L1-reorg prunes: every "was this failure a prune or a genuine failure?" predicate samples control-plane state that lags the data-plane world-state unwind, so it is racy by construction. The fix pushes the fact down to where it is unambiguous — **decouple "a checkpoint prover failed" (a fact about the prover) from "the epoch failed" (a decision)**, and never treat a *cancel* (prune / reap / shutdown) as a failure. A genuine failure is then told apart from a prune with no control-plane sampling. - **`CheckpointProver`** — gains `isFailed()`, set when its block proofs reject for a non-cancel reason (a sub-tree fault or a prune-induced fork fault). On failure it fires an `onFailed` callback so the owner can upload that single checkpoint's post-mortem. A cancel leaves `isFailed()` false. - **`EpochSession`** — a fault now settles in one of two terminal states, decided by the provers: - `stopped` — a `CheckpointProver` under it failed. May be a prune, so it is **not** uploaded and the session is dropped; the epoch recovers when a prune/re-add installs a fresh prover. - `failed` (`hasFailed()`) — the session's **own** top-tree/submit work failed while *every* prover was healthy. Healthy provers rule out a prune, so this is a genuine, race-free failure. - **`SessionManager`** — removed `lastTickEpoch`. Two mechanisms stop a doomed epoch being re-proved every tick: `openFullSessionIfReady` refuses to build over a set containing a failed prover, and `recreateInvalidSessions` **retains** a genuinely-`failed` full session as a do-not-re-prove marker, replacing it only when its canonical content changes (a re-add). A `failed` session fires `onSessionFailed` (session-level upload). Recovery from prune/reorg flows through the ungated `checkpoint`/`prune` triggers, reusing already-completed sub-proofs from the content-addressed broker. - **`ProverNode`** — uploads post-mortems at **two levels**: per checkpoint (`tryUploadCheckpointFailure`, wired to `CheckpointProver.onFailed` — fires for prune-induced faults too, on purpose) and per session (`tryUploadEpochFailure`, wired to `onSessionFailed`). `expireEpoch` only releases the chonk cache and reaps — it does **not** upload, since a missed-window epoch's provers may already be pruned by then. - **Re-proving** — add `rerunCheckpointProvingJob` to re-prove a single downloaded checkpoint in isolation (sub-tree only; no epoch top-tree or L1 submit), sharing offline setup with `rerunEpochProvingJob`. `downloadEpochProvingJob` is unchanged (same serialized format). - `epoch-session.test.ts` — a fault ends `stopped` (a prover failed) vs `failed` (top-tree/submit with healthy provers). - `checkpoint-prover.test.ts` — a mid-proof fork fault sets `isFailed()` and fires `onFailed` once; a cancel does neither. - `session-manager.test.ts` — skips building over a failed prover; retains + uploads a `failed` session and replaces it on content change; does not upload on `stopped`; data-plane fault + re-add (identical and different content) recovers and completes. - `prover-node.test.ts` — a registered checkpoint prover failure routes to `tryUploadCheckpointFailure`; `expireEpoch` reaps without uploading. - `upload_failed_proof.test.ts` (e2e) — the existing epoch upload/rerun test, plus a new checkpoint test: force a sub-tree failure via the `checkpointProveOverride` hook, capture the per-checkpoint upload, and re-prove it via `rerunCheckpointProvingJob`. - prover-node README rewritten to the failed-prover / two-level-upload model, with the "how a genuine failure is told apart from a prune" rationale. prover-node unit suite green; full `yarn build`; format + lint clean. Resolves A-1418. (cherry picked from commit 6795246)
…24802) Fixes the `Error: Timeout connecting to bb socket: unknown (retry=false)` failures reported by an operator since v5.0.0, which killed proving jobs during epoch top-tree and cost epochs. ## Root cause The NativeUnixSocket backend gave bb a hard 5s wall-clock budget that was **shared** between two phases: waiting for bb to create its socket file, and connecting to it. `connectWithRetry` reused the `startTime` captured before the file-wait poll loop, so when bb took close to 5s to create the socket (many bb processes spawning simultaneously during top-tree checkpoint/merge jobs, or the Node event loop starving the 50ms polls), the connect phase was entered with its budget already exhausted and threw without making a single connect attempt — that is what the `unknown` in the error message means (`lastErr` was never set). Two aggravating factors: - 5s is an arbitrary opinion about how fast a loaded machine should spawn a process. The pre-socket v4 CLI model had no such deadline — you waited on the child, and the only failures were real process events. - The resulting plain `Error` reached the proving agent, which only honours the retry flag on `ProvingError`s, so the failure was reported `retry=false` and the job failed permanently instead of being re-enqueued. ## Changes ### `barretenberg/ts` — socket backend restructure (`native_socket.ts`) - Replaced the constructor + deferred `connectionPromise` wiring with a `static async new()` factory, matching the shm and wasm backends. An instance can now only exist once connected, so `call()` no longer awaits a stashed connection promise (and is no longer `async` — its body has no awaits). - Startup is one flat wait-and-connect loop with the correct liveness condition: **retry for as long as the bb process is alive**. If bb dies, fail immediately with the real cause (exit code / signal); spawn failures are caught up front by awaiting the `spawn` event. Both 5s timers are deleted. - One generous 60s backstop remains for a bb that is alive but wedged before `listen()`. It kills the process (routing cleanup through the exit path) rather than leaving an orphan. It is a broken-process detector, not a performance expectation: the timed window ends at bb's `listen()`, which is reached after only exec + linking + minimal init (the expensive startup work comes after the socket is up), so firing it requires a machine degraded far beyond ordinary proving load. And if it ever does fire on a merely-distressed machine, the failure is retryable (see below), so the cost is a re-enqueue, not an epoch. - The four copy-pasted reject-pending-callbacks blocks (process error/exit, socket error/end) are consolidated into `failAllPending()`. Note one deliberate behavioural improvement: the socket `error`/`end` handlers now also destroy and null the socket, so subsequent `call()`s fail fast with `Socket not connected` instead of `write after destroy`. - New `native_socket.test.ts` covers: prompt startup, bb taking >5s to create its socket (the incident's failure mode — fails on the old code by construction, passes now), bb dying before the socket exists, and a nonexistent binary. ### `yarn-project/bb-prover` — make startup failures retryable - `BBJsInstance.create` wraps any `Barretenberg.new` failure as `ProvingError(..., retry: true)`: bb startup failures are environmental (machine load, wedged process), never a property of the proof inputs, so the job is always safe to retry. - The two catch sites in `bb_prover.ts` that re-wrap errors (`generateProof`, `verifyProof`) previously constructed a fresh `ProvingError` with the default `retry=false`, silently dropping the flag. They now propagate the inner error's retryability and attach it as `cause`. Errors that were non-retryable before remain non-retryable. The AVM paths don't wrap, so they needed no change. - New `bb_js_backend.test.ts` asserts a startup failure surfaces as a retryable `ProvingError`. ## Result for operators A loaded prover can take as long as it needs to spawn bb; a genuinely broken bb fails after 60s with a concrete, diagnosable cause instead of `Timeout ... unknown`; and if that ever happens the broker re-enqueues the job (up to its retry limit) instead of failing it permanently and costing the epoch. ## Testing - `barretenberg/ts`: new jest suite passes (including a 7s-delayed fake bb that the old implementation fails on). - `yarn-project`: full `yarn build`, bb-prover lint, and the new unit test pass. Root `./bootstrap.sh` green. ## Unrelated CI fix included `docs/examples/ts/aztecjs_runner/run.sh` installed `typescript` and `tsx` unpinned; the TypeScript 7.0.2 release (which drops `lib/_tsc.js`) crashes Yarn 4's builtin `compat/typescript` patch at install time, failing `docs/examples/bootstrap.sh execute` on every branch. Pinned to `typescript@^5.3.3` / `tsx@^4`, matching the existing pin in `docs/examples/ts/bootstrap.sh`. This was the only failure in this PR's first full CI run and needs forward-porting to the `next` line as well. # Conflicts: # barretenberg/ts/bb.js/src/bb_backends/node/index.ts # barretenberg/ts/bb.js/src/bb_backends/node/native_socket.test.ts # barretenberg/ts/bb.js/src/bb_backends/node/native_socket.ts
## Summary Ports #24802 to `next`. This keeps the same behavior change from the merged v5 PR: - bb.js native socket startup now waits while the bb process remains alive, with a 60s wedged-process backstop instead of the old shared 5s socket/connect deadline. - bb startup failures are wrapped as retryable `ProvingError`s in bb-prover. - retryability is preserved when bb-prover re-wraps proof generation and verification failures. ## Conflict resolution The automatic cherry-pick conflicted because `barretenberg/ts/src/...` on the source branch has moved to `barretenberg/ts/bb.js/src/...` on `next`. Resolved by adapting the socket backend changes into the `bb.js` path and preserving `next`'s `createAsyncBackend` API shape, which returns `IMsgpackBackendAsync` rather than constructing a `Barretenberg` wrapper there. Added a small follow-up commit to satisfy the `next` lint rule for `destroy()`. Refs #24802 ## Verification - `git diff --check origin/next...HEAD` - `yarn formatting:fix` in `barretenberg/ts/bb.js` - `yarn test bb_backends/node/native_socket.test.ts --runInBand` in `barretenberg/ts/bb.js` Blocked locally: - `./bootstrap.sh build` in `barretenberg/ts/bb.js` passed formatting, then stopped at codegen because this checkout lacks `barretenberg/cpp/build/bin/bb`. - `JEST_MAX_WORKERS=1 yarn workspace @aztec/bb-prover test src/bb/bb_js_backend.test.ts` could not complete in this cold checkout without built workspace exports for `@aztec/bb.js`/`@aztec/foundation`. --- *Created by [claudebox](https://claudebox.work/v2/sessions/f03a66a010266d34/jobs/1) · group: `slackbot` · [Slack thread](https://aztecprotocol.slack.com/archives/C0AGN2WT3CP/p1784890226762699?thread_ts=1784890226.762699&cid=C0AGN2WT3CP)*
…o next (#24933) Forward-ports the **prover** and **node / world-state / barretenberg** slices of the v5-next → next backlog (work merged to `v5-next` after the ~2026-07-08 cut that reshaped `next`). ## Applied (clean cherry-picks, chronological) - feat(prover): revive cancelled proving jobs from a persisted aborted state (#24578) - fix(prover-node): rebuild pruned checkpoint provers and recover the epoch (#24436) - feat: exported in-process testing network (#24629) - fix(validator): sync world state before forking in checkpoint proposal validation (#24694) ##⚠️ Needs owner conflict-resolution (conflict against reshaped `next`; not included here) Cherry-pick onto this branch and resolve: - [ ] `git cherry-pick -x a4e3a44` — feat(world-state): support prefilled nullifiers in genesis state (#24567) **[RESHAPE — v6 subsystem rewritten; re-implement, do not merge]** - [ ] `git cherry-pick -x c44e74e` — fix(prover-node): do not abort in-flight proving jobs on a clean shutdown (#24579) - [ ] `git cherry-pick -x 1039d38` — feat: allow custom proof submission target address (#24270) Part of the manual v5-next → next backlog sweep. Draft until conflicts are resolved and CI is green.
# Conflicts: # yarn-project/end-to-end/src/automine/phase_check.parallel.test.ts # yarn-project/sqlite3mc-wasm/src/index.ts
AztecBot
added
ci-draft
nchamo
approved these changes
Jul 24, 2026
nchamo
marked this pull request as ready for review
July 24, 2026 14:49
nchamo
enabled auto-merge (squash)
July 24, 2026 14:49
rangozd
pushed a commit
to rangozd/aztec-packages
that referenced
this pull request
Aug 5, 2026
BEGIN_COMMIT_OVERRIDE chore(port): forward-port remaining v5-next fairies backlog to next (AztecProtocol#24949) chore: remove unused l1-contracts repo mirror from release flow (AztecProtocol#24929) chore: merge next into merge-train/fairies (resolve conflicts, unblock AztecProtocol#24951) (AztecProtocol#24965) chore: merge next into merge-train/fairies (resolve conflicts, unblock AztecProtocol#24951) (AztecProtocol#24967) END_COMMIT_OVERRIDE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Merges the current
nextintomerge-train/fairiesand resolves the two merge conflicts that are currently blocking #24951 (mergeable_state: dirty). Merging this PR intomerge-train/fairiesbrings that branch up to date withnextconflict-free, which unblocks #24951 (merge-train/fairies→next).Conflicts resolved
yarn-project/sqlite3mc-wasm/src/index.ts— both branches independently rewrote the vendored-wasm loader entrypoint:nexthas #24937 (fix: respect wasm load config params when instantiating sqlite) — a full implementation that honorslocateFile/wasmBinary/instantiateWasmand keeps the bundler-friendly staticSQLITE3_WASM_URLreference, shipped with a test (index.test.ts).emscriptenLocateFilefor the bundler case.The
nextversion is a strict superset (itsdefaultLocateFileresolvessqlite3.wasmto the same staticSQLITE3_WASM_URL), so I tooknext's version — the v5-next fix is already superseded upstream.yarn-project/end-to-end/src/automine/phase_check.parallel.test.ts— comment-only conflict; the test code is identical on both sides. Kept the more descriptive comment.No production logic beyond the above was changed by the resolution; everything else is a straight merge of
next.Refs #24951
Created by claudebox · group:
slackbot· requested by Nico Chamo · Slack thread