feat(slasher): warn and expose metrics when own validators are slash targets by aminsammara · Pull Request #24923 · AztecProtocol/aztec-packages

aminsammara · 2026-07-22T21:54:27Z

Closes A-1443.

Operators currently get no signal when the network starts voting to slash their validators — during the v5 inactivity-slashing incident, operators found out only after stake was lost. This PR makes the node detect when its own validators are targeted by onchain slashing, at two points in the lifecycle:

Vote time (early warning): the slasher client subscribes to VoteCast events (only when the node runs validators), decodes each vote, and warns once per vote for each of the node's own validators it names. Each warning carries the running tally against the quorum — (7 of 65 votes needed to slash) — so successive warnings read as progress toward a slash rather than repeated noise. Because quorum requires a majority of a round's slots, it cannot be reached before roughly the first 78 minutes of a 2.5-hour round, which is the window an operator has to react.
Round execution: handleRoundExecuted filters the already-fetched Slashed events to own validators and emits a WARN plus slash count and amount. No additional L1 reads.

Five metrics, all node-level with no per-validator labels:

aztec.slasher.own_validator.targeted_count — cumulative count of votes naming an own validator, once per vote per validator
aztec.slasher.own_validator.current_round_votes_max — highest vote count against any committee position held by an own validator in the current round, reset each round
aztec.slasher.quorum_size — the threshold to compare it against
aztec.slasher.own_validator.slashed_count and .slashed_amount — executed slashes, the amount in whole staking-asset tokens

The alert is current_round_votes_max approaching quorum_size; the operator then finds which validator from the WARN, which carries the address as a structured log field. Keeping identity in logs rather than metric labels means each metric is one series per node regardless of how many attesters it runs, which in turn makes zero-seeding cheap enough to keep — increase() needs a prior sample or it misses the first increment, and a flat zero distinguishes "no slashing" from "node stopped reporting".

Implementation notes:

The round tally is kept per flattened committee position, not per validator address, because the contract tallies quorum per position: a validator sitting in several of the round's committees is named once per position by a single vote, and each position races quorum independently. Warnings and targeted_count are still per (vote, validator) — reporting a validator once at its highest position tally — since an operator cares about the validator, not which of its committee seats is being voted on.
Own validator addresses were already passed to createSlasherFacade but only folded into slashValidatorsNever; they are now also threaded as a first-class ownValidators parameter through the facade and factory chain into SlasherClient, independent of slashSelfAllowed (warnings fire regardless of that flag). They are deliberately not part of SlasherConfig, which is runtime-mutable via updateConfig.
The round tally is seeded from the votes already cast before the VoteCast subscription is attached, so a node restarted mid-round reports a correct tally instead of counting from zero for the rest of it. Seeding before subscribing means no vote can be counted by both the seed and the subscription; votes landing between the seed's reads and the subscription attaching are missed, which the tally tolerates as best-effort. The whole-round read is bounded to 32 parallel calls, and the round monitor's listener is registered before the monitor starts so a round boundary crossed during the seed is still announced.
listenToVoteCast collapses each log batch to the latest log per round. A vote is resolved by reading the round's most recent entry, so acting on every log in a backed-up batch would read that same entry once per log.
getSlashingAmounts is memoized — the amounts are Solidity immutable — which removes a repeated read from every vote decode; the settings loader makes the first, priming call at startup.
Nodes running zero validators skip the VoteCast subscription entirely, so they pay no extra L1 RPC.

Known limitations: the round tally is best-effort. Delayed L1 log delivery can miss votes, and because a vote is resolved by reading the round's most recent entry, two closely spaced deliveries can each read the same entry — counting it twice and never reading the one behind it — so the tally can drift a few votes low or high. Both directions are documented on the metrics. The exact fix is a per-event index cursor, which was considered and deliberately deferred; serializing the event handlers was also considered and rejected because both handlers would still read the same most-recent entry. The binary "am I being targeted" signal is unaffected, since reaching quorum takes dozens of votes, and the execution-time path reads Slashed events directly from the L1 block regardless.

Testing: 16 unit tests in slasher_client.test.ts covering per-vote warning (message and context pinned), per-position tallying for a validator sitting in several committees, multi-validator nodes, round rollover (event-driven, clock-driven, and the clock catching up to an event) and quiet-round reset, votes for closed rounds, foreign-validator votes, execution-time metrics, subscription gating, quorum export, and restart seeding (success, failure, and live votes counting on top). Plus 5 in slashing_proposer.test.ts for the log-batch collapse.

…targets The node now detects when its own validators are targeted by onchain slashing, at two points in the lifecycle: - On VoteCast events, it decodes the vote and emits a WARN (once per round and validator) plus an aztec.slasher.own_validator.targeted_count metric, early enough to react within the execution delay / veto window. - On RoundExecuted, it filters the Slashed events to own validators and emits a WARN plus aztec.slasher.own_validator.slashed_count and slashed_amount metrics. Metrics are labelled per validator address so nodes running multiple validators get per-validator series, seeded to zero at startup. Nodes running no validators skip the VoteCast subscription entirely. Fixes A-1443

spalladino

Looks good! Just left a few comments, and I'd appreciate @alexghr's thoughts on how to properly segment the votecount metric per round.

spalladino · 2026-07-24T14:25:37Z

+  /** Rounds mapped to own validators already warned about, so each (round, validator) pair warns only once. */
+  private readonly warnedSelfSlashVotes = new Map<bigint, Set<string>>();


Nit: SelfSlash sounds like it's this validator slashing itself. Not sure how to rename though, maybe SlashToSelf? Perhaps claude can think of something.

Done. Renamed to ownValidators something. Matches naming patterns elsewhere in the code.

spalladino · 2026-07-24T14:26:46Z

+  /** Records that an onchain slashing vote named one of the node's own validators as a target. */
+  public recordOwnValidatorTargeted(validator: EthAddress): void {
+    this.ownValidatorTargeted.add(1, { [Attributes.ATTESTER_ADDRESS]: validator.toString() });
+  }


Should we also scope this by round? Or decrement it when a round ends? Not sure how to best visualize that, but I imagine that an operator would want to see how many times they get targeted per round, to know how far they are from being actually slashed.

If we scope by round then it will explode the number of timeseries prometheus has to track.

Or decrement it when a round ends

Yes, this would be better. We should be able to keep track of how which validators were target in a round and when the round changes, reset the coutner to 0.

Added a per-round gauge rather than a round label.

spalladino · 2026-07-24T14:28:13Z

+      if (warned.has(validator.toString())) {
+        continue;
+      }


Why do we want to warn only once per round? I'd argue that, if you're getting multiple votes, you want to know on each vote you get, since that gets you closer to being slashed.

Agreed - dedupe removed. Every vote warns now adding the running count against quorum.

spalladino · 2026-07-24T14:30:12Z

    const vote = await this.contract.read.getVotes([round, voteCount - 1n]);
    const decoded = decodeSlashConsensusVotes(hexToBuffer(vote));
-    const slashAmounts = await this.getSlashingAmounts();
+    const slashAmounts = slashingAmounts ?? (await this.getSlashingAmounts());


It's probably easier to just add the memoize attribute to getSlashingAmounts and handle the caching in the SlashingProposerContract object (assuming it's not recreated every time).

I agree with Palla, that would be cleaner and would benefit other parts of codebase as well.

spalladino · 2026-07-24T14:37:27Z

+
+  /** Called when we see a VoteCast event. Warns once per round and validator when a vote names one of our own. */
+  protected async handleVoteCast(round: bigint, proposer: string) {
+    const votes = await this.slashingProposer.getLastVote(round, this.settings.slashingAmounts);


If for some reason the VoteCast events back up and the L1 node ends up returning more than one at the same time, this will read the last of them multiple times. It's a corner case, but check if there's a getVote for a given slot at a given round, so you can request the exact vote that corresponds to the event you're handling.

Fixed the repeated read by collapsing each onLogs batch to the latest log per round.

The complete fix is a per-vote index cursor, deliberately deferred; the metric is documented as best-effort in both directions. Tbh I just didn't want this PR to be swallowed to cater to an edge case.

I don't see why that's such a big lift, given the method is already there. Isn't it just a matter of changing getLastVote here to getVote with the vote index?

alexghr · 2026-07-27T10:47:36Z

    const vote = await this.contract.read.getVotes([round, voteCount - 1n]);
    const decoded = decodeSlashConsensusVotes(hexToBuffer(vote));
-    const slashAmounts = await this.getSlashingAmounts();
+    const slashAmounts = slashingAmounts ?? (await this.getSlashingAmounts());


I agree with Palla, that would be cleaner and would benefit other parts of codebase as well.

alexghr · 2026-07-27T10:54:12Z

+    const seedAttributes =
+      ownValidators.length > 0 ? { [Attributes.ATTESTER_ADDRESS]: ownValidators.map(v => v.toString()) } : [];


I don't think we should be seeding these counters to 0. There could potentially hundreds or thousands of attesters loaded into one node and each would create a timeseries at 0.

Better to just leave stuff unitialized and only increment it in the event of a slash.

As we discussed, I dropped the ATTESTER_ADDRESS label, so each metric is one series per node regardless of attester count. That made seeding cheap so I kept it. Will file a separate issue to look at other metrics - i.e. attestedEpochCount - that explode with time / attester count.

alexghr · 2026-07-27T10:55:55Z

+  /** Records that an onchain slashing vote named one of the node's own validators as a target. */
+  public recordOwnValidatorTargeted(validator: EthAddress): void {
+    this.ownValidatorTargeted.add(1, { [Attributes.ATTESTER_ADDRESS]: validator.toString() });
+  }


If we scope by round then it will explode the number of timeseries prometheus has to track.

Or decrement it when a round ends

Yes, this would be better. We should be able to keep track of how which validators were target in a round and when the round changes, reset the coutner to 0.

alexghr · 2026-07-27T11:00:39Z

+    for (const round of this.warnedSelfSlashVotes.keys()) {
+      if (round < oldestLiveRound) {
+        this.warnedSelfSlashVotes.delete(round);
+      }
+    }


I think we should not be editing the collection while iterating it.

Suggested change

for (const round of this.warnedSelfSlashVotes.keys()) {

if (round < oldestLiveRound) {

this.warnedSelfSlashVotes.delete(round);

}

}

const rounds = Array.from(this.warnedSelfSlashVotes.keys())

for (const round of rounds) {

if (round < oldestLiveRound) {

this.warnedSelfSlashVotes.delete(round);

}

}

replaced by a single current-round tally, so there's nothing to iterate or prune.

Quorum is tallied per committee position, not per validator address, so a validator sitting in several of a round's committees held several independent tallies that were being summed. Also drops the per-attester metric label in favour of node-level series, warns on every vote with the running count against quorum, memoizes the immutable slashing amounts, seeds the round tally at startup, and collapses batched VoteCast deliveries.

spalladino · 2026-07-27T21:35:37Z

@@ -297,6 +327,49 @@ export class SlashingProposerContract {
  }
 }

+/** Maximum number of parallel getVotes reads when fetching a whole round. */
+const VOTE_READ_BATCH_SIZE = 32;
+
+/**
+ * The validators a single slashing vote targets, with the amount voted for each. The position is the validator's
+ * index in the round's flattened slash target committees — the unit the contract tallies quorum by. A validator
+ * sitting in several of the round's committees holds several positions, each with its own tally.
+ */
+export type SlashVote = { validator: EthAddress; slashAmount: bigint; position: number }[];
+
+function decodeVote(vote: Hex, validators: EthAddress[], slashAmounts: [bigint, bigint, bigint]): SlashVote {
+  return decodeSlashConsensusVotes(hexToBuffer(vote))
+    .map((units, position) => ({
+      validator: validators[position],
+      slashAmount: slashAmounts[units - 1] ?? 0n,
+      position,
+    }))
+    .filter(v => v.slashAmount > 0n);
+}
+
+/** Arguments decoded from a VoteCast event. */
+export type VoteCastEventArgs = { round: bigint; slot: SlotNumber; proposer: string };
+
+/**
+ * Collapses a batch of VoteCast logs down to the latest log of each round.
+ *
+ * A vote is resolved by reading the round's most recent entry, so acting on every log in a batch would read that
+ * same entry once per log: the newest vote gets counted repeatedly while the ones behind it are never read at all.
+ * Batches only occur when L1 log delivery falls behind, but the resulting miscount lasts for the rest of the round.
+ */
+export function collapseVoteCastLogs(
+  logs: { args: { round?: bigint; slot?: bigint; proposer?: string } }[],
+): VoteCastEventArgs[] {
+  const latestPerRound = new Map<bigint, VoteCastEventArgs>();
+  for (const { args } of logs) {
+    const { round, slot, proposer } = args;
+    if (round !== undefined && slot !== undefined && proposer) {
+      latestPerRound.set(round, { round, slot: SlotNumber.fromBigInt(slot), proposer });
+    }
+  }
+  return [...latestPerRound.values()];
+}
+
 /**


I don't think this collapse is correct. We should be emitting every VoteCast event, without collapsing anything. Adding this collapse here is confusing to consumers of this method.

spalladino · 2026-07-27T21:37:55Z


    // Listen for VoteCast events to warn early when a vote names one of our own validators as a slash target
    if (this.ownValidators.length > 0) {
+      this.metrics.recordQuorumSize(this.settings.slashingQuorumSize);


How does grafana behave if you record this once during node startup, but then you need it several days later, when fetching data from the last few hours? Is it still "seen" by grafana? Or would we need to update this value on every new vote cast to ensure it's available?

spalladino · 2026-07-27T21:40:46Z

+
+  /** Called when we see a VoteCast event. Warns once per round and validator when a vote names one of our own. */
+  protected async handleVoteCast(round: bigint, proposer: string) {
+    const votes = await this.slashingProposer.getLastVote(round, this.settings.slashingAmounts);


I don't see why that's such a big lift, given the method is already there. Isn't it just a matter of changing getLastVote here to getVote with the vote index?

spalladino · 2026-07-27T21:44:08Z

+  /**
+   * Reads the votes already cast in the current round so the tally survives a restart mid-round. Without this a
+   * node restarted partway through a round would report a tally near zero for the rest of it, and never warn.
+   */
+  private async seedOwnValidatorVotes() {


Heads up some metrics here are incremented whenever this method is called, like recordOwnValidatorTargeted. So if the node is running and records its own validator as targeted, and is then restarted, it'll record it again, and grafana will think it got targeted twice.

I think easiest way around is to just remove it, and live with the fact that, if the node was offline, then votes that happened during it being offline won't be reported. Otherwise, we should properly store the last round we scanned in a db and resume from there.

spalladino · 2026-07-27T21:46:23Z

+      // The round monitor may have rolled the tally past this round while the votes were being read
+      if (round > this.ownValidatorVotes.round) {


Isn't this always true, since this is called on startup, when ownValidatorVotes.round is -1?

spalladino · 2026-07-27T21:51:21Z

+      countByPosition.set(position, count);
+      const entry = targeted.get(validator.toString());
+      if (!entry || count > entry.votes) {
+        targeted.set(validator.toString(), { validator, slashAmount, votes: count });


What happens if different votes set different slash amounts? Wouldn't this persist only the last valid value?

…targets (#25058) Closes A-1443. Reworked version of #24923 targeting the v5 line, addressing the review feedback there. Operators currently get no signal when the network starts voting to slash their validators — during the v5 inactivity-slashing incident, operators found out only after stake was lost. This PR makes the node detect when its own validators are targeted by onchain slashing, at two points in the lifecycle: - **Vote time (early warning)**: a new `OwnValidatorSlashMonitor` (following the `SlashRoundMonitor`/`SlashOffensesCollector` decomposition) subscribes to `VoteCast` events — only when the node runs validators — and warns on every vote that names one of them, with the running tally against the quorum: `Own validator 0x… targeted by slashing vote (7 of 65 votes needed to slash)`. Quorum needs a majority of a round's slots, so warnings start well inside the window an operator has to react. - **Round execution**: `handleRoundExecuted` filters the already-fetched `Slashed` events to own validators and emits a WARN with the exact slashed amount. No additional L1 reads. Five node-level metrics (no per-validator labels, so one series per node regardless of attester count): `aztec.slasher.own_validator.targeted_count`, `.current_round_votes_max` (highest tally against any committee position held by an own validator, reset each round), `aztec.slasher.quorum_size` to compare it against, and `.slashed_count` / `.slashed_amount` for executed slashes. The alert is `current_round_votes_max` approaching `quorum_size`; the WARN identifies which validator via a structured log field. Design notes: - **Votes are read by index, events are only triggers.** `VoteCast` carries no vote index, so the monitor keeps a per-round cursor: on each event it reads the round's `voteCount` and processes votes `[cursor, voteCount)` via a new `getVoteAt(round, index)`. Duplicate or batched deliveries are no-ops, backlogs are caught up by the next event, and a failed read is retried because the cursor only advances after a vote is successfully processed. - **All processing is serialized** through a single queue, with the round revalidated after every await, so concurrent event handlers cannot double-count, drop votes into the wrong round, or emit out-of-order tallies. `stop()` drains the queue and suppresses any late warns/metrics. - **Startup baseline instead of replay.** On start the monitor reads the current `voteCount` (before subscribing) and skips everything before it: replaying old votes would double-count cumulative counters across restarts. Votes cast while the node is offline or starting up are not counted, and L1 reorgs can cause small drift within a round — both documented on the metrics. - **The tally is per flattened committee position** because that is the unit the contract tallies quorum by: a validator sitting in several of the round's committees is named once per position by a single vote, each position racing quorum independently. Warnings and `targeted_count` remain per (vote, validator), reporting the validator at its highest position tally. - Own validator addresses were already passed to `createSlasherFacade` for `slashValidatorsNever`; they are now also threaded as `ownValidators` into `SlasherClient`, independent of `slashSelfAllowed`. `getSlashingAmounts` is memoized (the amounts are Solidity `immutable`), and `SlashVote` is renamed to the singular `SlashVoteTarget`. Testing: 22 unit tests for the monitor (per-vote warning with pinned message/context, per-position tallying, multi-validator nodes, round rollover in both directions and quiet-round reset, closed-round and foreign-validator votes, cursor semantics under duplicate/batched/missed events, failed-read retry, drain serialization, mid-drain rollover, startup baseline and its failure fallback, stop/restart lifecycle, subscription gating), 3 client wiring tests, and an anvil-backed `getVoteAt` decoding test in the ethereum package.

…index (#25068) Next-line counterpart of #25058 (itself a rework of #24923), which targets the v5 line. Closes A-1443 alongside it. Three commits, kept separate (`ci-no-squash`). ### `feat(slasher): warn and expose metrics when own validators are slash targets` - Warns when an onchain slashing vote names one of the node's own validators, and when an executed round slashes one of them. - Adds five node-level metrics: `own_validator.targeted_count`, `own_validator.current_round_votes_max`, `own_validator.slashed_count`, `own_validator.slashed_amount`, and `quorum_size`. - Tallies per flattened committee position, which is the unit the contract tallies quorum by: a validator sitting in several of a round's committees holds several positions, each racing quorum independently. Warnings report the validator's highest position tally. - The whole feature is skipped when the node runs no validators, so those nodes pay no extra L1 subscription. ### `feat(l1): emit vote index in SlashingProposer VoteCast event` - `VoteCast` gains a non-indexed `uint256 voteIndex` carrying the index of the vote it recorded within the round (the value `getVotes(round, index)` accepts). - The event becomes self-identifying: a listener can read the vote back without re-deriving the index from the round's vote count, which needed an extra L1 read per event and could only ever approximate which vote the event referred to. - This changes the event signature and therefore its topic0. External indexers filtering on `VoteCast` need to update their ABI. The unrelated `VoteCast` on `Governance` is untouched. - TS ripple: `SlashingProposerContract.listenToVoteCast` now hands `{ round, voteIndex, proposer }` to its callback. ### `refactor(slasher): drive own-validator vote tracking from the VoteCast vote index` - The monitor's cursor is now fed purely by event indices: no `getRound` read per event, and no startup baseline read (`start()` is synchronous, and the client no longer has to await it before subscribing). - A missed event delivery is healed by gap-filling from the cursor up to the latest event's index; a duplicate or out-of-order delivery is a no-op. A failed vote read leaves the cursor put, so the next event retries it. - Starting mid-round still counts only votes cast from the first event onwards, so cumulative counters are not double-counted across restarts. - Vote processing stays serialized through the existing queue, which is what keeps the cursor sound and the warning tallies monotonic. ### Tests - `l1-contracts`: 34 tests under `test/slashing/` pass, including a new `test_voteEmitsVoteIndex` asserting the index the event carries across two votes in a round. - `@aztec/ethereum`: 22 tests in `slashing_proposer.test.ts` pass; the anvil-backed vote test now also asserts the emitted event's `voteIndex`. - `@aztec/slasher`: 22 tests in `own_validator_slash_monitor.test.ts` (duplicate delivery, gap-fill, stale index, mid-round start, failed-read retry, drain serialization, rollover mid-drain, stop/restart) and 44 in `slasher_client.test.ts` pass.

AztecBot · 2026-08-03T20:18:03Z

This issue was automatically closed because it was referenced in PR #25068 which has been merged to the default branch.

View workflow run

aminsammara added the backport-to-v5-next label Jul 23, 2026

aminsammara requested a review from spalladino July 24, 2026 13:36

spalladino reviewed Jul 24, 2026

View reviewed changes

alexghr reviewed Jul 27, 2026

View reviewed changes

spalladino reviewed Jul 27, 2026

View reviewed changes

This was referenced Jul 30, 2026

feat(slasher): warn and expose metrics when own validators are slash targets #25058

Merged

feat(slasher): warn on own-validator slash votes using VoteCast vote index #25068

Merged

AztecBot closed this Aug 3, 2026

		/** Rounds mapped to own validators already warned about, so each (round, validator) pair warns only once. */
		private readonly warnedSelfSlashVotes = new Map<bigint, Set<string>>();

		const seedAttributes =
		ownValidators.length > 0 ? { [Attributes.ATTESTER_ADDRESS]: ownValidators.map(v => v.toString()) } : [];

		// The round monitor may have rolled the tally past this round while the votes were being read
		if (round > this.ownValidatorVotes.round) {

Conversation

aminsammara commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spalladino left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AztecBot commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

aminsammara commented Jul 22, 2026 •

edited

Loading