Skip to content

fix: reference-detection.test.js rename-scenario test leaks state via nested beforeEach #3452

Closed

Description

@eleshar

Problem

`scripts/validation/tests/reference-detection.test.js` › "End-to-End Integration" › "should handle rename scenario correctly" fails:

```
Expected: "issue-triage-agent"
Received: "issue-agent"
```

Root cause (verified, not assumed)

This is not a platform-specific issue — reproduces identically regardless of OS. Confirmed via direct instrumentation of FixSuggester.suggestFix().

The `describe('End-to-End Integration', ...)` block has a nested `beforeEach` (line 154-156):

```js
beforeEach(() => {
suggester.buildIndex(['issue-agent', 'pr-agent', 'release-agent'], 'agent');
});
```

This runs before every test in that describe block, including "should handle rename scenario correctly" (line 173), which then calls its own `buildIndex(['issue-triage-agent', 'pr-agent', 'release-agent'], 'agent')` on top. `buildIndex`/`addToIndex` only adds to the index (Map.set), never clears — so by the time `suggestFix('agents/issue-agent', ...)` runs, the index actually contains 4 entries: `issue-agent`, `pr-agent`, `release-agent`, `issue-triage-agent` — not the 3 the test intends.

Since the whole point of this test is "issue-agent was renamed to issue-triage-agent, so the old name should no longer be in the index," the leaked `issue-agent` entry (still present from the outer `beforeEach`) fuzzy-matches closer to the input `agents/issue-agent` (similarity 0.61) than `issue-triage-agent` does, so the test's own leaked fixture data defeats its purpose.

Fix

The "rename scenario" test needs a suggester instance whose index does not already contain `issue-agent` before it adds `issue-triage-agent`. Either:

  • Give this test its own FixSuggester instance instead of relying on the outer beforeEach's shared one, or
  • Have the outer beforeEach not apply to this specific test (restructure the describe blocks), or
  • Reset suggester.agentIndex explicitly at the start of this test before calling buildIndex again.

Verification

Confirmed via direct instrumentation (temporary console.error in findCandidates) that the index contains all 4 names at the point of failure, not just the 3 from this test's own buildIndex call.

Note on CI visibility

This is not currently caught by CI at all — no active workflow in .github/workflows/ runs the Jest suite (the workflows that did, checks.yml and testing.yml, were archived 2026-09-11 as part of the workflow consolidation; their replacements validation-unified.yml/testing-unified.yml don't exist yet — tracked in #2896 / PR #3359). Not raising that as a separate issue here since it's already an active initiative.

Related

Activity

  1. linear-code commented on Sep 22, 2026

    @linear-code
  2. eleshar commented on Sep 22, 2026

    @eleshar
    ContributorAuthor

    Follow-up: while preparing the fix, running the full test file (not just this one test filtered out) surfaced 4 more pre-existing failures unrelated to the nested-`beforeEach` leak documented here — filed separately as #3460.

    Important nuance for whoever reviews the fix PR: isolating this test's state (giving it its own `FixSuggester` instance) is necessary but not sufficient — even properly isolated, the test still fails, because `suggestFix('agents/issue-agent', ...)` against `['issue-triage-agent', ...]` scores 0.44 similarity (below the 0.6 threshold), for the reason documented as Root Cause 1 in #3460. The state-leak fix and the threshold/normalization fix are two separate, real bugs stacked on the same test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions