Skip to content

Test fidelity check should fail CI when upstream tests aren't ported #53

Description

@patrick-chinchill

Process gap — silent test coverage erosion.

The problem

scripts/verify_test_fidelity.py reports every TS it("...") that has no matching Python test_*():

```
MISSING: [Options Load] should call onOptionsLoad handler for a matching action ID
MISSING: [Options Load] should prefer specific handlers before catch-all handlers
...
TOTAL: 548/588 matched (93%), 40 missing, 3 absorbers
```

But it exits 0 regardless of how many are missing. CI prints "Run with --fix to generate stubs for missing tests" and moves on. New upstream tests land unported, undetected, for releases at a time.

Why it's bad

This is how we ended up with:

Every one of those shows up in fidelity output. Every one got ignored because the script exits 0.

Fix

Option A — strict: exit non-zero when `missing > 0`. Simple, forces discipline.

Option B — graduated: exit non-zero when `missing > K` (where K is the current count, i.e. treat today's gap as the baseline and ratchet down). Allows incremental adoption without a big-bang PR.

Option C — per-file threshold: allow configured gaps per file in a `fidelity-baseline.json` (like pyrefly's). New misses must be explicitly acknowledged.

I'd recommend Option A for post-4.26 PRs + a separate catch-up PR that either ports the 40 missing tests or explicitly marks them as "intentionally skipped" with a reason.

Acceptance

  • `verify_test_fidelity.py --strict` (or equivalent) exits non-zero on missing tests
  • CI workflow invokes the strict mode
  • Existing 40 gaps handled (either ported or baselined with rationale)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions