Repository navigation
covers globs + ghost-anchor coverage (action-at-a-distance) #5
Description
Activity
Research:
coversglobs + ghost-anchor coverage (issue #5)Status: research / decision-ready design. No implementation. #5 is labeled
deferred/research, milestone 0.7.0. This doc exists to decide whether and how to un-defer it.Context
Surface's deterministic gate makes exactly one promise (§7 of
docs/surface-proposal.md):the code a claim points at is unchanged since it was last verified.
It explicitly does not promise the documented invariant still holds across the system. The canonical failure (§7) is action at a distance:
rotateRefreshToken's span is byte-for-byte unchanged (hash green), but a newbypassAuthmiddleware insrc/server.tsexposes the rotation path. The gate passes; the doc is now a lie. No deterministic check can catch this — by definition the change is to code no hub declared a relationship with. §11 calls leaving this gap open "the ghost-anchor hole."Note on the issue's section refs: "action at a distance" actually lives in §7, and the open question ("ghost-anchor hole") in §11 (line ~213). There is no §11.1 in the current proposal. Same concept, just cite §7 + §11.
Issue #5 proposes the spec's own forward-declared escape hatch: an optional
coversglob per hub plus a louder coverage pass that looks at diffs falling inside a hub's declared scope — consumed by the (not-yet-built) reviewer plugin, advisory only, never part of the deterministic verdict.What the spec already commits to
- §9.1:
coversis deliberately absent from the MVP — "Forward-declared, not shipped." Asking authors to write a field that does nothing is the ceremony §4 warns against. - §9.3 (Deferred): the reviewer plugin is "LLM off
check --format json: suppress clean refactors, uphold suspected falsifications, look atcoversdiffs." Gated to "teams already running CI review bots. Ship advisory-only first; measure its false-negative rate before letting it suppress anything." - §5 / §8: the hard rule — "the fuzzy layer can only ever advise; it can never be load-bearing for the verdict." Fatigue wants the gate quieter; coverage wants it louder; the resolution is to keep the deterministic core narrow and make loud coverage opt-in.
So the design space is pre-constrained:
coversis additive, opt-in, and advisory. This research is about the mechanism, not whether to change the verdict (we must not).Current architecture (grounding for the design)
Concern Where Notes Hub schema surf-core/src/hub.rs→Frontmatter#[serde(deny_unknown_fields)]; hassummary,anchors,refs. A test (covers_field_is_rejected) currently assertscoversis rejected.Anchor/Span surf-core/src/anchor.rs,resolve.rsat:path → tree-sitterSpan(byte/line range).Verdict/gate surf-cli/src/check.rs→check_claim()hash compare → DivergenceKind::{Changed,Unverified,Unresolvable}; exit 0/1.JSON contract surf-core/src/report.rs→CheckReport/Divergenceversion: 1, additive-only.old_code/magnitudealready modeled as advisory fields — the existing precedent for non-verdict data.Globbing glob 0.3.3, used incheck.rs(--files),workspace.rs,suggest.rs,stats.rsglob::Patternmatching infra already exists.Change detection surf-cli/src/git.rs→changed_files(),show()merge-base diff vs --base; already advisory-only.Existing coverage notion surf-cli/src/suggest.rs→covered_anchors()already computes "which symbols are anchored." A coverspass is the file-glob analogue.Key insight: every primitive #5 needs already exists — glob matching, git diff of changed files, an advisory-field precedent in the JSON contract, and a "what's covered" computation. The work is wiring, not new infrastructure.
Recommended design
Two separable layers; ship in order, stop after L1 if signal is weak.
Layer 1 — schema: accept
coversglobs (small, in-core)- Add
#[serde(default)] pub covers: Vec<String>toFrontmatter(hub.rs). Globs are repo-root-relative, same dialect asconfig.hubs. - Flip
covers_field_is_rejected→covers_field_is_accepted(and add round-trip + default-empty tests). - Inert for the verdict.
check's pass/fail is untouched. Validate the globs insurf lint(reuseglob::Pattern::newerror path already incheck.rs) so a malformedcoversis a lint error, not a runtime surprise. - Optional nicety:
surf lintwarns if a hub's own anchored files are not matched by itscovers(catches authors who fat-finger the glob).
Layer 2 — advisory coverage pass (the "louder look")
- New advisory computation, emitted only in
--format json, never affecting exit code. Add an additive optional field to the report (e.g.coverage_touches: Vec<CoverageTouch>onCheckReport, or per-hub) —versionstays1(additive-only per the contract's stated rule). - Algorithm: for each hub with
covers, intersectgit::changed_files(base)with the hub'scoversglobs minus files already accounted for by green/blocked anchors. The remainder = "changes inside this hub's declared scope that the deterministic gate said nothing about" — exactly theserver.tscase from §7. - Surface these as advisory
CoverageTouch { hub, file, anchored: bool }records. The CLI human output mentions them as a quiet note ("N change(s) in covered scope — review"), distinct from blocking divergences. - This is the seam the future reviewer plugin reads; we are not building the LLM plugin here, only the data it consumes.
Explicitly out of scope (keep deferred)
- The reviewer plugin / LLM judgment itself (§9.3) — separate issue
#reviewer-plugin. - Any path where
coversinfluences the verdict. Non-negotiable per §5/§8.
Open questions to resolve before un-deferring (the actual research output)
- Does
coversearn its keep without the plugin? L2 alone produces an advisory "you changedserver.tswhich this hub covers" note. Is that useful to a human in CI, or just noise until the LLM plugin can triage it? → Decides whether L2 ships before the plugin or with it. - False-negative measurement (§9.3's gate). The spec says measure the false-negative rate before suppression. We can't measure suppression until the plugin exists — but we can measure recall: on a corpus of real action-at-a-distance bugs, how often is the offending file inside some hub's plausible
covers? If authors won't write broad-enough globs, the whole approach is a ghost itself. → Needs a small empirical study on this repo's own history (git log+ known incidents) or a willing design-partner repo. - Authoring burden vs §4 ("no ceremony").
coversis opt-in, but every opt-in field is a maintenance surface. Default-empty + lint-only validation keeps it honest, but: do we wantsurf suggestto proposecoversglobs (e.g., the dir of a hub's anchors)? Trades convenience for the risk of rubber-stamped, meaningless scopes. - Change ≠ falsification (§11). Even a perfect
covershit only proves a change happened in scope, never that the invariant broke. L2 must phrase its advisory output to reflect that — "worth a look," not "this is wrong." Wording matters for trust.
Verification (for whenever L1/L2 are actually built)
- L1:
cargo test -p surf-core(newcoversaccept/round-trip/default tests); author a hub withcovers:and confirmsurf checkexit code is unchanged vs. without it;surf lintflags a bad glob. - L2: fixture repo with a hub covering
src/**, make an unanchored change to a covered file, runsurf check --format json --base <ref>→ assert the advisorycoverage_touchesrecord appears AND exit code is still 0. Reproduce the §7bypassAuthscenario as the canonical test.
Recommendation
Keep #5 deferred as a feature, but this research clears it for a phased un-defer: L1 is a genuinely small, safe, in-core change (schema + lint, verdict untouched) that could land independently and stop being a forward-declared lie. L2 should not ship until Q1/Q2 above are answered — specifically a recall study on real action-at-a-distance cases — because an advisory pass nobody's globs actually cover is itself a ghost anchor. The LLM reviewer plugin stays a separate, later issue.
- §9.1:
L1 split out → #94. The schema-only slice (accept + lint
covers, verdict untouched) is now tracked separately in #94 under milestone 0.7.0, where it can land green on its own.This issue (#5) now scopes the deferred L2 work: the advisory "louder coverage pass" that actually consumes
coversglobs, still gated on the recall study described in the research comment above. Reviewer-plugin consumption remains a separate concern.- added a commit that references this issue
on Jun 18, 2026 Recall study: would
coversglobs catch this repo's real action-at-a-distance cases?This is the recall study Q2 gates un-deferring L2 on. Run against this repo's own 175-commit history and 17 dogfooded hubs.
Method
An advisory's value = recall (does the offending file land in some hub's plausible
covers?) × precision (when it fires, is it a real coupling or noise?). Corpus built fromdocs/dogfood-log.md— the repo's curated record of gate blind spots — with each offending symbol's real location confirmed in source, then measured against three plausible glob breadths: own-file, own-crate (surf-cli/src/**), and repo-wide (**/*.rs).Corpus — 2 documented real specimens
The log records exactly two genuine action-at-a-distance events (the §7
bypassAuthcase is hypothetical TypeScript, not this repo):# Claim that went / would-go false Offending change Lives in A #43:pick()is unused / deletableresolve_gostarted calling itsurf-core/src/resolve.rs(same file aspick)B cli-lint's coverage-nudge behaviorbroadening public_symbols's defaultsurf-core/src/resolve.rs(different crate from the hub's anchor insurf-cli/src/lint.rs)Recall result — depends entirely on glob breadth
- Case A: offending file = the anchored file, so caught at every breadth — but it's a whole-program "unused" property; the log's own lesson is the right gate is the compiler (
#[warn(dead_code)]), not a span/glob. - Case B: the dependency crosses a crate boundary. Own-file and own-crate globs both miss it; only a repo-wide glob catches it.
Glob breadth Recall own-file (the natural instinct) 1/2 (50%) own-crate surf-cli/src/**1/2 (50%) — B still missed repo-wide **/*.rs2/2 (100%) Precision result — the breadth that buys recall destroys precision
A repo-wide
coverson anysurf-clihub fires on every commit touchingsurf-core. From history (175 commits):- 23 commits touched
surf-core/src— every one would fire the advisory on a CLI hub. - Exactly 1 was a genuine coupling (Case B) — and the author caught it by hand and gated it behind
--all, so it never even falsified anything. - → precision ≈ 1/23 (~4%), i.e. ~96% noise, to recover that one case.
Structural finding (generalizes past n=2)
This isn't a small-sample fluke — it's baked into the dependency graph:
- All 11 CLI hubs depend cross-crate on
surf-core(hash_anchor,parse_anchor,public_symbols,Claim,Hub). A file/crate glob can't see the dependency that carries most of the claim's meaning. - 3 core hubs (
hash,resolve,rename) have cross-file deps even withinsurf-core— e.g.hash's claim depends on the identifier-family tables inlang.rs; acovers: …/hash.rsmisses a change there that genuinely alters what gets hashed. - Only 4 leaf hubs (
config,hub-format,lang,anchor) are self-contained — and for thosecoversis coextensive with the anchor, so it adds nothing.
So 13 of 17 hubs have meaning-bearing dependencies outside their own file, and the glob an author would naturally write (cover my own file/dir) systematically misses the cross-crate couplings that are the actual specimens.
Verdict on the gating questions
- Q2 (recall): You can hit 100% recall, but only at a breadth (
**/*.rs) where the advisory fires on essentially every code commit at ~4% precision. At the breadth authors realistically write, recall is ~50% and misses precisely the cross-crate couplings. Authors won't write broad-enough globs without drowning in noise — confirming the worry that "the whole approach is a ghost itself." - Q1 (does L2 earn its keep without the plugin?): No. A ~96%-noise human-facing advisory will be tuned out — the dogfood log already flagged this exact risk ("22 warnings risks being tuned out"). L2 only makes sense as data fed to an LLM reviewer that can triage the noise, not as a standalone CLI note.
Recommendation
Keep #5 (L2) deferred — the study empirically confirms the §7 warning. Specifically:
- Do not ship L2 as a standalone human advisory. Invert the dependency: L2 should be gated on the reviewer plugin, not shippable before it, because only the plugin can absorb the precision cost.
- Drop the dead-code family (Case A) from L2's scope — the log's own lesson is "match the claim to a gate that can see the property"; the compiler beats any glob there.
- One-line study output: on this repo, recall and precision are in direct opposition — every glob broad enough to catch the real (cross-crate) case is ~96% noise, so
covers-as-human-advisory does not earn its keep; it's only viable as plugin input.
Study run on history through
d8b10b8; numbers reproducible viagit log --format=%H -- surf-core/src | wc -land the cross-crate dependency grep oversurf-cli/src/*.rs.- Case A: offending file = the anchored file, so caught at every breadth — but it's a whole-program "unused" property; the log's own lesson is the right gate is the compiler (
- added a commit that references this issue
on Jul 28, 2026
Why
The honest limit (§7): Surface only checks the spans you anchored. A change to code no hub declared a relationship with can falsify a claim while the gate stays green — "action at a distance" (§11.1).
What
Optional
coversglobs per hub + a louder coverage pass that looks at diffs within a hub's declared scope. Consumed by the reviewer plugin (#reviewer-plugin). Advisory only — never part of the deterministic verdict.Unlock trigger
For teams already running CI review bots; ship advisory-first and measure false-negative rate before it suppresses anything.