Skip to content

The conformance corpus's guards can lose a row silently #357

Description

@blooop

Question

The conformance corpus from #310 is faithful, but its guards have holes the review of PR #353 found by mutation. A corpus whose rows can vanish silently is a guard that decays without anyone noticing — which is the same failure mode as the drifted fakes it replaced.

Three gaps, in severity order:

  1. A row can be deleted with everything green. the_corpus_carries_the_rows_the_decision_named checks that certain flag names are mentioned somewhere across all rows, not that specific rows exist. The reviewer deleted a single row — including the regression row for the very --ignore-not-found defect Fix --ignore-not-found and drive the corpus through both fakes #310 fixed — and both cargo test and pytest stayed fully green. Guard row identity, not aggregate mentions.
  2. The honesty guard is Rust-only. The test enforcing that every row's verified line opens with measured or unverified runs on the Rust side alone, so a corpus edited from the Python side can lose it silently. Both drivers read the same file; both should enforce its invariants.
  3. The value-flag tables are now parallel in two languages with nothing comparing them — five tables in Rust and Python, no test diffing them, and only ~14 of ~44 flag names have a corpus row. So the defect class One conformance corpus for both devpod fakes #309 set out to close is half-closed: the corpus proves the fakes agree on what it covers, and covers a third of the surface.

Also worth doing while here: the seven rows marked unverified are all measurable. The reviewer provisioned a docker provider and an alpine devcontainer under scratch state and measured every one — all matched what the corpus asserts. So the provenance lines overstate the obstacle: measurement was inconvenient, not impossible. Either upgrade those rows to measured with the method recorded, or reword them to say "not measured here" rather than implying it cannot be done.

Red first for each: the mutation that currently passes is the test to write.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions