You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The conformance corpus from #310 is faithful, but its guards have holes the review of PR #353 found by mutation. A corpus whose rows can vanish silently is a guard that decays without anyone noticing — which is the same failure mode as the drifted fakes it replaced.
Three gaps, in severity order:
A row can be deleted with everything green.the_corpus_carries_the_rows_the_decision_named checks that certain flag names are mentioned somewhere across all rows, not that specific rows exist. The reviewer deleted a single row — including the regression row for the very --ignore-not-found defect Fix --ignore-not-found and drive the corpus through both fakes #310 fixed — and both cargo test and pytest stayed fully green. Guard row identity, not aggregate mentions.
The honesty guard is Rust-only. The test enforcing that every row's verified line opens with measured or unverified runs on the Rust side alone, so a corpus edited from the Python side can lose it silently. Both drivers read the same file; both should enforce its invariants.
The value-flag tables are now parallel in two languages with nothing comparing them — five tables in Rust and Python, no test diffing them, and only ~14 of ~44 flag names have a corpus row. So the defect class One conformance corpus for both devpod fakes #309 set out to close is half-closed: the corpus proves the fakes agree on what it covers, and covers a third of the surface.
Also worth doing while here: the seven rows marked unverified are all measurable. The reviewer provisioned a docker provider and an alpine devcontainer under scratch state and measured every one — all matched what the corpus asserts. So the provenance lines overstate the obstacle: measurement was inconvenient, not impossible. Either upgrade those rows to measured with the method recorded, or reword them to say "not measured here" rather than implying it cannot be done.
Red first for each: the mutation that currently passes is the test to write.
Question
The conformance corpus from #310 is faithful, but its guards have holes the review of PR #353 found by mutation. A corpus whose rows can vanish silently is a guard that decays without anyone noticing — which is the same failure mode as the drifted fakes it replaced.
Three gaps, in severity order:
the_corpus_carries_the_rows_the_decision_namedchecks that certain flag names are mentioned somewhere across all rows, not that specific rows exist. The reviewer deleted a single row — including the regression row for the very--ignore-not-founddefect Fix --ignore-not-found and drive the corpus through both fakes #310 fixed — and bothcargo testandpyteststayed fully green. Guard row identity, not aggregate mentions.verifiedline opens withmeasuredorunverifiedruns on the Rust side alone, so a corpus edited from the Python side can lose it silently. Both drivers read the same file; both should enforce its invariants.Also worth doing while here: the seven rows marked
unverifiedare all measurable. The reviewer provisioned a docker provider and an alpine devcontainer under scratch state and measured every one — all matched what the corpus asserts. So the provenance lines overstate the obstacle: measurement was inconvenient, not impossible. Either upgrade those rows tomeasuredwith the method recorded, or reword them to say "not measured here" rather than implying it cannot be done.Red first for each: the mutation that currently passes is the test to write.