Skip to content

feat(test-infrastructure): keep test temp files under one run root - #698

Open
simonheimlicher wants to merge 5 commits into
mainfrom
work/change-667
Open

simonheimlicher wants to merge 5 commits into
mainfrom
work/change-667

Conversation

@simonheimlicher

@simonheimlicher simonheimlicher commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • A test run keeps every temp file under one run root inside the worktree and removes the root when the run ends.
  • The user temp folder holds the same entries after a pytest run as before it.

Background

Test runs left directories and files in the user temp folder, and a killed run left them there for good. The amended test-language decision rules that each run owns one root, that every temp path reaches it, that Git never searches above it, that a later run sweeps stale roots in its own worktree, and that no test retains a temp path.

Changes

  • spx/15-test-language.adr.md: five rules for the run-scoped temp root, its reach, its Git ceiling, the stale-root sweep, and the ban on retained temp paths.
  • spx/13-infrastructure.enabler/21-test-infrastructure.enabler/test-infrastructure.md: five assertions, each linked to its test.
  • outcomeeng_testing/harnesses/run_temp_root.py: a pytest plugin that creates the run root under .spx/worktree/test-runs/, points the temp location and the Git ceiling at it, sweeps stale roots, removes each tmp_path directory after its test, and restores the environment at run end.
  • conftest.py: registers the plugin.
  • outcomeeng_testing/harnesses/run_temp_root_observation.py and outcomeeng_testing/fixtures/run_temp_root/case_temp_runs.py: run a fixture module under pytest in a scratch worktree and return what the run left behind.
  • Two tests under the node's tests/ directory: one scenario and one compliance file.

Test plan

  • just test spx/13-infrastructure.enabler/21-test-infrastructure.enabler/tests/ passes.
  • just check passes on the head.
  • spx validation markdown, just eval-links and spx spec status --format json pass.

Known gap

The recipe runner in outcomeeng/validation/_engine.py writes outcomeeng-validation-summary-<recipe>-<id>.json to the user temp folder with delete=False before pytest starts, and a failing step leaves its log there. This changeset does not touch the runner. Read-only check: ls /var/folders/vg/stx1n5ys4_lgr08v49lg6t880000gp/T/ | grep outcomeeng-validation-summary.

Validation

  • Touched-scope deterministic verification passes
  • Plugin reload: not applicable, no plugin content changed

Refs

  • spx/15-test-language.adr.md
  • spx/13-infrastructure.enabler/21-test-infrastructure.enabler
  • outcomeeng/changes#667

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

The decision now keeps every temp file and directory a test run creates
under one run root in the worktree. The run creates the root before its
first test, points the temp location and the Git ceiling at it, removes
it when the run ends, sweeps roots a killed run left, and keeps no
tmp_path directory after a test ends.

Verification: not-run
Refs: spx/15-test-language.adr.md
The Node declares that a pytest run keeps every temp file under one run
root in its worktree: the user temp folder is unchanged after a run, the
temp location and the Git ceiling point at the root, a later run sweeps
roots a killed run left, and no tmp_path directory outlives its test.
The assertions await verification selection.

Verification: not-run
Refs: spx/13-infrastructure.enabler/21-test-infrastructure.enabler
The five run temp root assertions carry their test links, grouped under
the scenario and compliance headings. The scenario assertion links the
scenario test, and the four rules link the compliance test.

Verification: not-run
Refs: spx/13-infrastructure.enabler/21-test-infrastructure.enabler
A pytest plugin creates one run root under .spx/worktree/test-runs/ in
the worktree before the first test. It sweeps roots a killed run left,
points the temp location and the Git ceiling at the new root, removes
each tmp_path directory after its test, and removes the root and
restores the environment when the run ends. The root conftest registers
the plugin. An observation harness runs a fixture module in a scratch
worktree and returns what the run left behind.

Verification: not-run
Refs: spx/13-infrastructure.enabler/21-test-infrastructure.enabler
The scenario test shows a run with a passing and a failing test leaves
the user temp folder as it found it. The compliance tests show the run
root location and environment, the Git ceiling, the sweep that stays
inside the worktree, and the removal of each tmp_path directory.

Verification: not-run
Refs: spx/13-infrastructure.enabler/21-test-infrastructure.enabler
@claude

claude Bot commented Oct 21, 2026

Copy link
Copy Markdown

BLOCKING [consistency]: outcomeeng_testing/harnesses/run_temp_root.py:63

Reference: spx/15-test-language.adr.md:9 — "...the run removes the root when it ends, and the next run sweeps any root a killed run left." And spx/15-test-language.adr.md:16 — "ALWAYS: before its first test starts, a pytest run removes every run root left under .spx/worktree/test-runs/ in its own worktree, and never reads or removes a path outside that worktree."
Evidence: sweep_stale_roots (run_temp_root.py:63-69) removes every entry under .spx/worktree/test-runs/ unconditionally — for entry in roots.iterdir(): _remove(entry) — with no PID, lock, or other liveness check distinguishing a dead runs root from one still in use. pytest_configure (line 92-98, tryfirst=True) calls this sweep before creating its own root. If a second pytest invocation starts in the same worktree while a first is still running — e.g. two terminal sessions each running just test against the same checkout, a plausible local workflow this repo does not otherwise prevent — the second runs startup sweep deletes the firsts active run root (its live tmp_path directories, in-flight temp files, and GIT_CEILING_DIRECTORIES target) out from under it mid-run, contradicting the declared intent that only a killed runs leftovers are swept.
Required: give each run root a liveness marker (e.g. a PID file checked with a liveness probe) and have sweep_stale_roots remove only roots whose owning process is no longer alive.

BLOCKING [consistency]: outcomeeng_testing/harnesses/run_temp_root.py:92

Reference: spx/15-test-language.adr.md:13 — "ALWAYS: every temp file and directory that a test, a harness, a fixture, or a shipped script under test creates lives under one run root that the pytest run creates." CLAUDE.md documents the command surface as just test <pytest-target>..., and outcomeeng/validation/_steps.py:321 test_recipe(pytest_args) appends caller-supplied pytest_args verbatim to the pytest argv (PYTEST_ARGV) with no filtering.
Evidence: pytest_configure (run_temp_root.py:92-108) only ever sets the TMPDIR env var and tempfile.tempdir; it never inspects config.option.basetemp. pytests own tmp_path/tmp_path_factory honors a --basetemp CLI option directly, independent of tempfile.gettempdir()/TMPDIR, whenever that option is supplied. Because just test <pytest-target>... forwards arbitrary trailing args straight into the pytest invocation, running just test <target> -- --basetemp=/tmp/mine through this repos own sanctioned command surface sends every tmp_path-based temp file outside the run root entirely, silently breaking the "lives under one run root" guarantee while the plugin still believes it has bounded the run (TMPDIR and GIT_CEILING_DIRECTORIES are still set, masking the gap).
Required: detect config.option.basetemp in pytest_configure and either fail configuration with an actionable error or redirect it beneath the run root; add a test exercising --basetemp passthrough.

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant