Skip to content

test: Assemble wrap-empty-environment-list markers at run time - #181

Merged
leongdl merged 1 commit into
OpenJobDescription:mainlinefrom
leongdl:fix/wrap-empty-env-marker-collision
Sep 4, 2026
Merged

leongdl merged 1 commit into
OpenJobDescription:mainlinefrom
leongdl:fix/wrap-empty-env-marker-collision

Conversation

@leongdl

@leongdl leongdl commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

wrap-empty-environment-list has been failing on the Python conformance job's windows-latest leg since it was added. Neither implementation is at fault — WrappedAction.Environment is empty on Windows exactly as the fixture intends. The fixture cannot tell, and it also cannot tell whether its own script ran at all.

The problem

Both of the fixture's markers were literals inside its own python -c script:

- "for e in {{repr_py(WrappedAction.Environment)}}:\n print(f'ENV_PRESENT={e}')\nprint('ENV_DUMP_COMPLETE')"
expected:
  output:    [ENV_DUMP_COMPLETE]
  forbidden: [ENV_PRESENT=]

The runner searches for both in the same captured stream the action's command line is logged into. A conforming CLI may log the full argument list, and openjd-cli does on Windows: openjd-sessions renders list2cmdline(args) at INFO (_subprocess.py), where on POSIX the argument list is written into a temp .sh file and only its path is logged at INFO — the script body goes to DEBUG, which the CLI does not emit. The comment already in that code names the asymmetry.

So on Windows both markers are found in the echo of the script rather than in its output.

Measured, with the list demonstrably empty

Reproduced on macOS by flipping the _runner_base branch so the argument list is not hidden in a .sh file. Captured output:

0:00:00.002113	Running command python -c 'for e in []:
 print(f'"'"'ENV_PRESENT={e}'"'"')
print('"'"'ENV_DUMP_COMPLETE'"'"')'
0:00:00.004310	Output:
0:00:00.038368	ENV_DUMP_COMPLETE
0:00:00.040896	Process pid 99134 exited with code: 0 (unsigned) / 0x0 (hex)

for e in [] — repr_py(WrappedAction.Environment) rendered the empty list. The script ran, printed its sentinel, exited 0. The runner's verdict on that same text:

expected  'ENV_DUMP_COMPLETE'   present=True
forbidden 'ENV_PRESENT='        present=True

The list is empty, the implementation is correct, and the fixture fails anyway.

Corroborating at the code level: WrappedAction.Environment is _collect_session_env_list(), formatting _session_env_vars, which has four write sites (__init__, declarative variables:, an openjd_env token, an openjd_unset_env token). None is platform-conditional and none is reached by this fixture.

The second half, which fails silently

The expected marker has the same hole pointing the other way. A run with no python on PATH gave:

Process failed to start: [Errno 2] No such file or directory: 'python'
expected  'ENV_DUMP_COMPLETE'   present=True

The process never started and the expected-output check passed, satisfied entirely by the echo. A fixture in that state asserts nothing at all on Windows.

That is why this PR splits both markers, not just the forbidden one. Fixing the loud collision and leaving the silent one would have left the fixture passing on Windows even when nothing ran.

The change

One fixture. expected and forbidden are unchanged; the markers are assembled at run time:

- "m='ENV_'+'PRESENT='\nfor e in {{repr_py(WrappedAction.Environment)}}:\n print(m+str(e))\nprint('ENV_'+'DUMP_COMPLETE')"

The echoed command line now contains ENV_'+'PRESENT=' and ENV_'+'DUMP_COMPLETE', neither of which matches, while the process's own stdout still prints the assembled markers. Both assertions become assertions about stdout again.

The header comment carries both measurements at length, because the natural instinct on reading 'ENV_'+'PRESENT=' is to tidy it into a literal.

How this was tested

Five steps, all run:

Step Expected Result
1. unpatched, real POSIX pass pass
2. simulated Windows argv logging, before the fix fail fail — Found forbidden output: ENV_PRESENT=, matching CI verbatim
3. simulated Windows, after the fix pass pass
4. unpatched, after the fix pass pass
5. after the fix, one openjd_env seeded fail fail on both hosts

Step 5 is the control that matters: it prepends an environment whose onEnter prints openjd_env: SEEDED_VAR=yes, making the list genuinely non-empty, and the fixed fixture still catches it — on real POSIX and under the simulation. A forbidden entry that nothing can trigger is worse than no entry, because it reads as coverage.

Regression: full POSIX conformance suite on this branch, 1161 passed, 0 failed. WRAP_ACTIONS alone, 72 passed, 0 failed.

Mutation testing, the mutant being the fixture since there is no production code — both caught:

Mutant Verdict Result
forbidden marker back to a literal CAUGHT Found forbidden output: ENV_PRESENT= under simulated Windows
expected marker back to a literal CAUGHT run passes with no python on PATH — the vacuous pass

The second needs its own probe, because reverting that split breaks nothing on its own — it makes an assertion stop discriminating. With the split, a run with no python fails with Missing expected output: ENV_DUMP_COMPLETE; with the literal it passes though nothing ran. That inversion is the mutant being caught.

Confirmed on Windows. This PR's windows-latest Python conformance run reports ✓ wrap-empty-environment-list, with Total: 1104 passed, 12 failed — down from 13. The 12 that remain are exactly the known pre-existing POSIX-path-on-Windows set, unchanged by this PR.

That is also the strongest available evidence that the list really is empty: with the markers unwritable, a non-empty list would still print the assembled ENV_PRESENT=<entry> and still trip the forbidden check, as the step 5 control above demonstrates.

Two follow-ups I have not folded in

This is a class, not one fixture. Every WRAP_ACTIONS fixture whose expected.output marker also appears in its own -c source proves nothing on Windows — wrap-three-hooks-python's WRAP_TASK_CMD=, for instance. This is only the one where a forbidden marker made it visible. Detecting the rest is mechanical (does any expected.output entry appear in the template's rendered args?) so it belongs in the runner as a warning, but it will newly fail fixtures that pass today, so it wants its own PR and its own triage.

run_job discards the captured output on its forbidden-output branch (run_openjd_cli_tests.py), unlike the two missing-output branches which append --- Actual output ---. That single omission is why the CI log contains exactly one ENV_PRESENT hit — the pattern from the YAML — and no output at all, and why diagnosing this needed a local reproduction rather than a log read. One line, worth fixing on its own.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Both of this fixture's markers were literals inside its own `python -c` script,
and the runner searches for them in the same captured stream the action's command
line is logged into. A conforming CLI may log the full argument list, and
openjd-cli does on Windows: openjd-sessions renders `list2cmdline(args)` at INFO,
where on POSIX the argument list is hidden inside a temp .sh file and only its
path is logged at INFO (the script body goes to DEBUG, which the CLI does not
emit). So on Windows both markers were found in the echo of the script rather
than in its output.

That made the fixture fail on windows-latest with `Found forbidden output:
ENV_PRESENT=` while WrappedAction.Environment was in fact empty. Measured by
reproducing the argv logging on macOS: the echoed command line reads
`for e in []`, the script prints its sentinel, and the process exits 0 -- and the
forbidden check still trips. Nothing is wrong with either implementation, and the
CI log could not show this because `run_job` discards the captured output on its
forbidden-output branch.

The expected marker had the same hole in the other direction, and it fails
silently: also measured, `ENV_DUMP_COMPLETE` satisfied its expected-output check
on a run where the process failed to start at all. A fixture in that state
asserts nothing. Both markers are therefore assembled at run time, not just the
forbidden one, so each is again an assertion about this process's stdout.

`expected` and `forbidden` are unchanged. Verified over five steps: passes
unpatched before and after; fails under the simulated argv logging before the fix
with CI's exact message and passes after; and, the control that matters, still
fails when an `openjd_env` is seeded so the list is genuinely non-empty -- on
real POSIX and under the simulation. Full POSIX conformance suite 1161 passed,
0 failed. Both fixture mutants caught, the second by a probe with no `python` on
PATH, where the literal spelling passes and the split spelling correctly reports
a missing expected output.

The header comment records both measurements, because the instinct on reading
`'ENV_'+'PRESENT='` is to tidy it back into a literal.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
@leongdl
leongdl requested a review from a team as a code owner September 3, 2026 21:24
@leongdl
leongdl merged commit 8709e1f into OpenJobDescription:mainline Sep 4, 2026
8 of 24 checks passed
leongdl added a commit to leongdl/openjd-specifications that referenced this pull request Sep 11, 2026
The harness scans a job's whole output for forbidden substrings, and on Windows
openjd-sessions-for-python echoes the full child command line at INFO
(_subprocess.py logs list2cmdline(self._args); the POSIX path logs only the temp
.sh path). A fixture whose script SOURCE contains the literal FAIL therefore
matches its own forbidden marker even when every assertion passed.

That is why repr-json-roundtrip-adversarial, repr-py-roundtrip-adversarial and
repr-pwsh-roundtrip failed the Python lane on windows-latest only, with 'Found
forbidden output: FAIL' and never a missing expected line: every PXX:PASS was
present and the jobs succeeded. The Rust CLI is unaffected because it prints only
records tagged COMMAND_OUTPUT, and the command echo is tagged
FILE_PATH|PROCESS_CONTROL.

Split each marker so it is assembled at run time, the convention this repo
already uses for the same hazard (OpenJobDescription#181, and the timeout fixture's
'SHOULD_NOT' + '_PRINT'). Applied to repr-sh-roundtrip-adversarial and to
proposed/repr-cmd-roundtrip too: both carry the literal today and avoid the trap
only by being posix-scoped or unpromoted.

Verified the markers still catch a real failure: corrupting one expected value in
each of the three runnable adversarial fixtures makes the runner report failure
in all three. The four active fixtures still pass locally against openjd-rs.
Windows itself is unverified from here.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
leongdl added a commit to leongdl/openjd-specifications that referenced this pull request Sep 14, 2026
The harness scans a job's whole output for forbidden substrings, and on Windows
openjd-sessions-for-python echoes the full child command line at INFO
(_subprocess.py logs list2cmdline(self._args); the POSIX path logs only the temp
.sh path). A fixture whose script SOURCE contains the literal FAIL therefore
matches its own forbidden marker even when every assertion passed.

That is why repr-json-roundtrip-adversarial, repr-py-roundtrip-adversarial and
repr-pwsh-roundtrip failed the Python lane on windows-latest only, with 'Found
forbidden output: FAIL' and never a missing expected line: every PXX:PASS was
present and the jobs succeeded. The Rust CLI is unaffected because it prints only
records tagged COMMAND_OUTPUT, and the command echo is tagged
FILE_PATH|PROCESS_CONTROL.

Split each marker so it is assembled at run time, the convention this repo
already uses for the same hazard (OpenJobDescription#181, and the timeout fixture's
'SHOULD_NOT' + '_PRINT'). Applied to repr-sh-roundtrip-adversarial and to
proposed/repr-cmd-roundtrip too: both carry the literal today and avoid the trap
only by being posix-scoped or unpromoted.

Verified the markers still catch a real failure: corrupting one expected value in
each of the three runnable adversarial fixtures makes the runner report failure
in all three. The four active fixtures still pass locally against openjd-rs.
Windows itself is unverified from here.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
leongdl added a commit to leongdl/openjd-specifications that referenced this pull request Sep 16, 2026
The harness scans a job's whole output for forbidden substrings, and on Windows
openjd-sessions-for-python echoes the full child command line at INFO
(_subprocess.py logs list2cmdline(self._args); the POSIX path logs only the temp
.sh path). A fixture whose script SOURCE contains the literal FAIL therefore
matches its own forbidden marker even when every assertion passed.

That is why repr-json-roundtrip-adversarial, repr-py-roundtrip-adversarial and
repr-pwsh-roundtrip failed the Python lane on windows-latest only, with 'Found
forbidden output: FAIL' and never a missing expected line: every PXX:PASS was
present and the jobs succeeded. The Rust CLI is unaffected because it prints only
records tagged COMMAND_OUTPUT, and the command echo is tagged
FILE_PATH|PROCESS_CONTROL.

Split each marker so it is assembled at run time, the convention this repo
already uses for the same hazard (OpenJobDescription#181, and the timeout fixture's
'SHOULD_NOT' + '_PRINT'). Applied to repr-sh-roundtrip-adversarial and to
proposed/repr-cmd-roundtrip too: both carry the literal today and avoid the trap
only by being posix-scoped or unpromoted.

Verified the markers still catch a real failure: corrupting one expected value in
each of the three runnable adversarial fixtures makes the runner report failure
in all three. The four active fixtures still pass locally against openjd-rs.
Windows itself is unverified from here.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants