Conversation
Both of this fixture's markers were literals inside its own `python -c` script, and the runner searches for them in the same captured stream the action's command line is logged into. A conforming CLI may log the full argument list, and openjd-cli does on Windows: openjd-sessions renders `list2cmdline(args)` at INFO, where on POSIX the argument list is hidden inside a temp .sh file and only its path is logged at INFO (the script body goes to DEBUG, which the CLI does not emit). So on Windows both markers were found in the echo of the script rather than in its output. That made the fixture fail on windows-latest with `Found forbidden output: ENV_PRESENT=` while WrappedAction.Environment was in fact empty. Measured by reproducing the argv logging on macOS: the echoed command line reads `for e in []`, the script prints its sentinel, and the process exits 0 -- and the forbidden check still trips. Nothing is wrong with either implementation, and the CI log could not show this because `run_job` discards the captured output on its forbidden-output branch. The expected marker had the same hole in the other direction, and it fails silently: also measured, `ENV_DUMP_COMPLETE` satisfied its expected-output check on a run where the process failed to start at all. A fixture in that state asserts nothing. Both markers are therefore assembled at run time, not just the forbidden one, so each is again an assertion about this process's stdout. `expected` and `forbidden` are unchanged. Verified over five steps: passes unpatched before and after; fails under the simulated argv logging before the fix with CI's exact message and passes after; and, the control that matters, still fails when an `openjd_env` is seeded so the list is genuinely non-empty -- on real POSIX and under the simulation. Full POSIX conformance suite 1161 passed, 0 failed. Both fixture mutants caught, the second by a probe with no `python` on PATH, where the literal spelling passes and the split spelling correctly reports a missing expected output. The header comment records both measurements, because the instinct on reading `'ENV_'+'PRESENT='` is to tidy it back into a literal. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
AlexTranAmz
approved these changes
Sep 4, 2026
leongdl
added a commit
to leongdl/openjd-specifications
that referenced
this pull request
Sep 11, 2026
The harness scans a job's whole output for forbidden substrings, and on Windows openjd-sessions-for-python echoes the full child command line at INFO (_subprocess.py logs list2cmdline(self._args); the POSIX path logs only the temp .sh path). A fixture whose script SOURCE contains the literal FAIL therefore matches its own forbidden marker even when every assertion passed. That is why repr-json-roundtrip-adversarial, repr-py-roundtrip-adversarial and repr-pwsh-roundtrip failed the Python lane on windows-latest only, with 'Found forbidden output: FAIL' and never a missing expected line: every PXX:PASS was present and the jobs succeeded. The Rust CLI is unaffected because it prints only records tagged COMMAND_OUTPUT, and the command echo is tagged FILE_PATH|PROCESS_CONTROL. Split each marker so it is assembled at run time, the convention this repo already uses for the same hazard (OpenJobDescription#181, and the timeout fixture's 'SHOULD_NOT' + '_PRINT'). Applied to repr-sh-roundtrip-adversarial and to proposed/repr-cmd-roundtrip too: both carry the literal today and avoid the trap only by being posix-scoped or unpromoted. Verified the markers still catch a real failure: corrupting one expected value in each of the three runnable adversarial fixtures makes the runner report failure in all three. The four active fixtures still pass locally against openjd-rs. Windows itself is unverified from here. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
leongdl
added a commit
to leongdl/openjd-specifications
that referenced
this pull request
Sep 14, 2026
The harness scans a job's whole output for forbidden substrings, and on Windows openjd-sessions-for-python echoes the full child command line at INFO (_subprocess.py logs list2cmdline(self._args); the POSIX path logs only the temp .sh path). A fixture whose script SOURCE contains the literal FAIL therefore matches its own forbidden marker even when every assertion passed. That is why repr-json-roundtrip-adversarial, repr-py-roundtrip-adversarial and repr-pwsh-roundtrip failed the Python lane on windows-latest only, with 'Found forbidden output: FAIL' and never a missing expected line: every PXX:PASS was present and the jobs succeeded. The Rust CLI is unaffected because it prints only records tagged COMMAND_OUTPUT, and the command echo is tagged FILE_PATH|PROCESS_CONTROL. Split each marker so it is assembled at run time, the convention this repo already uses for the same hazard (OpenJobDescription#181, and the timeout fixture's 'SHOULD_NOT' + '_PRINT'). Applied to repr-sh-roundtrip-adversarial and to proposed/repr-cmd-roundtrip too: both carry the literal today and avoid the trap only by being posix-scoped or unpromoted. Verified the markers still catch a real failure: corrupting one expected value in each of the three runnable adversarial fixtures makes the runner report failure in all three. The four active fixtures still pass locally against openjd-rs. Windows itself is unverified from here. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
leongdl
added a commit
to leongdl/openjd-specifications
that referenced
this pull request
Sep 16, 2026
The harness scans a job's whole output for forbidden substrings, and on Windows openjd-sessions-for-python echoes the full child command line at INFO (_subprocess.py logs list2cmdline(self._args); the POSIX path logs only the temp .sh path). A fixture whose script SOURCE contains the literal FAIL therefore matches its own forbidden marker even when every assertion passed. That is why repr-json-roundtrip-adversarial, repr-py-roundtrip-adversarial and repr-pwsh-roundtrip failed the Python lane on windows-latest only, with 'Found forbidden output: FAIL' and never a missing expected line: every PXX:PASS was present and the jobs succeeded. The Rust CLI is unaffected because it prints only records tagged COMMAND_OUTPUT, and the command echo is tagged FILE_PATH|PROCESS_CONTROL. Split each marker so it is assembled at run time, the convention this repo already uses for the same hazard (OpenJobDescription#181, and the timeout fixture's 'SHOULD_NOT' + '_PRINT'). Applied to repr-sh-roundtrip-adversarial and to proposed/repr-cmd-roundtrip too: both carry the literal today and avoid the trap only by being posix-scoped or unpromoted. Verified the markers still catch a real failure: corrupting one expected value in each of the three runnable adversarial fixtures makes the runner report failure in all three. The four active fixtures still pass locally against openjd-rs. Windows itself is unverified from here. Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
wrap-empty-environment-listhas been failing on the Python conformance job'swindows-latestleg since it was added. Neither implementation is at fault —WrappedAction.Environmentis empty on Windows exactly as the fixture intends. The fixture cannot tell, and it also cannot tell whether its own script ran at all.The problem
Both of the fixture's markers were literals inside its own
python -cscript:The runner searches for both in the same captured stream the action's command line is logged into. A conforming CLI may log the full argument list, and openjd-cli does on Windows: openjd-sessions renders
list2cmdline(args)at INFO (_subprocess.py), where on POSIX the argument list is written into a temp.shfile and only its path is logged at INFO — the script body goes to DEBUG, which the CLI does not emit. The comment already in that code names the asymmetry.So on Windows both markers are found in the echo of the script rather than in its output.
Measured, with the list demonstrably empty
Reproduced on macOS by flipping the
_runner_basebranch so the argument list is not hidden in a.shfile. Captured output:for e in []—repr_py(WrappedAction.Environment)rendered the empty list. The script ran, printed its sentinel, exited 0. The runner's verdict on that same text:The list is empty, the implementation is correct, and the fixture fails anyway.
Corroborating at the code level:
WrappedAction.Environmentis_collect_session_env_list(), formatting_session_env_vars, which has four write sites (__init__, declarativevariables:, anopenjd_envtoken, anopenjd_unset_envtoken). None is platform-conditional and none is reached by this fixture.The second half, which fails silently
The expected marker has the same hole pointing the other way. A run with no
pythononPATHgave:The process never started and the expected-output check passed, satisfied entirely by the echo. A fixture in that state asserts nothing at all on Windows.
That is why this PR splits both markers, not just the forbidden one. Fixing the loud collision and leaving the silent one would have left the fixture passing on Windows even when nothing ran.
The change
One fixture.
expectedandforbiddenare unchanged; the markers are assembled at run time:- "m='ENV_'+'PRESENT='\nfor e in {{repr_py(WrappedAction.Environment)}}:\n print(m+str(e))\nprint('ENV_'+'DUMP_COMPLETE')"The echoed command line now contains
ENV_'+'PRESENT='andENV_'+'DUMP_COMPLETE', neither of which matches, while the process's own stdout still prints the assembled markers. Both assertions become assertions about stdout again.The header comment carries both measurements at length, because the natural instinct on reading
'ENV_'+'PRESENT='is to tidy it into a literal.How this was tested
Five steps, all run:
Found forbidden output: ENV_PRESENT=, matching CI verbatimopenjd_envseededStep 5 is the control that matters: it prepends an environment whose
onEnterprintsopenjd_env: SEEDED_VAR=yes, making the list genuinely non-empty, and the fixed fixture still catches it — on real POSIX and under the simulation. Aforbiddenentry that nothing can trigger is worse than no entry, because it reads as coverage.Regression: full POSIX conformance suite on this branch, 1161 passed, 0 failed. WRAP_ACTIONS alone, 72 passed, 0 failed.
Mutation testing, the mutant being the fixture since there is no production code — both caught:
Found forbidden output: ENV_PRESENT=under simulated WindowspythononPATH— the vacuous passThe second needs its own probe, because reverting that split breaks nothing on its own — it makes an assertion stop discriminating. With the split, a run with no
pythonfails withMissing expected output: ENV_DUMP_COMPLETE; with the literal it passes though nothing ran. That inversion is the mutant being caught.Confirmed on Windows. This PR's
windows-latestPython conformance run reports✓ wrap-empty-environment-list, withTotal: 1104 passed, 12 failed— down from 13. The 12 that remain are exactly the known pre-existing POSIX-path-on-Windows set, unchanged by this PR.That is also the strongest available evidence that the list really is empty: with the markers unwritable, a non-empty list would still print the assembled
ENV_PRESENT=<entry>and still trip the forbidden check, as the step 5 control above demonstrates.Two follow-ups I have not folded in
This is a class, not one fixture. Every WRAP_ACTIONS fixture whose
expected.outputmarker also appears in its own-csource proves nothing on Windows —wrap-three-hooks-python'sWRAP_TASK_CMD=, for instance. This is only the one where a forbidden marker made it visible. Detecting the rest is mechanical (does anyexpected.outputentry appear in the template's renderedargs?) so it belongs in the runner as a warning, but it will newly fail fixtures that pass today, so it wants its own PR and its own triage.run_jobdiscards the captured output on its forbidden-output branch (run_openjd_cli_tests.py), unlike the two missing-output branches which append--- Actual output ---. That single omission is why the CI log contains exactly oneENV_PRESENThit — the pattern from the YAML — and no output at all, and why diagnosing this needed a local reproduction rather than a log read. One line, worth fixing on its own.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.