[WatchTree-19]

Description

This answers open question 6 in the scorer contract design: "Which execution differences count as nondeterminism?"

The observation records one exact run, and replay repeats judgment, not execution, so a single stored run cannot show whether a program is deterministic. A program that reads the clock or a random source can pass once and fail the next time, and today that single pass becomes a confident true.

The answer this change takes: a difference counts as nondeterminism only when it changes a case's verdict. Output the case does not check, and duration, never count. That keeps the question inside the existing matcher instead of adding a second notion of "same output".

• ProgramBehaves takes repetitions: int = 1 (at least 1). Each case runs that many times in one acquisition, as consecutive invocations, so the existing source and sandbox need no change.
• A repeated case is settled only when every run reaches the same verdict. If runs disagree the case is undetermined, with a rationale such as case 0 gave different results across runs (2 of 3 failed). A repeated case with a missing run is also undetermined.
• Repetitions share one runtime profile, so they do not sample other platforms or architectures; the docstring says to compare those under separate runtime profiles.
• Replay judges the stored runs again and never repeats execution. A stored run can only be replayed against a condition with the same repetitions, because the invocations must match, which the scorer already enforces.
• With the default of 1, invocations, rationales and scores are unchanged, and conditions stored before this change load with one run.

The ProgramBehaves and ProgramExecutionScorer docstrings and the replay section of the program execution doc page (.py and .ipynb) describe the option.

Tests and Documentation

• tests/unit/models/test_program.py: invocations repeat per case in case order and round-trip through Condition; the default is one run and a stored condition without the field loads; zero and negative counts are rejected.
• tests/unit/score/test_program_execution_scorer.py: agreeing runs pass or fail as before; disagreeing runs in either direction are undetermined with the count; a missing run is undetermined; each case is read from its own runs (one variant passes only with the right index mapping); end to end, a drifting program is undetermined with repetitions, passes with one run, replays as undetermined without executing, and refuses replay under a condition with a different repetition count; output a case does not check is not treated as a difference.

How it was tested

python -m pytest -q -n 4 tests/unit/score tests/unit/models tests/unit/_code_execution tests/unit/common/test_lazy_package_imports.py
ruff check <touched files>; ruff format --check <touched files>
ty check <touched files>

Rebased on current main (dd3f593). The new and touched tests pass; the only failures in that run are 57 that fail the same way on main in this environment. ruff is clean, and ty reports nothing new in the touched pyrit/ files compared with main. The doc page's markdown cells were checked to match between .py and .ipynb; the page's code was not executed here because it needs the Docker daemon.

Each guard was removed in turn to confirm the tests catch it:

Mutation	Tests that fail
the first run decides the case	6
disagreement scored as a failure instead of undetermined	4
a missing run ignored	2
invocations not repeated	3
wrong run index (index + run)	1
only the first run of each case read	6
no lower bound on repetitions	2
repetition count missing from the rationale	3

This is independent of the other two program execution PRs; each applies to current main on its own.

[Copilot]

🟢 Approval recommended

No active review issues were identified for the optional repeated-execution scoring change.

No open issues 🎉

What changed in this PR

Adds optional repeated execution to PyRIT’s program-behavior scoring, allowing cases with inconsistent verdicts to remain undetermined while preserving single-run behavior by default.

Changes:

• Adds a positive repetitions setting to ProgramBehaves, repeating invocations in case order.
• Settles repeated cases only when all runs agree; disagreement or missing runs leave the case undetermined.
• Adds validation, scoring, and replay tests, plus documentation clarifying that replay reuses stored evidence without executing again.

File	Description
tests/​unit/​score/​test_program_execution_scorer.py	Tests repeated verdicts, missing runs, case indexing, and replay behavior.
tests/​unit/​models/​test_program.py	Tests repetition ordering, validation, serialization, and backward-compatible defaults.
pyrit/​score/​true_false/​program_execution_scorer.py	Evaluates repeated runs per case and updates verdict rationales.
pyrit/​models/​score/​condition.py	Adds repetitions and expands each case’s invocations.
doc/​code/​scoring/​7_program_execution_scorer.py	Documents repetition semantics and replay limitations.
doc/​code/​scoring/​7_program_execution_scorer.ipynb	Mirrors the replay documentation in the notebook.

_{Review effort: Balanced}