[primorLee]

What

The reproduction notebook now collects all 600 orig and 570 phi images into a unique run directory. The scorer rejects missing images by default; --allow_missing_images remains an explicit debugging option and reports coverage against the full benchmark.

Cached scores are reused only when both coverage and the saved run manifest match. The manifest identifies the judge model and endpoint, complete benchmark prompts and testpoint definitions, image contents, judge system prompt, and scorer/gateway source. Legacy caches without a manifest fail closed with a rescore instruction. API credentials are never saved.

Why

The notebook previously omitted the physical split, allowing an incomplete run to appear complete. Matching image filenames alone also allowed scores from a different judge, prompt set, generation, or scorer to be silently reused.

Fixes #363.

Validation

• Synced with upstream main 3e3c6d6.
• Python 3.12: python -m unittest discover -s evaluation/cosmos3/generator/unigenbench -p test_ugb_scorer.py -v — 8 tests passed, including an offline scorer round trip and eight cache identity changes.
• Regression coverage includes all 1,170 bundled prompts, explicit partial runs, missing images, prior-run isolation, a matching cache, and rejection of changed model/endpoint/prompt/testpoint/image/system-prompt/scorer or legacy caches before constructing the judge.
• Ruff checks and git diff --check passed.
• No VLM endpoint, GPU generation, or paid evaluation was used.

[sylvesterkaczmarek]

I think the new completeness check can still silently accept stale generations from an earlier run.

The collection loop only copies vision.jpg when dst.exists() is false. If all_images_dir already contains orig123_0.jpg or phi123_0.jpg from a previous checkpoint/run, a new generation will not replace it. missing_ids then sees the filename as present, and the scorer's cache validation also compares coverage by filename rather than run identity.

Could the collection step overwrite the destination files for the current run, or clear/use a run-specific output directory before collecting? Otherwise a nominally complete 1,170-image evaluation can mix outputs from different model runs.

[primorLee]

Addressed in 21a3e85. Each execution now uses a unique UTC run directory for generated outputs, collected images, and score cache. The new regression test creates an older sibling run and verifies that its image is ignored, preventing cross-run mixing.

[dvasquezlopez]

This covers the missing-image and cached-coverage failure well. Related #363 remains distinct: when the current image filenames match but the judge model, prompt definitions, or scorer version changes, validate_cached_results() still accepts the stale result. Including those run identities in the cached-result check would close that remaining path.

[sylvesterkaczmarek]

Re-reviewed current head 8753a29. The stale-cache identity gap is addressed: cached scores are now tied to the benchmark rows, image contents, judge model and endpoint, scorer source hashes, and system prompt, while the API credential is not persisted. The new regressions cover matching-cache reuse plus changes to model, endpoint, benchmark prompt/testpoint data, image bytes, system prompt, scorer version, and legacy caches without a manifest. I also ran the exact-head focused suite in an isolated environment: 8/8 tests pass, and both changed Python files compile cleanly. No remaining blocker from my earlier review.