[Joel (aspire488)]

Description

Tests and Documentation

Summary

Fixes #1912 by strengthening role-play prompt instructions to prevent conversational preambles and ensure generated messages begin directly with the in-character content.

Changes

• Updated all five role-play persona YAML templates with explicit no-preamble instructions.
• Updated role_play_next_message.yaml with the same requirement.
• Added regression tests covering all six prompt templates.
• Added checks to guard YAML block-scalar indentation.

Validation

• Regression tests have been added but have not yet been executed.
• The requested 50-run before-and-after comparison is still pending.
• No behavioral improvement is claimed until the comparison is completed.

Testing

• Run the new regression tests.
• Compare 50 baseline runs against 50 runs with the changes using the same model and settings.

Closes #1912

[Joel (aspire488)]

Hi Roman, the PR is open and the CLA check has passed. However, the CI workflows are showing "action_required" without executing any jobs. Could you or another maintainer approve the workflow runs so validation can proceed? The regression tests and 50-run before/after comparison are still outstanding. I'll update the PR with the results once validation is complete. Thanks!

[Roman Lutz (romanlutz)]

Thanks Joel (@aspire488)! Did you try it out as well? Any stats on whether it helped?

[Roman Lutz (romanlutz)]

Thanks for putting this together, Joel (@aspire488), and sorry for the unnecessary work. I wasn't aware that the July 11 role-play refactor in #2160 had already addressed the original issue when I encouraged work on #1912.

That refactor removed the old RolePlayAttack raw-text rephrasing path and replaced it with simulated-conversation techniques. The replacement prompts already require an in-character message, and the shared adversarial JSON parsing from #2053 forwards only next_message rather than the entire model reply.

I verified this with GPT-4o-unsafe and a harmless origami objective: the original June-era prompts produced preambles in 6 of 9 completed generations; one additional historical generation was truncated and excluded. The existing implementation before this PR produced 0 preambles in 12 generations, and this PR also produced 0 in 12, with matched settings within each comparison.

The sample is small, but it reproduces the old failure and supports that the July replacement already addressed it. This PR adds further prompt guidance, but we haven't demonstrated an additional improvement on the current implementation. I'm closing this PR without merging and resolving #1912 as already addressed by the July refactor. Thank you again for the contribution.