[MODSetter]

What

The contract redline no longer writes a model's formatting into a contract. When a change's new words hold a mark the contract never uses, the model is asked once more and shown the words that hold the mark. If the mark is still there, the point goes to a person as MARKUP and that part of the contract is left as it is.

• The check. modules/jobs/word_clauses/markup.py runs in decide.py beside the quote and placement checks, on the new words of a replace or an insert. It refuses:
  • a tag (</parameter>, </br>);
  • a Markdown link (](#), ](...));
  • an elision (...);
  • a brace or a backtick;
  • a closing bracket that closes nothing, beyond those the replaced words already close.
• Only marks the contract never writes. A kind of mark counts only when no paragraph of the contract has one. A contract's own {Company} placeholders or a) lists can still be written. [__] blanks, (i) items and defined terms in brackets are never refused. On the 6 dev NDAs, every paragraph of each contract, checked as new text in each of the other five (955 checks), passes.
• The ask reaches the model through the pack's existing repair.md as $problem, so pack 0.1.0 is unchanged: Your new_text is not only contract words: it holds a brace, "Agreement.},". Give new_text as the words the contract will read and nothing else: no Markdown, tags, code or JSON.
• The person's note reads "SurfSense could not settle : Its new words hold formatting the contract does not use, such as a tag or a link."
• docs: docs/architecture/jobs.md describes the new check under the redline's checks. A new Known gaps line says comments are not checked this way: across all eval runs, 1 of 1,350 comments held a mark, a } from Sonnet 5.5.

Why

Sometimes Haiku 5.5's reply runs on past the end of new_text, and the runner wrote whatever followed into the contract as tracked text. That text reached the "With changes" download. While the redline guide was being captured, both Haiku runs on dev-406 did this (...result from them.](...), ...so retained.</parameter>, ...this Agreement.}, twice). On the 6 Nov test-set run, Sonnet 5.5 did it twice (}).

The eval below used the dev set: 6 ContractNLI NDAs, with the Bonterms Mutual NDA as the playbook. It ran pack 0.1.0 on Haiku 5.5 through OpenRouter (openrouter:anthropic/claude-haiku-5.5@anthropic), 3 runs per arm, 288 scored points each:

	dev (before)	this branch (after)
Changes whose tracked insertion holds model markup	12 of 96, in 8 of 18 case runs and 5 of 6 NDAs	0 of 92
Replies the check sent back	–	7 (5 braces or JSON tails, 1 Markdown link, 1 stray ])
What the second reply did	–	6 clean changes and 1 delete; none left to a person as MARKUP
Outcomes right	237/288 (82.3%)	243/288 (84.4%)
Verdicts right	220/288 (76.4%)	225/288 (78.1%)
Changes right, unneeded, missed	69, 32, 18	71, 27, 16
Left to a person	3	2 (CHANGES_OVERLAP, NEEDS_REVIEW)
Integrity problems	0	0
Model calls	291	296
$ for 18 case runs	0.183	0.174

The 12 insertions in the before arm ended with:

• .}, (4 times);
• a backtick and a comma (2);
• .](...) and .](#);
• ."},;
• ','','']},;
• ’,’external_comment’:’’ ,’_’:’’},;
• a stray ) (years.) ,).

How markup was counted. Every tracked insertion in each redlined copy was scanned for tags, ](, braces, backticks, **, ..., escapes and trailing brackets. That list is wider than the check's, and each flag was read by hand. After the change, the only flags are 6 trailing ) in dev-411. Each closes a parenthesis its own clause opens, so it is contract text. No paragraph the after arm changed was left with a bracket that closes nothing.

Outcomes. The before arm's own three runs ranged from 77 to 81 of 96 outcomes right; each after run scored 81. So this PR claims no loss, not a gain. The gold is still a draft: the maintainer has not yet checked firm/stance.json.

Part of #2414. No issue was filed for this bug.

How to test

cd surfsense_local/backend
PYTHON_DOTENV_DISABLED=1 uv run pytest tests/unit/jobs/test_word_clauses.py

The new tests in tests/unit/jobs/test_word_clauses.py all go through review_contract:

• test_new_text_holding_the_models_formatting_is_asked_again[...] covers the shapes Haiku and Sonnet wrote (.](...), .</parameter>, .},, a backtick and a comma, ','external_comment':''},, a stray )) and an elision. The model is asked again with the words that hold the mark, and the clean second change is made.
• test_formatting_the_model_writes_again_goes_to_a_person: when the mark is still there, the code is MARKUP, nothing is inserted, and the person's note says why.
• test_a_closing_bracket_the_replaced_words_hold_is_kept: an old_text ending in ) is replaced by new words ending in ).
• test_brackets_a_contract_writes_in_new_text_are_kept[...]: [__] blanks, (each, an “Oral Disclosure”), (i) and (ii).
• test_marks_the_contract_itself_writes_are_kept_in_new_text: a contract with a) lists and {Company}.

On dev's decide.py, the 7 leak cases and the person test fail. The tests that check marks are kept pass there, since they guard against false refusals.

Each guard was checked by removing it. Without either half of the rule about the contract's own marks, test_marks_the_contract_itself_writes_are_kept_in_new_text fails. Without the comparison with the replaced words, test_a_closing_bracket_the_replaced_words_hold_is_kept fails.

The eval runs from surfsense_local/backend against the private surfsense-jobs repo. The results are in contract-redline/eval/results/2026-11-09-dev-haiku-before-markup-check and 2026-11-09-dev-haiku-markup-check, not committed. The after arm ran through a wrapper that only logged each check, which is where the count of 7 comes from. Each arm ran:

PYTHONPATH=. python scripts/run_job_eval.py run --pack <surfsense-jobs>/contract-redline/pack --eval <surfsense-jobs>/contract-redline/eval/dev --embedder none --model openrouter:anthropic/claude-haiku-5.5@anthropic --runs 3 --out <results>
PYTHONPATH=. python scripts/run_job_eval.py report <results>

The test set was not run, since it runs once per frozen pack.

High-level PR Summary

This PR prevents model-generated formatting artifacts from being written into contract documents during the redline process. When a language model's response overruns its intended string boundary, it sometimes includes JSON syntax, Markdown links, or HTML tags in the tracked text. The fix adds a validation check that detects these formatting marks in proposed changes, asks the model again with the problematic words highlighted, and if the marks persist, escalates the point to human review with a MARKUP code. The check is intelligent about contract-native formatting, only rejecting marks the contract itself never uses, so legitimate elements like placeholder braces {Company}, list markers a), and bracketed items (i) are preserved. Evaluation on 6 NDA contracts showed this eliminated all 12 instances of leaked formatting (braces, links, tags) that appeared in the baseline, with only 7 retry requests needed and no changes incorrectly flagged.

⏱️ Estimated Review Time: 5-15 minutes

💡 Review Order Suggestion

Order	File Path
1	surfsense_local/backend/modules/jobs/word_clauses/markup.py
2	surfsense_local/backend/modules/jobs/word_clauses/decide.py
3	surfsense_local/backend/tests/unit/jobs/test_word_clauses.py
4	docs/architecture/jobs.md

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
surf-sense-frontend	Ignored		Nov 10, 2026 7:22am UTC

[coderabbitai]

Caution

Review failed

The pull request is closed.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn