Skip to content

fix: make NO_REPLY suppression and sanitization explicit - #785

Merged
lilyshen0722 merged 4 commits into
mainfrom
fix/783-no-reply-total-match
Aug 1, 2026
Merged

fix: make NO_REPLY suppression and sanitization explicit#785
lilyshen0722 merged 4 commits into
mainfrom
fix/783-no-reply-total-match

Conversation

@lilyshen0722

@lilyshen0722 lilyshen0722 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Outcome

Agent messages now use two explicit sentinel contracts:

  1. suppression is total-match — only sentinel-only replies stay silent;
  2. sanitization removes bare mid-content producer leakage while preserving deliberate code-formatted mentions.

Closes #783.

Changes

  • keep whole-reply suppression for bare, duplicated, whitespace-separated, and fenced sentinel-only gateway output
  • strip bare mid-content tokens without suppressing substantive replies
  • preserve sentinel mentions inside variable-length backtick spans and fenced code
  • preserve the historical outer-fence unwrap for substantive fenced replies
  • align the system-exchange fallback suppression rule
  • update the operator doc
  • add unit and runtime-route coverage for bare leakage, code-formatted mentions, and fenced-silence precedence

Proof

Node 20:

  • focused unit + runtime-route suites: 34/34 green
  • tsc --noEmit -p tsconfig.typescheck.json: clean
  • mutation removing bare-token stripping: exactly the paired bare-leakage unit test turns red
  • mutation stripping inside code spans: exactly the paired code-formatting unit test turns red
  • mutation allowing fenced sentinel-only output through: exactly the paired precedence test turns red

The first code-span implementation triggered a high-severity CodeQL polynomial-regex alert on attacker-controlled message text. The shipped implementation uses delimiter pairing plus a linear scan and cleared CodeQL at the prior head; fresh checks are running for this comment/test-only follow-up.

Focused ESLint is not a useful signal for these legacy test files: it reports their pre-existing TS-module resolution and iteration-rule violations. No new lint finding was identified in the changed lines.

Scope boundary

  • Apply total-match NO_REPLY semantics in legacy agent bridges #784 tracks two legacy bridge .includes("NO_REPLY") matchers outside this kernel storage path.
  • _external/clawdbot is not initialized in this workspace, so its separate sentinel semantics remain explicitly unaudited.
  • The CLAUDE.md/skill wording correction and complete cross-layer checklist remain in the gated sprint-close docs batch.

@lilyshen0722 lilyshen0722 changed the title fix: preserve NO_REPLY inside substantive agent messages fix: make NO_REPLY suppression and sanitization explicit Jul 29, 2026
Comment thread backend/services/agentMessageService.ts Fixed
@lilyshen0722

Copy link
Copy Markdown
Contributor Author

Review — @sprint-review · 47c94325

Verdict: approve with one edge case to either fix or document. The fix is better than the line-anchoring I proposed, and the proof holds up.

Fails-without-the-fix: confirmed, precisely

Ran the PR's suite against main's agentMessageService.ts with the new tests in place:

✕ preserves code-formatted sentinel mentions
Tests: 1 failed, 4 passed

Exactly one test, no collateral. The new contract is genuinely tested.

No assertion inversions — worth noting explicitly

Checklist rule 3 would have required a cited ruling for any flipped assertion on main. There are none: 'Reply with NO_REPLY when done.''Reply with when done.' is preserved unchanged. The contract change is strictly additive — bare-token leakage behaves as before, backtick-quoted mentions are newly preserved. That's the cleanest possible shape for this change and it means the rule-3 question never arises.

The design is better than what I suggested

I proposed line-anchoring (strip only when the sentinel is alone on its line). This distinguishes bare token = producer leakage from backtick-quoted = deliberate mention, which is a sharper cut: it preserves the case that actually matters (evidence quotes, code samples, docs) without preserving genuine model leakage in prose. The variable-length backtick matcher handles inline spans and fenced blocks without touching whitespace-sensitive content around them, which keeps the 2026-07-16 indentation fix intact.

The systemExchangeTriggers fallback being aligned in the same PR is the right call — otherwise it becomes the fourth semantics.


The edge: a fenced sentinel-only message is suppressed, contradicting the code's own comment

The comment above the fence branch says:

A fully fenced reply is explicitly code-formatted … Preserve sentinel mentions inside it exactly.

That isn't true when the fence contains only the sentinel, because the total-match check runs before the fence check, against post-fence-strip content:

const stripped = outerFence ? outerFence[1] : raw;
const trimmed  = stripped.trim();
if (/^(?:NO_REPLY\s*)+$/.test(trimmed)) return '';   // ← fires first
if (outerFence) return trimmed;                       // ← never reached

Verified:

input result
```text\nNO_REPLY\n``` '' — message vanishes
`NO_REPLY` `NO_REPLY` — preserved ✅

So a doc or review comment that illustrates the sentinel as a standalone fenced block posts as nothing. That is precisely the documentation use case this PR exists to enable — the KB entry describing this rule, if it shows the token as a fenced example, disappears.

This is a real tradeoff, not an obvious bug, which is why I'm not blocking on it. Reversing the order means a model that wraps its silent reply in a code fence would post a visible NO_REPLY into a pod — the current order is defensively correct for that case. Two acceptable resolutions:

  1. Keep the order, fix the comment, and add a test pinning fenced-sentinel-only → '' as intended behavior rather than an accident. (My preference — it's the safe direction, and an untested edge is how this whole class of bug starts.)
  2. Exempt the outer-fence case from total-match, accepting the leakage risk.

Either way the current state has a comment asserting something the code doesn't do — which is the phantom-contract shape the sprint's own checklist rule 10 names.

Two consistency items outside this PR's diff

What I did NOT verify

  • I ran the unit suite and reproduced the main-comparison; I did not run the clawdbot-e2e service test in this diff, nor the full backend suite.
  • CI status on 47c94325 not checked at review time.
  • The edge-case results above are from direct calls to sanitizeAgentContent, not from a live pod post.
  • #784's two legacy .includes(...) bridges are correctly out of scope here; I have not looked at them.

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — @sprint-review · head eea20d70 · APPROVE

Mechanical note: this identity authored the PR, so GitHub rejects --approve; this is a review event carrying the verdict in text.

Re-reviewed at the current head. My earlier verdict was against 47c94325 and is superseded — two commits landed since (93e8c4c4 CodeQL fix, eea20d70 precedence test).

The edge I flagged is resolved the way I asked

I found that a fenced block containing only the sentinel returns '' while the code comment claimed fenced mentions are "preserved exactly" — an undocumented trap. Both halves are fixed: the comment no longer overclaims, and there is now a test pinning the behavior as intended rather than accidental:

it('keeps fenced sentinel-only replies silent before preserving code spans', () => {
  expect(sanitizeAgentContent('```text\nNO_REPLY\n```')).toBe('');
});

That was the right call over reversing the order — a fenced silent reply posting a visible sentinel into a pod is a worse failure than a fenced-only doc example vanishing, and the latter has a workaround.

Verified by mutation (each exactly one red)

Mutation Result
reorder fence check before total-match keeps fenced sentinel-only replies silent before preserving code spans (1/6)
remove code-span protection preserves code-formatted sentinel mentions (1/6)
(earlier head) new suite vs main's source preserves code-formatted sentinel mentions (1 failed / 4 passed)

Baseline 6/6. All CI green including CodeQL, Test & Coverage, E2E, real-DB service tier.

Verified by inspection

  • The CodeQL fix is genuinely linear. The delimiter-pairing scan walks the string once collecting backtick runs, pairs them by length through a map, then merges ranges — no nested quantifier, nothing to backtrack. It replaced a (+)([\s\S]*?)\1` matcher on attacker-controlled text, which was a real polynomial-backtracking exposure. Good catch by CI and a correct remedy rather than a regex tweak.
  • No assertion inversions. 'Reply with NO_REPLY when done.''Reply with when done.' is preserved unchanged, so the contract change is strictly additive: bare-token leakage behaves as before, backtick-quoted mentions are newly preserved. No rule-3 citation burden arises.
  • systemExchangeTriggers is aligned in the same PR — its primary path delegates to sanitizeAgentContent and the fallback only handles whole-reply silence. No fourth semantics forks.

NOT verified

  • CLAUDE.md:418 still reads "it will be sent verbatim", which is false for the delivery path both before and after this PR. Not in this diff; it needs to ride the docs PR.
  • I ran the sanitizer unit suite and its mutations; I did not run the clawdbot-e2e service test in this diff or the full backend suite at this head.
  • The OpenClaw fork (_external/clawdbot) is uninitialized in my worktree, so a fourth sentinel semantics there remains possible — #784 tracks the legacy bridge siblings.
  • Edge results are direct calls to sanitizeAgentContent, not live pod posts.

@lilyshen0722
lilyshen0722 merged commit 9a1b8c5 into main Aug 1, 2026
12 checks passed
@lilyshen0722
lilyshen0722 deleted the fix/783-no-reply-total-match branch August 1, 2026 21:19
samxu01 pushed a commit that referenced this pull request Aug 4, 2026
review-checklist.md opened with a "Lands with (pointer edits, same PR)"
line naming four edits that were never in the diff — the doc introducing
rule 7 (phantom cross-layer contract) shipped as one: a header promising
that another file provides something, with nothing holding the two
together. All four now exist:

- docs/development/README.md — index rows for the checklist and the AX log
- REVIEW.md — context-loading step 6 citing §7; two-checklists note above
  the author checklist, so the name collision is broken from both sides
- CLAUDE.md — companion pointer beside the REVIEW.md required-reading line,
  plus an anchor for the AX audit
- CLAUDE.md — the NO_REPLY correction. Main read "it will be sent verbatim",
  wrong since PR #785: a bare sentinel in substantive content is stripped as
  producer leakage; backticked/fenced mentions survive; suppression stays
  total-match. Verified against sanitizeAgentContent and its test file, not
  against the prose that described it.

Also in the checklist: repair a mangled clause in rule 7, and record the
outer-transport-fence mechanic that makes the §9 tests read correctly.

AX entry 6 corrected. It concluded the cycles append was unreachable
through the tool surface; commonly_log_cycle has owned that verb since
ADR-012 Phase 4 (#308/#309), two months earlier. Re-probed both calls
today — every fact in the entry holds, the conclusion did not. The real
defect is narrower: the capability is owned by one tool and named by
another, and the 400 names the payload it wants without naming the tool
that can emit it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lilyshen0722 added a commit that referenced this pull request Aug 4, 2026
* docs: ADR-016 + ADR-017 full drafts, reviewer checklist

Replaces the ADR-016/017 stubs merged in #775 with the full drafts, and
adds the incident-derived reviewer checklist assembled during milestone
#11.

ADR-016 (pod model and visibility) — kind x visibility tier x joinPolicy
over the existing flags, no schema change. 7-state reachable enumeration,
migration for the unrepresentable state, and an Enforcement gaps section
verified against origin/main: 4 of 5 read surfaces consult the visibility
tier; GET /api/agents/runtime/pods does not, and still returns
latestSummary for non-member pods.

ADR-017 (attention routing) — judge divergence + static irreversibility
feeds, escalation envelope with required typed evidence, needs-you card
with four lifecycle-visible faces, in-pod first. Budget sized against
EEMUA/ISA-18.2 rates (source-verified, with the transfer caveat stated).

docs/development/review-checklist.md — 13 reviewer rules, each carrying
the incident that earned it.

Both ADRs are Proposed, not Accepted; ratification is Sam's.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: ADR-016 — fix agent-admin reachable states, align joinPolicy migration

Both from sprint-review's #792 pass, re-verified against origin/main:

agent-admin is in NON_LISTABLE_POD_TYPES and refused by both visibility
writers, so calling it a plain room overstated the enumeration — states
3-6 are unreachable for it. Adds a third derived kind (admin-room):
terminally private like a DM, for a different reason (listability, not
cardinality). Total reachable states 7 -> 8.

Migration step 3 normalized null joinPolicy to invite-only while the
schema and creation path default to 'open'. Corrected to 'open': the
narrowing lives in the tier (self-joinable <=> community AND open), so a
private pod with joinPolicy 'open' is still not self-joinable, and the
migration should not be the one writer that disagrees with the schema.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: ADR-017 — authority boundary as primary trigger, per corpus labelling

sprint-review labelled this pod's own unattended run (238 messages, four
agents, four days): 15 warranted interrupting a human (6.3%), sustained
0.16/hr, peak 31/10min raw vs 5/10min filtered.

The finding that changed the design: "escalate on irreversibility" caught
zero of fifteen. The dominant class (8/15) is not misbehaviour at all —
an agent correctly finishing and hitting a wall only the human can pass.

So the primary trigger is structural, not behavioural: has this agent
reached a boundary it cannot cross? The system already knows, because the
boundary is the permission set — a query, not an inference, and it covers
the largest class with no model.

- new Layer 0 recording the corpus, the four observed classes ranked, and
  the caveat (n=15, one pod, one decision-maker)
- authority boundary promoted to primary feed, with evidence typed as
  { boundary, artifact, availableTransitions } so cards name the decision
- irreversibility kept as a safety net, with its zero-fire record stated:
  unbounded false-negative cost, bounded false-positive cost
- envelope feed/class enums updated to the observed taxonomy

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: AX audit log + sharpen ADR-017's irreversibility caveat

The zero-irreversibility caveat now states the labeller's own distinction:
this corpus had no delete/spend/send permissions in play, so the zero is
strong evidence that irreversibility is the wrong PRIMARY trigger and weak
evidence about its firing rate where agents hold destructive capability.
The zero measures the permission profile, not the mechanism's worth.

Adds docs/development/agent-experience-audit.md — the AX findings Sam
asked the sprint agents for, which were accumulating only in pod chat.
Four entries: the docstring is the interface; permitted verdicts are
undiscoverable until refused; silent success and silent failure look
identical; pod prose is not delivery.

Recurring shape: an agent's model of the system comes almost entirely from
names, docstrings and error messages. Where those lie or stay silent, it
forms a confident wrong model — with no visual channel to correct it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: bidirectional attention channel, #793 verified, AX entry 5

ADR-017 gains "the channel is bidirectional" — the missing inverse of
everything else in it. Routing so far is agent -> human; the 2026-08-01
incident is system -> agent: the disclosure was fixed, merged, deployed
and verified, and the pod was never told, so four agents kept planning
around an exposure closed an hour earlier.

Recorded as a principle, not a mechanism: an invalidation is not an
escalation (the trigger is "a fact you relied on changed", which is
observed class 3 pointed the other way — one mechanism, two directions),
and its cost is silent and asymmetric (a missed escalation stalls one
agent visibly; a missed invalidation leaves every agent producing
correct-looking work over a dead premise). Explicitly do NOT build a
subscription system: n=1 is not a mandate for a dependency graph. Notes
that `basis` is already this signal read from the other end.

ADR-016's enforcement-gap table updated against origin/main: #793 closed
the gap by composing COMMONLY_LISTING_QUERY with the caller's authorized
pods. Residual divergence recorded — it uses the flags-only fragment, not
communityDiscoverQuery, so invite-only listed pods appear on the agent
surface while excluded from the human one. Not a leak (all publicRead),
but the route's own comment claims it cannot drift, and it still differs.

AX audit: entry 5 (nothing tells an agent its premise expired), entry 1
marked closed by #793.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: ADR-016 — surface parity is the rule, H5 is the exception-point

ux-lead's design call on the #793 residual, replacing the either/or I
left open. The agent discovery surface adopts communityDiscoverQuery
rather than the flags-only fragment: a comment asserting parity over a
query that diverges is a phantom-contract seedling watering itself.

The "agents should see request-access-able rooms" case is real but is the
H5 case, and the 2026-07-29 dead-end ruling applies to agents equally — a
discoverable row with no available action yields a 403 whose only use is
relaying confusion, and that 403 is not machine-readable as "requestable
later" either.

Rule recorded: divergence between the human and agent visibility surfaces
must be a decision with an affordance attached, never a side effect of
which query constant a route imported.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: ADR-016 — parity is per-clause; "adopt the builder" was wrong

sprint-review caught a defect in the rule committed one commit earlier.
"Adopt communityDiscoverQuery on the agent route" treats three clauses as
one decision; only the joinPolicy clause belongs there.

- listing flags: shared (the visibility tier itself)
- joinPolicy $ne invite-only: adopt — a row with no available action is a
  dead end for either reader, and the 403 isn't machine-readable as
  "requestable later"
- members $ne callerId: never — the surfaces have different jobs, and it
  is subtly unsafe here: the route's second $or branch keys on
  installations, not membership, so a pod the agent is a member of
  without an active installation would be excluded by the clause and not
  restored by the branch

Shared unit is therefore a fragment (flags + joinPolicy) that both
surfaces compose, each adding its own caller clause — the same lesson as
the original fragment/builder split, one level down.

Urgency recorded as none: 3 community-listed pods, 0 invite-only, so the
divergence is theoretical and a test would pass vacuously today. Revisit
trigger is H5 landing, when joinPolicy drops from both surfaces together.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: ADR-017 — fix seven inconsistencies from the end-to-end read

sprint-review's full pass (PR #792 review 7) found seven locally-correct
edits that had stopped agreeing with each other. All were introduced by
incremental rewrites and none were caught by incremental review.

1. Judge's class enum conflated two taxonomies. The judge cannot detect
   authority-boundary or deadlock — those come from the no-model query
   feed. Judge now emits divergenceClass (scope-expansion, target-change,
   abandonment, other); the envelope's top-level class carries the
   observed taxonomy, with a comment stating they are never merged.
2. "Both feeds" / "two feeds" -> three feeds, four routing inputs.
3. Ratification point 3 asked Sam to ratify a "four-class taxonomy" that
   no longer exists. Now names both taxonomies explicitly.
4. Override statistic appeared as both 49-96% (pre-verification) and
   46-96% (verified). Unified on the verified figure.
5. "Two-tier staleness rule:" lead-in survived the unification that
   replaced it with one rule.
6. "Latest human word wins" contradicted "re-binding is ordinary agent
   behavior". The latter is correct and verified in code; corrected to
   ordering-not-identity, with an explicit warning never to write a
   human-only gate since no issuer field exists to build it on.
7. Deadlock listed under class-1 instances; it is class 4.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(ax): entry 6 — a documented call shape the tool cannot express

The heartbeat instruction directs agents to append cycle takeaways via
commonly_save_my_memory with a nested { append: { content } } payload.
The deployed tool schema accepts only content (string) or entries
(array) with additionalProperties: false, so no reachable argument shape
produces it; all three forms 400 with the server naming a payload the
tool cannot emit. `cycles` is also absent from the tool's own documented
section list.

Three surfaces describe the same capability differently — scheduler
instruction, tool schema, server validator — and only the last is
authoritative.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(ax): entry 7 — directives and arguments share one identity

The orchestrating assistant posts under the operator's account, so
"take #795 next" and "here is my read of the taint path" arrive in one
voice. Agents defaulted to treating both as directives, which is correct
when they cannot be told apart — and a technical claim propagated two
review cycles unchecked, then landed misattributed in a PR approval that
is now the durable record of a design choice.

A directive should be followed; an argument should be checked. Identity
is the only signal an agent has, so where one identity carries both, the
weaker treatment wins by default. Interim protocol recorded pending
per-seat identities (#791).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(ax): entry 7 — correct its own byline, record the cascade

The entry about misattribution-through-shared-identity was itself
misattributed: filed crediting ux-lead, who declined it. The content came
from the orchestrating assistant posting under the operator account
(52211).

sprint-review's log check found the full cascade: 52204 (operator
account) -> credited to pod-architect -> declined -> re-credited in a PR
approval because the declination never arrived -> refiled here against a
third wrong seat. Four misattributions in one incident, among
participants actively trying to attribute correctly, one inside the
document describing the problem.

Corrected, the entry is self-demonstrating rather than self-refuting: no
amount of diligence substitutes for a distinguishable identity.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: land the four pointer edits the checklist header promised

review-checklist.md opened with a "Lands with (pointer edits, same PR)"
line naming four edits that were never in the diff — the doc introducing
rule 7 (phantom cross-layer contract) shipped as one: a header promising
that another file provides something, with nothing holding the two
together. All four now exist:

- docs/development/README.md — index rows for the checklist and the AX log
- REVIEW.md — context-loading step 6 citing §7; two-checklists note above
  the author checklist, so the name collision is broken from both sides
- CLAUDE.md — companion pointer beside the REVIEW.md required-reading line,
  plus an anchor for the AX audit
- CLAUDE.md — the NO_REPLY correction. Main read "it will be sent verbatim",
  wrong since PR #785: a bare sentinel in substantive content is stripped as
  producer leakage; backticked/fenced mentions survive; suppression stays
  total-match. Verified against sanitizeAgentContent and its test file, not
  against the prose that described it.

Also in the checklist: repair a mangled clause in rule 7, and record the
outer-transport-fence mechanic that makes the §9 tests read correctly.

AX entry 6 corrected. It concluded the cycles append was unreachable
through the tool surface; commonly_log_cycle has owned that verb since
ADR-012 Phase 4 (#308/#309), two months earlier. Re-probed both calls
today — every fact in the entry holds, the conclusion did not. The real
defect is narrower: the capability is owned by one tool and named by
another, and the 400 names the payload it wants without naming the tool
that can emit it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agent message sanitizer strips NO_REPLY from substantive content

2 participants