feat(shared): bare-mention scanner, code-fence normalizer, structural plain text (#193)

[patrick-chinchill]

Summary

Ports upstream's shared text utilities into chat_sdk/shared and brings ast_to_plain_text in line with upstream toPlainText. The new functions are utilities only; no adapter uses them yet.

• chat_sdk.shared.mentions: a character-for-character port of adapter-shared/src/mentions.ts. It provides replace_bare_mentions(text, replacer), mask_code_spans(text, replacement=" ") and MentionReplacer. The scanner skips inline and fenced code, http(s):// URLs (any case), schemeless host.tld/… paths, emails and <…> tokens.
• chat_sdk.shared.code_fences: a port of code-fences.ts, providing normalize_code_fences(text, *, convert_text=None, convert_code=None).
• All three functions and MentionReplacer are exported from chat_sdk.shared.
• ast_to_plain_text is rewritten as a single recursive _plain_text_node. It walks children with an explicit loop, which keeps one Python frame per nesting level, so deep blockquotes/lists do not overflow the stack before parse_markdown would:
  • root children are joined with \n\n;
  • list, listItem and blockquote children are joined with \n;
  • tableRow cells are joined with \t, keeping empty cells;
  • table rows are joined with \n, dropping rows that are empty after JS trim();
  • a string value/alt is returned as-is.
• New private shared/_js_compat.py holds the JS trim() whitespace set (JS_WHITESPACE). The scanner's boundary test, trimStart and the table-row check all use it.

Upstream commits mapped

Upstream	Release	What was ported
d4c52cad (#652)	chat@4.33.0	replaceBareMentions → replace_bare_mentions
5c926f19 (#604)	chat@4.34.0	toPlainText structural whitespace
764e4759 (#817)	chat@4.38.1	core markdown.ts half only (table cells/rows). The Slack half is #210
e71bfead (#843)	chat@4.39.0	normalizeCodeFences. Adoption is #209/#239
683eadc1 (#947)	chat@4.41.0	maskCodeSpans. Slack adoption is #209

Tests ported

• markdown.test.ts → tests/test_markdown_faithful.py: all 8 named tests. The 6 toPlainText tests assert exact values; the 2 markdownToPlainText tests use upstream's regex.
• chat.test.ts → tests/test_chat_faithful.py: "should call onNewMention for newline-separated GitHub bot mentions" and "should not call onNewMention for concatenated GitHub bot mention text".
• thread.test.ts: "should preserve double newlines (paragraph breaks) in streamed text" now expects "hello.\n\nhow are you?".
• adapter-shared/src/mentions.test.ts → new tests/test_shared_mentions.py: all 17 replaceBareMentions cases and all 6 maskCodeSpans cases.
• adapter-shared/src/code-fences.test.ts → new tests/test_shared_code_fences.py: all 11 cases.
• GitHub adapter:
  • "should preserve whitespace in newline-separated issue comment mentions" and "…review comment mentions" are in tests/test_github_webhook.py.
  • "should preserve whitespace after newline-separated @mentions" is in tests/test_github_format.py.
• Python-specific tests:
  • index-boundary sweeps: @a at index 0, a trailing @, a lone backtick, an empty string, an unclosed < at end of string;
  • the ASCII word class (é@x → é<@x>);
  • HTTPS://host/@x and other mixed-case schemes are left untouched;
  • JS vs Python whitespace: U+FEFF ends a URL, U+001C–U+001F do not, and the same holds for the blockquote trimStart and the table-row check;
  • the code-fence patterns: $ vs \Z, and \d vs [0-9] (an Arabic-Indic digit is not an ordered-list marker);
  • table_to_ascii output is unchanged.
  • stack depth: a 600-deep blockquote and a 300-deep nested list extract without RecursionError;
  • closed <...> tokens that contain an @ (<!subteam^S1|@team>);
  • exact-output pins for the fence/newline bookkeeping: a fence already at line start, adjacent fences, blockquote-line scope, an inline span that crosses a newline, and close >= end in the mention code scanner. All values were checked against the chat@4.41.1 TS under Node.

Differential check (not committed): I ran the upstream TS sources at chat@4.41.1 directly under Node against the Python ports.

• mentions.ts and code-fences.ts: about 130k random inputs, built from the delimiter alphabet plus JS/Python whitespace edge characters. 0 mismatches.
• Upstream plainTextNode, run on 20k Python-parsed ASTs: 0 mismatches.

Fidelity

Delta vs committed report (HEAD): missing 282 -> 273 (-9). markdown.test.ts goes 7 → 0 and chat.test.ts goes 33 → 31. Strict at the pin is still 733/733.

Divergences

None in behavior. JS string semantics (ASCII classes, the trim() whitespace set, case-insensitive startsWith, undefined for out-of-range indexes, $, \d) are reproduced explicitly and documented in docs/UPSTREAM_SYNC.md ("Shared text utilities"). That section also adds the two module-mapping rows.

Two test-level notes:

• Upstream's "keeps a token after an unterminated fence" uses toContain. The port asserts the exact TS output, "` <@U1> help".
• The review-comment whitespace test uses "@test-bot\n\nhi there" instead of upstream's single newline. The Python parser always kept soft breaks, so a paragraph break actually exercises #604 on that code path.

Consumer impact (high)

message.text / extract_plain_text output changes wherever the base extractor runs:

• a blank line between paragraphs (was one \n);
• a newline between the blocks of a list item (was a space);
• newline-separated blockquote children;
• tab-separated table cells and newline-separated table rows. Tables used to collapse into run-together text.

This affects:

• Teams inbound messages;
• GitHub issue and review comments;
• Discord fetch_messages / parse_message;
• Telegram plain text (the sent/streamed message text, and the plain-text fallback bodies);
• thread.post(...) results (SentMessage.text).

Not affected: Slack and Google Chat (both override extract_plain_text) and Discord live-gateway text (raw content). Slack text changes later, in #209/#210. Teams must not adopt the mention scanner (#216).

Closes #193
Part of #184

Merge gate

Review findings (2 independent reviewers, 5 findings): 5 fixed, 0 declined.

• major: plain-text extraction used two frames per nesting level (_plain_text_node -> _child_plain_text). It hit RecursionError at about half the depth parse_markdown accepts: '>' * 600 + ' x' crashed, while main returned 'x'. Fixed: the child walk is now an explicit loop inside _plain_text_node. The regression tests test_deeply_nested_blockquote_does_not_overflow_the_stack and test_deeply_nested_list_does_not_overflow_the_stack fail on the previous HEAD and pass now.
• minor: no test covered the closed <...> skip in mentions.py. Fixed: TestReplaceBareMentionsClosedAngleTokens kills the "drop the angles branch" mutant.
• minor: the code-fence and mention bookkeeping mutants (C4, C6, C8, C15, M8) survived. Fixed: exact-output tests with TS-verified expectations. All six mutants are now killed.
• minor: hyphenated upstream names registered only as fuzzy matches. Fixed: renamed to newlineseparated / paragraphseparated, so they are now exact matches.
• minor: scripts/fidelity_target.json was stale. Fixed: regenerated on the final HEAD. Missing is unchanged at 273; the PR-level delta vs main is still missing 282 -> 273 (-9).

gpt-6-astra: 1 round, on d0140cf. Verdict: No actionable defects found.

Bots: CodeRabbit skipped the draft and was then rate-limited, so it left no inline review comments. No gemini comments.

CI (on d0140cf): all green. That covers Tests 3.12/3.13, CodeQL, and Lint & Type Check (ruff, audit, pin drift, strict fidelity, pyrefly).

Local validation: all green. ruff check/format; test-quality audit (0 hard failures); --check-docs; strict fidelity 733/733; pytest 5672 passed / 13 skipped; pyrefly 0 errors.

[coderabbitai]

Warning

Review limit reached

Next included review available in 42 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: f5181825-c8dc-428f-ba9a-0a8d5abcafd5

📥 Commits

Reviewing files that changed from the base of the PR and between 2faf6b7 and 7499a6e.

📒 Files selected for processing (15)

• CHANGELOG.md
• docs/UPSTREAM_SYNC.md
• scripts/fidelity_target.json
• src/chat_sdk/shared/__init__.py
• src/chat_sdk/shared/_js_compat.py
• src/chat_sdk/shared/code_fences.py
• src/chat_sdk/shared/markdown_parser.py
• src/chat_sdk/shared/mentions.py
• tests/test_chat_faithful.py
• tests/test_github_format.py
• tests/test_github_webhook.py
• tests/test_markdown_faithful.py
• tests/test_shared_code_fences.py
• tests/test_shared_mentions.py
• tests/test_thread_faithful.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[patrick-chinchill]

Merge gate: CI green (Lint & Type Check, test (3.12), test (3.13), Analyze (actions), Analyze (python), CodeQL) on 7499a6e (d0140cf + clean merge of origin/main: #202 cards/modals, #240 state-pg, #252 github; only CHANGELOG and the generated fidelity_target.json conflicted, the latter regenerated: missing 273 -> 256); local Codex review (gpt-6-astra, xhigh, --base origin/main) on d0140cf: "No actionable defects found. The changes match the targeted upstream implementations"; 2 astra rounds (no re-review after the merge: no src/tests conflicts and no overlapping modules with the newly merged PRs; full local validation green, 5759 passed, strict 733/733, pyrefly 0 errors); CodeRabbit rate-limited, no other bot findings. Merging with --admin (Protect Main requires a code-owner approval).