Skip to content

fix: strip stray code fences and separate bare URLs from non-ASCII text - #76

Open
hoanle0126 wants to merge 1 commit into
webdriverio:mainfrom
hoanle0126:fix/fa-ja-markdown-and-links-33
Open

hoanle0126 wants to merge 1 commit into
webdriverio:mainfrom
hoanle0126:fix/fa-ja-markdown-and-links-33

Conversation

@hoanle0126

Copy link
Copy Markdown
Contributor

Summary

Fixes #33

Addresses the two Markdown translation issues reported in #33 that caused Docusaurus MDX compilation and link parsing to fail for fa, ja, and other locales:

  1. Stray code fences (```) and section labels (Body: / بخش body:) in translated Markdown:

    • When the model wraps either the frontmatter or the entire response in a ```markdown / ```yaml block, stripping only the preamble before the opening --- delimiter left either:
      • a closing ``` fence (and optional translated Body: label) immediately after the frontmatter --- (as seen in fa/docusaurus-plugin-content-docs/current/Capabilities.md), or
      • an unclosed/duplicate trailing ``` fence on the final line of the file (present in 12 translated .md files across ar, de, fa, fr, ja, ru, and ta).
    • Updated sanitizeMarkdown and validateFrontmatter in src/translate.ts (plus TRANSLATION_INSTRUCTIONS in src/constants.ts) to strip and reject stray post-frontmatter fences/labels and unbalanced trailing code fences, and cleaned up the 12 affected Markdown files.
  2. Bare URLs glued directly to non-ASCII text / fullwidth punctuation in CJK translations:

    • In languages without word spaces (ja, zh, ko), bare http:// or https:// URLs followed immediately by non-ASCII characters or fullwidth punctuation ((, )) without a space caused GFM autolink (remark-gfm) to swallow the trailing text (and any subsequent [ from a Markdown link), either failing MDX compilation (Can't parse URL ... with base unspecified:// as in ja/.../BoilerplateProjects.md) or producing corrupted URLs.
    • Updated sanitizeMarkdown and validateFrontmatter to ensure bare URLs outside code blocks, inline code, and [text](url) / <url> links are separated by a space before trailing non-ASCII characters, and fixed the 6 affected files across ja, ko, and zh.

Verification

  • Added unit tests in test/frontmatter.test.ts covering post-frontmatter fences/labels, outer trailing fences, bare URLs followed by non-ASCII text, and repository-wide validation across all 18 locales (npm test).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[🐛 Bug]: FA and JA cause issues for deploying docs

1 participant