feat: add MD→HTML context-tree migration script (Python, one-shot) - #700
Closed
RyanNg1403 wants to merge 11 commits into
Closed
feat: add MD→HTML context-tree migration script (Python, one-shot)#700RyanNg1403 wants to merge 11 commits into
RyanNg1403 wants to merge 11 commits into
Conversation
Standalone Python migrator under scripts/migrate-context-tree-py/. Walks .brv/context-tree/, converts every .md topic to a single <bv-topic> HTML document matching the format curate writes on proj/html-mem-conversion, and archives originals to .brv/_migrations/context-tree-md-<YYYY-MM-DD>/ for full reversibility via --rollback. _archived/ subtree is skipped (no <bv-archive-stub> in the vocabulary). After one run the live tree contains zero .md outside _archived/.
Refactor YAML loading to use FrontmatterLoader and enhance frontmatter parsing.
Summary of changes to migrate_context_tree.py: Refactor: drop non-vocabulary <p data-md-section> emission. The brv HTML reader (html-reader.ts:77) and LLM renderer (html-renderer.ts:155) both silently skip non-bv-* tags, making preservation in <p>/<section> dead content. Output now uses ONLY the closed bv-* vocabulary; orphan content is mapped to existing bv-* targets via a heading-name heuristic, or dropped with a per-file warning so the operator sees what was lost. Edge-case fixes: - H1 body title falls back into <bv-topic title> when frontmatter title is empty/missing - Body prose between H1 and first ## hoists into <bv-topic summary> when summary attr is empty - Orphan section heuristic: ## Overview/Purpose -> <bv-reason>, ## Architecture/Structure/Scope -> <bv-structure>, ## Evidence -> bv-fact bullets, ## Rules -> split bv-rule siblings, ## Patterns -> bv-pattern, ## Decisions -> bv-decision, ## Abstract/Summary -> bv-topic summary attr - Unknown frontmatter keys emit dropped-frontmatter-key warnings; runtime-signal keys (importance, recency, maturity, accessCount, updateCount) are allow-listed and dropped silently per spec - Multi-rule blocks with 'Rule N:' prefix split correctly (line-start and after sentence-end + whitespace) - Plural raw-concept labels tolerated (Tasks, Flows, Changes, Files, Patterns, Timestamps, Authors); unknown labels warned - Unknown ### subsections under ## Narrative routed via heuristic: ### Patterns -> bv-pattern siblings, ### Decisions -> bv-decision, ### Overview -> bv-structure - ## Facts / ## Raw Concept Changes/Files lists accept any bullet style (-, *, +, 1.) — previously only - was matched - All fenced code blocks anywhere in body promoted to <bv-diagram>; language tag drives type (mermaid/plantuml/ascii/dot/graphviz, else other) - YAML scalar values containing ' #' emit truncation-hazard warning - _index.md descriptors with only ## Purpose / ## Scope now produce populated <bv-topic> instead of empty stubs Validation: zero regressions on 246-file golden baseline (real 220-file production archive + 26 synthetic and LLM-generated fixtures); zero canonical bv-* elements lost. Live validation across two LLM providers (Gemini 3 Flash Preview + OpenAI gpt-5.4-mini) confirmed 0 failures and 0 non-bv-* tag emissions on freshly-generated corpora plus a snapshot of the developer's live context tree.
Contributor
|
Claude finished @RyanNg1403's task in 7m 14s —— View job Claude Code PR Review — #700Overall score: 4 / 5 — comprehensive, well-documented, and well-tested. One real correctness bug (multi-dot filenames) and a handful of correctness/UX nits worth a pass before merge. Todo
SummaryThe PR delivers exactly what the description promises: orphan section content that would otherwise be silently dropped by The codebase is unusually well-commented for a script — case numbering tying back to the ticket, schema enums mirrored from TS as Python constants, explicit conflict-resolution policies in Significant findings (one per inline)
Things I checked and was happy with
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5acb808d44
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
P1 (correctness bugs):
- Path.with_suffix(".html") misuse breaks multi-dot filenames
(e.g. node.js.md). New _html_sibling_path() helper uses string
concatenation so the topic path attribute and on-disk filename
agree for all topic names. Fixes 3 sites: _html_sibling_exists,
_process_file, rollback.
- Rollback unconditionally deleted .html siblings for every
restored .md — destructive on mixed trees that already had .html
files before migration. Migration now writes a manifest of pre-
existing siblings (_pre_existing_html_siblings.json) under the
archive root; rollback reads it and skips deletion for entries
listed there.
P2 (correctness):
- Malformed YAML frontmatter is now surfaced as a
malformed-frontmatter warning instead of silently dropping
title/summary/tags to defaults.
- _RULE_PREFIX_LINE split was capturing pre-prefix prose as a
leading rule entry (the intro paragraph became <bv-rule> #1).
Drop parts[0] explicitly.
- _SECTION_REGEX / _parse_section now mask fenced code blocks
before matching so a literal `## ...` line inside ```bash …
``` doesn't terminate the enclosing section. Same masking via
new _mask_fenced_blocks helper.
- _FENCED_BLOCK_REGEX title group line-anchored — a `**bold**`
mid-sentence before a fence no longer becomes a spurious
diagram title.
- Rules splitter and bullet-list parsers now use
_collect_bullet_items_with_continuations so indented
continuation lines on multi-line list items are preserved
rather than silently discarded.
P3 (UX / minor):
- --rollback honors --dry-run (preview restore/delete counts) and
requires --yes (or stdin TTY confirmation) before executing the
destructive action.
- _classify_entry only marks <name>.abstract.md / <name>.overview.md
as derived when the base <name>.md sibling exists. Empty sidecars
without a base archive as "empty-sidecar"; non-empty orphans are
migrated as real topics. Avoids silently archiving user-authored
topics whose filenames happen to end in those suffixes.
- bv-rule id dedup now shared across canonical + orphan-routed
rules via a topic-scoped seen-ids set; new _uniquify_id helper.
- _parse_narrative and _diagrams_section_span use (?ms) anchoring
so back-to-back `## A\n## B` (no blank line) terminates the
Narrative block correctly.
- Frontmatter values of the wrong type (title: 42, tags: not-a-list)
emit frontmatter-type-mismatch warnings instead of silent
coercion via the new _opt_str_typed / _str_list_typed helpers.
- Inline comment near createdat=/updatedat= emission documents the
bv-topic schema's RESERVED_TOPIC_ATTRIBUTES hazard for downstream
curate prompt builders.
Validation: zero regressions on 246-file golden baseline (zero
canonical bv-* elements lost on any previously-migrated file). 37
newly-migrated topics from the case-12 sidecar promotion. Live-val
across 6 workspaces holds at 79 migrated / 0 failed. End-to-end
preserve-list test confirms rollback no longer deletes pre-existing
HTML siblings; end-to-end multi-dot test confirms node.js.md
migrates to node.js.html with matching <bv-topic path>.
Contributor
|
Claude finished @RyanNg1403's task in 7m 29s —— View job Claude Code PR Review — #700 (follow-up)Overall score: 4 / 5 — the follow-up commit cleanly addresses every P1/P2/P3 finding from the previous round, plus the two Codex findings. One residual bug from the same class as a previous P2 (fence-mask incompleteness), one data-safety concern around manifest write timing, and a few low-priority nits. Todo
SummaryThe follow-up commit ( The biggest residual issue is that the fence-masking fix was applied to Significant findings
Things I checked and was happy with
|
RyanNg1403
changed the title
P1 (data-safety, bug): - Preserve-list manifest is now written BEFORE archiving begins. Computing the list of .md topics whose .html siblings predate the migration up front (rather than after the archive loop) means a Ctrl+C / OOM / power loss mid-run still leaves a usable manifest on disk; rollback won't silently delete pre-existing siblings just because the script was killed before the manifest write. - _parse_narrative and _diagrams_section_span now run against a fence-masked body so a literal `## ...` line inside a fenced code block within `## Narrative` doesn't truncate the section early and miscompute the diagrams-exclude span (which would otherwise produce duplicate <bv-diagram> emissions in _extract_all_fenced_blocks). - Rollback warns loudly on stderr when the preserve manifest is missing or unreadable, instead of silently falling back to an empty preserve set. P2 (correctness): - _FENCED_BLOCK_REGEX now requires the optional **Title** prefix to be immediately followed by the fence opener (`[ \t]*\n`) so a standalone bolded paragraph + blank line before a fence is no longer captured as a spurious diagram title. - _FENCED_BLOCK_REGEX matches both ``` and ~~~ fence markers, mirroring _FENCE_MASK_REGEX. Previously a ~~~-fenced diagram was masked from section detection but never promoted to bv-diagram, so its content vanished from the migrated HTML. - _parse_section now uses the same `(?m)^##\s[^#]` anchoring as _parse_narrative so back-to-back H2s with no blank line between (e.g. `## Reason\n## Facts`) terminate the section correctly. Consistency / UX: - Empty `.abstract.md` / `.overview.md` files with no base sibling now surface as `failed` (empty-file) instead of being silently archived as `empty-sidecar`. Aligns with case-12: non-empty standalone sidecars migrate as real topics, so empty ones are real topics with empty content — the operator can decide whether to delete or fill. - Rollback TTY confirmation prompt and dry-run preview are both written to stderr; the prompt itself is read via sys.stdin.readline rather than input() (which writes to stdout). Stream consistency for `--rollback 2>preview.log`. Validation: - Re-ran migration on all 9 baseline workspaces (246 file golden diff): 137 changed, 109 unchanged, 37 new files (case-12 sidecar promotions), 0 lost files, 0 canonical bv-* losses, 0 non-bv-* tag emissions. - 3 sub-agent corpora freshly generated via Gemini gemini-3-flash-preview (36 topics across realistic engineering domains + an explicit stress-test agent): 0 migration failures, 0 non-bv-* tag emissions, all 285 tag emissions are from the closed bv-* vocabulary plus <li> legitimately nested in bv-changes/bv-files.
Contributor
|
Claude finished @RyanNg1403's task in 6m 26s —— View job Claude Code PR Review — #700 (post-fix verification)Overall score: 5 / 5 — the Todo
SummaryThe follow-up commit ( The biggest residual is symmetry, not correctness — The non-TTY rollback path's Significant findings
Things I checked and was happy with
|
Correctness: - split_rules_block runs its bullet / numbered / Rule-prefix / paragraph detection against a fence-masked copy of the input. Same class of bug the prior commit cleaned up elsewhere: a `- some code` line or a `Rule 1:` comment inside a fenced sample within `### Rules` used to flip the detector and spawn a spurious rule entry. Item extraction still operates on the original text via byte-span slicing so the bullet collector keeps multi-line continuations intact. UX (safety): - `--rollback` without `--yes` now ERRORS OUT when stdin is not a TTY (CI / piped invocations), instead of silently proceeding with the destructive operation. Help text updated to match. The previous behavior — "assume non-interactive caller accepted the risk" — is unsafe for a typo'd CI command; explicit consent is required. Observability: - Preserve-list manifest is now written unconditionally (even when the list is empty) so rollback's "no preserve-list manifest" stderr warning only fires on genuine anomalies (archive predates feature, prior run killed before write, manifest deleted) — not on the common case of a clean migration with no pre-existing siblings. Validation: - 9 baseline workspaces vs golden: 137 changed, 109 unchanged, 37 new (case-12 sidecar promotions), 0 lost, 0 canonical bv-* lost, 0 non-bv-* tag emissions. - 2 fresh sub-agent corpora via Gemini gemini-3-flash-preview, one explicitly stress-testing the new fence-mask-in-split-rules-block fix (fenced "Rule N:" comments, mixed ``` / ~~~ fences, bullet rules with fenced pseudocode): 25 topics migrated, 0 failures, 0 non-bv-* tag emissions, 200 tag emissions all from the closed bv-* vocabulary plus <li>/<pre>/<code> as documented children.
Contributor
|
Claude finished @RyanNg1403's task in 4m 7s —— View job Claude Code PR Review — #700 (4th-round verification)Overall score: 5 / 5 — commit Todo
SummaryThe follow-up commit (
The one residual is Significant findings
Things I checked and was happy with
|
…terator symmetry) Close the last fence-mask symmetry gap: `_parse_narrative`'s inner `### X` subsection iterator now runs against a fence-masked copy of `section` (mirroring what the outer narrative regex, `_parse_section`, `_list_orphan_sections`, and `_diagrams_section_span` already do). Without this, a fenced `### Foo` inside `## Narrative > ### Structure` would truncate Structure's body and emit a phantom subsection routed via the case-8 heuristic. Also upgrade the inner lookahead from `\n##\s` to `^##\s[^#]` (multiline) so a `### X` followed IMMEDIATELY by another H2 (no blank line) terminates correctly — consistent with the same upgrade previously applied to `_parse_section` and `_parse_narrative`'s outer regex. Validation: - Baseline (246 files, 9 workspaces): 137 changed, 109 unchanged, 37 new (case-12 sidecar promotions), 0 lost, 0 canonical bv-* lost, 0 non-bv-* tag emissions. - 2 fresh sub-agent corpora via Gemini gemini-3-flash-preview, one explicitly stress-testing the inner-iterator fix with `### Foo` embedded inside narrative fenced code samples: 28 topics migrated, 0 failures, 0 non-bv-* tag emissions, 211 tag emissions all from the closed bv-* vocabulary + <li> as a documented child.
Contributor
|
Claude finished @RyanNg1403's task in 2m 31s —— View job Claude Code PR Review — #700 (5th-round verification)Overall score: 5 / 5 — commit Todo
SummaryThe diff is minimal and surgical — 12 lines added, 2 removed, all in
This is the same uniform fix the rest of the PR has been applying across all six section-walkers ( Inline comments record the verification and two non-blocking symmetry nits I noticed while spot-checking:
Both are low priority and unrelated to the core fix in this commit. Significant findings
Things I checked and was happy with
|
…mmetry)
Both findings labeled low-priority by the reviewer ("not a primary
data-loss path", "essentially defensive"), but applying for full
pattern uniformity: every section walker in the file now uses the
same fence-mask + `(?m)^##\s[^#]` multiline-anchor idiom, so future
edits can't accidentally drift back into the old literal-`\n##\s`
behavior in one place while the rest of the file stays consistent.
_extract_snippets_from_body:
- Heading-strip loop and orphan-section strip now operate on a
fence-masked copy of the body so a literal `## X` line inside a
code fence within (say) `## Reason` can't terminate the strip
early and produce a false-positive `dropped-snippets` warning.
- Strip is now span-based — drop spans are collected from the masked
text, merged, then the residual is rebuilt from the ORIGINAL body
via the inverse span set. Fenced content inside the residual
(i.e., outside any canonical section) survives byte-for-byte.
- Lookahead also upgraded to `(?ms)^##\s[^#]|\n---\n|\Z` for the
back-to-back-H2 case.
_diagrams_section_span inner `### Diagrams` regex:
- Inner lookahead upgraded from `\n###\s|\n##\s|$` to `(?ms)^###\s|^##\s[^#]|\Z`,
matching the outer narrative regex, _parse_section, the inner
narrative iterator, and `split_rules_block`. Benign in practice
(outer regex already bounds `section`), purely a consistency fix.
Validation:
- Baseline (246 files, 9 workspaces): 137 changed, 109 unchanged,
37 new, 0 lost, 0 canonical bv-* losses, 0 non-bv-* tag emissions.
- 2 fresh sub-agent corpora via Gemini gemini-3-flash-preview,
one explicitly stress-testing the snippet extractor (fenced `## X`
inside narrative) and the inner Diagrams anchor (back-to-back
`### Diagrams\n### Implementation Notes` with no blank line):
18 topics migrated, 0 failures, 0 non-bv-* tag emissions.
Contributor
|
Claude finished @RyanNg1403's task in 4m 46s —— View job Claude Code PR Review — #700 (6th-round verification)Overall score: 5 / 5 — commit Todo
SummaryThe diff is minimal (+42 / −6, both in 1. 2. The PR's golden-baseline corpus (246 files) wouldn't catch this — the test asserts "0 canonical Significant findings
Things I checked and was happy with
|
The previous commit's parallel-span collection in `_extract_snippets_from_body` introduced a subtle regression: the orphan-section loop walked `_SECTION_REGEX` (which has NO `\n---\n` terminator), so a canonical `## Reason` heading adjacent to a `---`-separated snippet would match the WHOLE `## Reason\n...\n---\n snippet...` span, the merge step picked the larger span over the canonical-loop's `\n---\n`-terminated span, and the snippet vanished from the residual — no `dropped-snippets` warning fired. The pre-refactor sequential `.sub()` chain dodged this because the second strip operated on the already-cleaned residual where the canonical heading was gone. The parallel-span refactor lost that ordering guarantee. Fix: filter out canonical headings in the orphan-section loop (same filter that `_list_orphan_sections` already uses at the `KNOWN_SECTION_ HEADINGS` check). Canonical spans are added by the dedicated loop above with the correct `\n---\n` terminator; the orphan loop should only contribute spans for truly-unknown headings. Added an inline regression test in the commit message verification: ``` case 1: ## Reason\\nbecause foo\\n---\\nlegacy snippet content → ['legacy snippet content'] ✓ (was [] before fix) case 2: ## Reason\\nfoo\\n---\\nsnippet\\n## Facts\\nbar → ['snippet'] ✓ (was [] before fix) ``` Validation: - Baseline (246 files, 9 workspaces): 137 changed, 109 unchanged, 37 new, 0 lost, 0 canonical bv-* losses, 0 non-bv-* tag emissions. - 2 fresh sub-agent corpora via Gemini gemini-3-flash-preview, one explicitly producing the ruler-separated snippet shape the bot flagged: 26 topics migrated, 0 failures, 0 non-bv-* tag emissions.
Contributor
|
Claude finished @RyanNg1403's task in 2m 56s —— View job Claude Code PR Review — #700 (7th-round verification)Overall score: 5 / 5 — commit Todo
SummaryMinimal, surgical diff (+10 lines, all in for m in _SECTION_REGEX.finditer(masked):
if m.group(1).strip() in KNOWN_SECTION_HEADINGS:
continue
drop_spans.append((m.start(), m.end()))Canonical headings now contribute spans only through the dedicated loop (line 1473–1479) which carries the Trace check (both worked examples from the commit message):
The pre-fix behavior would have returned Significant findings
Things I checked and was happy with
|
…try) Make the orphan-section canonical-heading filter case-insensitive so it stays consistent with the canonical-heading loops that use `re.IGNORECASE`. Add `KNOWN_SECTION_HEADINGS_LOWER` constant and apply the lowered comparison in both `_list_orphan_sections` and `_extract_snippets_from_body`. Without this, a lowercase `## reason` adjacent to a `---`-separated snippet would slip past the orphan-walker filter and emit an unterminated drop span that swallows the snippet on merge — the same regression class the previous round closed for the canonical-vs-orphan walker. Theoretical for current production trees (all Title Case) but removes the asymmetry.
Contributor
|
Claude finished @RyanNg1403's task in 5m 46s —— View job Claude Code PR Review — #700 (8th-round verification)Overall score: 5 / 5 — commit Todo
SummarySurgical diff (+15 / −2, all in 1. New constant 2. 3. The inline comments at both call sites explicitly explain the why (consistency with Significant findings
Things I checked and was happy with
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/migrate-context-tree-py/migrate_context_tree.py— a one-shot Python migrator that converts a project's.brv/context-tree/from the legacy Markdown-with-YAML-frontmatter format to thebv-*HTML format this branch'scurateflow writes today. Offline, no daemon, three flags (--project-root,--dry-run,--rollback), PyYAML as sole runtime dep.bv-*vocabulary defined insrc/server/infra/render/elements/registry.ts. Orphan markdown content (non-canonical## Xsections like## Overview/## Evidence/## Architecture/## Patterns) is routed to the nearest semantically-fittingbv-*target via a heading-name heuristic; canonical content always wins and unmapped headings surface asdropped-orphan-sectionwarnings. The rationale: the HTML reader (src/server/infra/render/reader/html-reader.ts:77) and LLM renderer (html-renderer.ts:155) silently skip non-bv-*tags, so any preservation in<p data-md-section="…">or<section>would be dead content from brv's pipeline perspective.title:falls back to body# H1, lede paragraph (between H1 and first##) hoists into<bv-topic summary>, "Rule N:" prefix splitter (line-start or after sentence-end), plural**Subsection:**tolerance under## Raw Concept, mixed bullet style (-/*/+/1.) in## Factsand Changes/Files lists with indented continuation preservation, fenced code blocks anywhere in body promoted to<bv-diagram>with language-driventype, multi-dot filename support (e.g.node.js.md→node.js.html), rollback that preserves pre-existing.htmlsiblings via an archive-root manifest,--rollback--dry-runpreview +--yesconfirmation gate, and per-topic warnings for malformed YAML / unknown frontmatter keys / type-mismatched frontmatter values / YAML' #'truncation hazards.Test plan
bv-*elements lost, 0 files disappeared, every changed file matches an expected output patterngpt-5.4-mini) on freshly-generated context trees + a developer's live tree snapshot: 0 migration failures, 0 non-bv-*tag emissions in any output HTML.htmlremoved, pre-existing.htmlsiblings preserved per the archive manifest, archive folder removed)node.js.md) produces matchingnode.js.html+<bv-topic path="node.js">; preserve-list test (pre-existingfoo.htmlalongsidefoo.mdsurvives migrate → rollback unchanged)malformed-frontmatter,dropped-frontmatter-key:<key>,frontmatter-type-mismatch:<key>,yaml-comment-truncation:<key>,dropped-orphan-section:<name>,dropped-raw-concept-subsection:<label>,dropped-narrative-subsection:<label>,dropped-snippets,missing-timestamps) — these are the operator's only signal for what was lost or guessed