Language request: Astro (.astro) #67

Activity

1. joyful-ii-V-I commented on Sep 10, 2026

   

   joyful-ii-V-I

   on Sep 10, 2026

   Collaborator

   More actions

   This is the most complete language request the project has had, and two things in it are worth naming before anything else.

   You found the cost before we did. The grammar not being self-contained is the whole decision, and you flagged it in the second bullet. nvim-treesitter's requires = { 'html', 'html_tags' } means tree-sitter-astro parses the .astro skeleton and delegates the template to HTML and the frontmatter to TypeScript. We vendor 23 grammars and none of them is HTML, so the full grammar is a two-grammar adoption plus injection plumbing.

   Your reduced scope is the right call, and it is cheaper than you pitched it. Extracting only the frontmatter as TypeScript is not a stripped-down version of the full job — it is a different, much smaller job:

   • ts_parser_set_included_ranges() is already in our vendored tree-sitter (api.h:263). ripwire has never called it — zero hits in src/ — so this would be the first use, but it is an API we already ship, not machinery to invent.
   • Frontmatter-only is that call with one range, parsed with the TypeScript grammar we already vendor. No HTML. No second grammar. The --- fences are findable without any parse at all.
   • It builds the injection primitive that Vue SFC, Svelte and MDX would each reuse, so the value outlives Astro.

   Your own measurement is the argument: 114 of 172 exported service functions have callers only in .astro files, and the frontmatter is where those calls live. The template mostly interpolates what the frontmatter already computed.

   If you want to take it

   prompts/add-a-language.md is written for exactly this, aimed at a coding agent, and it is honest about where the afternoons go. Point Claude Code (or your agent of choice) at it from a checkout.

   Do STEP 0 first and be willing to stop there. It asks for a parse rate before any code, and it says plainly that a low rate is a finding rather than a failure. For Astro there is a version of STEP 0 that costs almost nothing and answers the real question: map .astro to the TypeScript grammar wholesale and measure the parse rate. The TS grammar will choke on the template half, so the number will be poor — but how poor, and which nodes survive, tells you whether one-range injection recovers the call edges or whether the template carries more than we think. That is a one-line experiment against withastro/docs and your private monorepo, and it is worth more than any amount of reasoning about it.

   Astro-specific things that will save you time

   • The extension table already supports what you need. src/ingest_crawl.h maps extension → (Lang, grammar, label), and it is normal there for an extension to use a grammar that is not its namesake: .metal → tree_sitter_cpp, .cu/.cuh → tree_sitter_cuda, .mm → ObjC. .astro → TypeScript is the same shape.
   • The Lang enum is serialized into the cache — append only. Reordering invalidates every cache in existence. The prompt says this; it is the trap most likely to bite.
   • Two coordination points, live right now. kParserVer in src/ingest_cache.h is at 85 and moves with any extraction change — kIngestParserVerMirror in src/quality.h MUST move in the same commit (three separate collisions on that pair happened in one day recently). And PR Dart: a 23rd grammar, and the span arm its tree shape requires #75 adds Dart as a 23rd grammar; it also fixes six sites that sized per-language arrays with Lang::Elixir + 1 instead of kLangCount, so if Astro lands after Dart that class of bug is already gone from under you. Rebase and re-derive those numbers rather than carrying them.
   • The gate is written RED first (STEP 7). That is not a formality here — it is how every recent contribution has been reviewed, and it makes review fast because the failure is demonstrated before the fix.

   One thing already resolved

   You filed the silent count="0" separately as #66 — that landed and is closed, along with #62 and #63, in the confident-zero fix. So the "answers zero when the callers live somewhere unindexed" half of your report is fixed independently of whether Astro is ever added, exactly as you argued it should be.

   And if you would rather not

   Filing this issue, at this quality, was already a contribution — the cost analysis in it is the part we would have had to do anyway. If someone else picks it up, they will be starting from your numbers. If you do take it, say so here and ask whatever you need; the frontmatter-only route in particular is small enough to be a first PR, and we will review it properly rather than quickly.
2. joyful-ii-V-I commented on Sep 11, 2026

   

   joyful-ii-V-I

   on Sep 11, 2026

   Collaborator

   More actions

   Marking this help wanted. Astro support would be a great contribution, and the path is well-trodden now.

   prompts/add-a-language.md walks an AI-assisted port end to end:

   • vendoring a pinned tree-sitter grammar;
   • every per-language registration surface;
   • a red-first gate;
   • measuring on real repositories.

   The recent Dart and Kotlin ports are good templates to read alongside it.

   The interesting design question for Astro is likely the component script: the --- frontmatter block holds JavaScript/TypeScript, and reusing the existing TypeScript extraction for it would be the natural route rather than re-deriving symbols.

   If you pick this up, say so here so nobody duplicates the work. Maintainers are happy to review early drafts.
3. joyful-ii-V-I

   added

   help wantedExtra attention is neededExtra attention is needed

   on Sep 11, 2026
4. snrmwg commented on Sep 15, 2026

   

   snrmwg

   on Sep 15, 2026

   Author

   More actions

   A follow-up that changes the size of the frontmatter-only route.

   --callers does not see a call that has no named enclosing function. Astro frontmatter is module-level code. Parsing only the frontmatter would bring those calls into --uses, but not into --callers, --impact or --affected.

   Reproduced on 0.6.1 (release binary, linux-x64). Plain TypeScript, no Astro involved:

   mkdir -p src
   cat > src/svc.ts <<'X'
   export async function getFair(id: number) { return { id }; }
   X
   cat > src/named.ts <<'X'
   import { getFair } from "./svc";
   export async function load() { return getFair(3); }
   X
   cat > src/route.ts <<'X'
   import { getFair } from "./svc";
   type APIRoute = (ctx: unknown) => Promise<unknown>;
   export const GET: APIRoute = async (ctx) => { return getFair(2); };
   X
   cat > src/toplevel.ts <<'X'
   import { getFair } from "./svc";
   const fair = await getFair(1);
   export default fair;
   X
   cat > src/callback.ts <<'X'
   import { getFair } from "./svc";
   [1, 2].forEach(async (n) => { await getFair(n); });
   X
   ripwire . --no-cache --callers=getFair --json
   ripwire . --no-cache --uses=getFair --legend=compact

   {"of":"getFair","defs":1,"count":2,"root":".","hop_tested":0,"hop_untested":2,"graph_ambiguous":0,"graph_unresolved":0,"counts_floor":true,"callers":[{"t":"fn","n":"load","p":"src/named.ts:2"},{"t":"fn","n":"GET","p":"src/route.ts:3"}]}

   <u role="call" p="src/callback.ts:2"/>
   <u role="call" p="src/named.ts:2" in_id="load"/>
   <u role="call" p="src/route.ts:3" in_id="GET"/>
   <u role="call" p="src/toplevel.ts:2"/>

   --uses finds all four. --callers reports count=2 with graph_unresolved=0, so nothing in the answer hints at the two missing sites.

   A frontmatter-only port would therefore also have to attribute module-level call sites to something the caller walk reaches, for example a file-level symbol. That looks like the same design question as #60: a call inside a node:test arrow callback has no enclosing symbol either.

   Measured on the monorepo from my original report, TypeScript side only. Ground truth was TypeScript's findReferences over the 178 exported service functions in apps/www, with Astro frontmatter loaded as virtual .ts files. --callers found 58 of the 113 non-Astro files that reference one of them. The 55 misses are all 52 test files and 3 API route files; I have not looked into why the routes are missed.

   I won't take the frontmatter-only PR on for now. If someone builds a branch, I'm glad to run it against the monorepo.
5. joyful-ii-V-I commented on Sep 16, 2026

   

   joyful-ii-V-I

   on Sep 16, 2026

   Collaborator

   More actions

   Thank you, this measurement is very useful. Those misses have the same root cause as #60: a call outside any named function is dropped from the call graph. We're fixing that first under #60, which should make the frontmatter part much smaller. When there's a branch, we'd be glad to take you up on running it against your monorepo; we'll ping you here.
6. sclyde commented on Sep 22, 2026

   

   sclyde

   on Sep 22, 2026

   More actions

   Taking this, frontmatter-only, along the route you and @snrmwg converged on. STEP 0 first, as prompts/add-a-language.md asks — here are the numbers before the code.

   STEP 0

   Corpus: 1902 real .astro files — withastro/docs, withastro/astro, withastro/starlight, onwidget/astrowind, satnaing/astro-paper, plus two private Astro sites. Shallow clones, not committed, in the spirit of bench/recalleval/extcorpus.lock.

   You asked for the one-line experiment — map .astro to the TypeScript grammar wholesale and see how poor it is. It is poor:

   config	files degraded	median ERROR-byte ratio
   baseline (today)	— all 1902 invisible	—
   wholesale TypeScript	1878 / 1902 (98.7%)	0.33 – 1.00
   frontmatter-only, one included range	1 / 1902 (0.05%)	0.000

   Against .metal's shipped 0.0081 and the 0.123 the C grammar was rejected at for CUDA, wholesale is 27×–120× worse than the option this project already turned down. So it is refused rather than error-recovered.

   The single degraded file under frontmatter-only is packages/astro/e2e/fixtures/errors/src/pages/astro-frontmatter-syntax-error.astro — a file Astro ships on purpose to test its own error reporting.

   What it recovers

   repo	before	after
   astrowind	files=54 symbols=539 edges=88	files=145 symbols=645 edges=137
   astro-paper	files=63 symbols=1949 edges=40	files=90 symbols=1993 edges=75
   starlight	files=777 symbols=8905 edges=1351	files=866 symbols=8981 edges=1381
   docs	files=2729 symbols=24939 edges=2276	files=2811 symbols=25011 edges=2309

   On the grammar

   I checked virchau13/tree-sitter-astro at the revision you pinned, 213f6e69 — still HEAD, so the ecosystem really has not moved. It does not help, for a reason worth recording: grammar.js:81 is frontmatter: $ => seq(token(prec(1,'---')), optional($.frontmatter_js_block), '---'), and frontmatter_js_block is an external token. The grammar lexes the whole frontmatter as one opaque blob and parses none of it, and queries/ ships only highlights.scm and injections.scm — no tags.scm. Since ripwire extraction is tags-query driven, it would yield zero definitions. Every symbol still has to come from the TypeScript grammar over that range. PRRPCHT/tree-sitter-astro-next has the identical design and 0 stars / 0 forks.

   One small correction to the thread, since the cost analysis leaned on it: the grammar is self-contained as a build artifact — grammar.js does grammar(require('tree-sitter-html/grammar'), …), which resolves at generation time, so the committed parser.c already has HTML merged in and injections.scm never injects HTML. Vendoring would not have needed a second grammar. It is still the wrong move, for the tags reason above.

   #60

   Confirmed fixed on main before building anything: --callers=sum_of_squares on test/jslangfix returns the module-scope owner. So frontmatter top-level calls mint real --callers edges, and @snrmwg's 2026-09-15 objection no longer applies. They read as <s t="modscope" n="&lt;file-scope&gt;" p="src/pages/index.astro:1"/> rather than a named function, which the PR discloses.

   @snrmwg — I could not reproduce your 3 API-route misses. Every route spelling I could construct resolves on current main: export async function GET, const GET: APIRoute = async (ctx) =>, satisfies APIRoute, a wrapper call, and a handler inside an object literal — as do ~/* path aliases and workspace package imports. You measured on 0.6.1, and the object-literal shape in particular was a #60 case, so I think #60 took them. I would rather say that than file a speculative issue. The 52 test-file misses were certainly #60.

   Branch is up: #320.

   @snrmwg — this is the branch you offered to run against the monorepo, and I would very much like that. Your ground truth was 114 of 172 exported service functions reachable only from .astro; what I would most like to know is how many of those --callers finds now, and whether the 3 API routes come back. Build is cmake -S . -B build && cmake --build build -j, and --callers=<a service fn> on apps/www is the whole test.
7. joyful-ii-V-I

   closed this as completedin 8cffe4aon Sep 26, 2026
8. kapoorsunny

   added a commit that references this issue on Sep 26, 2026

   feat(lang): Astro — .astro frontmatter on the TypeScript grammar, via…

   ...

   This issue will close once commit 8cffe4a is merged into the 'main' branch.

   8cffe4a
9. snrmwg commented on Sep 27, 2026

   

   snrmwg

   on Sep 27, 2026

   Author

   More actions

   Ran 0.6.5 (release binary) against the monorepo from the original report. Ground truth: TypeScript findReferences over the 179 exported service functions in apps/www, with the Astro frontmatter loaded as virtual .ts files. That gives 349 (export, file) pairs.

   • Frontmatter calls: 178 of 182 found. Non-Astro files: 116 of 124. On 0.6.1 it was 0 for Astro and 58 of 113 for the rest.
   • Every miss outside the templates has one of two causes. Either track has 6 same-named definitions, which the answer discloses as unproven_defs=3. Or a function is passed as a value in a table-driven test instead of being called.
   • What remains is silent: 43 pairs are calls made only in the template half, e.g. href={createFairWebsiteLink(...)}. For 5 exports those are the only callers, and --callers/--uses answer count=0 with nothing in the answer hinting at it. graph_unindexed still carries the repo-wide total on every answer. Would a per-answer marker be feasible, e.g. when the symbol is imported by an .astro file whose template is unread?

   On the 3 API-route misses: on 0.6.5, only 3 route files are missed, all because of the track ambiguity. I can't tell whether they are the same 3 as in my earlier measurement.