You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
aspire docs api: read the API catalog from the sitemap index - #21044
aspire docs api list|search|get built its catalog from a single hardcoded https://aspire.dev/sitemap-0.xml. microsoft/aspire.dev#1874 adds an opt-in hybrid build that renders API pages on demand, which moves the API URLs out of sitemap-0.xml and into a new sitemap-api.xml linked from sitemap-index.xml. Without this change, list and search return almost nothing against such a site.
This PR makes the CLI ingest the catalog from the sitemap index:
DefaultSitemapUrl is now https://aspire.dev/sitemap-index.xml; docs:api:sitemapUrl still overrides it.
The fetched document is resolved by its root element. sitemapindex fetches every child <sitemap><loc>; urlset behaves exactly as before, so existing overrides and deployments that list every API route in one sitemap keep working.
Each child is fetched with its own ETag/content cache entry, so an unchanged sitemap-0.xml stays a 304 when only the API sitemap changes.
Child URLs are rebased onto the configured host (same as page URLs), so a staging/local override never reads production's child sitemaps.
Child entries are unioned, de-duplicated and sorted, so the index fingerprint is deterministic and changes whenever any child changes. IndexSchemaVersion is bumped to 2 so indexes persisted by older CLIs are rebuilt.
If a child cannot be fetched and has no cached copy, the refresh fails (logged as a warning) and the existing index fallback applies, instead of building a catalog missing a language.
get is unchanged (<page-url>.md).
docs/specs/api-docs-commands.md, the AspireConfigFile doc example and the generated VS Code config schema description are updated.
Open question carried over from the issue: released CLIs only read sitemap-0.xml, so the aspire.dev side must decide whether to keep API URLs there for a transition period before a hybrid build serves production. This change de-duplicates, so it works either way.
Not covered: this has not yet been run against a real hybrid-built site.
Default to sitemap-index.xml, merge child sitemaps (rebased onto the configured host, per-child ETag caching), and keep plain urlset sitemaps working. Bump the index schema version so older cached indexes are rebuilt. Fixes#20967.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Tested At: 2026-10-16, about 14:02Z–14:27Z (Windows, local mode, isolated temp dir <testDir>)
Test target: aspire.dev production does not serve the two-child sitemap structure this PR expects yet, so the main scenarios ran against the staging build at https://aspiredevendpoint-et5z2uzad5afe-fff6bthkbvduasfc.z01.azurefd.net/ (called <staging> below), selected through docs:api:sitemapUrl. Production was also tested for compatibility (Scenarios 3 and 4c).
Installed Version:17.0.0-pr.21044.gabf08f7a (dogfood CLI from workflow run 38836394587, channel pr-21044)
Status: ✅ Verified. The PR head was unchanged when testing finished.
Changes Analyzed
Files Changed (9 files, +378/−19)
src/Aspire.Cli/Documentation/ApiDocs/ApiDocsFetcher.cs (+133/−2) - Modified: resolves a <sitemapindex>, rebases child URLs onto the configured host, fetches children in parallel (one ETag cache entry each) and merges them into a deterministic synthetic <urlset>. A plain <urlset> root passes through unchanged.
src/Aspire.Cli/Documentation/ApiDocs/ApiDocsSourceConfiguration.cs (+7/−2) - Modified: default docs:api:sitemapUrl is now https://aspire.dev/sitemap-index.xml.
src/Aspire.Cli/Documentation/ApiDocs/ApiDocsIndexService.cs (+3/−2) - Modified: persisted-index schema version 1 → 2.
Hosting integration, Dashboard, Template or Client changes
CI infrastructure changes
VS Code extension source changes (only a default-URL string in the JSON config schema)
Test and docs changes
Test Scenarios Executed
All runs used the PR dogfood CLI with -l Debug --non-interactive --nologo. The cache and logs lived under <testDir>, not the real ~\.aspire. In total there were 44 recorded CLI runs: 36 exit 0, 8 exit 1 (analyzed below), 0 timeouts.
Scenario 1: Staging two-child sitemap index
Objective: The CLI follows the index to both children, merges them and serves the full API catalog. Coverage Type: Happy path Status: ✅ Passed
Steps:
Set docs:api:sitemapUrl to <staging>/sitemap-index.xml in a scenario-local aspire.config.json.
Ran docs api list csharp (cold), list typescript, list csharp/aspire.hosting, search DistributedApplicationBuilder, get csharp/aspire.hosting/distributedapplicationbuilder and search addRedis --language typescript.
Cold run: 3.9 s, exit 0. It fetched the index (245 chars), then both children from the staging host: sitemap-0.xml (14.5M chars) and sitemap-api.xml (3.6M chars). Result: "Indexed 24205 API reference items". No request went to aspire.dev.
list csharp returned 186 packages, list typescript 146, and list csharp/aspire.hosting 387 types. search ranked csharp/aspire.hosting/distributedapplicationbuilder first (score 202). get fetched the .md page from the staging host.
Ground truth from curl: staging sitemap-0.xml has 8,131 <loc> entries and 0 API routes. sitemap-api.xml has 24,205 entries, all API routes. The union is 24,205 unique API routes (4,913 C# + 19,292 TypeScript), which equals the CLI count.
Scenario 2: Cache and ETag behavior
Objective: Each file in the index flow is cached and revalidated independently. Coverage Type: Happy path / boundary Status: ✅ Passed
Observations:
Warm runs send If-None-Match for the index and each child, then log "Loaded 24205 … from cache" (Scenarios 3 and 5 show the 304s).
The staging Azure Front Door endpoint ignores If-None-Match: it always answers 200 with Cache-Control: no-cache and X-Cache: TCP_MISS. Every invocation therefore re-downloads about 18 MB (2.3–2.4 s per warm run). This is an endpoint limitation, not a PR defect.
Cache keys are host-qualified for non-aspire.dev hosts (for example <staging-host>-sitemap-api.txt plus .etag.txt), so they cannot collide with production entries.
Scenario 3: Production default (no config), today's published structure
Objective: The new default URL works against what aspire.dev serves today. Coverage Type: Compatibility Status: ✅ Passed
Observations:
Production's index currently lists one child, sitemap-0.xml (18.2M chars). Cold run 2.2 s: "Indexed 24205".
Warm run 1.0 s: HTTP 304 for both the index and sitemap-0.xml, then "Loaded 24205 … from cache".
Scenario 4: Plain <urlset> roots (back-compat)
Objective: A configured URL that is not a sitemap index keeps its previous behavior. Coverage Type: Happy path / boundary Status: ✅ Passed
Case
Root configured
Result
4a
<staging>/sitemap-api.xml
One fetch, no child fetches, "Indexed 24205" ✅
4b
<staging>/sitemap-0.xml (models the pre-PR default against the new structure)
"Indexed 0", ❌ No API entries found under 'csharp'., exit 0 (see Finding 3)
4c
production sitemap-0.xml, pinned explicitly
"Indexed 24205", 304s on the warm run ✅ (existing pinned configs keep working)
Scenario 5: Unhappy paths
Objective: Check partial, missing, empty, unreachable and malformed responses. Coverage Type: Unhappy path / boundary Setup: A local fake origin on loopback (small 6-item catalog, 7 after child B changes) serving a two-child index. Every row was verified against the origin's per-request log (server-requests.jsonl). The indexes list canonical https://aspire.dev/... child URLs; the log shows the CLI rebased them onto the loopback origin.
A. Caching and resilience
ID
Case
Observed
Status
F1
Cold run
exit 0, 515 ms; child requests overlapped on the server (parallel fetch)
✅
F2–F4
Warm list, search, search
3 × 304 (index, A, B), each with If-None-Match; "Loaded 6 … from cache"
✅
F5
Child B returns 404, B cached
Warning "Failed to fetch content … 404", cached B reused, exit 0
✅
F6
Child B content changed
B 200, re-index to 7 items, new entry visible
✅
F7
Index lists children in swapped order
Index 200, children 304, merged content identical, so "Loaded 7" (no needless re-index)
✅
F8
Index lists a child twice
Exactly one request per child
✅
F10
Origin serves corrected B after F9
B 200 with a new ETag, cached bad body replaced, exit 0
✅
F11
B 404, no cached B, good cached index
Warnings "API catalog is incomplete" and "Using cached index with 7 items", exit 0
✅
B. Cold-start failures (no cache)
ID
Case
Observed
Status
M1, M2
A child returns 404
Warning "catalog is incomplete", then an empty catalog (not a partial one): ❌ No API entries found, exit 0. M2 re-requests the missing child on each run.
✅
E1
Index with zero children
Warning "does not list any child sitemaps", ❌ No API entries found, exit 0
✅
U1
Origin refused connection
Warning (SocketException 10061), ❌ No API entries found, exit 0, about 2.2 s (latency not investigated)
✅
C. Malformed or hostile XML
ID
Case
Observed
Status
F9
Child B returns HTTP 200 + truncated XML, a good cached index exists
exit 1: XmlException from ApiDocsFetcher.MergeSitemaps; no fallback to the cached index
❌ new code path (Finding 1)
B1
Same, cold
exit 1, same stack
❌ new code path
R2
Same, minimal static repro (see Finding 1)
exit 1, same stack
❌ new code path
D2
Child contains a DOCTYPE with an external entity
exit 1 from MergeSitemaps ("DTD is prohibited"); 0 requests reached the entity URL
❌ resilience only, no XXE
N1
Root body is not XML
exit 1 in ApiSitemapParser.Parse
⚠️ pre-existing path
D1
Root contains a DOCTYPE
exit 1 in ApiSitemapParser.Parse; DTD prohibited
⚠️ pre-existing path
P1
Control: truncated file configured directly as the root
exit 1 in ApiSitemapParser.Parse; reused B1's cached ETag, so the 304 replayed the cached truncated body
⚠️ pre-existing; shows a cached bad body is sticky
D. CLI surface on a good site (G1–G6) ✅ — cold parallel index "Indexed 6"; get csharp/aspire.hosting/distributedapplicationbuilder returned a url rebased onto the origin and fetched its .md page; get typescript/aspire.hosting.redis/addredis worked; an unknown list scope printed ❌ No API entries found, exit 0; an unknown get id printed a friendly "not found" message, exit 1 (by design); a search with no match printed ❌ No API results found, exit 0.
Objective: An index persisted under the old schema is rebuilt, not trusted. Coverage Type: Boundary Status: ✅ Passed (with a simulated old fingerprint)
Steps:
Used the pinned production sitemap-0.xml flow from Scenario 4c.
Recomputed the fingerprint offline (XxHash3-64 over int32-LE schema version + cached content). The v2 value f609d7a369eb8bc2 matched the CLI's persisted fingerprint byte for byte, which validates the method.
Overwrote the fingerprint file with the v1 value (9f850bb428e52a43).
Observations:
S1 (2.1 s): HTTP 304 for the sitemap, then "Indexed 24205 API reference items in 00.932 seconds" (a re-index, not a cache load). The fingerprint was rewritten to f609d7a369eb8bc2.
S2 (1.1 s): "Loaded 24205 API reference items from cache".
Caveat: the stale fingerprint was simulated; it was not written by a real older CLI.
Summary
Scenario
Status
Notes
1. Staging two-child index
✅ Passed
24,205 items = union of both children; no request to aspire.dev
2. Cache and ETag
✅ Passed
Per-child revalidation; staging Front Door ignores If-None-Match (endpoint limitation)
3. Production default
✅ Passed
One-child production index; 304s on the warm run
4. Plain <urlset> roots
✅ Passed
4b documents the rollout caveat (Finding 3)
5A–5B, 5D. Resilience, cold failures, CLI surface
✅ Passed
304 fallback, dedupe, order independence, self-heal, safe empty failure
5C. Malformed or hostile child (F9, B1, R2, D2)
❌ exit 1, new code path
Crash instead of cached-index fallback (Finding 1); no XXE
5C. Malformed or hostile root (N1, D1, P1)
⚠️ exit 1, pre-existing
Unchanged by the PR
6. Schema bump
✅ Passed
Simulated v1 fingerprint triggers a re-index, then loads from cache
Findings
Finding 1 (⚠️ low to medium, non-blocking): a malformed child sitemap returned with HTTP 200 crashes aspire docs api instead of falling back to the cached index
Where:src/Aspire.Cli/Documentation/ApiDocs/ApiDocsFetcher.cs, return MergeSitemaps(childContents!) (L87). MergeSitemaps and its XML reader have no try/catch, so an XmlException escapes FetchSitemapAsync. That bypasses the cached-index fallback that every null-return path uses (404, network error, missing child), and the command ends with ❌ An unexpected error occurred: … and exit 1. The comment at L76–77 states the intent: treat a failed child as a failed refresh and let the caller fall back to its cached index.
Observed (F9):XmlException: Unexpected end of file while parsing Name has occurred. Line 1, position 255. at ApiDocsFetcher.MergeSitemaps, called from ApiDocsIndexService.EnsureIndexedAsync and BaseCommand.HandleCommandAsync. stdout was empty even though a good cached index existed.
Not a regression by itself: a malformed 200 at the root already crashed before this PR in ApiSitemapParser.Parse (N1, D1, P1), and the PR deliberately preserves that. What is new is one more unguarded parse site per child, which defeats the fallback the PR otherwise implements.
Sticky: the malformed body and its ETag are cached, so later 304s replay the crash (P1). It heals only when the origin serves a corrected body with a new ETag (F10) or the cache is cleared.
Likely triggers: a proxy, captive portal or CDN error page served with HTTP 200 (D2 shows a DOCTYPE variant also throws), or a partially written file during a deploy. A literal truncated body is less likely because HttpClient normally throws on it, and that case is handled gracefully.
Security: DTD/XXE handling is correct. D2 produced 0 requests to the entity URL.
Suggested fix: wrap MergeSitemaps(childContents!) in try/catch (XmlException), log a warning (for example "Failed to parse API child sitemaps from {SitemapUrl}; the API catalog is incomplete") and return null so the existing cached-index fallback applies. Add unit tests for a malformed child (HTTP 200 + invalid XML) with and without a cached index. Optionally, avoid caching bodies that fail to parse in the shared CachedHttpDocumentFetcher.
Minimal repro:
# site\ holds three static files: sitemap-index.xml (lists https://aspire.dev/sitemap-0.xml and# https://aspire.dev/sitemap-api.xml), sitemap-0.xml (2 non-API urls), sitemap-api.xml (3 API urls)
python -m http.server 18766--bind 127.0.0.1--directory site
# aspire.config.json: {"docs":{"api":{"sitemapUrl":"http://127.0.0.1:18766/sitemap-index.xml"}}}
aspire docs api list csharp --non-interactive --nologo # R1: exit 0, "Indexed 3 API reference items"# truncate site\sitemap-api.xml (the server still answers 200), then rerun:
aspire docs api list csharp --non-interactive --nologo # R2: exit 1, XmlException from ApiDocsFetcher.MergeSitemaps
Finding 2 (ℹ️ environment): the staging Front Door endpoint ignores If-None-Match
Every invocation re-downloads about 18 MB (including the new 3.6 MB sitemap-api.xml) and takes 2.3–2.4 s. Production returns 304 correctly (Scenario 3). When the split structure ships, production should keep ETag revalidation working for sitemap-api.xml.
Finding 3 (ℹ️ rollout): already-shipped CLIs will see an empty API catalog once the site switches
Case 4b shows that a CLI whose default is sitemap-0.xml gets "Indexed 0" and ❌ No API entries found (exit 0) once the API routes move to sitemap-api.xml. Today's production works with the new default (Scenario 3). Coordinate the site cut-over with a CLI release, or keep API routes in sitemap-0.xml during a transition.
Finding 4 (ℹ️ test gap): the PR's unit tests do not cover two behaviors verified above
ApiDocsFetcherTests covers the merge, rebase, per-child ETag, order independence, failed child with and without cache, and empty index. It has no test for a malformed child (Finding 1) and none for the IndexSchemaVersion bump (Scenario 6).
Overall Result
✅ PR VERIFIED functionally: the staging two-child index, today's production, plain <urlset> roots, caching and the schema bump all behave as intended. One non-blocking resilience finding (⚠️ Finding 1): a small, local fix that makes the new per-child parse follow the cached-index fallback the PR already implements.
Recommendations
Fix Finding 1 (try/catch around MergeSitemaps) and add unit tests for it and for the schema bump.
Plan the aspire.dev cut-over against the CLI release (Finding 3) and keep production ETag revalidation working for sitemap-api.xml (Finding 2).
Re-run the failed macOS CI job (see Notes). The PR's own scenarios do not point at it.
Notes
Unit tests were not run locally (CI covers them). PR CI at the head commit: 180 SUCCESS, 16 SKIPPED, 3 FAILURE. The only failing job is Tests / No-package tests (regular, Aspire.Cli.Tests) / Cli (macos-latest), which cascades into "Final Test Results" and "Final Results". It hit the 10-minute hang-dump timeout (exit 137) after 1,880 passing tests. The three tests still running were TelemetryShutdownTests.Shutdown_FailedDrainLogsWarningAndRestoresFailureBudget, ProcessGuestLauncherTests.LaunchAsync_WithGracefulServices_ProcessIgnoresSignal_ExpireEscalatesToTreeKill and RunCommandTests.RunCommand_DetachedChild_WhenLauncherDiesBeforeReadiness_CancelsRun, none ApiDocs related. It looks unrelated but is not proven flaky.
The branch is 2 commits behind upstream/main (unrelated: an AKS AcrPull change and a MessagePack pin). It was tested as published, not rebased.
Consistency check: the default URL matches in ApiDocsSourceConfiguration, AspireConfigFile and the JSON schema (valid JSON). sitemap-0 remains only in comments, docs and tests that pin it explicitly.
Evidence under <testDir>: per-run <Label>.stdout.txt, <Label>.stderr.txt, runs.jsonl and aspire.config.json in scenario-1-staging/, scenario-3-prod-default/, scenario-4-urlset/, scenario-5-unhappy/ and repro-malformed-child/, plus server-requests.jsonl (the fake origin's request log with per-run markers).
A child sitemap can answer HTTP 200 with a truncated download or an HTML
error page. Merging it threw an XmlException that escaped the fetcher, so
`aspire docs api` failed even when a good cached index was available.
Treat a child that is not well-formed XML like a missing child: log a
warning that names the child and return null so the caller keeps using its
cached index. The cached child body is replaced once its ETag changes, so
the catalog recovers when the child is fixed.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 291c0f5c-d93c-43fa-838d-705c8ead807c
An HTTP 200 child response such as <html><body>Bad gateway</body></html> is well-formed XML, so this loop accepts it as an empty sitemap. The merged catalog then succeeds and can overwrite the complete cached index with partial or empty results. Require a urlset root for each child and return null otherwise to preserve the cached-index fallback. Add a regression test for well-formed HTML without a DOCTYPE; the existing DOCTYPE case only exercises XML rejection.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
aspire docs api list|search|getbuilt its catalog from a single hardcodedhttps://aspire.dev/sitemap-0.xml. microsoft/aspire.dev#1874 adds an opt-in hybrid build that renders API pages on demand, which moves the API URLs out ofsitemap-0.xmland into a newsitemap-api.xmllinked fromsitemap-index.xml. Without this change,listandsearchreturn almost nothing against such a site.This PR makes the CLI ingest the catalog from the sitemap index:
DefaultSitemapUrlis nowhttps://aspire.dev/sitemap-index.xml;docs:api:sitemapUrlstill overrides it.sitemapindexfetches every child<sitemap><loc>;urlsetbehaves exactly as before, so existing overrides and deployments that list every API route in one sitemap keep working.sitemap-0.xmlstays a 304 when only the API sitemap changes.IndexSchemaVersionis bumped to 2 so indexes persisted by older CLIs are rebuilt.getis unchanged (<page-url>.md).docs/specs/api-docs-commands.md, theAspireConfigFiledoc example and the generated VS Code config schema description are updated.Open question carried over from the issue: released CLIs only read
sitemap-0.xml, so the aspire.dev side must decide whether to keep API URLs there for a transition period before a hybrid build serves production. This change de-duplicates, so it works either way.Not covered: this has not yet been run against a real hybrid-built site.
Fixes #20967
Checklist
<remarks />and<code />elements on your triple slash comments?Validation
.\.dotnet\dotnet.exe test --project tests\Aspire.Cli.Tests\Aspire.Cli.Tests.csproj --no-launch-profile -- --filter-class "*.ApiDocsFetcherTests" --filter-class "*.ApiDocsIndexServiceTests" --filter-class "*.ApiDocsSourceConfigurationTests" --filter-class "*.ApiDocsCacheTests" --filter-class "*.ConfigCommandTests" --filter-class "*.AspireConfigFileTests" --filter-not-trait "quarantined=true"— passed, 163 tests (new cases coversitemapindexmerge,urlsetback-compat, child rebasing, per-child ETag/304, de-duplication and order-independence, fingerprint change when a child changes, and child failure with and without a cache).