Repository navigation
test(evals): site cases check list_public_tokens_tool, not export_map_html_tool - #311
Merged
Merged
Conversation
…_html_tool The site-building cases still expected export_map_html_tool, which #279 reverted, so they could never pass on main. They now expect the agent to get a token from list_public_tokens_tool (#304) and embed it. - Stub list_public_tokens_tool with a realistic-looking pk. token (one that reads as fake makes models leave a placeholder) and add a token_embedded metric: the written page uses that token. - New site-chatgpt-app case: an Apps SDK widget with a Mapbox map. - site-walksheds no longer requires isochrone_tool; pages may call the Isochrone API themselves with the embedded token. - EVAL_CASES=<ids> runs only those cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 task done
…c-tokens # Conflicts: # CHANGELOG.md
zmofei
approved these changes
Nov 5, 2026
zmofei
left a comment
zmofei
left a comment
Member
There was a problem hiding this comment.
Eval-only change and the harness hashes match. One small thing about the new metric.
| ref_ok: refs && !refs.ok ? 0 : 1, | ||
| no_token_ask: token && !token.ok ? 0 : 1 | ||
| no_token_ask: token && !token.ok ? 0 : 1, | ||
| token_embedded: embedded && !embedded.ok ? 0 : 1 |
Member
There was a problem hiding this comment.
token_embedded is a new grade, but it is not in metrics in .claude/hillclimb/tool-selection/_state.json or tool-selection-openai/_state.json, so the report does not list it next to ref_ok and no_token_ask. Please add it to both files.
mattpodwysocki
added a commit
that referenced
this pull request
Nov 5, 2026
* fix: review follow-ups for #278, #309 and #311 - render_map_tool: clear the base map config on teardown so the Satellite button can't bring back an earlier render's config; the description says traffic and satellite carry over and to re-pass showTraffic with the content. - resource_reader_tool: return an image blob once (as an image under 700 KB, else a short note) instead of base64 text plus a structured copy; point mapbox://selffetch/ refs to render_map_tool. - Eval: list token_embedded in both hillclimb metrics. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: add PR number to CHANGELOG Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: review follow-ups for #312 - Say when the 1x fallback replaced a too-large @2x image. - Test that the large-image ref's decoded data= has no access token. - docs/mcp-ui.md describes the static map viewer and public-token link. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The tool-selection eval's site-building cases still expected
export_map_html_tool, which #279 reverted, so on main they could never pass. This points them at what we shipped instead: the agent gets a public token fromlist_public_tokens_tool(#304) and embeds it in the page it writes.site-walkshedsandsite-routenow expectlist_public_tokens_toolandwrite_file. Newsite-chatgpt-appcase: a ChatGPT Apps SDK widget with a Mapbox GL JS map, which runs outside ourrender_map_toolUI and so needs its own token.token_embeddedmetric: the page the agent writes uses the tokenlist_public_tokens_toolreturned, not a placeholder or a made-up one.pk.token that looks real but belongs to no account. My first stub saidEvalStubNotARealToken, and OpenAI's model correctly refused to embed it and leftYOUR_MAPBOX_PUBLIC_TOKENinstead, which looked like a tool problem but wasn't.site-walkshedsno longer requiresisochrone_tool. Claude's pages call the Isochrone API themselves with the embedded token, which is a fine way to build a site.EVAL_CASES=<ids>runs just those cases, so a focused check like this costs about $1.50 instead of about $8 for the full suite._state.jsonare updated for the changedharness.tsandcases.json.No server code changes.
Results
Main's
list_public_tokens_tool, the three site cases:list_public_tokens_toolchat-latest(eager, 5 reps, 15 rows)I also tried a reworded tool guidance ("put the recommended token straight into the page; never leave a placeholder"). It scored the same, so I left the tool alone.
Testing
EVAL_CASES=site-walksheds,site-route,site-chatgpt-app npm run eval:tools -- --variant v2 --reps 3EVAL_MODES=eager EVAL_CASES=... npm run eval:tools:openai -- --variant v2 --reps 5 --model chat-latestnpx eslint evals/tool-selection/harness.tsChecklist
🤖 Generated with Claude Code