Repository navigation
feat(local): add the chat eval, judge its runs, and fix the compact chat prompt - #1912
Conversation
- fixed model connection invalidation blocking whole app
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # surfsense_local/frontend/src/app/app-bootstrap.test.tsx # surfsense_local/frontend/src/app/app-bootstrap.tsx
Rules say a reply failed; they cannot say why. A judge on OpenRouter reads the messages the model received, its reasoning and its answer beside the answer key, returns a verdict held to a schema with a likely cause and a fix per failure, and writes a report of the patterns across a run. Verdicts record the judge and a fingerprint of the rubric, so two runs are only compared under the same both. The rubric and the report prompt are markdown beside the code, so a Studio format or an artifact edit adds a rubric rather than a second judge. Co-authored-by: Cursor <cursoragent@cursor.com>
Measured on Qwen3 1.7B with the chat eval: the compact prompt left it citing passages that did not hold the answer in 6 of 33 replies, and never giving an answer it knew when the sources fell short. It now answers from its own knowledge only where that is general knowledge, never guesses a detail of the user's own documents, labels neither sentence, and has an example of each case rather than one the model could copy. Judged against the same baseline: grounded 24/33 to 30/33, citations 21/33 to 27/33, admits gaps 3/9 to 7/9. Co-authored-by: Cursor <cursoragent@cursor.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
What
Adds the chat eval from
docs/proposals/chat-eval.md, gives it a frontier judge, and makes the first prompt fix the eval paid for.scripts/run_chat_eval.py,scripts/chat_eval/). 11 cases covering a single fact, two passages, a superseded policy, two unanswerable questions, no passages at all, a follow-up with history, Hindi, Chinese, a prompt injection, and one right passage among five. Every request is built with chat's ownbuild_context,build_messagesandclassify, so the eval cannot drift from the app: same prompt tier, same 1,024-token answer cap, same sampling from the manifest. Local runs ask the dev app's own llama-server through--base-url; Featherless needs only a key.score.py): cut off by the cap, empty, invented citations read through chat's ownresolve_citations, whether every supporting passage was cited and whether any other was, the expected facts, and whether the answer is in the question's script.chat_eval/judge/). Rules say a reply failed; they cannot say why. The judge is shown the messages the model received, its reasoning and its answer beside the answer key, and returns a verdict held to a schema with a likely cause and a fix for each failure, then a report of the patterns across the run. Each verdict records the judge and a fingerprint of the rubric, so two runs are compared only under the same both. The rubric and the report prompt are markdown beside the code, so a Studio format or an artifact edit later adds a rubric rather than a second judge.Why
The model catalog proposal left the agent loop to "a later chat rework with its own proposal", and that rework needs a way to tell whether a change helped. Before this, a prompt or pipeline change on a small model could only be judged by reading replies.
The judge is what makes it useful beyond pass and fail: the first report named the causes behind four failure patterns, and acting on one of them produced the prompt fix in this PR.
Fixes #
How to test
Install a curated chat model in the dev app, then from
surfsense_local/backend, with the app running (its[llamacpp]port is in the dev console):The judge needs
OPENROUTER_API_KEYand costs roughly $0.02 a run onopenai/gpt-6-luna, $0.70 onanthropic/claude-opus-5.5:Measured on Qwen3 1.7B, judged by
openai/gpt-6-lunaunder one rubric, before and after the prompt change: grounded 24/33 → 30/33, citations right 21/33 → 27/33, admits gaps 3/9 → 7/9, expected facts 21/24 → 23/24.Worth knowing: the eval scores the answer, not retrieval — cases hand the model their passages, so
retrieve()is never exercised. The cases are written for the eval, so nothing a user wrote leaves the machine.capable.mdandfrontier.mdkeep the old wording until a larger model has been measured, and the no-hits case is still a Known gap inchat.md: with nothing retrieved the prompt still asks for[n]labels.The first commit here also carried a fix to the startup path for an unusable chat model;
devfixed the same bug more completely in #1911, and the merge keepsdev's version.High-level PR Summary
This PR introduces a comprehensive chat evaluation framework for SurfSense's local models, including 11 test cases covering various scenarios (single facts, multiple passages, policy updates, unanswerable questions, multilingual support, and prompt injections). It implements rule-based scoring to validate answers against expected criteria (grounding, citations, facts, language), a frontier model judge on OpenRouter that provides actionable feedback on failures with likely causes and suggested fixes, and a prompt improvement for the compact tier that was validated on Qwen3 1.7B, improving grounding from 24/33 to 30/33 and citation accuracy from 21/33 to 27/33. The eval runs against the dev app's own llama-server or Featherless, using chat's own
build_context,build_messages, andclassifyfunctions to prevent drift between evaluation and production.⏱️ Estimated Review Time: 5-15 minutes
💡 Review Order Suggestion
docs/proposals/chat-eval.mdsurfsense_local/backend/scripts/chat_eval/cases.jsonsurfsense_local/backend/scripts/chat_eval/cases.pysurfsense_local/backend/scripts/chat_eval/model.pysurfsense_local/backend/scripts/chat_eval/request.pysurfsense_local/backend/scripts/chat_eval/send.pysurfsense_local/backend/scripts/chat_eval/score.pysurfsense_local/backend/scripts/chat_eval/judge/verdict.mdsurfsense_local/backend/scripts/chat_eval/judge/verdict.pysurfsense_local/backend/scripts/chat_eval/judge/report.mdsurfsense_local/backend/scripts/chat_eval/judge/report.pysurfsense_local/backend/scripts/chat_eval/judge/openrouter.pysurfsense_local/backend/scripts/chat_eval/summary.pysurfsense_local/backend/scripts/run_chat_eval.pysurfsense_local/backend/tests/unit/scripts/test_chat_eval_request.pysurfsense_local/backend/tests/unit/scripts/test_chat_eval_score.pysurfsense_local/backend/tests/unit/scripts/test_chat_eval_verdict.pysurfsense_local/backend/modules/chat/prompts/compact.mddocs/architecture/chat.mdsurfsense_local/backend/scripts/chat_eval/__init__.pysurfsense_local/backend/scripts/chat_eval/judge/__init__.py