Repository navigation
fix(chat): send no citation rules when retrieval finds nothing - #2028
Conversation
With no hits, build_context returned the tier instruction, which asks for [n] labels there are no sources for. Send a short no-sources instruction instead, for every tier: say the knowledge base does not cover the question, then answer from general knowledge without guessing about the user's own data. Fixes MODSetter#1947
|
@MONISMALIK1 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
What
When retrieval returns no hits,
build_context()now returns a short no-sources instruction instead of the tier prompt. It carries no citation rules and asks the model to say the knowledge base does not cover the question, then answer from general knowledge if it can, never guessing about the user's own documents, products or people. It keeps the language and code-fence rules.surfsense_local/backend/modules/chat/prompt.py:_NO_SOURCESconstant, returned on the no-hits branch for every tier. The tier prompt is loaded only when there are hits.tests/unit/chat/test_prompt.py:test_no_hits_leaves_the_instruction_aloneasserted the old behaviour, so it is replaced bytest_no_hits_asks_for_no_citations, parametrised over all three tiers: no[n]or[1]-style label, no "bracket label" rule, and the "knowledge base does not cover" line present.docs/architecture/chat.md: Known gaps line deleted; the "No hits, no context block" Grounding shape bullet updated.Why
The no-hits system message was the tier instruction alone, which still asks for
[n]labels with no sources to label. In the chat eval Qwen3 1.7B invented a[1]in all three such answers, and one never said the sources did not cover the question.I took the shorter-diff route from the issue rather than a
casesubfolder: one place to change, no new prompt files, so the PyInstallerprompts/*.mdglobs stay as they are, andcompact.mdis untouched. The tier contract holds because the same instruction goes to every tier;test_prompts_ship.pyis unaffected.Tradeoff: on no hits,
capableandfrontierno longer get their answer-shaping guidance (work order, "lead with the answer"). Most of it is about using sources, which do not exist in this branch. Happy to switch to a per-tierno_sources/case if you would rather keep that.Fixes #1947
How to test
The new test fails for all three tiers on
devand passes with the change. I have not rerun the chat eval (no local model here), so the claim about Qwen3 1.7B's no-hits answers is not re-measured.High-level PR Summary
This PR fixes a citation hallucination issue in the chat module where small language models would invent citations when no sources were retrieved. When retrieval returns no hits, the system now sends a simplified instruction that explicitly tells the model to say the knowledge base doesn't cover the question and answer from general knowledge if possible, without any citation rules or bracket label formatting requirements. The fix replaces the old behavior where the tier-specific prompt (which included citation instructions) was sent even when there were no sources to cite.
⏱️ Estimated Review Time: 5-15 minutes
💡 Review Order Suggestion
surfsense_local/backend/tests/unit/chat/test_prompt.pysurfsense_local/backend/modules/chat/prompt.pydocs/architecture/chat.md