Skip to content

fix(chat): send no citation rules when retrieval finds nothing - #2028

Merged
MODSetter merged 1 commit into
MODSetter:devfrom
MONISMALIK1:fix/no-sources-citation-prompt
Sep 29, 2026
Merged

MODSetter merged 1 commit into
MODSetter:devfrom
MONISMALIK1:fix/no-sources-citation-prompt

Conversation

@MONISMALIK1

@MONISMALIK1 MONISMALIK1 commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

What

When retrieval returns no hits, build_context() now returns a short no-sources instruction instead of the tier prompt. It carries no citation rules and asks the model to say the knowledge base does not cover the question, then answer from general knowledge if it can, never guessing about the user's own documents, products or people. It keeps the language and code-fence rules.

  • surfsense_local/backend/modules/chat/prompt.py: _NO_SOURCES constant, returned on the no-hits branch for every tier. The tier prompt is loaded only when there are hits.
  • tests/unit/chat/test_prompt.py: test_no_hits_leaves_the_instruction_alone asserted the old behaviour, so it is replaced by test_no_hits_asks_for_no_citations, parametrised over all three tiers: no [n] or [1]-style label, no "bracket label" rule, and the "knowledge base does not cover" line present.
  • docs/architecture/chat.md: Known gaps line deleted; the "No hits, no context block" Grounding shape bullet updated.

Why

The no-hits system message was the tier instruction alone, which still asks for [n] labels with no sources to label. In the chat eval Qwen3 1.7B invented a [1] in all three such answers, and one never said the sources did not cover the question.

I took the shorter-diff route from the issue rather than a case subfolder: one place to change, no new prompt files, so the PyInstaller prompts/*.md globs stay as they are, and compact.md is untouched. The tier contract holds because the same instruction goes to every tier; test_prompts_ship.py is unaffected.

Tradeoff: on no hits, capable and frontier no longer get their answer-shaping guidance (work order, "lead with the answer"). Most of it is about using sources, which do not exist in this branch. Happy to switch to a per-tier no_sources/ case if you would rather keep that.

Fixes #1947

How to test

cd surfsense_local/backend
uv run pytest -m unit      # 3051 passed, 8 skipped (llama.cpp not staged)
uv run ruff check .
cd ../.. && python scripts/check_docs.py

The new test fails for all three tiers on dev and passes with the change. I have not rerun the chat eval (no local model here), so the claim about Qwen3 1.7B's no-hits answers is not re-measured.

High-level PR Summary

This PR fixes a citation hallucination issue in the chat module where small language models would invent citations when no sources were retrieved. When retrieval returns no hits, the system now sends a simplified instruction that explicitly tells the model to say the knowledge base doesn't cover the question and answer from general knowledge if possible, without any citation rules or bracket label formatting requirements. The fix replaces the old behavior where the tier-specific prompt (which included citation instructions) was sent even when there were no sources to cite.

⏱️ Estimated Review Time: 5-15 minutes

💡 Review Order Suggestion
Order File Path
1 surfsense_local/backend/tests/unit/chat/test_prompt.py
2 surfsense_local/backend/modules/chat/prompt.py
3 docs/architecture/chat.md

Need help? Join our Discord

With no hits, build_context returned the tier instruction, which asks for [n]
labels there are no sources for. Send a short no-sources instruction instead,
for every tier: say the knowledge base does not cover the question, then answer
from general knowledge without guessing about the user's own data.

Fixes MODSetter#1947
@vercel

vercel Bot commented Sep 29, 2026

Copy link
Copy Markdown

@MONISMALIK1 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel.

A member of the Team first needs to authorize it.

@coderabbitai

coderabbitai Bot commented Sep 29, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 1e749a65-3aaf-4c64-af73-345e231eb904

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@MODSetter
MODSetter merged commit c59f91e into MODSetter:dev Sep 29, 2026
1 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants