Skip to content

With no retrieved passages the prompt still asks the model to cite #1947

Closed

Description

@MODSetter

The gap

docs/architecture/chat.md lists this under Known gaps:

With no hits, the system message is the instructions alone, which still ask for [n] labels. In the chat eval, Qwen3 1.7B wrote a [1] in all three such answers, which chat drops, and one of them never said the sources did not cover the question.

Why it matters

When retrieval returns nothing — an empty workspace, a question about something the user never saved, or every source unticked in the sources panel — the model is still handed instructions that tell it to put a [n] label after each claim and cite only what the sources back. There are no sources. The small local models this app is built around do as they are told, and the eval caught Qwen3 1.7B inventing a [1] in every one of those answers.

resolve_citations() drops the invented number, so nothing broken reaches the screen. The damage is upstream of that: a model spending its instruction budget obeying a rule that cannot apply is a model not following the rule that matters, which is to say plainly that the knowledge base did not cover the question. One of those three answers never said it.

Where

What done looks like

  • With no hits, the model is not asked to label claims it cannot source, and is asked to say the knowledge base did not cover the question before answering from its own knowledge.
  • How you get there is yours to choose, and the two options differ in cost. A case subfolder means three more prompt files and full control per tier; appending or swapping a paragraph in build_context() means one place to change and no new files. The house style favours the shorter diff, but compact.md was tuned against measured eval behaviour, so do not quietly rewrite prose the eval is holding steady.
  • The prompt-tier contract is unchanged: whatever you add exists for all three tiers, because tests/unit/llm/prompting/test_prompts_ship.py asserts a case ships every tier and a missing file fails the job on the user's machine.
  • Delete the Known gaps line from docs/architecture/chat.md, and update the Grounding shape bullet that says "No hits, no context block: the system message is the instruction alone".

What not to change

If you add prompt files in a subfolder, the PyInstaller specs will not pick them up. Both bundling/api.spec and bundling/worker.spec collect chat's prompts with includes=["prompts/*.md"], which is one level deep and will not match prompts/<case>/<tier>.md. Nothing imports a prompt, so nothing fails at build time — it fails on a packaged install, at the moment a user asks a question with no hits. Widen both globs if you take that route.

How to verify

surfsense_local/backend/tests/unit/chat/test_prompt.py covers build_context() and is where the failing test goes: with no hits, the system message does not ask for [n]. tests/unit/llm/prompting/test_prompts_ship.py covers the tier completeness half.

cd surfsense_local/backend
uv run pytest -m unit
uv run ruff check .
# from the repo root, because the PR edits a doc
cd ../.. && python scripts/check_docs.py

Before you start

Read CONTRIBUTING.md, the chat doc and the chat eval proposal, which is where the measurement in this gap line came from and where a claim about prompt behaviour gets settled. Open your PR against dev, not main. Comment here to claim this.

Activity

  1. added
    bugSomething isn't working
    help wantedExtra attention is needed
    desktopSurfSense desktop app (surfsense_local)
    on Sep 28, 2026
  2. MONISMALIK1 commented on Sep 29, 2026

    @MONISMALIK1
    Contributor

    I'd like to take this one. Going with the shorter-diff route in build_context(): a no-sources instruction for every tier, no new prompt files, compact.md untouched. PR incoming.

  3. added a commit that references this issue on Sep 29, 2026
    c69f86c
  4. MODSetter commented on Sep 29, 2026

    @MODSetter
    OwnerAuthor

    Fixed by #2028, fix(chat): send no citation rules when retrieval finds nothing, by @MONISMALIK1. Merged into dev.

    Closing by hand rather than by keyword: Fixes #1947 only auto-closes on a merge to the default branch, and this repo merges through dev (CONTRIBUTING), so this would have stayed open and claimable while the work was already done. It reaches main with the next release.

    Nothing left to pick up here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdesktopSurfSense desktop app (surfsense_local)help wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions