Skip to content

Agent mishandles questions about the user's knowledge: Research & Search not routed, no search/read before answering, search returns the chat itself #3265

Closed

Description

@malibio

Problem

The local AI chat handles questions about the user's knowledge badly. Four observed failures on a fresh DB with nodespace-docs imported (all traced in the daemon log):

  1. "could you find the skill on how to find nodes in nodespace?" Skill routing (find_skills "find nodes in nodespace") returned Organization 0.895, Node Creation 0.868, Node Deletion 0.842. Research & Search wasn't in the top 3. The model then ran search_semantic with the default Knowledge scope, which by design excludes system types (skill, tool, agent-guidance, play aren't in KNOWLEDGE_CORE_TYPES), got nothing, and answered "I couldn't find a specific skill…".
  2. "how our front end persistence layer works exactly?" Routing: Creating Implementation Tasks / Node Creation / Schema Creation. One search_semantic, then it answered from the top hit's snippet without opening it. That hit was a stale ADR (decisions/002-dual-persistence-architecture.md, still marked Accepted, describing a dead design). The current doc, architecture/frontend-state-and-persistence.md, wasn't used.
  3. "how is the debounce logic applied?" Routing: Node Deletion / Node Merge / Creating Implementation Tasks. Zero tool calls: "I don't see … in this conversation history. Could you provide more context?" The clarification re-prompt didn't fire.
  4. "what does our shared data layer on the front end" First reply was again zero tools plus "provide more context". The clarification contract re-prompted, leading to one search_nodes whose first result was the ai-chat node itself. It then listed two docs and asked "Would you like me to look up details?" instead of reading them.

Root causes

  • Skill description vocabulary. The skills that win routing literally say "nodes" and list the action verbs ("Create new nodes, records, entries…", "Delete, remove, erase…"). Research & Search says "Search and explore the knowledge graph to find relevant information, discover connections, and answer questions about stored knowledge." It loses on both "find X" and "how does X work" questions. The seeded template is in packages/agent/src/skill_pipeline.rs.
  • No search-before-answering rule. A question about the user's knowledge can be answered in prose from the transcript, and the clarification re-prompt only catches some of these turns.
  • No read-before-answering rule. The agent answers from a single snippet and offers to read instead of reading.
  • Search returns the active conversation (ai-chat nodes shouldn't be search results; ADR-029 already stops embedding them).
  • Meta-questions about skills hit the Knowledge scope's system-type exclusion.

Acceptance criteria

  • Research & Search's description carries retrieval/question trigger vocabulary (find, look up, locate, list, search for nodes/records of any type; how/what/why/explain questions about stored knowledge), mirroring how Node Creation/Deletion are phrased
  • eval:decisions shows Research & Search routed for both "find …" and question-style prompts ("how does X work", "what is Y", "explain Z"), without regressing other skills' routing (don't use the stale agent-matrix eval)
  • Guidance/contract: a question about the user's knowledge always searches before answering or asking for context; the clarification re-prompt covers the zero-tool "provide more context" case
  • For explanatory questions the agent reads the top results' content (e.g. include_content) and synthesizes across them rather than answering from one snippet, and doesn't offer to read when reading is the task
  • search_nodes / search_semantic exclude the current ai-chat node (and ai-chat nodes generally)
  • Questions about the agent's own skills ("what skills do you have", "find the skill for X") are answered from the routed/available skill set rather than by a Knowledge-scope search
  • Prompt goldens regenerated (UPDATE_GOLDEN=1, prompt_assembly_snapshot) and the diff reviewed

Activity

  1. malibio commented on Oct 1, 2026

    @malibio
    CollaboratorAuthor

    Frontend baseline: 298 test files passed, 5510 tests passed

  2. self-assigned this
    on Oct 1, 2026
  3. malibio commented on Oct 1, 2026

    @malibio
    CollaboratorAuthor

    Note on the sixth acceptance criterion (PR #3480). 'What skills do you have' is answered from the skill list with no search. 'Find the skill for X' is different: it reads as a find request, Stage 1 routes it as a lookup, and a lookup is always searched. No wording of the routing tool kept it out of a lookup without also routing 'which skills do you have?' into one (measured on the locked model). So that turn is searched and then answered from the skill list, which is in the prompt either way and which now states that a search does not return skills. eval:decisions scores the reply (outcome-find-own-skill): it must name the skill.

  4. added 2 commits that reference this issue on Oct 2, 2026
    790c37b
    07f8fd8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions