Repository navigation
Closed
Labels
Description
Problem
The local AI chat handles questions about the user's knowledge badly. Four observed failures on a fresh DB with nodespace-docs imported (all traced in the daemon log):
- "could you find the skill on how to find nodes in nodespace?" Skill routing (
find_skills "find nodes in nodespace") returned Organization 0.895, Node Creation 0.868, Node Deletion 0.842. Research & Search wasn't in the top 3. The model then ransearch_semanticwith the default Knowledge scope, which by design excludes system types (skill,tool,agent-guidance,playaren't inKNOWLEDGE_CORE_TYPES), got nothing, and answered "I couldn't find a specific skill…". - "how our front end persistence layer works exactly?" Routing: Creating Implementation Tasks / Node Creation / Schema Creation. One
search_semantic, then it answered from the top hit's snippet without opening it. That hit was a stale ADR (decisions/002-dual-persistence-architecture.md, still marked Accepted, describing a dead design). The current doc,architecture/frontend-state-and-persistence.md, wasn't used. - "how is the debounce logic applied?" Routing: Node Deletion / Node Merge / Creating Implementation Tasks. Zero tool calls: "I don't see … in this conversation history. Could you provide more context?" The clarification re-prompt didn't fire.
- "what does our shared data layer on the front end" First reply was again zero tools plus "provide more context". The clarification contract re-prompted, leading to one
search_nodeswhose first result was the ai-chat node itself. It then listed two docs and asked "Would you like me to look up details?" instead of reading them.
Root causes
- Skill description vocabulary. The skills that win routing literally say "nodes" and list the action verbs ("Create new nodes, records, entries…", "Delete, remove, erase…"). Research & Search says "Search and explore the knowledge graph to find relevant information, discover connections, and answer questions about stored knowledge." It loses on both "find X" and "how does X work" questions. The seeded template is in
packages/agent/src/skill_pipeline.rs. - No search-before-answering rule. A question about the user's knowledge can be answered in prose from the transcript, and the clarification re-prompt only catches some of these turns.
- No read-before-answering rule. The agent answers from a single snippet and offers to read instead of reading.
- Search returns the active conversation (ai-chat nodes shouldn't be search results; ADR-029 already stops embedding them).
- Meta-questions about skills hit the Knowledge scope's system-type exclusion.
Acceptance criteria
- Research & Search's description carries retrieval/question trigger vocabulary (find, look up, locate, list, search for nodes/records of any type; how/what/why/explain questions about stored knowledge), mirroring how Node Creation/Deletion are phrased
-
eval:decisionsshows Research & Search routed for both "find …" and question-style prompts ("how does X work", "what is Y", "explain Z"), without regressing other skills' routing (don't use the stale agent-matrix eval) - Guidance/contract: a question about the user's knowledge always searches before answering or asking for context; the clarification re-prompt covers the zero-tool "provide more context" case
- For explanatory questions the agent reads the top results' content (e.g.
include_content) and synthesizes across them rather than answering from one snippet, and doesn't offer to read when reading is the task -
search_nodes/search_semanticexclude the current ai-chat node (and ai-chat nodes generally) - Questions about the agent's own skills ("what skills do you have", "find the skill for X") are answered from the routed/available skill set rather than by a Knowledge-scope search
- Prompt goldens regenerated (
UPDATE_GOLDEN=1,prompt_assembly_snapshot) and the diff reviewed
Activity
malibio commentedon Oct 1, 2026
Frontend baseline: 298 test files passed, 5510 tests passed
malibio commentedon Oct 1, 2026
Note on the sixth acceptance criterion (PR #3480). 'What skills do you have' is answered from the skill list with no search. 'Find the skill for X' is different: it reads as a find request, Stage 1 routes it as a lookup, and a lookup is always searched. No wording of the routing tool kept it out of a lookup without also routing 'which skills do you have?' into one (measured on the locked model). So that turn is searched and then answered from the skill list, which is in the prompt either way and which now states that a search does not return skills. eval:decisions scores the reply (outcome-find-own-skill): it must name the skill.
Route questions about stored knowledge as lookups, and search before …
Route questions about stored knowledge as lookups, and search before …