AI-Chat context is neither targeted nor grounded: too much that does not bear on the message, too little on what it is about

Problem

AI-Chat's context is neither well-targeted nor well-grounded. It is told too much that does not bear on the message, and not reliably told what the message is about: what a core type such as task is, or how it connects to other types. Whether the model sees the task schema depends on which skills retrieval happens to pick for the message, and the relationships a lookup needs (for a task: its assignee, project, plan, spec) are described nowhere the model reliably sees. The same question succeeds on one model and fails on another.

Evidence

Same pipeline, two models, a question about tasks assigned to the current user ("status of tasks assigned to Michael" / "How many tasks are assigne to Michael Libio?"). Gemma's full request was captured with daemon payload logging; Mistral's was not (logging was off).

Gemma answered in 2 rounds (~7s) with one call: search_nodes with {"type":"relationship","path":["assignee"],"node_id":<current user's id>}. It inferred assignee from the filters parameter docs; nothing names it as the way from a task to a person.
Mistral Large took 5 rounds (~96s): search_semantic first, then search_nodes filtering type:"property" on custom:assignee, then the same call again until the duplicate-call guard cut the turn off and it asked the user instead of answering.

What the model was given in Gemma's request (36K-character system prompt, 10 tools):

The EXISTING SCHEMAS block appears five times, once under each of four candidate procedures plus the top. Four list only customer and invoice (custom types). Only the block under "Implementing a Task" lists checkbox, task and text, because that skill alone links to them (applies_to). Core types are excluded from every other block by design.
No person schema appears anywhere. A person arrives only as ids (the current-user line and the mentioned-entities block).
The task line shows its fields and ~> blocks; relates_to; duplicates; decisions. A task's assignee is the derived inverse of person.tasks, so assignee is not on the line.
The Research & Search procedure documents only type:"property" filters (status, due_date) and tells the model to call search_semantic first. It has no example of a relationship filter.
The search_nodes description says filters only accepts a type's own fields and to use get_related_nodes for relationships. get_related_nodes is not among the 10 tools (Research & Search does not grant it), and the same tool's filters parameter docs say type:"relationship" works.
About 15K of the 36K characters is the Schema Creation procedure and its rules, which a lookup does not use.

How much of the request bore on the question

Gemma's main request for "How many tasks are assigne to Michael Libio?" was about 77K characters: a 35,985-character system prompt and 40,981 characters of tool schemas. Roughly 6K of the system prompt bore on the question.

System prompt part	Chars	Bears on the question?
Opening, current user, collections, mentioned entities	~800	Yes (the person's id)
Research & Search procedure	~5,000	Partly (it points at `search_semantic` first)
`task` schema line	~430	Yes, the only relevant schema
`customer` / `invoice` schema lines, repeated four times	~3,000	No
Seven active playbooks	~2,100	No (they govern writes)
Node Deletion procedure	~2,000	No
Schema Creation procedure and rules	~16,300	No (45% of the prompt)
Implementing a Task procedure	~4,400	No, for a count question
General tool and response rules	~3,400	Mostly

Tools: search_nodes (5.5K) is the one the question needs. create_schema and update_schema are 18K together and create_node is 3.9K, for a read-only question. A mentioned entity the message does not name is also included.

Two causes visible in the code: the router passes four candidate procedures with "use whichever applies and IGNORE the rest", so low-scoring candidates (0.87 to 0.90 against 1.056 for the best) ride along in full; and the schemas shown follow the retrieved skills' applies_to links, not the message's subject (task and person).

Approach

The work should check thoroughly how best to improve this, with measurements, using a combination of prompt tuning and deterministic tools that hand the model only what bears on the message: the relevant schemas (task and person for a task lookup), skills and tools, and the node content that matters, chosen by code rather than left to retrieval scores. Try both, measure each against the test prompts below on the same models, and combine what works. The local (on-device) model path appears to assemble the same context, where a repeated large prompt costs more; measure it too. Measurement tooling (per-request timing and token counts) is #3704.

Test prompts

The prompts used in manual testing, from the chat transcripts, with what the chat answered (the model varied between runs, and is not recorded for all of them). They form the starting set; add cases where a gap shows.

Prompt	What happened
Tell me how many tasks are assigned Anoop	"Could not find any tasks currently assigned to Anoop Nair"
How many tasks do we have assigned to Anoop?	Upstream 429
How many tasks are assigned to Anoop?	Correct: one task
Let's assign that task to Michael	Said it was assigned; wrote a stray `custom:assignee` property instead (#3705)
How many tasks are assigned to Norbert	"Could not find a relationship type called 'assigned to' for Norbert", listed `reported_tasks` and `tasks`
How about tasks assigned to me (Michael)	Asked the user for Michael's node id
Could you check how many tasks are assigned to Michael Libio?	Provider 400
Could you check the status of tasks assigned to Michael	Mistral Large: 5 rounds, ~96s, guessed `custom:assignee`, duplicate-call guard fired, asked the user
How many tasks are assigne to Michael Libio?	Gemma 31b: correct in 2 rounds, ~7s, one `type:"relationship"` filter on `["assignee"]`

The same question was answered correctly by one model and not by another on the same assembled context, so cases like these should run against at least one reasoning and one non-reasoning remote model and one local model.

Acceptance Criteria

For a message about tasks assigned to a person, the request carries the procedures, schemas and tools that bear on it and not the ones that do not: a read-only lookup does not carry the schema-creation procedure or the schema-writing tools.
The prompt's size for that message is measured before and after and recorded on this issue.
For a message about tasks, the model is given what a task is in this system: its fields and its relationships, including the ones derived from the other side (assignee, project, plan, spec), regardless of which skills retrieval picks.
A lookup that goes through a relationship ("tasks assigned to X", "tasks in project Y") has one documented, tool-consistent way to be asked, and no tool description contradicts another about it.
A test (prompt-assembly or eval case) asserts the above for a task lookup by person, and the same case passes on at least two models, one of them Mistral Large.
The same assertion is made for one other core type with a derived relationship, so the fix is not special-cased to task.
Every test prompt above gets a correct answer on the models measured (at least one reasoning remote, one non-reasoning remote and one local model), or the gap is recorded with its cause.
The approaches tried (prompt changes, deterministic selection of schema, skill, tool and content), and their before/after request size and results on the same prompts, are recorded on this issue.

Measurement 1: baseline on the local model, through the real turn

Corrects an earlier version of this comment, which ran nodespace skill guidance on the raw prompt. That is not what the agent does: Stage 1 first classifies the message and, for a lookup, rewrites the retrieval query to "find, look up, or search stored knowledge for ". So the earlier table (Research & Search absent for 4 of 6 prompts) did not describe the agent and is withdrawn.

Setup: a new eval, bun run eval:grounding (scripts/eval/fixtures/grounding.ts), run against an isolated daemon and database seeded with two people (Anoop Nair, Norbert Weber) and three tasks (two assigned to Anoop through person.tasks, one unassigned), model gemma-4-e4b-q4km, 1 rep. The seed was checked with nodespace relationship get: Anoop has 2 tasks.

Scenario	Prompt	Result	Latency
count-first-name	How many tasks are assigned to Anoop?	FAIL: "I found 0 tasks assigned to Anoop Nair."	20.5s
count-missing-preposition	Tell me how many tasks are assigned Anoop	FAIL: same	19.5s
count-typo-full-name	How many tasks are assigne to Anoop Nair?	FAIL: same	21.5s
count-none	How many tasks are assigned to Norbert	pass (0 is correct, so this proves nothing)	20.5s
status-of-assigned	Could you check the status of tasks assigned to Anoop	FAIL: "couldn't find any tasks"	39.0s

Pass: 1 of 5 (0 of 4 where the answer is not zero).

What the daemon log shows for those turns:

Every turn calls search_nodes once with a property filter, {"property":"assignee","operator":"equals","value":"Anoop Nair's id=<uuid>"} on node_type:"task". assignee is not a property, so it returns count: 0, and the model reports 0. It never tries the relationship filter that works.
The main request is about 10,000 prompt tokens (system prompt 17.9 to 18.6K characters plus 10 to 11 tool schemas). Generating the 77-token tool call took about 17s; the follow-up round took about 0.7s because the prompt is cached. So on the local model about 85% of a turn's time is reading the prompt.
Stage 1 itself is cheap (1,091 prompt tokens, about 2s).
The wording decides the path: "How many tasks are assigned to Anoop?" is routed as a lookup (Research & Search 1.005) with 10 tools; the typo variant is routed as a query (Breaking a Plan into Tasks .898, Research & Search not a candidate) with 11 tools including get_related_nodes. The model makes the same wrong call on both.
Four candidate procedures ride along even for the lookup: Research & Search 1.005, Implementing a Task .896, Node Deletion .850, Organization .841.

Reading: for this model the failure is not that the right skill is missing; it is that nothing it was given says that a task's assignee is reached through a relationship, and the property filter it guesses at returns an empty result that looks like an answer.

Measurement 2: what changed the local model's answers, and what did not

All runs: bun run eval:grounding (5 lookups, isolated seeded database), local gemma-4-e4b-q4km, one chat per scenario. "pass^3" is a scenario passing in all 3 reps. Only the local model is measured so far (no remote endpoint was called, to avoid cost); the criteria that name a reasoning and a non-reasoning remote model are still open.

Variant	Result	Notes
Baseline	1 of 5 (1 rep; the one pass is the zero-task case)	Property filter `assignee = "<name>'s id=<uuid>"`, returns `count: 0`, the model reports 0. ~10,200 prompt tokens, ~20s
`query_ops` refuses a relationship named as a property, with the filter to use	3 of 5 (1 rep)	The error reaches the model; in 2 of 4 turns it answers "I cannot filter by assignee" instead of retrying
`search_nodes` runs that call as the relationship filter it meant (the id is in the value), says so on the result	15 of 15 (5/5 pass^3)	Same ~10,200 prompt tokens, 20 to 22s
+ lookup turns drop skills and tools that write (Stage 1 `lookup`)	3 of 5 (3 reps)	Prompt down to ~6,300 tokens, 13 to 17s, but the model now invents `assigned_to` as a property with the id as value, which matches nothing
+ the tool reads an invented name as the relationship that reaches that kind of node (`assigned_to` is read as `assignee`, `creator` does not begin like it)	15 of 15	Lookup turns: ~6,200 to 6,500 prompt tokens (was ~10,000), 15 to 17s (was ~22s), 5 tools (was 11)
Description names the relationship filter, with a JSON example	1 of 5	The model builds the filter but leaves out `node_id` or `node_type`
Same, compact wording, no example	4 of 5 (`status-of-assigned` 0/3)	The model starts reasoning about a relationship named "assigned_to"
Description unchanged except its false sentence, and `get_related_nodes` added to Research & Search	15 of 15	Removes the contradiction (the description sent the model to a tool the skill did not offer)

Findings:

For this model, wording that tells it about relationship filters makes it worse; a tool that accepts the call the model already writes, and reports what it ran, makes it right. A fix that depends on the model emitting an exact shape is not reliable here.
The prompt narrowing is a time win on the local model (about 30% less time, 40% fewer prompt tokens on a lookup), because nearly all of a turn is reading the prompt. It also makes a wrong guess more likely to be an empty result, so it needed the resolver above to stay correct.
Not narrowed yet: a message Stage 1 classifies as a query rather than a lookup (the typo variant "assigne") still carries all four candidate procedures and 11 tools, ~9,950 prompt tokens.
The eval's uniform-result guard refuses a run where every scenario passes unless it is a partial run, so these were run with --only over all five scenarios.

Tell the model how other nodes reach a mentioned entity (refs #3706)

Address review: keep the search rewrite from changing what was asked …

Address re-review: guard the id-repair pass the same way (refs #3706)

Answer "tasks assigned to X" correctly on the local model; narrow loo…

Tell the model how other nodes reach a mentioned entity (refs #3706)

Address review: keep the search rewrite from changing what was asked …

Address re-review: guard the id-repair pass the same way (refs #3706)

Answer task lookups by person correctly on the local model; narrow lo…

PR #3709 (local model, gemma-4-e4b-q4km, 3 reps): project lookups now ground (the task via project link, core test + eval scenario) and answer 3/3; assigning writes the relationship 3/3 (update_node refuses custom:assignee and names the create_relationship call). Found and fixed: the fabricated-id guard replaced a correct project answer because the project id came from the prompt, not a tool result. Read-only questions with no question mark ("Tell me how many…", "How many tasks are assigned to X") were routed as queries: 18.5K prompt tokens and ~42s, against 6.9K and ~4s as a lookup; they now route as lookups unless a word names a change. Gap: status-of-assigned finds the right tasks but the 4B model writes 'started today' for in_progress (0/3); it passed on the larger query-route prompt. Still open: reasoning and non-reasoning remote-model measurements (the harness only drives a loaded local model; needs a provider key in the isolated database).

Address review: fail closed on the lookup override, share the request…

Ground project lookups and refuse relationship names as field values;…

Address review: fail closed on the lookup override, share the request…

Ground project lookups, refuse relationship names as field values, lo…

Address review: keep the hint to the types it relates, hold the entit…

Remote-model measurements (PR #3716)

The grounding eval now runs against a remote provider (NS_MODEL=openai-compat:<config-id>:<model>). 8 scenarios × 3 reps each, on OpenRouter, final tree: the person lookups (first name, missing preposition, typo, none), status of assigned, tasks in a project, assign, and "tasks assigned to me".

Model	Kind (measured)	Stable across 3 reps
Claude Haiku 5.5	reasoning, adaptive (5,657 reasoning tokens over 84 requests)	8 of 8
Gemma 4 31B	non-reasoning (0 reasoning tokens over 78 requests)	8 of 8

The local Gemma 4 e4b (4B) result from the earlier comment still stands, including the status-of-assigned gap, which is a small-model paraphrase: both remote models answer it.

What the first remote run found (Haiku 5.5, before any fix: 4 of 7 stable)

Every failure had a cause in the tools or the context:

search_nodes searched a query of quote marks alone ("" written as JSON text) literally: a correct relationship filter then returned 0 rows and the reply said "no tasks". Hit count-first-name and count-missing-preposition.
create_relationship with a task's assignee failed with a message naming neither working form (the call is made from the person, as tasks).
Graph Editing and Node Creation refuse a relationship written as a field and say to call create_relationship, but did not grant it, so on a routed write the tool was not in the surface at all ("I don't have a tool for creating relationships").
"me": the entity tier resolves names, so "me" got no id and no links; the model searched semantically for the owner's name and found nothing. And a blank owner name/email dropped the current-user line altogether, which is the "asked for Michael's node id" row.

All five are fixed in #3716, each with a test that fails without it.

Test prompts

Covered with a correct answer on both remote models: counts by person (four phrasings, including no tasks), tasks in a project, status of assigned tasks, assign (writes the relationship; the eval now requires a successful create_relationship), and "tasks assigned to me". Not run: Mistral Large (excluded for cost), so the "at least Mistral Large" criterion is not evidenced. The prompts that returned a provider 429 or 400 in the original transcripts were not re-run either.

Still open on this issue

A before/after request-size record for the remote path: the local record is on the earlier comment (heavy read-only turn ~18.5K → ~6.9K prompt tokens). On the remote path a lookup turn is three requests of roughly 1.1K (routing), 6.6K and 6.9K prompt tokens.
Mistral Large, if that model is still wanted as the third.

Skip the Stage-1 request for a settled lookup; read ids through the p…

Update on the final tree (PR #3716, `af216c1`)

Re-measured after the review fixes, the Stage-1 skip and the id tolerance: 8 scenarios × 3 reps each.

	Stable across 3 reps
Claude Haiku 5.5 (reasoning, adaptive)	8 of 8
Gemma 4 31B (non-reasoning)	8 of 8
Gemma 4 e4b, local (4B)	7 of 8

Correction

The local assign-to-person result reported earlier as 3 of 3 was an overstatement: the check counted any create_relationship call, errored ones included. With the check requiring a successful call it is 0 of 3 on the 4B local model. It gets the update_node refusal and the create_relationship hint (both spell out relationship_type 'tasks', from the person's id to the task's) and sends an invented assigned_to anyway. Both remote models pass it 3 of 3. This is recorded as a gap of the 4B model, not worked around: making update_node create the edge itself would reverse the decision that a relationship is refused as a field value.

status-of-assigned (the earlier local gap) and count-for-me pass 3 of 3 on the local model.

Request size

Local, lookup turns: the first request carries ~6.7K prompt tokens (system prompt ~11K bytes, six tool schemas ~16.6K bytes) and takes a median 15.3s, 13.9s of it before the first token. The follow-up after the tool result re-sends the same tools and takes 2.5s (prefix cache). Remote: three requests became two (routing request removed): ~6.6K and ~6.9K prompt tokens.

Criteria

All of this issue's criteria are met by #3709, #3716 and the comments above, with the Mistral Large requirement dropped. The gap that remains (the 4B model on assign) is recorded here.

Address re-review: keep compound questions with Stage 1, make id repa…

Make the lookup and assign paths hold on remote models; run the groun…

Address review: keep the hint to the types it relates, hold the entit…

Skip the Stage-1 request for a settled lookup; read ids through the p…

Address re-review: keep compound questions with Stage 1, make id repa…

Make the lookup and assign paths hold on remote models; run the groun…

Navigation Menu

Repository navigation

AI-Chat context is neither targeted nor grounded: too much that does not bear on the message, too little on what it is about #3706

Problem

Evidence

How much of the request bore on the question

Approach

Test prompts

Acceptance Criteria

Measurement 1: baseline on the local model, through the real turn

Measurement 2: what changed the local model's answers, and what did not

Remote-model measurements (PR #3716)

What the first remote run found (Haiku 5.5, before any fix: 4 of 7 stable)

Test prompts

Still open on this issue

Update on the final tree (PR #3716, `af216c1`)

Correction

Request size

Criteria

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

AI-Chat context is neither targeted nor grounded: too much that does not bear on the message, too little on what it is about #3706

Description

Problem

Evidence

How much of the request bore on the question

Approach

Test prompts

Acceptance Criteria

Activity

malibio commented on Oct 8, 2026

Measurement 1: baseline on the local model, through the real turn

malibio commented on Oct 9, 2026

Measurement 2: what changed the local model's answers, and what did not

malibio commented on Oct 9, 2026

malibio commented on Oct 10, 2026

Remote-model measurements (PR #3716)

What the first remote run found (Haiku 5.5, before any fix: 4 of 7 stable)

Test prompts

Still open on this issue

malibio commented on Oct 10, 2026

Update on the final tree (PR #3716, af216c1)

Correction

Request size

Criteria

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Update on the final tree (PR #3716, `af216c1`)