diff --git a/packages/agent/src/agent_guidance.rs b/packages/agent/src/agent_guidance.rs index 190220ce5..219acb830 100644 --- a/packages/agent/src/agent_guidance.rs +++ b/packages/agent/src/agent_guidance.rs @@ -18,8 +18,8 @@ /// `skill_pipeline.rs` (used only by the skill-based schema-creation path) and /// should be consolidated here when that path is unified. pub const SCHEMA_CREATION_RULES: &str = "NODE MODEL: Everything is a node. Built-in types: task, text, date. Custom types need a schema first (create_schema). Once a schema exists, create instances with create_node(node_type=). Never call create_schema for a type already in RELEVANT ENTITY TYPES.\n\ - \"DATABASE\" = SCHEMA: When the user says \"create a database\", \"set up a tracker\", or \"track X\", call create_schema IMMEDIATELY — no confirmation, no search_skills, no planning text. The entity name is the schema (\"a contacts database\" → call create_schema with name=\"Contact\").\n\ - INSTANCE vs TYPE: \"Add an invoice for $500\", \"add a contact named Jane Doe\", \"log a project called X\" — these ask for a specific INSTANCE of an existing type. Call search_skills(query=\"add contact\") then call create_node(node_type=\"contact\", ...). Never ask for confirmation — just execute.\n\ + \"DATABASE\" = SCHEMA: A request to start tracking a kind of thing — worded as a database, a tracker, a list, or \"track X\" — means call create_schema IMMEDIATELY: no confirmation, no search_skills, no planning text. Name the schema after the single entity being tracked, in singular form, stripped of the tracking wording itself.\n\ + INSTANCE vs TYPE: A request that supplies the particulars of ONE record — a name, an amount, a title, a date — asks for an INSTANCE of a type that already exists, not a new type. Call search_skills, then create_node(node_type=, ...) with those particulars as properties. Never ask for confirmation — just execute. Only a request to start tracking a KIND of thing calls for create_schema.\n\ NO CONFIRMATION FOR KNOWN TYPES: If a type appears in RELEVANT ENTITY TYPES, you have all the information you need. Do NOT say \"Could you confirm\" or \"I want to make sure\" or \"Would you like me to\" — just call search_skills then create_node immediately. Confirmation is NEVER required when the schema already exists.\n\ SCHEMA SUCCESS/FAILURE: After create_schema returns a schema object, respond to the user and STOP — do NOT call create_schema again. If create_schema returns an \"already exists\" error, stop immediately and tell the user the type already exists.\n\ TITLE TEMPLATE: Every {field_name} in title_template MUST appear in the fields array. If you want {invoice_number} in a template, you MUST add a field named invoice_number. Never reference a placeholder that is not in fields — this causes a validation error.\n\ @@ -33,7 +33,8 @@ pub const SCHEMA_CREATION_RULES: &str = "NODE MODEL: Everything is a node. Built /// search_semantic, and canonical create/update/connect patterns. Parameter-level /// detail is intentionally deferred to tool schemas to avoid duplication. pub const TOOL_STRATEGY_RULES: &str = "TOOL STRATEGY:\n\ - - CONVERSATIONAL TURNS USE NO TOOLS. Greetings (\"hi\", \"hello\"), thanks, small talk, capability questions (\"what can you do?\"), and meta questions about yourself — answer directly in text. Do NOT call any tool.\n\ + - NODE_TYPE COMES FROM RELEVANT ENTITY TYPES: Every node_type argument (search_nodes, create_node, update_node, resolve_query) MUST be an id copied exactly from the RELEVANT ENTITY TYPES block in this prompt — character for character, including any underscores. Never shorten, singularize, paraphrase, or guess it from the user's wording: the user's noun is usually NOT the id, because the id was derived from whatever the type was named when its schema was created. If the type you need is not listed in RELEVANT ENTITY TYPES, it does not exist yet — do not invent an id for it.\n\ + - CONVERSATIONAL TURNS USE NO TOOLS. Greetings, thanks, small talk, questions about your own capabilities or limits, and other meta questions about yourself — answer directly in text. Do NOT call any tool: nothing in the user's graph needs to be read to answer them.\n\ - META QUESTIONS (\"how did you check?\", \"what tool did you use?\", \"did you look up X?\"): answer ONLY from what is visible in this conversation's tool call history. Do NOT fabricate tool names, arguments, or results. If you cannot see a tool call in the history that matches the claim, say so honestly — \"I did not make that search\" or \"I don't see a record of that in this conversation.\"\n\ - SCHEMA CLAIMS WITHOUT VERIFICATION: Never state that a node type has or lacks a specific property (e.g. \"task has no due_date field\") without first calling a tool to verify. If you have not called get_node or search_nodes on the schema in this turn, you do not know its fields — say so.\n\ - IDENTICAL TOOL CALLS: Never call the same tool with the exact same arguments twice in one turn. If a tool returned a result and you are about to call it again identically, you already have the answer — produce your response using the result you have.\n\ @@ -46,11 +47,11 @@ pub const TOOL_STRATEGY_RULES: &str = "TOOL STRATEGY:\n\ - BLAST-RADIUS GATE: deletion is irreversible — only call delete_node or delete_schema when the user explicitly and unambiguously asks to delete. Never clarify before create_schema, create_node, or update operations. \"Could you confirm?\" and \"I want to make sure\" are FORBIDDEN before any non-delete operation.\n\ - ALWAYS search_nodes first before update_node or update_task_status — even if a node ID appeared earlier in the conversation. Never skip the search step.\n\ - READ-THEN-WRITE TURN COMPLETION: an instruction to change something (\"mark X as Y\", \"update X\", \"set X's status to Y\") is not finished when the search that finds X returns — it is finished when the write (update_node or update_task_status) succeeds. Once search_nodes returns a clear single match for a write instruction, your NEXT action MUST be that write call in the SAME turn — not a summary, not a question, not stopping. Searching and then stopping is a FAILED turn, not a completed one.\n\ - - search_nodes is the ONE tool for finding, listing, and filtering nodes. By keyword/title: search_nodes(query, node_type). To LIST ALL OF A TYPE (e.g. \"list all my invoices\", \"show all tasks\"): search_nodes(query=\"\", node_type=) — empty query lists every node of that type. To FILTER BY A TYPED PROPERTY (status, due_date, amount, operators like gt/lt): add filters, e.g. search_nodes(node_type=, filters=[{\"type\":\"property\",\"operator\":\"equals\",\"property\":\"status\",\"value\":\"open\"}]). search_nodes returns each node's properties. Use search_semantic(query, node_types, scope, threshold, graph_boost) ONLY for meaning-based / fuzzy questions.\n\ + - search_nodes is the ONE tool for finding, listing, and filtering nodes. By keyword/title: search_nodes(query, node_type). To LIST EVERY NODE OF A TYPE — any request naming a type with no narrowing condition — call search_nodes(query=\"\", node_type=); the empty query is what makes it return the whole type, so do NOT invent a keyword to put there. To FILTER BY A TYPED PROPERTY (status, due_date, amount, operators like gt/lt): add filters, e.g. search_nodes(node_type=, filters=[{\"type\":\"property\",\"operator\":\"equals\",\"property\":\"status\",\"value\":\"open\"}]). search_nodes returns each node's properties. Use search_semantic(query, node_types, scope, threshold, graph_boost) ONLY for meaning-based / fuzzy questions.\n\ - search_nodes filter \"type\" values: use \"property\" for schema/node fields (e.g. status, due_date, priority — anything defined on the node type). Use \"metadata\" ONLY for created_at, modified_at, node_type, or content. Using \"metadata\" for a property field (e.g. status) always fails with \"Invalid metadata field\".\n\ - search_semantic result: if 'markdown' is non-empty, summarize from it directly — skip get_node.\n\ - To get full content: get_node(id, format=markdown). To get connections: get_related_nodes(id).\n\ - - To update a CUSTOM schema node's property (e.g. mark invoice as paid): if the request identifies the target by a paraphrased description, an implicit property reference (e.g. a dollar amount, without naming which field it is), or a relative date/status word (\"next Friday\", \"overdue\", \"recent\") — call resolve_query(request=, node_type=) FIRST, then pass its returned \"query\" and \"filters\" directly into search_nodes, then update_node(id=, properties={...}). Do NOT hand-write the search_nodes query yourself in these cases — resolve_query has looked up the schema's real fields and today's date, which you have not. Example: \"Mark the $500 invoice as paid\" → resolve_query(request=\"Mark the $500 invoice as paid\", node_type=\"invoice\") → use its output in search_nodes → update_node. Skip resolve_query only when the identifier is already simple and unambiguous (e.g. \"mark the Acme invoice as paid\" → search_nodes(node_type=\"invoice\", query=\"Acme\") directly). Use update_task_status ONLY for built-in task nodes — NOT for custom types like invoice, contact, book.\n\ + - To update a CUSTOM schema node's property: if the request identifies the target indirectly — by a paraphrased description, an implicit property reference (a bare value such as an amount or code, without naming which field it belongs to), or a relative date/status word (a weekday, \"overdue\", \"recent\") — call resolve_query(request=, node_type=) FIRST, then pass its returned \"query\" and \"filters\" directly into search_nodes, then update_node(id=, properties={...}). Do NOT hand-write the search_nodes query yourself in these cases — resolve_query has looked up the schema's real fields and today's date, which you have not. Skip resolve_query only when the target is already named directly and unambiguously (a proper name or exact title you can pass straight to search_nodes as query). Use update_task_status ONLY for built-in task nodes — for every custom schema type, use update_node.\n\ - To create a node: call search_skills first to get schema_metadata, then call create_node(content, node_type=, properties=). For built-in types (task, text, date), call create_node directly with no properties unless the user provides them.\n\ - To add/modify an entity type: create_schema or update_schema(schema_id).\n\ - To connect nodes: create_relationship with names from schemas above.\n\ @@ -129,10 +130,314 @@ mod tests { ); } + /// `node_type` must be taught as a lookup into the injected RELEVANT ENTITY + /// TYPES block, never as a literal. A hardcoded type id is wrong for + /// essentially every real workspace, because the id is derived from whatever + /// the schema happened to be named at creation time. + #[test] + fn tool_strategy_rules_bind_node_type_to_relevant_entity_types() { + assert!( + TOOL_STRATEGY_RULES.contains("NODE_TYPE COMES FROM RELEVANT ENTITY TYPES"), + "must state where node_type values come from" + ); + for tool in [ + "search_nodes", + "create_node", + "update_node", + "resolve_query", + ] { + assert!( + TOOL_STRATEGY_RULES.contains(tool), + "node_type guidance must cover {tool}" + ); + } + assert!( + !TOOL_STRATEGY_RULES.contains("node_type=\"invoice\""), + "node_type must never be taught as a hardcoded literal" + ); + } + #[test] fn node_reference_format_specifies_bare_uri() { assert!(NODE_REFERENCE_FORMAT.contains("nodespace://")); assert!(NODE_REFERENCE_FORMAT.contains("no markdown links")); assert!(NODE_REFERENCE_FORMAT.contains("no backticks")); } + + // ------------------------------------------------------------------ + // Eval-contamination guard + // ------------------------------------------------------------------ + + /// Every guidance string that reaches the model, paired with its constant + /// name for error reporting. Add new guidance constants here. + fn guidance_corpus() -> Vec<(&'static str, String)> { + let mut corpus: Vec<(&'static str, String)> = vec![ + ("SCHEMA_CREATION_RULES", SCHEMA_CREATION_RULES.to_string()), + ("TOOL_STRATEGY_RULES", TOOL_STRATEGY_RULES.to_string()), + ("NODE_REFERENCE_FORMAT", NODE_REFERENCE_FORMAT.to_string()), + ]; + // skill_rules.rs `imperative` text is seeded into the DB as prompt + // content by seed_skill_nodes(), so it is guidance too. + for r in crate::skill_rules::SCHEMA_RULES { + corpus.push(("skill_rules::SCHEMA_RULES", r.imperative.to_string())); + } + for r in crate::skill_rules::INTERACTION_RULES { + corpus.push(("skill_rules::INTERACTION_RULES", r.imperative.to_string())); + } + // Every seeded skill's markdown_content is injected as skill + // instructions when search_skills routes to it, so it is model-facing + // guidance and equally contaminable. Pulling these from + // seed_skill_nodes() rather than naming constants means a skill added + // later is covered automatically — a hand-maintained file list is + // exactly how the skill_pipeline.rs contamination went unnoticed. + for t in crate::skill_pipeline::seed_skill_nodes() { + if !t.markdown_content.is_empty() { + corpus.push(("skill_pipeline::seed_skill_nodes", t.markdown_content)); + } + } + // The seeded prompt nodes ARE the base system prompt. Some interpolate + // the constants above, but others carry their own literal text that + // nothing else covers. + for t in crate::prompt_assembler::PromptAssembler::seed_prompt_nodes() { + if !t.markdown_content.is_empty() { + corpus.push(("prompt_assembler::seed_prompt_nodes", t.markdown_content)); + } + } + corpus + } + + /// Extract the `prompt: "..."` / `prompt: '...'` literals from an eval + /// script. Parsing the real file (rather than keeping a copy here) is the + /// point: a reworded scenario is picked up automatically, so this guard + /// cannot silently go stale against the eval it protects. + /// Returns (parsed prompts, number of `prompt:` literal sites seen). The + /// caller compares the two: a parser that silently skips a literal shape + /// it does not understand fails OPEN — that prompt simply goes unchecked — + /// so coverage is asserted rather than assumed. + fn eval_prompts(source: &str) -> (Vec, usize) { + let mut prompts = Vec::new(); + let mut sites = 0usize; + for (idx, _) in source.match_indices("prompt:") { + // Only treat this as an object key. `prompt:` also appears inside + // identifiers/field access (e.g. `fixture.prompt:`) and prose. + if source[..idx] + .chars() + .next_back() + .is_some_and(|c| c.is_alphanumeric() || c == '_' || c == '.') + { + continue; + } + let rest = source[idx + "prompt:".len()..].trim_start(); + let quote = match rest.chars().next() { + Some(q @ ('"' | '\'' | '`')) => q, + // e.g. `prompt: fixture.prompt` — a reference, not a literal; + // nothing to parse, so it is not counted as a site. + _ => continue, + }; + sites += 1; + let body = &rest[quote.len_utf8()..]; + // Locate the closing quote, honoring backslash escapes so an + // embedded \" does not truncate the literal early. + let mut end = None; + let mut escaped = false; + for (i, c) in body.char_indices() { + if escaped { + escaped = false; + continue; + } + match c { + '\\' => escaped = true, + c if c == quote => { + end = Some(i); + break; + } + _ => {} + } + } + let Some(end) = end else { continue }; + let literal = &body[..end]; + if !literal.is_empty() { + prompts.push(literal.to_string()); + } + } + (prompts, sites) + } + + fn normalize(s: &str) -> Vec { + s.to_lowercase() + .split(|c: char| !c.is_alphanumeric()) + .filter(|w| !w.is_empty()) + .map(|w| w.to_string()) + .collect() + } + + /// Longest run of consecutive words shared between an eval prompt and a + /// guidance string. Catches near-verbatim reuse (a planted example lightly + /// reworded), not just exact substring matches. + fn longest_shared_run(prompt_words: &[String], guidance_words: &[String]) -> (usize, String) { + let mut best = 0usize; + let mut best_phrase = String::new(); + for i in 0..prompt_words.len() { + for j in 0..guidance_words.len() { + let mut k = 0; + while i + k < prompt_words.len() + && j + k < guidance_words.len() + && prompt_words[i + k] == guidance_words[j + k] + { + k += 1; + } + if k > best { + best = k; + best_phrase = prompt_words[i..i + k].join(" "); + } + } + } + (best, best_phrase) + } + + /// The agent's guidance must not contain the eval's own scenario prompts. + /// + /// If a prompt is planted in guidance alongside the tool call it should + /// produce, a passing scenario proves only that the model can copy a + /// memorized example — the eval stops measuring generalization and prompt + /// tuning acquires a degenerate solution (add more worked examples that + /// match more scenarios). This is trivially easy to reintroduce while + /// tuning prompts, which is exactly when it does the most damage. + /// + /// Fix a failure by rewording the GUIDANCE into a rule that teaches the + /// shape ("pass the id exactly as shown in RELEVANT ENTITY TYPES") rather + /// than by deleting the eval scenario. + #[test] + fn guidance_is_not_contaminated_by_eval_prompts() { + // Longest run of consecutive shared words tolerated between an eval + // prompt and a guidance rule. Ordinary instruction-following overlap + // ("all my", "set it to") is short; a planted example is not. + const MAX_SHARED_RUN: usize = 4; + + let repo_root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../.."); + let eval_scripts = ["scripts/aichat-matrix.ts", "scripts/routing-eval.ts"]; + + let corpus = guidance_corpus(); + let tokenized: Vec<(&str, Vec)> = corpus + .iter() + .map(|(name, text)| (*name, normalize(text))) + .collect(); + + let mut violations: Vec = Vec::new(); + let mut checked = 0usize; + + for script in eval_scripts { + let path = repo_root.join(script); + let source = std::fs::read_to_string(&path) + .unwrap_or_else(|e| panic!("failed to read {}: {e}", path.display())); + + let (prompts, sites) = eval_prompts(&source); + assert!( + !prompts.is_empty(), + "no `prompt:` literals parsed out of {script} — the parser has \ + drifted from the eval's format and this guard is now vacuous; \ + fix eval_prompts() rather than deleting this assertion" + ); + // A parser that skips a literal shape it does not understand fails + // OPEN: the prompt goes unchecked and the guard still passes. So + // every site it saw must have produced a prompt. + assert_eq!( + prompts.len(), + sites, + "parsed only {} of {sites} `prompt:` literals in {script} — the \ + unparsed ones are silently exempt from this guard; teach \ + eval_prompts() the shape it missed", + prompts.len() + ); + + for prompt in prompts { + checked += 1; + let prompt_words = normalize(&prompt); + if prompt_words.is_empty() { + continue; + } + for (const_name, guidance_words) in &tokenized { + let (run, phrase) = longest_shared_run(&prompt_words, guidance_words); + // A short prompt can be quoted in full without ever + // exceeding a fixed run length, so also flag any prompt + // reproduced in its entirety. + // Below three words this degenerates into "guidance + // mentions this word anywhere", which flags ordinary + // vocabulary and reports it as contamination. + let quoted_whole = run == prompt_words.len() && run >= 3; + if run > MAX_SHARED_RUN || quoted_whole { + let why = if quoted_whole { + "the ENTIRE prompt appears" + } else { + "shares a long phrase" + }; + violations.push(format!( + " {const_name} {why} ({run} consecutive words) from \ + {script} prompt {prompt:?}\n shared phrase: {phrase:?}" + )); + } + } + } + } + + assert!(checked > 0, "no eval prompts were checked"); + assert!( + violations.is_empty(), + "agent guidance contains eval scenario prompts — the eval would be \ + measuring recall of a planted example, not generalization:\n{}\n\n\ + Reword the GUIDANCE into a rule that teaches the shape rather than \ + a memorized literal; do not weaken this test or delete the scenario.", + violations.join("\n") + ); + } + + /// The guard above is only meaningful if it actually fires — this pins the + /// detector against the real contamination that motivated it (the former + /// `agent_guidance.rs:53` worked example built on scenario 6's prompt). + #[test] + fn contamination_guard_detects_a_planted_example() { + let planted = normalize( + "Example: \"Mark the $500 invoice as paid\" \ + → resolve_query(request=\"Mark the $500 invoice as paid\", node_type=\"invoice\")", + ); + let prompt = normalize("Mark the $500 invoice as paid"); + let (run, _) = longest_shared_run(&prompt, &planted); + assert!( + run > 4, + "detector failed to flag a known-contaminated example (run={run})" + ); + } + + #[test] + fn eval_prompt_parser_extracts_literals() { + let src = r#" + { scenario: "a", prompt: "Hi there", expect: x }, + { scenario: "b", prompt: 'Add a book called "Dune"', expect: y }, + { scenario: "c", prompt: fixture.prompt }, + "#; + let (got, sites) = eval_prompts(src); + // The single-quoted literal keeps its embedded double quotes; the + // non-literal `fixture.prompt` reference is skipped and not counted + // as a site, so coverage stays exact. + assert_eq!(got, vec!["Hi there", "Add a book called \"Dune\""]); + assert_eq!(sites, got.len(), "every counted site must be parsed"); + } + + #[test] + fn eval_prompt_parser_handles_backticks_and_escapes() { + let src = r#" + { prompt: `a template literal prompt` }, + { prompt: "an \"escaped\" quote inside" }, + "#; + let (got, sites) = eval_prompts(src); + assert_eq!(sites, 2, "both literal shapes must be counted as sites"); + assert_eq!( + got, + vec![ + "a template literal prompt", + "an \\\"escaped\\\" quote inside" + ], + "backticked and escaped literals must parse in full, not truncate" + ); + } } diff --git a/packages/agent/src/skill_pipeline.rs b/packages/agent/src/skill_pipeline.rs index a438f2e89..d15bd3970 100644 --- a/packages/agent/src/skill_pipeline.rs +++ b/packages/agent/src/skill_pipeline.rs @@ -296,7 +296,7 @@ STRUCTURED PROPERTY QUERIES: To filter by property values (status, due_date, etc CALL create_node NOW: You received this instruction because search_skills was called. Your NEXT action MUST be create_node — do not output any planning text. Gather all needed values from the user message and call create_node immediately. -TYPE MAPPING FROM schema_metadata: When search_skills returned schema_metadata for this skill, set node_type to the type_id from that metadata (e.g. "invoice", "customer", "project"). For generic text notes use node_type="text". For tasks use node_type="task". +TYPE MAPPING FROM schema_metadata: When search_skills returned schema_metadata for this skill, set node_type to the type_id from that metadata, copied exactly as written — never the user's noun for it, and never a shortened or paraphrased form. For generic text notes use node_type="text". For tasks use node_type="task". REQUIRED FIELDS: Read the fields array from schema_metadata. Required fields (required=true) MUST be included in the properties map. Optional fields should be included if the user provided a value for them. @@ -304,16 +304,18 @@ TITLE: The node title is the content field. If schema_metadata has a title_templ PROPERTY KEYS: Use the field name exactly as it appears in schema_metadata fields[].name. Do NOT add namespace prefixes for schema-defined fields. -EXAMPLE — create an invoice for $500 due next Friday (schema_metadata type_id="invoice"): +EXAMPLE — the shape of the call, NOT the values. Copy the structure; take every value from schema_metadata and the user's message. Suppose schema_metadata has type_id="widget" and fields named label, quantity, received_on, condition: { - "node_type": "invoice", - "content": "Invoice #001", + "node_type": "widget", + "content": "Shipment 24", "properties": { - "amount": 500, - "due_date": "2026-06-14", - "status": "draft" + "label": "Shipment 24", + "quantity": 12, + "received_on": "2026-03-04", + "condition": "sealed" } } +Never reuse "widget" or these field names — they are placeholders. Your node_type is the type_id from schema_metadata, and your property keys are that metadata's field names. SUCCESS: After create_node returns a node ID, confirm to the user what was created and STOP. Do NOT call search_skills, get_node, or any other tool — the create response is sufficient. The task is complete."#.to_string(), }, diff --git a/scripts/aichat-matrix.ts b/scripts/aichat-matrix.ts index d93934054..ea8ee9635 100644 --- a/scripts/aichat-matrix.ts +++ b/scripts/aichat-matrix.ts @@ -34,8 +34,21 @@ * baseline and fail on regression (a scenario that was passing and is now failing). * * Scenarios 4/6/8 depend on earlier turns, so the script keeps groups of turns on - * the same chat node. Scenario 6 ("Mark invoice ... as paid") has no stable id up - * front, so it is phrased to let the agent search-then-update by description. + * the same chat node. Scenario 6 has no stable id up front, so it is phrased to let + * the agent search-then-update by description. + * + * CONTAMINATION: scenario prompts must NOT appear in the agent's compiled guidance + * (agent_guidance.rs, skill_rules.rs, and every seeded skill's markdown_content in + * skill_pipeline.rs — all of it reaches the model) — otherwise a pass measures + * recall of a planted example rather than generalization. The Rust test + * `guidance_is_not_contaminated_by_eval_prompts` parses the prompts out of this file + * and fails the build if any reappears in guidance. When editing prompts here, keep + * them phrased unlike the guidance rules; when editing guidance, don't paste prompts + * from this file in as worked examples. + * + * Scenario 8's sub-labels name each step's ROLE in the multi-type flow (first type, + * second type, instance of each), not the entity in the prompt. They are stable IDs + * for baseline diffing; the prompt wording underneath them is free to change. */ import { dirname, join, resolve } from "node:path"; @@ -212,59 +225,65 @@ export function assertExpectation( const GROUPS: ScenarioStep[][] = [ [{ scenario: "1. Greeting", prompt: "Hi there", expect: { kind: "noTools" } }], [{ scenario: "2. Capability", prompt: "What can you do?", expect: { kind: "noTools" } }], - // Invoice CRUD chain (scenarios 3-7) shares one chat node. + // Single-custom-type CRUD chain (scenarios 3-7) shares one chat node. + // Wording here must stay independent of packages/agent/src/agent_guidance.rs — + // guidance_is_not_contaminated_by_eval_prompts (agent_guidance.rs) enforces it. [ { scenario: "3. Schema creation", - prompt: "Create an invoice tracking database", + // Mentions checked-out vs returned so the schema plausibly carries a + // status field — scenario 6 then tests resolve_query routing on an + // indirect reference, not whether the model guessed a status value. + prompt: + "I want to keep a record of the equipment my team checks out and whether it's been returned", expect: { kind: "noExtraTypes" }, }, { scenario: "4. Instance creation", - prompt: "Add an invoice for $500 due next Friday", + prompt: "Log a laser cutter checked out on the 12th, replacement cost 2400", expect: { kind: "toolOnce", tool: "create_node" }, }, { scenario: "5. List/query", - prompt: "List all my invoices", + prompt: "What equipment is on the books?", expect: { kind: "toolOnce", tool: "search_nodes" }, }, { scenario: "6. Update", - prompt: "Mark the $500 invoice as paid", + prompt: "The 2400 one came back — set it to returned", expect: { kind: "toolSequence", tools: ["search_nodes", "update_node"] }, }, { scenario: "7. Empty-result query", - prompt: "Find my invoice for one million dollars", + prompt: "Do we have anything worth 90000 sitting out?", expect: { kind: "noRetry", tool: "search_nodes" }, }, ], // Multi-custom-type CRUD (scenario 8) shares its own chat node. [ { - scenario: "8a. Create type: book", - prompt: "Create a database to track books I want to read", + scenario: "8a. Create type: first", + prompt: "Start tracking albums I mean to listen to", expect: { kind: "toolOnce", tool: "create_schema" }, }, { - scenario: "8b. Create type: contact", - prompt: "Create a database for my contacts", + scenario: "8b. Create type: second", + prompt: "I also need a tracker for the venues I book", expect: { kind: "toolOnce", tool: "create_schema" }, }, { - scenario: "8c. Instance: book", - prompt: 'Add a book called "Dune" by Frank Herbert', + scenario: "8c. Instance: first type", + prompt: "Put down Kind of Blue, it's by Miles Davis", expect: { kind: "toolOnce", tool: "create_node" }, }, { - scenario: "8d. Instance: contact", - prompt: "Add a contact named Jane Doe, email jane@example.com", + scenario: "8d. Instance: second type", + prompt: "New venue: the Blue Note, they can be reached at booking@example.com", expect: { kind: "toolOnce", tool: "create_node" }, }, { scenario: "8e. Query across types", - prompt: "Show me all my books", + prompt: "Run through the albums for me", expect: { kind: "toolOnce", tool: "search_nodes" }, }, ], diff --git a/scripts/results-e4b.json b/scripts/results-e4b.json new file mode 100644 index 000000000..ab5e4874a --- /dev/null +++ b/scripts/results-e4b.json @@ -0,0 +1,178 @@ +{ + "label": "e4b-decontaminated", + "model": "gemma-4-e4b-q4km", + "recorded_at": "2026-07-28", + "guidance_commit": "da126293", + "eval_commit": "805a60e4", + "n_ctx": 32768, + "n_ctx_note": "Chat context granted for this run, read from the daemon log ('Creating chat LlamaContext (n_ctx=32768...)'). Well above the ~6600-token agent system prompt, so this run was not context-starved - the failure mode where every turn dies on 'Context window exceeded' and no-tool assertions score as passes.", + "host_memory_gb": 16, + "note": "Honest baseline: measured AFTER eval-prompt contamination was removed from agent guidance. Recorded on a purged database so every prompt/skill node was reseeded from the compiled constants - a reused DB keeps old seeded guidance and silently measures the pre-fix prompts. Not comparable to the earlier 11/12, which was measured with eval prompts planted in the guidance. EFFECTIVE SCORE IS 8/12: scenario 3 passes only because the assertion cannot see that its create_schema call failed validation (see caveats[0]). When #1775 lands and scenario 3 passes for real, this headline may not move - compare effective_passed, not the raw count.", + "caveats": [ + "Scenario 3 is a FALSE PASS in this run. Its assertion (noExtraTypes) only checks that create_schema was called once; it does not check the schema persisted. Here create_schema returned is_error=true (title_template placeholders with no fields array) and no equipment type was registered. Scenario 4 then correctly reported the type did not exist and was scored a failure for it. Tracked in #1775.", + "Scenarios 4 and 6 are downstream of scenario 3's silent failure, not independent defects. 8c is independent and real, but is NOT a blind-retry / IDENTICAL TOOL CALLS violation: the log shows create_node called first with EMPTY arguments (is_error=true, 'missing field node_type'), then corrected and retried successfully. The two calls were not identical, and the recovery itself was reasonable. The actual defect is emitting a tool call with no arguments - the same argument-construction failure class as scenario 3's title_template-without-fields. The toolOnce assertion counts tool NAMES, so it cannot tell a duplicate write from a successful recovery. Tracked in #1775.", + "Control experiments run after this baseline, recorded so the 8a/8b wording is not re-litigated from git archaeology. (a) The DATABASE=SCHEMA rule is not a literal lookup on the four words database/track/tracker/list: two phrasings lacking all four ('Set up somewhere to note down albums I mean to listen to', 'I want a place to jot down albums worth hearing') both fired create_schema. Do NOT read this as 'the rule generalizes' broadly - both probes are still canonical record-keeping English, n=2 with a stochastic sampler, and neither reaches semantically oblique phrasing. Critically, the FIRST probe string is the exact wording that FAILED as scenario 8a in this run: same prompt, both outcomes. That makes scenario 8a non-deterministic, which is itself a finding. The probes ran single-turn on a fresh chat while 8a runs as the first turn of a multi-turn group, so they do not reproduce the failing condition and cannot attribute the difference to wording vs. context vs. sampling noise. (b) Scenario 4's wording is not the cause - any instance phrasing fails when no schema exists to instantiate." + ], + "results": [ + { + "scenario": "1. Greeting", + "prompt": "Hi there", + "expect": { + "kind": "noTools" + }, + "toolsCalled": [], + "passed": true, + "latencyMs": 19136 + }, + { + "scenario": "2. Capability", + "prompt": "What can you do?", + "expect": { + "kind": "noTools" + }, + "toolsCalled": [], + "passed": true, + "latencyMs": 7146 + }, + { + "scenario": "3. Schema creation", + "prompt": "I want to keep a record of the equipment my team checks out and whether it's been returned", + "expect": { + "kind": "noExtraTypes" + }, + "toolsCalled": [ + "create_schema" + ], + "passed": true, + "latencyMs": 4089 + }, + { + "scenario": "4. Instance creation", + "prompt": "Log a laser cutter checked out on the 12th, replacement cost 2400", + "expect": { + "kind": "toolOnce", + "tool": "create_node" + }, + "toolsCalled": [ + "search_skills", + "create_schema" + ], + "passed": false, + "failure": "Expected 'create_node' exactly once, got 0 (tools: create_schema)", + "latencyMs": 10195 + }, + { + "scenario": "5. List/query", + "prompt": "What equipment is on the books?", + "expect": { + "kind": "toolOnce", + "tool": "search_nodes" + }, + "toolsCalled": [ + "search_nodes" + ], + "passed": true, + "latencyMs": 3118 + }, + { + "scenario": "6. Update", + "prompt": "The 2400 one came back \u2014 set it to returned", + "expect": { + "kind": "toolSequence", + "tools": [ + "search_nodes", + "update_node" + ] + }, + "toolsCalled": [], + "passed": false, + "failure": "Expected sequence [search_nodes,update_node] as a subsequence, got: ", + "latencyMs": 2103 + }, + { + "scenario": "7. Empty-result query", + "prompt": "Do we have anything worth 90000 sitting out?", + "expect": { + "kind": "noRetry", + "tool": "search_nodes" + }, + "toolsCalled": [ + "search_nodes" + ], + "passed": true, + "latencyMs": 18386 + }, + { + "scenario": "8a. Create type: first", + "prompt": "Start tracking albums I mean to listen to", + "expect": { + "kind": "toolOnce", + "tool": "create_schema" + }, + "toolsCalled": [ + "create_schema" + ], + "passed": true, + "latencyMs": 2109 + }, + { + "scenario": "8b. Create type: second", + "prompt": "I also need a tracker for the venues I book", + "expect": { + "kind": "toolOnce", + "tool": "create_schema" + }, + "toolsCalled": [ + "create_schema" + ], + "passed": true, + "latencyMs": 20585 + }, + { + "scenario": "8c. Instance: first type", + "prompt": "Put down Kind of Blue, it's by Miles Davis", + "expect": { + "kind": "toolOnce", + "tool": "create_node" + }, + "toolsCalled": [ + "search_skills", + "create_node", + "create_node" + ], + "passed": false, + "failure": "Expected 'create_node' exactly once, got 2 (tools: create_node,create_node)", + "latencyMs": 26453 + }, + { + "scenario": "8d. Instance: second type", + "prompt": "New venue: the Blue Note, they can be reached at booking@example.com", + "expect": { + "kind": "toolOnce", + "tool": "create_node" + }, + "toolsCalled": [ + "search_skills", + "search_nodes", + "create_node" + ], + "passed": true, + "latencyMs": 28521 + }, + { + "scenario": "8e. Query across types", + "prompt": "Run through the albums for me", + "expect": { + "kind": "toolOnce", + "tool": "search_nodes" + }, + "toolsCalled": [ + "search_nodes" + ], + "passed": true, + "latencyMs": 3164 + } + ], + "score": "9/12 - but see caveats: scenario 3 is a confirmed FALSE PASS, so 8/12 turns actually did what was asserted", + "effective_passed": 8 +}