Repository navigation
feat(product-tree): ship the eval and probe skills - #723
simonheimlicher wants to merge 27 commits into
Conversation
BLOCKING [consistency]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/prompt.mdReference: spx/local/generated-sources.toml declares spx//evals//prompt.md as a generated output whose sources include dist/claude/** and dist/codex/, generator = ["outcomeeng_evals/"], and regenerate = "just eval-materialize-prompts spx". CLAUDE.md Generated Sources in Verification Scope section states: Deterministic verification covers the complete changeset; each relation regeneration command backs regeneration parity, rerunning it leaves the committed generated extents byte-identical, and names this relation in scope for judgment once the changeset touches its declared sources. |
BLOCKING [consistency]: src/plugins/product-tree/.claude-plugin/plugin.json:3Reference: CLAUDE.md states: "Plugin version bumps: the PR opens without a version commit; BLOCKING [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/history.jsonlReference: the new |
|
Disposition of the findings in #723 (comment), head 6a91bc5:
|
BLOCKING [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/ISSUES.md:3Reference: the known issue "Eval run history is stale for the routing suite" in that ISSUES.md says the prior staleness delta was frontmatter only, and sets the settlement condition as one passing run of the routing eval.toml with its row committed to history.jsonl. The new eval-evidence-standards skill (property 5, Run evidence) states a run for a stale definition proves nothing about the current one. DEBT [consistency]: src/plugins/product-tree/skills/probe/SKILL.md:47Reference: verify/SKILL.md line 94 states the routers output status vocabulary exhaustively as routed, capability-required, or blocked. |
BLOCKING [consistency]: spx/21-product-tree.enabler/35-evidence.enabler/39-eval-skill.enabler/eval-skill.md:14Reference: src/plugins/product-tree/skills/understand/SKILL.md, assertion_model: "Assertions declare observable product output at the layer that owns the behavior, derived from decisions and specs." DEBT [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/verify-skill.md:14Reference: src/plugins/product-tree/skills/eval-evidence-standards/SKILL.md, properties: run evidence requires "a complete, passing run for the current definition, case set, threshold, and producer"; assertion alignment requires "every expected-verdict field corresponds to a claim in the assertion." |
fac166c to
9f071a1
Compare
|
Reviewed the full diff (README, both |
BLOCKING [consistency]: src/plugins/product-tree/.claude-plugin/plugin.json:3Reference: CLAUDE.md states: Plugin version bumps: the PR opens without a version commit; spx/local/merging.md runs just bump ... as the last commit before the merge, after base synchronization and once review and checks are green on the final content ... Ordinary commits never initiate or request a bump. NEVER hand-edit a manifest version field. spx/local/open-pr.md repeats this: The opening protocol writes no version. A changeset that warrants a bump opens its PR on verified content, and the version commit follows when the content is final. DEBT [evidence]: src/plugins/product-tree/skills/verify/SKILL.md:75Reference: src/plugins/product-tree/skills/eval-evidence-standards/SKILL.md, added by this PR, property 3: Assertion alignment. Every expected-verdict field corresponds to a claim in the assertion, negative cases target the assertion failure mode, and the assertion cannot be unfulfilled while every case passes. Property 4: Falsifiability. One concrete mutation to the producer, never to the prompt or cases, must fail at least one case. |
|
Disposition of the findings in #723 (comment), head 40a9591:
|
40a9591 to
1d59365
Compare
BLOCKING [consistency]: src/plugins/product-tree/skills/audit-eval-evidence/SKILL.md:19Reference: this same file states "NEVER issue a finding the evidence model does not support — drop an unbacked finding rather than reject the eval evidence for it" (line 29) and its success criteria require the suite be "judged on all five evidence properties with none skipped — producer coupling, oracle independence, assertion alignment, falsifiability, and run evidence" (line 233); the workflow body likewise says to "Check the five evidence properties in strict order" (line 38). DEBT [consistency]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/ISSUES.md:9Reference: |
1d59365 to
70f7a28
Compare
|
Reply to the review at #723 (comment), head 1d59365. BLOCKING consistency, DEBT consistency, The version commit follows as the last commit, 70f7a28, and |
BLOCKING [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/verify-skill.mdReference: eval-evidence-standards (shipped in this PR), Run evidence property: "The committed history holds a complete, passing run for the current definition, case set, threshold, and producer ... a run for a stale definition proves nothing about the current one." Also src/plugins/product-tree/skills/understand/SKILL.md, touched_file_debt: "ALWAYS: fix debt that the current change causes, surfaces, or invalidates ... Record and proceed only for work independent of the current change in a surface the change neither touches nor invalidates." DEBT [consistency]: spx/21-product-tree.enabler/68-audit.enabler/32-audit-eval-evidence.enabler/audit-eval-evidence.mdReference: Compliance assertion in that spec: "ALWAYS: check oracle independence, assertion alignment, falsifiability, and run evidence after producer coupling passes ([audit])." |
70f7a28 to
b715427
Compare
|
Reply to the review at #723 (comment), head 70f7a28. BLOCKING evidence, DEBT consistency, The version commit follows as the last commit, b715427, and |
The eval-skill Node declares that the product-tree plugin ships the generic eval specialist as `/eval`, the skill `/verify` routes an evaluate assertion to. Verification: not-run Refs: outcomeeng/changes#868
Add `/eval`, the generic eval specialist `/verify` routes an evaluate assertion to, and `eval-evidence-standards`, the standard it loads. The generated runtime trees follow the sources. Verification: not-run Refs: outcomeeng/changes#868
`/audit-eval-evidence` requires `eval-evidence-standards`, the standard `/eval` loads, so eval authoring and eval-evidence auditing judge from one source. The generated runtime trees follow the sources. Verification: not-run Refs: outcomeeng/changes#868
Regenerate the plugin catalog section of the README with `just docs` after `/eval` and `eval-evidence-standards` joined the product-tree plugin. Verification: passing Refs: outcomeeng/changes#868
State the objective as the output and add success criteria that name the five properties a sound eval suite holds. The generated runtime trees follow the sources. Verification: passing Refs: outcomeeng/changes#868
Add the probe-skill Node, which declares that the product-tree plugin ships `/probe`, the skill `/verify` routes a probe assertion to, and states what `/probe` writes and records. Verification: passing Refs: outcomeeng/changes#868
Add `/probe`, the specialist `/verify` routes a probe assertion to. It writes the probe protocol, records the Author's verdict with the retained artifacts of the attested run, and lists in the plugin catalog. The generated runtime trees follow the sources. Verification: passing Refs: outcomeeng/changes#868
The verify-skill Node maps the probe verification type to `/probe`, the specialist the probe-skill Node declares. Verification: passing Refs: outcomeeng/changes#868
`/verify` invokes `/eval` and `/probe` when the runtime skill catalog carries them and reports a capability gap only when a specialist or a producer kind is absent. The result example shows the routed rows. The generated runtime trees follow the sources. Verification: passing Refs: outcomeeng/changes#868
`/eval` classifies the producer kind and authors the suite under `eval-evidence-standards`; no producer specialist ships, so the skill composes none. `/verify` reports a capability gap only when `/eval` is absent. The generated runtime trees follow the sources. Verification: passing Refs: outcomeeng/changes#868
Materialize the routing eval prompt from the verify skill again after the skill began routing evaluate and probe assertions to `/eval` and `/probe`, so `eval-prompts` finds no drift from the producer source. Verification: passing Refs: outcomeeng/changes#868
The eval and probe skills ship in the product-tree plugin, so the exemptions for /eval, /probe, product-tree:eval and product-tree:probe no longer name skills that are absent. The exemption mapping is empty. Refs: spx/15-validation.enabler/32-skill-reference.enabler
/probe reports a claim that a test, eval, or audit settles as misrouted, naming the verification type to select. /verify now reclassifies that assertion to the named type, routes it once to that type's specialist, and never invokes /probe for it again, so the result keeps its three statuses. The result example gains a routed row for such a claim, the plugin changelog records the rule, and the routing eval prompt is regenerated from the changed producer. Refs: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler
The product-tree plugin ships /probe, so the entry that records a probe route with no specialist no longer holds. The entry now records the remaining gap, a routing eval with no committed passing run against the shipped specialist, and keeps one settlement condition. Refs: spx/21-product-tree.enabler/35-evidence.enabler
Name the object in the first assertion of the eval skill and the probe skill in plain order: the skill to which /verify routes the assertion. Verification: passing (spx validation markdown, just eval-links, spx spec status --format json, just fmt)
Adds a routing case where the probe specialist reports an assertion misrouted and names the test type, so the reroute branch of the verify skill has a case that fails when the branch is removed. Verification: passing
The audit prerequisite names six rules from the shared standard, but the audit body and its rule catalog carried five properties and no case-provenance rule. Step 3b now judges case provenance beside oracle independence, and the evidence model gains the procedure and the case-provenance rule slug. Verification: passing (check-skills, docs-check, validation markdown, eval-links)
The routing eval note named a frontmatter-only delta since its newest passing run. The probe reroute and its new case lie in the delta too. Verification: not-run
The eval-evidence audit judges case provenance beside oracle independence, so the node declares the check. Verification: passing (validation markdown, spec status, eval-links)
b715427 to
a1c6ece
Compare
BLOCKING [consistency]: src/plugins/product-tree/.claude-plugin/plugin.json:3Reference: CLAUDE.md:321 — "Plugin version bumps: the PR opens without a version commit; DEBT [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/cases.jsonl:10Reference: |
The routing prompt named the available_specialists field and left the specialist_report field of the reroute case to be guessed. The template and its materialized prompt now name the field and its members. Verification: passing (materialize-prompts check, validation markdown, eval-links)
a1c6ece to
7cd7b63
Compare
|
Reply to the review at #723 (comment), head a1c6ece. BLOCKING consistency, DEBT evidence, |
DEBT [consistency]: spx/21-product-tree.enabler/65-apply.enabler/ISSUES.md:33Reference: the entry " DEBT [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/cases.jsonlReference: DEBT [consistency]: src/plugins/product-tree/skills/audit-eval-evidence/SKILL.md:38Reference: the new |
The eval-evidence audit named skill, agent, classifier, and script as producer kinds, while its producer table and the shared standard also name command. All four sentences of the bundle now list the same kinds. Verification: passing (check-skills, docs-check)
The apply note said no eval skill ships. The plugin ships /eval, so the entry now records the guarded name and its settlement condition. Verification: passing (validation markdown, eval-links)
7cd7b63 to
f649915
Compare
DEBT [consistency]: spx/21-product-tree.enabler/68-audit.enabler/32-audit-eval-evidence.enabler/audit-eval-evidence.md:21Reference: spx/13-infrastructure.enabler/25-eval-harness.enabler/57-producer-coupled-skill-evals.adr.md:33 says evals this harness runs for skill, agent, classifier, script, or command behavior are coupled to the real producer -- the producer-kind taxonomy already includes command, unchanged by this PR. src/plugins/product-tree/skills/understand/SKILL.md:202 says assertions declare observable product output at the layer that owns the behavior, derived from decisions and specs, never from tests or code. DEBT [consistency]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/verify-skill.md:16Reference: src/plugins/product-tree/skills/understand/SKILL.md:202 says assertions declare observable product output at the layer that owns the behavior, derived from decisions and specs, never from tests or code. |
The audit-eval-evidence evidence model restated the case-provenance rule that eval-evidence-standards owns. The section now defers the rule to the standard and keeps the audit procedure and the rejection value. Verification: passing
DEBT [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/eval.tomlReference: src/plugins/product-tree/skills/eval-evidence-standards/SKILL.md property 5, "Run evidence": "The committed history holds a complete, passing run for the current definition, case set, threshold, and producer ... a run for a stale definition proves nothing about the current one." DEBT [architecture]: src/plugins/product-tree/skills/eval/SKILL.md:6Reference: src/plugins/instructions/skills/skill-standards references/command-capabilities.md: "ALWAYS: grant the narrowest allowed-tools the skill task needs, restricting bash to specific verb patterns." |
The audit-eval-evidence skill loads the shared standard in its audit workflow step instead of a separate prerequisites section. Its evidence-model reference cites the standard for the property definitions and artifact roles, and keeps the audit procedures and the two audit-only artifacts. Verification: passing
The audit-eval-evidence evidence model and workflow restated the oracle-independence property that eval-evidence-standards owns. Both now cite the standard and keep the audit-level case and the rejection value. Verification: passing
DEBT [consistency]: spx/21-product-tree.enabler/35-evidence.enabler/39-probe-skill.enabler/probe-skill.md:11Reference: the sibling node spx/21-product-tree.enabler/35-evidence.enabler/39-eval-skill.enabler/eval-skill.md ships in this same PR for /eval. Its compliance-style assertions sit under a ### Compliance subsection and each carries a routed ([audit]) tag, e.g. "the generic eval specialist owns product command binding, producer-kind routing, freshness inspection, serial one-case convergence, and deterministic aggregation ([audit])". src/plugins/product-tree/skills/understand/SKILL.md line 215 states the model: "Authoring declarations await selection untagged directly under ## Assertions ... alongside routed subsections if present ... Once routed, a spec-malleable assertion may omit its tag; every harder assertion carries exactly one." |
Name the five gates in the audit-eval-evidence objective. Cut the restatements of coupling, alignment, falsifiability, and run-evidence properties from the skill and its evidence model, and point each section at the shared standard's property. Verification: passing
|
Reply to the review of
|
Verification: not-run
|
DEBT [consistency]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/verify-skill.mdReference: CLAUDE.md, "Product tree navigation: declaration and inverse" — "The methodology declares forward: PDR/ADR → spec assertion → test link → test file → import → implementation file" and "An implementation file in the diff that no test imports has no governing spec assertion — a coverage gap the PR is shipping. Specs declare; tests verify; code complies. Surface the gap; do not invent a node to load." DEBT [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/eval.tomlReference: src/plugins/product-tree/skills/eval-evidence-standards/SKILL.md, |



Summary
/evaland/probeskills in the product-tree plugin, so a verify round that needs eval or probe authoring reaches a skill that exists./verifyroutes an evaluate assertion to/evaland a probe assertion to/probe.Background
The eval-skill Node and
/verifyalready declared/eval, and no skill shipped. The evidence Node's known issue records that the probe route has no specialist. A verify round for a Change that needs an[eval]or[probe]assertion therefore stopped withcapability-requiredand could not write or attest the evidence.Changes
/eval: classifies the producer kind, binds the repository's own eval commands, checks freshness, converges one case at a time, and aggregates one deterministic suite result. It loads the neweval-evidence-standardsskill.eval-evidence-standards: the shared standard for what makes eval evidence sound;/audit-eval-evidencenow requires it, so authoring and auditing judge from one source./probe: writesprobes/{probe-slug}/probe.mdfrom the probe template, links the retained artifacts of the attested run, and records the Author's verdict without an actor identity./verify: routes evaluate to/evaland probe to/probe, and reports a capability gap only when the specialist is absent.spx/21-product-tree.enabler/35-evidence.enabler/39-eval-skill.enablerdeclares the shipped/eval; the newspx/21-product-tree.enabler/35-evidence.enabler/39-probe-skill.enablerdeclares/probe;spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enablermaps probe to/probe.Test plan
spx validation markdownjust eval-linksjust check-skillsjust docs-checkeval,eval-evidence-standards,audit-eval-evidence,probe, andverify; spec audits approved the three Nodes; assertion audits approved the eval-skill and verify-skill Nodes.Refs
Versioning
Validation
/reload-pluginsconfirms the change loads in a running sessionNeed help on this PR? Tag
@codesmith-botwith what you need.