Skip to content

feat(product-tree): ship the eval and probe skills - #723

Open
simonheimlicher wants to merge 27 commits into
mainfrom
work/change-868
Open

simonheimlicher wants to merge 27 commits into
mainfrom
work/change-868

Conversation

@simonheimlicher

@simonheimlicher simonheimlicher commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Ship the /eval and /probe skills in the product-tree plugin, so a verify round that needs eval or probe authoring reaches a skill that exists.
  • /verify routes an evaluate assertion to /eval and a probe assertion to /probe.

Background

The eval-skill Node and /verify already declared /eval, and no skill shipped. The evidence Node's known issue records that the probe route has no specialist. A verify round for a Change that needs an [eval] or [probe] assertion therefore stopped with capability-required and could not write or attest the evidence.

Changes

  • /eval: classifies the producer kind, binds the repository's own eval commands, checks freshness, converges one case at a time, and aggregates one deterministic suite result. It loads the new eval-evidence-standards skill.
  • eval-evidence-standards: the shared standard for what makes eval evidence sound; /audit-eval-evidence now requires it, so authoring and auditing judge from one source.
  • /probe: writes probes/{probe-slug}/probe.md from the probe template, links the retained artifacts of the attested run, and records the Author's verdict without an actor identity.
  • /verify: routes evaluate to /eval and probe to /probe, and reports a capability gap only when the specialist is absent.
  • Specs: spx/21-product-tree.enabler/35-evidence.enabler/39-eval-skill.enabler declares the shipped /eval; the new spx/21-product-tree.enabler/35-evidence.enabler/39-probe-skill.enabler declares /probe; spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler maps probe to /probe.
  • The product-tree changelog and the README plugin catalog list the new skills; the generated runtime trees follow the sources.

Test plan

  • spx validation markdown
  • just eval-links
  • just check-skills
  • just docs-check
  • Typed skill audits approved eval, eval-evidence-standards, audit-eval-evidence, probe, and verify; spec audits approved the three Nodes; assertion audits approved the eval-skill and verify-skill Nodes.

Refs

  • spx/21-product-tree.enabler/35-evidence.enabler/39-eval-skill.enabler
  • spx/21-product-tree.enabler/35-evidence.enabler/39-probe-skill.enabler
  • spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler
  • Change outcomeeng/changes#868

Versioning

  • product-tree: MINOR (new skills), written before merge

Validation

  • Touched-scope deterministic verification passes
  • /reload-plugins confirms the change loads in a running session

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@claude

claude Bot commented Oct 28, 2026

Copy link
Copy Markdown

BLOCKING [consistency]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/prompt.md

Reference: spx/local/generated-sources.toml declares spx//evals//prompt.md as a generated output whose sources include dist/claude/** and dist/codex/, generator = ["outcomeeng_evals/"], and regenerate = "just eval-materialize-prompts spx". CLAUDE.md Generated Sources in Verification Scope section states: Deterministic verification covers the complete changeset; each relation regeneration command backs regeneration parity, rerunning it leaves the committed generated extents byte-identical, and names this relation in scope for judgment once the changeset touches its declared sources.
Evidence: this PR edits dist/claude/product-tree/skills/verify/SKILL.md and dist/codex/product-tree/skills/verify/SKILL.md (both declared sources of the prompt-materialization relation and the producers named in evals/routing/eval.toml), changing the evaluate bullet to read: use skill product-tree:eval when the runtime skill catalog carries it ... /eval owns product command binding, producer-kind classification, and eval authoring ... When /eval is not installed, preserve the target artifact evaluate evidence shape and report capability-required with the subject. It also updates the result-example table to show a routed /probe row and an /eval-not-installed row. The committed evals/routing/prompt.md was left unchanged and still embeds the old producer text verbatim: use skill product-tree:eval with that canonical node target plus a JSON array ... /eval owns product command binding and producer-specialized eval authoring ... When the required capability is unavailable, preserve the target artifact evaluate evidence shape and report capability-required with the subject and required producer kind. Its embedded result table still shows the pre-PR two-row example (Node B producer rule | evaluate | /eval | path-bearing | eval capability required for the structured producer | ... | capability-required) instead of the new routed-/eval, routed-/probe, and /eval-not-installed rows now present in the producer. Rerunning just eval-materialize-prompts spx would not leave this file byte-identical, so the CI eval-prompts gate step (outcomeeng/validation/_steps.py:322, EVAL_PROMPTS_ARGV) fails against this commit, and the committed eval suite backing the Scenarios/Mappings assertions in verify-skill.md no longer evidences the current /verify producer.
Required: run just eval-materialize-prompts spx (or the equivalent regeneration) to refresh evals/routing/prompt.md from the updated dist/claude and dist/codex verify/SKILL.md, and include the regenerated file in this changeset.

@claude

claude Bot commented Oct 28, 2026

Copy link
Copy Markdown

BLOCKING [consistency]: src/plugins/product-tree/.claude-plugin/plugin.json:3

Reference: CLAUDE.md states: "Plugin version bumps: the PR opens without a version commit; spx/local/merging.md runs just bump, followed by just build-skills, as the last commit before the merge, after base synchronization and once review and checks are green on the final content... Ordinary commits never initiate or request a bump. NEVER hand-edit a manifest version field." spx/local/merging.md GitHub PR version finalization adds: "The PR opens without a version commit. /manage-pr applies this entry once the PR's content is final... and before the mutation-point guard for the merge."
Evidence: the PR already carries the version bump as its own commit, present at PR open — gh pr view 723 --json commits shows commit 1a19ab2 "build(product-tree): version 0.110.0" bumping "version": "0.109.5" to "version": "0.110.0" in src/plugins/product-tree/.claude-plugin/plugin.json, .codex-plugin/plugin.json, and both dist/ mirrors. The PR body's "Versioning" section confirms this was hand-written: "product-tree: MINOR (new skills), written before merge."
Required: drop the version-bump commit from this PR. Let the merge-time /manage-pr flow (spx/local/merging.md "GitHub PR version finalization") run just bump and just bump-check as the last commit once review has converged on final content, instead of shipping the bump inside the content PR.

BLOCKING [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/history.jsonl

Reference: the new eval-evidence-standards skill this PR ships requires run evidence to be "a complete, passing run for the current definition, case set, threshold, and producer... a run for a stale definition proves nothing about the current one." The new /eval skill this PR ships defines freshness as: a row is current only when "its recorded commit is an ancestor of HEAD and the changes since it are the suite's own history appends only."
Evidence: this PR edits the Scenario and Mapping assertions in verify-skill.md that carry the evals/routing/eval.toml eval link as their evidence, changing probe routing from "a probe protocol link" to "/probe", and regenerates evals/routing/prompt.md from the changed verify/SKILL.md producer files (commit "regenerate the routing eval prompt"). The newest row in evals/routing/history.jsonl still carries timestamp 2026-09-08T09-22-31Z and git_sha 4a8fdd8, dated about seven weeks before this PR's commits and recorded against the pre-change prompt and producer. The PR's test plan lists just eval-links (checks link existence, not run freshness) but no just eval run for this suite.
Required: run just eval spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/eval.toml against the updated prompt and producer and commit the resulting passing history.jsonl row before merge, or show why the changed Scenario and Mapping assertions' cited eval evidence is still current despite the prompt and producer change.

@simonheimlicher

Copy link
Copy Markdown
Contributor Author

Disposition of the findings in #723 (comment), head 6a91bc5:

  1. Consistency, blocking, the version commit pushed before the review of the final content: fixed. Commit 7f59841 reverts 1a19ab2. The version step returns as the last commit once the review and the checks are green on the final content.
  2. Evidence, blocking, the routing eval history of the verify-skill Node predating the skill and assertion edits: dropped under the finding disposition of the merging standards. The Operator ruled that evals run on demand only, so a history row older than the edits is no defect. No eval run and no Fixer round.
  3. Quality Gate failure at pytest in spx/15-validation.enabler/32-skill-reference.enabler/tests/test_skill_reference.mapping.l1.py: fixed in 6a91bc5, which empties the skill-reference exemption mapping now that /eval and /probe ship. The Node's tests pass, and so do the markdown, skill, catalog, eval-prompt and eval-link lanes.

@claude

claude Bot commented Oct 28, 2026

Copy link
Copy Markdown

BLOCKING [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/ISSUES.md:3

Reference: the known issue "Eval run history is stale for the routing suite" in that ISSUES.md says the prior staleness delta was frontmatter only, and sets the settlement condition as one passing run of the routing eval.toml with its row committed to history.jsonl. The new eval-evidence-standards skill (property 5, Run evidence) states a run for a stale definition proves nothing about the current one.
Evidence: this PR rewrites the real routing-rule prose of the producer src/plugins/product-tree/skills/verify/SKILL.md (the evaluate bullet and the worked example table, where Node B changes from capability-required to routed and new rows are added) and regenerates evals/routing/prompt.md to match, since eval.toml prompt_source.producers names exactly these producer files. That is not a frontmatter-only delta like the one the ISSUES.md entry currently describes. history.jsonl is untouched by this diff and the PR test plan only lists spx validation markdown, just eval-links, just check-skills, and just docs-check, none of which reruns an eval suite or checks freshness; eval_links.py only confirms the linked file exists. The eval-tagged scenario, mapping, and compliance assertions in verify-skill.md therefore rest on a run against the prior producer and prompt, not the one this PR ships.
Required: run the routing eval.toml on this PRs head, commit the passing row to history.jsonl, and update or close the ISSUES.md entry.

DEBT [consistency]: src/plugins/product-tree/skills/probe/SKILL.md:47

Reference: verify/SKILL.md line 94 states the routers output status vocabulary exhaustively as routed, capability-required, or blocked.
Evidence: probe/SKILL.md line 47 and line 104 has /probe report a fourth status, misrouted, back to its caller, naming the assertion and the verification type verify should select instead. /eval has no equivalent concept. Before this PR shipped /probe, misrouted was unreachable; now that verify actually invokes a shipped /probe, a misrouted report is a live outcome that verifys own three-value status enumeration and routing-table example have no row or handling rule for.
Required: document how verify incorporates a misrouted report from /probe into its own status output, or remove misrouted from /probe if verifys classification is meant to be treated as always authoritative by the time /probe runs.

@claude

claude Bot commented Oct 28, 2026

Copy link
Copy Markdown

BLOCKING [consistency]: spx/21-product-tree.enabler/35-evidence.enabler/39-eval-skill.enabler/eval-skill.md:14

Reference: src/plugins/product-tree/skills/understand/SKILL.md, assertion_model: "Assertions declare observable product output at the layer that owns the behavior, derived from decisions and specs."
Evidence: The diff adds the bullet: the product-tree plugin ships the generic eval specialist as the skill /eval, the skill /verify routes an evaluate assertion to -- the sentence ends on the dangling preposition "to" with no object, so it declares no verifiable claim. The identical pattern appears in the new file spx/21-product-tree.enabler/35-evidence.enabler/39-probe-skill.enabler/probe-skill.md line 14: the product-tree plugin ships the probe specialist as the skill /probe, the skill /verify routes a probe assertion to -- same truncation.
Required: Complete both sentences into a single declarative claim (for example, name what the assertion is routed to it for), so each assertion is parseable product truth rather than a cut-off fragment.

DEBT [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/verify-skill.md:14

Reference: src/plugins/product-tree/skills/eval-evidence-standards/SKILL.md, properties: run evidence requires "a complete, passing run for the current definition, case set, threshold, and producer"; assertion alignment requires "every expected-verdict field corresponds to a claim in the assertion."
Evidence: The Scenarios and Mappings assertions here cite the eval at evals/routing/eval.toml and now claim probe routes directly to /probe and that a misrouted report from /probe triggers reclassification. The PR own ISSUES.md diff in this same node states the routing eval history predates the shipped /probe skill, so there is no current passing run. In addition, evals/routing/cases.jsonl is unchanged by this PR and has no case exercising the new misrouted-reclassification branch that verify-skill.md now documents as Node F in its routing table, so even a fresh run of the existing case set would not cover that new behavior.
Required: Add a case covering the misrouted reclassification path to cases.jsonl, then record a current passing run in history.jsonl against the shipped /probe and /eval specialists, matching the settlement condition already named in ISSUES.md.

@claude

claude Bot commented Oct 28, 2026

Copy link
Copy Markdown

Reviewed the full diff (README, both dist/ trees, src/plugins/product-tree/skills/{eval,probe,eval-evidence-standards,audit-eval-evidence,verify}, the three spx/21-product-tree.enabler/35-evidence.enabler/* spec nodes, and outcomeeng/validation/skill_reference.py).
No findings. Checked: the new untagged top-level assertions in eval-skill.md/probe-skill.md match the documented spec-malleability routing-pending pattern (understand/SKILL.md: "Authoring declarations await selection untagged directly under ## Assertions"); the EXEMPTED_REFERENCES removal in skill_reference.py has no other derivation elsewhere in the repo to disagree with; the duplicated producer block in evals/routing/prompt.md is a declared generated extent (spx/local/generated-sources.toml) regenerated consistently for both dist copies; Claude vs. Codex tool-name templating (Skill/AskUserQuestion vs request_user_input) in the new eval/probe skills matches the established test skill convention exactly; and the misrouted reclassification path /verify adds for /probe is applied identically across dist/claude, dist/codex, and src. The stale routing-eval history is honestly self-disclosed in ISSUES.md with a concrete settlement condition, not a hidden gap.

@claude

claude Bot commented Oct 28, 2026

Copy link
Copy Markdown

BLOCKING [consistency]: src/plugins/product-tree/.claude-plugin/plugin.json:3

Reference: CLAUDE.md states: Plugin version bumps: the PR opens without a version commit; spx/local/merging.md runs just bump ... as the last commit before the merge, after base synchronization and once review and checks are green on the final content ... Ordinary commits never initiate or request a bump. NEVER hand-edit a manifest version field. spx/local/open-pr.md repeats this: The opening protocol writes no version. A changeset that warrants a bump opens its PR on verified content, and the version commit follows when the content is final.
Evidence: This content commit already changes the version field from 0.109.7 to 0.110.0 in src/plugins/product-tree/.claude-plugin/plugin.json and .codex-plugin/plugin.json, and carries the same bump into dist/claude/product-tree/.claude-plugin/plugin.json, dist/claude/product-tree/.codex-plugin/plugin.json, dist/codex/product-tree/.claude-plugin/plugin.json, and dist/codex/product-tree/.codex-plugin/plugin.json. This is not a version-only recovery PR since 990 additions of feature content accompany the bump, so the opens without a version commit exception for that path does not apply.
Required: Revert the manifest version fields to 0.109.7 in this changeset, both src and dist, both plugin.json variants, and let the merge protocol just bump step write the version commit as the last commit before merge, per spx/local/merging.md.

DEBT [evidence]: src/plugins/product-tree/skills/verify/SKILL.md:75

Reference: src/plugins/product-tree/skills/eval-evidence-standards/SKILL.md, added by this PR, property 3: Assertion alignment. Every expected-verdict field corresponds to a claim in the assertion, negative cases target the assertion failure mode, and the assertion cannot be unfulfilled while every case passes. Property 4: Falsifiability. One concrete mutation to the producer, never to the prompt or cases, must fail at least one case.
Evidence: This PR adds a misrouted reclassification branch to the probe route in verify/SKILL.md: when probe reports an assertion misrouted with the verification type to select, reclassify the assertion to that type, route it once to that type specialist, and report its row under that type, never invoking probe again and never surfacing misrouted as a status. The spec Mapping-table example reflects it as Node F claim probe reported misrouted routing to evaluate with status routed. The linked eval suite at spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/cases.jsonl has zero cases mentioning misrouted, so no case exercises this new branch. Deleting the branch from verify/SKILL.md would not fail any committed case, so the suite is not falsifiable for this behavior. Separately, and already tracked in spx/21-product-tree.enabler/35-evidence.enabler/ISSUES.md under The routed probe case has no committed eval run, the newest history.jsonl row predates this PR changes to probe and verify, so even the existing routed-probe case has no current passing run against the producer this PR ships.
Required: Add a case to cases.jsonl that exercises the probe-reports-misrouted reroute, expecting the assertion to land under the named verification type specialist with status routed and no misrouted status in the result, and record a fresh passing history.jsonl run of the suite against the current producer before treating the routing scenario and mapping assertions as evidenced.

@simonheimlicher

Copy link
Copy Markdown
Contributor Author

Disposition of the findings in #723 (comment), head 40a9591:

  1. Consistency, blocking, the manifest version fields: dropped. spx/local/merging.md, section "GitHub PR version finalization", makes the version commit the last commit before the merge, written after the review and the checks are green on the final content. Commit 40a9591 is that commit, and just bump-check origin/main passes on it. The version step stays after every content commit.
  2. Evidence, debt: the missing case is fixed. evals/routing/cases.jsonl gains one case for the probe misrouted reroute, and the version commit follows it on the next push. The stale routing eval history stays dropped, because evals run on demand only.

@claude

claude Bot commented Oct 28, 2026

Copy link
Copy Markdown

BLOCKING [consistency]: src/plugins/product-tree/skills/audit-eval-evidence/SKILL.md:19

Reference: this same file states "NEVER issue a finding the evidence model does not support — drop an unbacked finding rather than reject the eval evidence for it" (line 29) and its success criteria require the suite be "judged on all five evidence properties with none skipped — producer coupling, oracle independence, assertion alignment, falsifiability, and run evidence" (line 233); the workflow body likewise says to "Check the five evidence properties in strict order" (line 38).
Evidence: the prerequisite this PR adds says to load product-tree:eval-evidence-standards and "Apply its complete producer-coupling, oracle-independence, assertion-alignment, falsifiability, run-evidence, and case-provenance rules" — six items, not five. The bundled references/evidence-model.md rejection_categories table (unchanged by this PR) has no rule slug for a case-provenance violation (only missing-artifact, missing-target, producer-coupling, oracle-leakage, assertion-alignment, falsifiability, run-evidence, no-eval-assertions), and no workflow step checks case provenance. A real case-provenance violation (a case input copied from the producers own text/output, which eval-evidence-standards's <case_provenance> section calls out as invalid) therefore has no supported rule to file it under, so the NEVER constraint forces the auditor to drop it — leaving the newly declared case-provenance requirement unenforceable even though /eval authoring is told the standard covers it.
Required: add a case-provenance rule to the rejection_categories table (and a corresponding check step) in references/evidence-model.md, and reconcile the "five evidence properties" wording in the workflow body and success criteria with the six-item prerequisite list, so /eval and /audit-eval-evidence judge from the same definition of sound evidence as the prerequisite itself claims.

DEBT [consistency]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/ISSUES.md:9

Reference: src/plugins/product-tree/skills/understand/SKILL.md states "A change invalidates another file when it removes a symbol that file references, enforces a rule it violates, falsifies its guidance, or causes a gate, audit, or review to expose its imperfection. Location never licenses deferral."
Evidence: this node's ISSUES.md still says the only uncovered delta since the last passing eval run is "frontmatter only and touches no routing rule" (lines 6-10). This PR's own diff to evals/routing/prompt.md and evals/routing/cases.jsonl in this same node adds real routing logic — the probe-bullet "except where the probe bullet reclassifies it" reroute behavior and the new reroutes-probe-misrouted-report case — none of which is covered by any committed history.jsonl row. The sibling note in 35-evidence.enabler/ISSUES.md already states the broader, accurate reason ("the newest row of its history.jsonl predates the /probe skill"), but this node's own ISSUES.md was left with the narrower, now-false "frontmatter only" explanation.
Required: update this ISSUES.md entry to describe the current staleness reason (the delta now includes the probe-misrouted reroute routing logic and a new case, not just frontmatter), or point it at the 35-evidence.enabler/ISSUES.md entry that already states this correctly.

@simonheimlicher

Copy link
Copy Markdown
Contributor Author

Reply to the review at #723 (comment), head 1d59365.

BLOCKING consistency, audit-eval-evidence/SKILL.md: fixed in 4eedb82. Step 3b now judges case provenance beside oracle independence, and a copied case input is a REJECT finding under the new case-provenance rule. references/evidence-model.md gains the <case_provenance> procedure and the case-provenance row in its rejection table. The body and the success criteria name the five properties plus the case-provenance rule, which matches the six-rule prerequisite. A typed skill audit of the committed head 050981a sealed approved with zero findings (run 2026-10-28_11-13-57-926-bd478d041645).

DEBT consistency, 69-verify-skill.enabler/ISSUES.md: fixed in 050981a. The entry now names both deltas since the newest passing routing run: the earlier frontmatter rewrite and the probe-bullet reroute with its new case.

The version commit follows as the last commit, 70f7a28, and just bump-check origin/main passes on it.

@claude

claude Bot commented Oct 28, 2026

Copy link
Copy Markdown

BLOCKING [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/verify-skill.md

Reference: eval-evidence-standards (shipped in this PR), Run evidence property: "The committed history holds a complete, passing run for the current definition, case set, threshold, and producer ... a run for a stale definition proves nothing about the current one." Also src/plugins/product-tree/skills/understand/SKILL.md, touched_file_debt: "ALWAYS: fix debt that the current change causes, surfaces, or invalidates ... Record and proceed only for work independent of the current change in a surface the change neither touches nor invalidates."
Evidence: This PR edits evals/routing/prompt.md (adds the probe-reroute bullet) and evals/routing/cases.jsonl (adds the reroutes-probe-misrouted-report case), both of which back the eval tag on the Scenarios and Mappings assertions in verify-skill.md that this PR also rewrites to claim /probe routing and the new reroute behavior. The 69-verify-skill.enabler/ISSUES.md entry added by this PR states: "No committed run covers the current materialized prompt or that case," and 35-evidence.enabler/ISSUES.md discloses the same gap against the now-shipped /probe. This change is itself the diff that touched prompt.md and cases.jsonl and invalidated the prior run, so the touched_file_debt deferral-to-ISSUES.md path does not apply, since that path is reserved for debt in a surface the change does not touch.
Required: Run the routing eval suite (just eval spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/eval.toml) against the shipped /probe and /eval, including the new reroute case, and commit a passing history.jsonl row before merge, or withhold the [eval] tag on the affected assertions until that evidence exists.

DEBT [consistency]: spx/21-product-tree.enabler/68-audit.enabler/32-audit-eval-evidence.enabler/audit-eval-evidence.md

Reference: Compliance assertion in that spec: "ALWAYS: check oracle independence, assertion alignment, falsifiability, and run evidence after producer coupling passes ([audit])."
Evidence: src/plugins/product-tree/skills/audit-eval-evidence/SKILL.md and its references/evidence-model.md are changed in this PR to add a sixth checked property, case provenance: "Then read each case input against the producers text and its recorded output, applying the evidence models case_provenance procedure. A case input copied from either is REJECT." The governing spec at 32-audit-eval-evidence.enabler/audit-eval-evidence.md is untouched by this PR and still enumerates only oracle independence, assertion alignment, falsifiability, and run evidence, with no mention of case provenance.
Required: Add a Compliance assertion to 32-audit-eval-evidence.enabler/audit-eval-evidence.md naming the case-provenance check, so the governing spec enumerates the audits actual checked properties.

@simonheimlicher

Copy link
Copy Markdown
Contributor Author

Reply to the review at #723 (comment), head 70f7a28.

BLOCKING evidence, 69-verify-skill.enabler/verify-skill.md: dropped. Evals run only on operator dispatch, so a missing history row for the changed routing prompt and case is no defect this changeset owes, whatever severity a review gives it. The ISSUES.md entry of 69-verify-skill.enabler records the unrun suite, the two deltas since its newest passing run, and the settlement condition.

DEBT consistency, 32-audit-eval-evidence.enabler/audit-eval-evidence.md: fixed in 6a2d6a0. The spec now declares that the audit checks case provenance beside oracle independence. A typed spec audit of that head returned APPROVED with no findings.

The version commit follows as the last commit, b715427, and just bump-check origin/main passes on it.

The eval-skill Node declares that the product-tree plugin ships the
generic eval specialist as `/eval`, the skill `/verify` routes an
evaluate assertion to.

Verification: not-run
Refs: outcomeeng/changes#868
Add `/eval`, the generic eval specialist `/verify` routes an evaluate
assertion to, and `eval-evidence-standards`, the standard it loads. The
generated runtime trees follow the sources.

Verification: not-run
Refs: outcomeeng/changes#868
`/audit-eval-evidence` requires `eval-evidence-standards`, the standard
`/eval` loads, so eval authoring and eval-evidence auditing judge from
one source. The generated runtime trees follow the sources.

Verification: not-run
Refs: outcomeeng/changes#868
Regenerate the plugin catalog section of the README with `just docs`
after `/eval` and `eval-evidence-standards` joined the product-tree
plugin.

Verification: passing
Refs: outcomeeng/changes#868
State the objective as the output and add success criteria that name
the five properties a sound eval suite holds. The generated runtime
trees follow the sources.

Verification: passing
Refs: outcomeeng/changes#868
Add the probe-skill Node, which declares that the product-tree plugin
ships `/probe`, the skill `/verify` routes a probe assertion to, and
states what `/probe` writes and records.

Verification: passing
Refs: outcomeeng/changes#868
Add `/probe`, the specialist `/verify` routes a probe assertion to. It
writes the probe protocol, records the Author's verdict with the
retained artifacts of the attested run, and lists in the plugin
catalog. The generated runtime trees follow the sources.

Verification: passing
Refs: outcomeeng/changes#868
The verify-skill Node maps the probe verification type to `/probe`,
the specialist the probe-skill Node declares.

Verification: passing
Refs: outcomeeng/changes#868
`/verify` invokes `/eval` and `/probe` when the runtime skill catalog
carries them and reports a capability gap only when a specialist or a
producer kind is absent. The result example shows the routed rows. The
generated runtime trees follow the sources.

Verification: passing
Refs: outcomeeng/changes#868
`/eval` classifies the producer kind and authors the suite under
`eval-evidence-standards`; no producer specialist ships, so the skill
composes none. `/verify` reports a capability gap only when `/eval` is
absent. The generated runtime trees follow the sources.

Verification: passing
Refs: outcomeeng/changes#868
Materialize the routing eval prompt from the verify skill again after
the skill began routing evaluate and probe assertions to `/eval` and
`/probe`, so `eval-prompts` finds no drift from the producer source.

Verification: passing
Refs: outcomeeng/changes#868
The eval and probe skills ship in the product-tree plugin, so the
exemptions for /eval, /probe, product-tree:eval and product-tree:probe
no longer name skills that are absent. The exemption mapping is empty.

Refs: spx/15-validation.enabler/32-skill-reference.enabler
/probe reports a claim that a test, eval, or audit settles as
misrouted, naming the verification type to select. /verify now
reclassifies that assertion to the named type, routes it once to that
type's specialist, and never invokes /probe for it again, so the
result keeps its three statuses. The result example gains a routed row
for such a claim, the plugin changelog records the rule, and the
routing eval prompt is regenerated from the changed producer.

Refs: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler
The product-tree plugin ships /probe, so the entry that records a probe
route with no specialist no longer holds. The entry now records the
remaining gap, a routing eval with no committed passing run against the
shipped specialist, and keeps one settlement condition.

Refs: spx/21-product-tree.enabler/35-evidence.enabler
Name the object in the first assertion of the eval skill and the probe
skill in plain order: the skill to which /verify routes the assertion.

Verification: passing (spx validation markdown, just eval-links,
spx spec status --format json, just fmt)
Adds a routing case where the probe specialist reports an assertion
misrouted and names the test type, so the reroute branch of the
verify skill has a case that fails when the branch is removed.

Verification: passing
The audit prerequisite names six rules from the shared standard, but
the audit body and its rule catalog carried five properties and no
case-provenance rule. Step 3b now judges case provenance beside oracle
independence, and the evidence model gains the procedure and the
case-provenance rule slug.

Verification: passing (check-skills, docs-check, validation markdown, eval-links)
The routing eval note named a frontmatter-only delta since its newest
passing run. The probe reroute and its new case lie in the delta too.

Verification: not-run
The eval-evidence audit judges case provenance beside oracle
independence, so the node declares the check.

Verification: passing (validation markdown, spec status, eval-links)
@claude

claude Bot commented Oct 28, 2026

Copy link
Copy Markdown

BLOCKING [consistency]: src/plugins/product-tree/.claude-plugin/plugin.json:3

Reference: CLAUDE.md:321 — "Plugin version bumps: the PR opens without a version commit; spx/local/merging.md runs just bump ... as the last commit before the merge ... Ordinary commits never initiate or request a bump. NEVER hand-edit a manifest version field." Also spx/local/merging.md:76 — "The PR opens without a version commit."
Evidence: The PR's commit list includes a final commit a1c6ece78e1846f4fbade3c7eb8b94f740bf68b5 titled "build(product-tree): version 0.110.0" that hand-edits version from 0.109.8 to 0.110.0 in src/plugins/product-tree/.claude-plugin/plugin.json, src/plugins/product-tree/.codex-plugin/plugin.json, and their generated dist/claude and dist/codex mirrors. This is exactly the hand-authored version/build commit the merge policy says must not exist when a PR opens — the bump is reserved for /merge's own just bump step, run after base synchronization and once review is green, as the literal last commit before merge.
Required: Drop the version-bump commit from this PR (restore the manifests to 0.109.8 and the corresponding dist/ output), and let the /merge flow apply just bump / just build-skills / just bump-check as its own last commit per spx/local/merging.md.

DEBT [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/cases.jsonl:10

Reference: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/prompt.md:8 explicitly introduces available_specialists to the producer in the prompt preamble ("The request's available_specialists array is the authoritative projection of the runtime skill catalog for this invocation.") because the embedded SKILL.md body only speaks of it abstractly ("the runtime skill catalog").
Evidence: The new case reroutes-probe-misrouted-report adds an input field specialist_report: {"specialist":"/probe","status":"misrouted","verification_type_to_select":"test"}, but neither the embedded verify/SKILL.md body nor the prompt preamble ever names this field or its shape — the SKILL body only describes the concept in prose ("When /probe reports an assertion misrouted with the verification type to select..."). Unlike available_specialists, which got an explicit preamble sentence precisely because the producer text alone was too abstract to bind to a JSON key, specialist_report gets no equivalent introduction, so the producer must guess the wire shape of a brand-new input channel from prose alone. Both spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/ISSUES.md and spx/21-product-tree.enabler/35-evidence.enabler/ISSUES.md already disclose that no committed run covers this case, so the ambiguity has not yet been exercised.
Required: Add a preamble sentence in prompt.md naming specialist_report and its fields, matching the treatment already given to available_specialists, before the routing eval's committed run is recorded.

The routing prompt named the available_specialists field and left the
specialist_report field of the reroute case to be guessed. The template
and its materialized prompt now name the field and its members.

Verification: passing (materialize-prompts check, validation markdown, eval-links)
@simonheimlicher

Copy link
Copy Markdown
Contributor Author

Reply to the review at #723 (comment), head a1c6ece.

BLOCKING consistency, src/plugins/product-tree/.claude-plugin/plugin.json: dropped. spx/local/merging.md, section "GitHub PR version finalization", makes the version commit the last commit before the merge, written by just bump, followed by just build-skills, once review and checks are green on the final content. It states that the PR opens without a version commit, and the merge step applies the commit after content review. Commit 7cd7b63 is that commit, written by just bump origin/main and checked by just bump-check origin/main, and it is the last commit on the branch.

DEBT evidence, 69-verify-skill.enabler/evals/routing/cases.jsonl: fixed in 45d4c4f. prompt.template.md and the materialized prompt.md now name the specialist_report field and its members specialist, status, and verification_type_to_select, as they name available_specialists. just eval-materialize-prompts-check passes on the node's evals.

@claude

claude Bot commented Oct 28, 2026

Copy link
Copy Markdown

DEBT [consistency]: spx/21-product-tree.enabler/65-apply.enabler/ISSUES.md:33

Reference: the entry "apply Step 5 names a skill the product-tree plugin does not ship" (line 33) states as its Evidence (line 35) that src/plugins/product-tree/skills/apply/SKILL.md:206 names /eval while "no such skill ships in the product-tree plugin", and its Settlement condition (line 39) is "Step 5 names only /verify's own eval routing, or the unshipped skill name is gone." Evidence: this changeset ships /evalin the product-tree plugin (newsrc/plugins/product-tree/skills/eval/SKILL.md), which falsifies the entry's central evidentiary claim, yet neither the entry nor apply/SKILL.md:206("It routes eval work through/evalwhen that capability is installed") was touched. The same PR narrows an analogous entry elsewhere once its premise no longer held —spx/21-product-tree.enabler/35-evidence.enabler/ISSUES.mdwas rewritten from "The probe route has no specialist" to "The routed probe case has no committed eval run" after/probeshipped — but the sibling65-apply.enablerentry about/evalwas left as if/evalstill does not ship. Required: update or resolve the65-apply.enabler/ISSUES.mdentry (or Step 5's wording) now that/evalships, the same way the35-evidence.enabler/ISSUES.md` probe entry was narrowed in this same changeset.

DEBT [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/cases.jsonl

Reference: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/ISSUES.md:3-15 ("Eval run history is stale for the routing suite") states the newest passing row of history.jsonl "predates two deltas," the second being that "the probe-route change ... added the probe-bullet reroute to the same skill, rematerialized prompt.md again, and added the reroutes-probe-misrouted-report case to cases.jsonl. No committed run covers the current materialized prompt or that case."
Evidence: the diff adds the reroutes-probe-misrouted-report case (new last line of cases.jsonl) and rewrites the route-specialist probe bullet in verify/SKILL.md and its materialized prompt.md/prompt.template.md to add the reroute branch, but history.jsonl carries no run exercising this new branch or case, so the behavioral change this PR ships to /verify's routing has no suite-run evidence backing it.
Required: run just eval spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/eval.toml and commit the passing row, or continue tracking this explicitly as the already-filed known issue until that run lands.

DEBT [consistency]: src/plugins/product-tree/skills/audit-eval-evidence/SKILL.md:38

Reference: the new eval-evidence-standards skill this PR ships enumerates the producer kind as "skill, agent, classifier, script, or command" (src/plugins/product-tree/skills/eval-evidence-standards/SKILL.md:25), matching the producer table this same file carries two lines below the touched line (audit-eval-evidence/SKILL.md:63: "The skill, agent, classifier, script, or command the assertion claims emits the structured verdict"). The new <prerequisites> block states that where /eval-evidence-standards and this audit "both speak to a property, the standard governs."
Evidence: this PR edits line 38 of audit-eval-evidence/SKILL.md (appending the case-provenance sentence) but leaves its producer-coupling enumeration as "the actual producing skill, agent, classifier, or script" — omitting "command," which the sibling table in the same file and the newly shipped shared standard both include as a valid producer kind for /eval to author against.
Required: add "or command" to the producer-coupling sentence at audit-eval-evidence/SKILL.md:38 so the audit's own instruction text agrees with its producer table and with /eval-evidence-standards.

The eval-evidence audit named skill, agent, classifier, and script as
producer kinds, while its producer table and the shared standard also
name command. All four sentences of the bundle now list the same kinds.

Verification: passing (check-skills, docs-check)
The apply note said no eval skill ships. The plugin ships /eval, so the
entry now records the guarded name and its settlement condition.

Verification: passing (validation markdown, eval-links)
@claude

claude Bot commented Oct 28, 2026

Copy link
Copy Markdown

DEBT [consistency]: spx/21-product-tree.enabler/68-audit.enabler/32-audit-eval-evidence.enabler/audit-eval-evidence.md:21

Reference: spx/13-infrastructure.enabler/25-eval-harness.enabler/57-producer-coupled-skill-evals.adr.md:33 says evals this harness runs for skill, agent, classifier, script, or command behavior are coupled to the real producer -- the producer-kind taxonomy already includes command, unchanged by this PR. src/plugins/product-tree/skills/understand/SKILL.md:202 says assertions declare observable product output at the layer that owns the behavior, derived from decisions and specs, never from tests or code.
Evidence: This PR edits audit-eval-evidence/SKILL.md and references/evidence-model.md in several places to add command to the producer-kind enumeration -- SKILL.md now reads NEVER approve prompt-only simulation as evidence for skill, agent, classifier, script, or command behavior, and evidence-model.md now reads the actual producing skill, agent, classifier, script, or command. This PR also adds a new top-level assertion to this same spec file at line 14. The three pre-existing audit-tagged assertions it left untouched still enumerate only 3 or 4 of the 5 producer kinds: line 9 names only producing skill, agent, or classifier; line 21 names only skill, agent, classifier, or script; line 22 names only skill, agent, or classifier behavior. The ADR already licensed command before this PR, so the spec was already stale on script and command, and this PR widens the gap between what the SKILL.md now does and what the spec that governs it says it does.
Required: Update lines 9, 21, and 22, and the SO THAT line, to enumerate the full producer-kind set -- skill, agent, classifier, script, command -- matching the ADR and the SKILL.md this PR already updated.

DEBT [consistency]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/verify-skill.md:16

Reference: src/plugins/product-tree/skills/understand/SKILL.md:202 says assertions declare observable product output at the layer that owns the behavior, derived from decisions and specs, never from tests or code.
Evidence: This PR adds a new routing rule to the probe bullet of src/plugins/product-tree/skills/verify/SKILL.md: when probe reports an assertion misrouted with the verification type to select, verify reclassifies the assertion to that type, routes it once to that type specialist per these bullets, and reports its row under that type with the status that route yields; it never invokes probe for it again, and no misrouted status reaches the result. This rule is backed by a new eval case, reroutes-probe-misrouted-report, added to evals/routing/cases.jsonl. But verify-skill.md own Scenario assertion at line 16 and Mapping assertion at line 20, the only assertions linked to this eval.toml, describe only verify initial classification of an assertion into test, eval, probe, or audit; neither states the distinct correction rule where a specialist own report overrides and reroutes an already-routed probe classification to test or evaluate. The new case is evidence for a rule the spec never states as an assertion.
Required: Add an assertion, Scenario or Compliance, to verify-skill.md stating the misrouted-reroute rule, linked to evals/routing/eval.toml, so the new eval case has a governing assertion to attach to.

The audit-eval-evidence evidence model restated the case-provenance
rule that eval-evidence-standards owns. The section now defers the
rule to the standard and keeps the audit procedure and the rejection
value.

Verification: passing
@claude

claude Bot commented Oct 28, 2026

Copy link
Copy Markdown

DEBT [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/eval.toml

Reference: src/plugins/product-tree/skills/eval-evidence-standards/SKILL.md property 5, "Run evidence": "The committed history holds a complete, passing run for the current definition, case set, threshold, and producer ... a run for a stale definition proves nothing about the current one."
Evidence: The Scenarios and Mappings assertions of 69-verify-skill.enabler carry eval, and this PR both rematerializes prompt.md (the probe-bullet reroute text) and adds the reroutes-probe-misrouted-report case to cases.jsonl. The ISSUES.md added by this PR for 69-verify-skill.enabler states plainly: "No committed run covers the current materialized prompt or that case." The PR description test-plan checklist lists spx validation markdown, just eval-links, just check-skills, and just docs-check, but names no just eval run of the routing suite.
Required: Run just eval spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/eval.toml against the shipped /probe and /eval and commit the passing history.jsonl row, or state the gap explicitly in the PR description so reviewers see it at merge time (it is already tracked in ISSUES.md).

DEBT [architecture]: src/plugins/product-tree/skills/eval/SKILL.md:6

Reference: src/plugins/instructions/skills/skill-standards references/command-capabilities.md: "ALWAYS: grant the narrowest allowed-tools the skill task needs, restricting bash to specific verb patterns."
Evidence: Line 6 frontmatter grants the ask_user tool (rendered as request_user_input in the Codex copy), but no step in the eval workflow ever asks the operator anything. Steps 1 through 8 cover argument parsing, context loading, assertion extraction, command binding, producer-kind routing, freshness inspection, serial convergence, and reporting, with no ask_user or blocked-pending-question path anywhere in the file. Contrast with the sibling probe skill (src/plugins/product-tree/skills/probe/SKILL.md Step 4), which grants the same tool and actually exercises it: "ask through ask_user when that tool is available; otherwise stop with blocked."
Required: Either add the missing step that uses ask_user (for example, when a repository declares no eval command or a producer-kind classification is ambiguous), or drop the ask_user grant from /eval allowed-tools.

The audit-eval-evidence skill loads the shared standard in its audit
workflow step instead of a separate prerequisites section. Its
evidence-model reference cites the standard for the property
definitions and artifact roles, and keeps the audit procedures and the
two audit-only artifacts.

Verification: passing
The audit-eval-evidence evidence model and workflow restated the
oracle-independence property that eval-evidence-standards owns. Both
now cite the standard and keep the audit-level case and the rejection
value.

Verification: passing
@claude

claude Bot commented Oct 28, 2026

Copy link
Copy Markdown

DEBT [consistency]: spx/21-product-tree.enabler/35-evidence.enabler/39-probe-skill.enabler/probe-skill.md:11

Reference: the sibling node spx/21-product-tree.enabler/35-evidence.enabler/39-eval-skill.enabler/eval-skill.md ships in this same PR for /eval. Its compliance-style assertions sit under a ### Compliance subsection and each carries a routed ([audit]) tag, e.g. "the generic eval specialist owns product command binding, producer-kind routing, freshness inspection, serial one-case convergence, and deterministic aggregation ([audit])". src/plugins/product-tree/skills/understand/SKILL.md line 215 states the model: "Authoring declarations await selection untagged directly under ## Assertions ... alongside routed subsections if present ... Once routed, a spec-malleable assertion may omit its tag; every harder assertion carries exactly one."
Evidence: probe-skill.md has four assertions, including three compliance-shaped ones (the protocol-writing rule, the verdict-recording rule, and the NEVER rule against standing in for a deterministic or agentic verdict), and all four sit directly under plain ## Assertions with no ### Compliance subsection and no tag at all, i.e. still awaiting selection, unlike the parallel eval-skill.md assertions shipped in the same changeset. The PR description test plan confirms the gap by omission: it states that assertion audits approved the eval-skill and verify-skill Nodes, naming only two of the three new or changed Nodes; probe-skill.md is absent from that list. No ISSUES.md entry inside 39-probe-skill.enabler or in the parent 35-evidence.enabler/ISSUES.md tracks this, unlike the other known gaps this PR explicitly discloses, such as the stale routing-eval history and the unrun probe case.
Required: route probe-skill.md compliance-shaped assertions the same way eval-skill.md routes its own, with a ### Compliance subsection and ([audit]) tags, or record the deferral explicitly in 35-evidence.enabler/ISSUES.md the way the sibling gaps are already recorded.

Name the five gates in the audit-eval-evidence objective. Cut the
restatements of coupling, alignment, falsifiability, and run-evidence
properties from the skill and its evidence model, and point each
section at the shared standard's property.

Verification: passing
@simonheimlicher

Copy link
Copy Markdown
Contributor Author

Reply to the review of 7cd7b63070d6da22167597e929fbe19aa632af85.

  • Finding 1: fixed in f649915541e572a50e6dd1ce603d810fdc2c9b2a.
  • Finding 3: fixed in 44662b546daa51618b5ecc34a0a3260a23f1c53d.
  • Finding 2: tracked. The review asks to "run just eval spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/eval.toml and commit the passing row, or continue tracking this explicitly as the already-filed known issue until that run lands." This PR continues tracking it as the already-filed known issue, in the entry under the heading at line 3 of spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/ISSUES.md.

@sonarqubecloud

Copy link
Copy Markdown

@claude

claude Bot commented Oct 28, 2026

Copy link
Copy Markdown

DEBT [consistency]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/verify-skill.md

Reference: CLAUDE.md, "Product tree navigation: declaration and inverse" — "The methodology declares forward: PDR/ADR → spec assertion → test link → test file → import → implementation file" and "An implementation file in the diff that no test imports has no governing spec assertion — a coverage gap the PR is shipping. Specs declare; tests verify; code complies. Surface the gap; do not invent a node to load."
Evidence: src/plugins/product-tree/skills/verify/SKILL.md adds a whole new decision branch — "When /probe reports an assertion misrouted with the verification type to select, reclassify the assertion to that type, route it once to that type's specialist per these bullets, and report its row under that type with the status that route yields; never invoke /probe for it again, and no misrouted status reaches the result" — and a new eval case reroutes-probe-misrouted-report in evals/routing/cases.jsonl exercises exactly this reroute. spx/.../69-verify-skill.enabler/verify-skill.md Assertions were edited in this PR (only the existing Scenarios/Mappings wording swapped "the protocol link or missing probe-authoring capability" for "/probe") but no assertion anywhere states that /verify reclassifies and reroutes an assertion when a specialist reports it misrouted. grep -rln misrouted spx/ finds the term only in 69-verify-skill.enabler/ISSUES.md, evals/routing/prompt.md, and evals/routing/cases.jsonl — never in an Assertions section of any spec file. The companion spx/.../39-probe-skill.enabler/probe-skill.md has the same gap: its four Assertions (shipped /probe, protocol write, verdict recording, attested-result boundary) never state that /probe reports misrouted for a claim it does not own, even though /probe's own success criteria make that a first-class contract: "Every probe assertion's claim is one only an executed observation settles, or the assertion is reported misrouted with the verification type to select."
Required: Add an Assertion to verify-skill.md (and/or probe-skill.md) declaring the misrouted-reroute contract, so the behavior /verify and /probe implement and the new eval case exercises has a governing spec assertion rather than outrunning the spec.

DEBT [evidence]: spx/21-product-tree.enabler/35-evidence.enabler/69-verify-skill.enabler/evals/routing/eval.toml

Reference: src/plugins/product-tree/skills/eval-evidence-standards/SKILL.md, <properties> item 5 — "Run evidence. The committed history holds a complete, passing run for the current definition, case set, threshold, and producer... a run for a stale definition proves nothing about the current one."
Evidence: this PR's own 69-verify-skill.enabler/ISSUES.md edit states the routing suite's history.jsonl newest passing row predates two deltas since: the probe-route change of this very Change "added the probe-bullet reroute to the same skill, rematerialized prompt.md again, and added the reroutes-probe-misrouted-report case to cases.jsonl. No committed run covers the current materialized prompt or that case." The PR ships /probe and the reroute logic /verify now depends on, and adds the eval case meant to prove it, but does not run just eval .../evals/routing/eval.toml to produce a passing history.jsonl row against the shipped specialist — the PR's own test-plan checklist lists only spx validation markdown, just eval-links, just check-skills, and just docs-check, none of which execute the suite.
Required: Run the routing eval suite against the shipped /probe and /eval and commit the resulting passing history.jsonl row, or keep this explicitly deferred per the stated settlement condition before relying on the routing behavior as verified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant