[Copilot]

Evaluations could select task and judge models but not their reasoning effort, preventing reproducible model-and-effort comparisons. This adds validated effort configuration for Copilot SDK task, responder, and LLM-as-judge sessions.

• Configuration

  • Add config.reasoning_effort for task and responder sessions.
  • Add config.judge_reasoning_effort as the prompt-grader default.
  • Allow prompt graders in eval, task, and checkpoint definitions to override that default with config.reasoning_effort; agent effort remains eval-level.
  • Restrict values to low, medium, high, xhigh, and max.
  • Reject effort settings for non-copilot-sdk executors in Go loading and eval/task schema checking.

• Execution

  • Forward effort on Copilot session creation and resumption, including follow-ups, responders, independent/continued-session/checkpoint/pairwise judges.
  • Validate explicit hosted-model effort against the Copilot SDK model catalog before creating or resuming a session. Unknown models, unavailable metadata, and unsupported values fail with actionable errors. Custom providers bypass the hosted catalog and retain direct forwarding.
  • A live probe demonstrated that embedded CLI 1.0.64-0 accepted gpt-5-mini/max despite advertising only low, medium, high; the metadata guard prevents that silent mismatch.

• Reproducibility

  • Record selected task and judge effort in evaluation output metadata.
  • Include both settings and judge model in cache keys.
  • Regrading updates judge effort metadata, clearing a previously pinned value when omitted.

• Coverage and docs

  • Add YAML/schema/executor validation, precedence, create/resume forwarding and runtime-error propagation, live unsupported-effort rejection, BYOM, pairwise, responder, cache-separation, and regrade metadata coverage.
  • Update README and published eval/schema/grader references; remove unrelated generated dashboard hash changes.
  • Verified Go vet, full and short Go suites, golangci-lint, docs build, and gated live unsupported-effort test.

config:
  executor: copilot-sdk
  model: gpt-5
  reasoning_effort: high
  judge_model: gpt-5-mini
  judge_reasoning_effort: low

graders:
  - type: prompt
    config:
      reasoning_effort: medium # overrides judge_reasoning_effort
      prompt: Evaluate the response.

Resumed-session effort remains enabled as required by #571 and supported by ResumeSessionConfig.ReasoningEffort; it supersedes the earlier proposal to exclude continued-session graders.

Fixes #571

[Copilot]

Copilot wasn't able to review any files in this pull request.

[Copilot]

Review details

• Files reviewed: 23/23 changed files
• Comments generated: 3
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (3)

Previously missed (2) — in code that hasn't changed since the last review.

internal/cache/cache.go:57

• CacheKey now includes reasoning_effort fields but still omits config.judge_model. Changing judge_model can change prompt-grader results, so the cache key should include judge_model as well to avoid reusing cached outcomes graded by a different judge model.

	if err := writeString(h, spec.Config.ModelID); err != nil {
		return "", err
	}
	if err := writeString(h, spec.Config.ReasoningEffort); err != nil {
		return "", err

README.md:1164

• This sentence says a prompt grader overrides the judge default with config.reasoning_effort, which is ambiguous with the eval-level config.reasoning_effort. The override is graders[].config.reasoning_effort (prompt grader config).

Pin `reasoning_effort` and `judge_reasoning_effort` to `low`, `medium`, `high`, `xhigh`, or `max` when benchmarking model-and-effort combinations. Omit either setting to preserve the Copilot SDK/model default. A `prompt` grader can override the judge default with `config.reasoning_effort`.

README.md:1071

• The example comment suggests using the mock executor when not pinning reasoning effort, but copilot-sdk is still valid without reasoning_effort/judge_reasoning_effort. This reads like an incorrect requirement and may confuse readers.

  executor: copilot-sdk   # use mock when not pinning reasoning effort

• Files reviewed: 23/23 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Shayne Boyer (spboyer)]

Owner review: requesting changes because the existing reasoning-effort blockers remain unresolved. Do not apply judge_reasoning_effort to continue_session graders, reject prompt-grader reasoning_effort under non-copilot-sdk executors, include judge_model in the relevant cache contract, and correct the misleading README executor guidance. This PR is also still draft and currently has only the CLA check.

[Copilot]

Owner review: requesting changes because the existing reasoning-effort blockers remain unresolved. Do not apply judge_reasoning_effort to continue_session graders, reject prompt-grader reasoning_effor...

Addressed in fe6e7ae:

• applyDefaults no longer injects judge_reasoning_effort into prompt graders with continue_session: true (the resumed task session's effort is fixed at creation).
• EvalSpec.Validate now rejects grader-level config.reasoning_effort unless executor: copilot-sdk, mirrored by a new conditional rule in schemas/eval.schema.json.
• CacheKey now includes spec.Config.JudgeModel alongside the effort settings.
• README executor comment now reads "required when using reasoning_effort/judge_reasoning_effort", and the reasoning-effort docs note the continue_session exception.

Tests added for each: grader default skip, executor rejection (Go + schema), and judge-model cache separation. go test ./internal/... passes.

[Copilot]

🟡 Changes recommended

Unresolved issues affect regrade metadata, continued-session defaults, task-level validation, runtime error coverage, and generated dashboard references.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (2)

internal/execution/copilot.go:380

• The new execution tests use mocked SDK clients and only verify that the effort field is copied into create/resume configs. They do not cover the required runtime behavior for an unsupported model/effort combination, so an SDK rejection could regress without detection. Add a gated live Copilot test (matching the existing live-test convention) that asserts the surfaced error is actionable.

			ReasoningEffort: req.ReasoningEffort,

internal/models/grader_params.go:204

• PromptGraderParameters is shared with task ValidatorInlines, and RunAll applies this field to those validators, but ValidatorInline.Validate never checks ValidReasoningEffort and schemas/task.schema.json still rejects the field via additionalProperties: false. A valid task-level override is therefore rejected by schema validation, while an invalid value can reach the runtime when the task is loaded directly (also bypassing the executor restriction). Either add matching task-level support and validation, or reject the field consistently.

	ReasoningEffort string           `yaml:"reasoning_effort,omitempty" json:"reasoning_effort,omitempty"`

• Files reviewed: 23/24 changed files
• Comments generated: 2
• Review effort level: Lite

[Shayne Boyer (spboyer)]

Reviewed reasoning-effort configuration and follow-up fixes at d71e561f with two independent review passes.

No new, non-duplicate findings. Previously reported blockers remain at this head, so changes are still requested. See the existing review; no duplicate inline comments are added.

[Copilot]

🔵 Needs a closer look

Unresolved validation/schema gaps and inaccurate regrade metadata block approval.

Review details

Suppressed comments (4)

Previously missed (2) — in code that hasn't changed since the last review.

internal/models/spec.go:373

• This validation only runs for GraderConfig (eval-level graders). Task-level and checkpoint prompt graders use ValidatorInline, whose validation has no corresponding ValidReasoningEffort or non-Copilot check; because LoadTestCase does not apply the eval schema, an invalid effort or an effort paired with executor: mock can reach RunAll and be forwarded instead of rejected. Share this validation with ValidatorInline and perform the executor-aware check when tasks are attached to an eval.
  schemas/eval.schema.json:1291
• The new field is added only to the eval schema's prompt config, but the same PromptGraderParameters is used by task-level and checkpoint graders and RunAll applies judge defaults on those paths. schemas/task.schema.json still has additionalProperties: false without reasoning_effort, so schema validation rejects a task/checkpoint override even for copilot-sdk, creating a mismatch between the supported execution path and the schema.

internal/orchestration/outcome.go:297

• runGrade always passes the current spec's JudgeReasoningEffort, including an empty string when the setting is omitted. If an older results file contains a non-empty effort, this conditional preserves that stale value even though the regrade ran with the SDK/model default, so the saved metadata is inaccurate; overwrite the field unconditionally for regrades (or explicitly clear it when the effective setting is empty).

	if judgeReasoningEffort != "" {
		setup.JudgeReasoningEffort = judgeReasoningEffort
	}

site/src/content/docs/guides/eval-yaml.mdx:125

• The public site grader reference remains stale: its Prompt grader table omits reasoning_effort, and its schema reference omits both new eval-level effort fields. Since these pages are the user-facing references for grader options and eval configuration, update them alongside this new configuration documented here.

| `judge_reasoning_effort` | string | judge default | Copilot SDK default effort for `prompt` graders, including `continue_session` graders; the grader can override it |

• Files reviewed: 25/26 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

Unresolved schema validation, nested-grader validation, documentation, and generated-asset issues remain.

Review details

Suppressed comments (4)

Previously missed (2) — in code that hasn't changed since the last review.

internal/models/spec.go:480

• This validation only iterates over s.Graders, but task prompt graders use TestCase.Validators and checkpoint prompt graders use TestCase.Checkpoints[].Graders; neither path has an equivalent reasoning-effort or executor check. Consequently task/checkpoint YAML can pass Go loading with an invalid effort or with an effort under the mock executor, and RunAll will still forward it. Validate these nested graders with the same rules and executor context.
  schemas/eval.schema.json:1291
• This adds reasoning_effort only to the eval schema's prompt-grader definition. Task-level graders are validated against the separate schemas/task.schema.json definition, whose promptGraderConfig still has additionalProperties: false without this property, so the documented override is rejected as an unknown field whenever it is placed in a task YAML. Add the matching property there (and cover it with a task-schema validation test).

docs/graders/prompt.md:24

• The public prompt-grader guide at site/src/content/docs/guides/graders.mdx still has no reasoning_effort row or example, while this change documents the override in the local reference. Update the published guide too so users see the new grader configuration and supported values.

| `reasoning_effort` | string | Override the eval's `judge_reasoning_effort` (`low`, `medium`, `high`, `xhigh`, or `max`); requires `executor: copilot-sdk` |

internal/models/grader_params.go:204

• This field is also used by ValidatorInline and graders.RunAll for task and checkpoint prompt graders, but schemas/task.schema.json still omits reasoning_effort from promptGraderConfig (which has additionalProperties: false). A task can therefore decode this new field in Go while waza check/JSON-RPC schema validation rejects the same YAML; update the task schema and add matching validation coverage.

	ReasoningEffort string           `yaml:"reasoning_effort,omitempty" json:"reasoning_effort,omitempty"`

• Files reviewed: 25/26 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Shayne Boyer (spboyer)]

Reviewed the reasoning-effort implementation and follow-up fixes with independent Opus 4.7 and GPT 5.5 passes, deduplicated against all prior feedback. No remaining findings. The fixes cover nested judge validation, accurate regrade metadata, published references, and hosted-model capability checks; resumed-session forwarding follows #571 and the SDK contract. Go vet, full/short tests, lint, docs build, and the gated live unsupported-effort check passed. This supersedes my earlier change requests after verifying the fixes; repository latest-push and codeowner approval rules still apply.

[Copilot]

Copilot review overview

🟡 Changes recommended

Address catalog preflight reuse, expose effort metadata through the run API, and add runner-level propagation coverage.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Cache model catalog preflight requests · New

Resolved since last review (1)

• runGrade now uses spec.Config.JudgeReasoningEffort to grade, but RegradeOutcome only receives…

[Shayne Boyer (spboyer)]

Reviewed the current diff, CI, repository compatibility, and existing feedback. No new issues found.