Skip to content

routed runs fail when the router selects effort none or max #64831

Closed

Description

@SivaKesava1

Title: Model routing: Copilot harness rejects router-selected reasoning efforts none and max

Summary

With engine.model-routing enabled (#64593), the Copilot harness refuses to start Copilot
whenever AWF's routing selection uses reasoning effort none or max:

unexpected error: AWF model routing failed: AWF /reflect returned unsupported reasoning effort max; refusing to start Copilot

Both values are valid end to end:

  • gh-aw-router and AWF's routing candidate pool use the effort set
    none | minimal | low | medium | high | xhigh | max.
  • The pinned Copilot CLI (1.0.89) accepts all seven values:
    copilot --help → --reasoning-effort <level> [possible values: none, minimal, low, medium, high, xhigh, max].

Only gh-aw's allow-lists are narrower: they accept minimal | low | medium | high | xhigh.

Impact

This affects any routed workflow whose candidates include models that support max or none.
In the published gh-aw-router tables, the top-ranked Copilot choice uses max or none in:

Profile Share of label cells
cost / balanced 30%
cost-speed / economy 22%
cost / robust 21%
cost / economy 14%

When this happens, the agent step fails before any work is done. This is not a routing failure
(exit 78). A local AWF run selected gpt-6-astra with effort max, and the Copilot API served
that request successfully.

Analysis

Two allow-lists in actions/setup/js are the only mismatch:

  1. copilot_harness.cjs, resolveAWFModelRoutingSelection (around line 509):
    if (effort && !["minimal", "low", "medium", "high", "xhigh"].includes(effort)) {
      return { selection: null, error: `AWF /reflect returned unsupported reasoning effort ${effort}` };
    }
  2. copilot_sdk_driver.cjs (around line 113) only forwards reasoningEffort to the SDK session
    when GH_AW_COPILOT_ROUTING_EFFORT is one of minimal | low | medium | high | xhigh.
    Otherwise it silently drops the effort.

The other code paths already pass the value through unchanged:

  • applyCopilotRoutingArgs (--reasoning-effort <effort>)
  • applyCopilotRoutingSelection (GH_AW_COPILOT_ROUTING_EFFORT)
  • buildCopilotSDKChildEnv (COPILOT_REASONING_EFFORT for sub-agent sessions)

Note: @github/copilot-sdk 1.0.13 types SessionConfig.reasoningEffort as
"low" | "medium" | "high" | "xhigh" | "max". The RPC layer uses string, and
copilot_sdk_session.cjs already passes the value as any. The headless server is the same CLI
that accepts all seven values.

Implementation plan

  1. Add one shared constant in actions/setup/js (in copilot_harness.cjs, or a small shared
    module that the SDK driver also imports):
    const ROUTING_REASONING_EFFORTS = Object.freeze(["none", "minimal", "low", "medium", "high", "xhigh", "max"]);
    These are the values the pinned Copilot CLI lists for --reasoning-effort.
  2. Use it in resolveAWFModelRoutingSelection in place of the inline list. Keep
    lower-casing and trimming. Keep the distinction between effort: null (no flag, no env) and
    effort: "none" (an explicit value that is passed through).
  3. Use it in copilot_sdk_driver.cjs for the reasoningEffort forwarding check, so max and
    none reach runWithCopilotSDK.
  4. Add tests:
    • copilot_harness.test.cjs:
      • resolveAWFModelRoutingSelection accepts max and none, and still rejects an unknown
        value such as "extreme".
      • applyCopilotRoutingArgs with effort: "max" produces --reasoning-effort max.
      • effort: "none" produces --reasoning-effort none.
      • effort: null produces no flag.
      • applyCopilotRoutingSelection sets GH_AW_COPILOT_ROUTING_EFFORT to max and none.
    • SDK driver: reasoningEffort is forwarded for max and none when GH_AW_MODEL_ROUTING=1.
  5. Optional hardening: if the Copilot CLI version pin changes, a test or comment should tie
    ROUTING_REASONING_EFFORTS to the CLI's documented --reasoning-effort values.
  6. Run make agent-finish.

Verification

  • Unit tests above.
  • End to end:
    • Compile a routed workflow with engine.model-routing, using mode: robust or balanced and
      allowed-models that include a model offering max (for example gpt-6-astra).
    • Confirm the agent step logs
      inference routing: mode=awf-routed model=<model> effort=max and Copilot starts.
    • Confirm the AWF api-proxy logs a model_routing request event with routed: "as_selected".

Out of scope

No AWF or gh-aw-router change is needed. Both already use the full effort set.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions