Route Pi models to compatible inference APIs by Copilot · Pull Request #65877 · github/gh-aw

Copilot · 2026-10-05T14:50:08Z

Pi workflows can fail before producing safe outputs when a selected model is unavailable through the configured proxy endpoint or requires the Responses API instead of chat completions.

Preflight model availability: When AWF reflection confirms model discovery is complete, fail before Pi starts if the selected model is not advertised by its configured endpoint. The error names the model and points to the reflected inventory.
Select a compatible API: Honor model catalog metadata marking a model as Responses-only, and reject conflicting engine.config.model.api overrides.

For example, a reflected Copilot model with wire_api: "responses" is registered in Pi’s generated models.json as:

{
  "api": "openai-responses",
  "models": [{ "id": "gpt-5.5" }]
}

Fixes [AW Top 10] 04 Prevent Pi engine runs on chat-incompatible models #65772

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2026-10-05T15:14:05Z

✅ Ponytail Reviewer completed successfully!

Lean already. Ship.

Generated by Ponytail Reviewer for #65877

github-actions · 2026-10-05T15:14:11Z

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff

🧪 Test quality analysis by Test Quality Sentinel

github-actions · 2026-10-05T15:14:17Z

🧠 Matt Pocock Skills Reviewer is reviewing this pull request using Matt Pocock's engineering skills...

github-actions · 2026-10-05T15:14:33Z

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

pelikhan · 2026-10-05T15:15:09Z

@copilot add logging

github-actions · 2026-10-05T15:15:36Z

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

No ADR enforcement needed for PR #65877: no 'implementation' label and 0 new lines in business logic directories (2 files changed, threshold 100, no custom .design-gate.yml).

🏗️ ADR gate enforced by Design Decision Gate 🏗️

Copilot

Copilot review overview

🟡 Changes recommended

Preflight incorrectly rejects valid model aliases and skips Google/Gemini endpoint validation.

Review effort: Balanced
Findings: 1

Open (1)

Missing Google/Gemini provider aliases skip availability checks · New

What changed in this PR

Adds Pi preflight model validation and model-specific API routing using AWF reflection and catalog metadata.

Changes:

Rejects unavailable models before Pi starts.
Routes Responses-only models correctly and rejects incompatible overrides.
Adds focused unit and integration tests.

File	Description
`actions/setup/js/pi_models_json.cjs`	Implements availability validation and API selection.
`actions/setup/js/pi_models_json.test.cjs`	Tests routing and preflight failures.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Copilot · 2026-10-05T16:08:46Z

+  const normalizedProvider = normalizeReflectProviderName(provider);
+  const aliases = REFLECT_PROVIDER_ALIASES[normalizedProvider] || new Set([normalizedProvider]);
+  const endpoint = reflectData.endpoints?.find(endpoint => endpoint?.configured === true && aliases.has(normalizeReflectProviderName(endpoint.provider)));
+  if (!endpoint || !Array.isArray(endpoint.models)) return;


Added the canonical google/gemini aliases and a preflight test for Google models advertised by a Gemini endpoint. Committed as 9044112.

github-actions · 2026-10-05T15:24:15Z

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)

reviewed_at: 2026-10-05T15:16:17Z
review_event: REQUEST_CHANGES
top_themes:
  - stale catalog fallback misroutes new Copilot GPT models to chat completions
files_reviewed:
  - actions/setup/js/pi_models_json.cjs
  - actions/setup/js/pi_models_json.test.cjs
comment_count: 1

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 56.5 AIC · ⌖ 8.47 AIC · ⊞ 19.2K · ◷
_{Comment /review to run again}

github-actions

Request changes

The new preflight checks are headed in the right direction, but the API-selection fallback still regresses stale-catalog cases: a new Copilot GPT/Responses-only model will silently drop back to openai-completions whenever models.json has not caught up yet.

Blocking theme

resolvePiApiForModel() now treats the local catalog as the only source of Responses-vs-Completions truth for GitHub models.
This repo already has code paths that operate with a missing or stale merged catalog, and those paths need the same GPT-5+ wire-API heuristic the rest of the harness uses.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 56.5 AIC · ⌖ 8.47 AIC · ⊞ 19.2K
_{Comment /review to run again}

github-actions · 2026-10-05T15:24:18Z

+    throw new Error(`Pi model "${modelId}" requires the OpenAI Responses API, but engine.config.model.api is "${overrideApi}"`);
+  }
+  if (requiresResponses) return "openai-responses";
+  return overrideApi || model?.api || resolvePiApiForProvider(provider);


This fallback regresses Copilot GPT-5+ models to openai-completions whenever models.json is missing or stale, so new Responses-only models will still boot Pi against the wrong endpoint and fail before safe outputs.

💡 Why this blocks merge

loadModelsJson() is explicitly allowed to return the bundled catalog or null when the merged catalog artifact is absent, and runtime.getModel() does not guarantee an api field for Copilot entries. In that common stale-catalog path, wireApi is empty here and line 199 falls back to resolvePiApiForProvider("github"), which hard-codes openai-completions.

That means the first newly-added GPT-5+/Responses-only Copilot model will still be misrouted unless the local catalog has already been updated, which is exactly the failure mode this PR is trying to eliminate. Please reuse the existing GPT heuristic/shared wire-API inference (inferWireApiForModel / applyCopilotWireAPI) instead of defaulting GitHub models straight to completions when the catalog lookup misses.

Suggested shape:

const inferredWireApi = inferWireApiForModel("openai", modelId, catalogEntry); if (inferredWireApi === "responses") return "openai-responses"; if (inferredWireApi === "completions") return "openai-completions";

Pi now uses the shared wire-API inference helper for Copilot models, so GPT-5+ still selects Responses when the catalog is missing or stale; added coverage for both cases. Committed as 9044112.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-10-05T15:31:33Z

@copilot add logging

Added secret-safe diagnostics for reflected model availability checks and API selection, with tests. Committed as 2ec0610.

gh-aw-bot · 2026-10-05T15:52:30Z

@copilot address the following outstanding work in one pass:

Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
Review (actions/setup/js/pi_models_json.cjs:167): Google-backed Pi runs use GH_AW_LLM_PROVIDER=google, while AWF reflection identifies this endpoint as gemini. Since the shared alias table has no Google/Gemini mapping, this lookup never finds the configured endpoint and silently skips the new availability check for every Google model. Add google/gemini aliases to the canonical reflect provider mapping and cover this provider in the tests. - Route Pi models to compatible inference APIs #65877 (comment)
Review (actions/setup/js/pi_models_json.cjs:199): This fallback regresses Copilot GPT-5+ models to openai-completions whenever models.json is missing or stale, so new Responses-only models will still boot Pi against the wrong endpoint and fail before safe outputs. - Route Pi models to compatible inference APIs #65877 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 2ec0610
Sous-chef work: 3c08c4024e7edd1482dca0dbdf02e6efc7a312501a2974dfd92eca7f0503372e b1037d40ed90747c89b3a56cd7419552318a3bc927a1f5cb205a5ff101f7d772
Sous-chef state: a8fd160d502714ea8fcb23bd97c359e4b60c31b32d9ec0bbef6275f5d333f868

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 4.07 AIC · ⌖ 7.15 AIC · ⊞ 3.2K · ◷
_{Comment /souschef to run again}

done

github-actions · 2026-10-05T23:07:11Z

🎉 This pull request is included in a new release.

Release: v0.91.0

Initial plan

ef4f40e

Copilot AI assigned Copilot and pelikhan Oct 5, 2026

Copilot started work on behalf of pelikhan October 5, 2026 14:50

View session

Copilot AI linked an issue Oct 5, 2026 that may be closed by this pull request

[AW Top 10] 04 Prevent Pi engine runs on chat-incompatible models #65772

Closed

2 tasks

Copilot AI and others added 2 commits October 5, 2026 15:01

Prevent Pi routing to unsupported model APIs

8941da0

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Clarify Pi routing test description

30dbc86

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add validation to prevent Pi engine runs on chat-incompatible models~~ Route Pi models to compatible inference APIs Oct 5, 2026

Copilot finished work on behalf of pelikhan October 5, 2026 15:03

Copilot AI requested a review from pelikhan October 5, 2026 15:03

pelikhan marked this pull request as ready for review October 5, 2026 15:12

Copilot AI balanced review requested due to automatic review settings October 5, 2026 15:12

Copilot started reviewing on behalf of pelikhan October 5, 2026 15:13

View session

Copilot AI reviewed Oct 5, 2026

View reviewed changes

Copilot started work on behalf of pelikhan October 5, 2026 15:21

View session

github-actions Bot previously requested changes Oct 5, 2026

View reviewed changes

Add diagnostics for Pi model routing

2ec0610

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan October 5, 2026 15:32

This was referenced Oct 5, 2026

[aw] Matt Pocock Skills Reviewer timed out #65890

Closed

[aw] Impeccable Skills Reviewer timed out #65891

Closed

github-actions Bot mentioned this pull request Oct 5, 2026

[pr-sous-chef] Sous Chef Report: 1 PR nudged #65759

Closed

Copilot started work on behalf of gh-aw-bot October 5, 2026 15:52

View session

Merge branch 'main' into copilot/aw-top-10-prevent-pi-engine-runs

d0bb0a2

pelikhan merged commit 1eb2653 into main Oct 5, 2026
12 checks passed

pelikhan deleted the copilot/aw-top-10-prevent-pi-engine-runs branch October 5, 2026 16:02

Copilot AI requested a review from gh-aw-bot October 5, 2026 16:10

Copilot finished work on behalf of gh-aw-bot October 5, 2026 16:10

Conversation

Copilot AI commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelikhan commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Copilot AI Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 5, 2026

Comment Memory

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Request changes

Uh oh!

github-actions Bot Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Oct 5, 2026

Uh oh!

gh-aw-bot commented Oct 5, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Oct 5, 2026 •

edited

Loading

github-actions Bot commented Oct 5, 2026 •

edited

Loading

github-actions Bot commented Oct 5, 2026 •

edited

Loading

github-actions Bot commented Oct 5, 2026 •

edited

Loading

github-actions Bot commented Oct 5, 2026 •

edited

Loading