Add VSC Model Override#5060
Conversation
There was a problem hiding this comment.
Pull request overview
Adds an advanced debug setting to force the “VSC model” variant used by prompt routing (and intended capability classification), and records the override in request logs for debugging/diagnostics.
Changes:
- Introduces
github.copilot.chat.debug.vscModelOverride(A/B/C/D/null) with schema + localization. - Extends
isVSCModelA/B/C/Dto accept an optional override and updates prompt routing to honor the override. - Adds request-log metadata (
vscModelOverrideFamily) and unit tests for override semantics + metadata preservation.
Show a summary per file
| File | Description |
|---|---|
| src/platform/requestLogger/test/node/requestLogger.spec.ts | Adds a test ensuring customMetadata values are preserved on logged requests. |
| src/platform/endpoint/test/node/chatModelCapabilities.spec.ts | Adds tests validating the new override parameter behavior for VSC model checks. |
| src/platform/endpoint/common/chatModelCapabilities.ts | Adds VSCModelVariant and optional override parameters to isVSCModel* helpers. |
| src/platform/configuration/common/configurationService.ts | Adds ConfigKey.Advanced.DebugVSCModelOverride with enum + nullable validation. |
| src/extension/prompts/node/agent/promptRegistry.ts | Makes prompt resolver selection honor the debug override and logs when used. |
| src/extension/prompt/node/chatMLFetcher.ts | Injects override info into request logger customMetadata. |
| package.nls.json | Adds localized description for the new debug setting. |
| package.json | Registers the new setting schema (github.copilot.chat.debug.vscModelOverride). |
Copilot's findings
- Files reviewed: 8/8 changed files
- Comments generated: 3
| export type VSCModelVariant = 'A' | 'B' | 'C' | 'D'; | ||
|
|
||
| export function isVSCModelA(model: LanguageModelChat | IChatEndpoint, override?: VSCModelVariant | null) { | ||
| if (override !== undefined && override !== null) { | ||
| return override === 'A'; | ||
| } |
There was a problem hiding this comment.
The VSC model override parameter is added to isVSCModel*(), but none of the capability helpers in this module pass an override value (they still call isVSCModelA/B/C/D with a single argument). As a result, the new config-driven override won’t actually affect capability checks unless the override is plumbed through at the call sites (or these helpers are updated to accept/pass an override).
| "github.copilot.config.completionsFetcher": "Sets the fetcher used for the inline completions.", | ||
| "github.copilot.config.nesFetcher": "Sets the fetcher used for the next edit suggestions.", | ||
| "github.copilot.config.debug.overrideChatEngine": "Override the chat model. This allows you to test with different models.\n\n**Note**: This is an advanced debugging setting and should not be used while self-hosting as it may lead to a different experience compared to end-users.", | ||
| "github.copilot.config.debug.vscModelOverride": "Override the hidden VSC model family classification used for prompt routing and capability checks. When set to `A`, `B`, `C`, or `D`, only that family returns `true` and the others return `false`. Set to `null` to use normal detection.", |
There was a problem hiding this comment.
The setting description says the override affects both prompt routing and capability checks, but in this PR the override is only used for prompt routing (PromptRegistry) and request log metadata; capability checks are not wired up to read/pass the override. Please either update the description to match current behavior, or plumb the override into the capability-check code paths.
| "github.copilot.config.debug.vscModelOverride": "Override the hidden VSC model family classification used for prompt routing and capability checks. When set to `A`, `B`, `C`, or `D`, only that family returns `true` and the others return `false`. Set to `null` to use normal detection.", | |
| "github.copilot.config.debug.vscModelOverride": "Override the hidden VSC model family classification used for prompt routing. When set to `A`, `B`, `C`, or `D`, only that family returns `true` and the others return `false`. Set to `null` to use normal detection.", |
| const modelOverride = instantiationService.invokeFunction(accessor => accessor.get(IConfigurationService).getConfig(ConfigKey.Advanced.DebugVSCModelOverride)); | ||
| const promptResolverCtor = await this.getPromptResolver(endpoint, modelOverride); | ||
| const agentPrompt = promptResolverCtor ? instantiationService.createInstance(promptResolverCtor) : undefined; | ||
|
|
||
| if (modelOverride) { |
There was a problem hiding this comment.
New behavior: PromptRegistry now selects a resolver based on the DebugVSCModelOverride setting. There isn’t test coverage validating that this setting forces selection of the expected VSC prompt resolver (and bypasses matchesModel/family-prefix detection). Consider adding a focused unit test that sets the config and asserts the resolved SystemPrompt/ReminderInstructions come from the chosen VSC model variant.
No description provided.