fix(drift): flag new voice/audio model families beyond the "realtime" substring#292
Merged
Merged
Conversation
… substring
The ws-realtime known-models canary filtered GET /v1/models with
`models.filter((m) => m.includes("realtime"))`, so a NEW full-duplex voice
family whose id lacks the "realtime" substring (e.g. OpenAI's gpt-live-1 /
gpt-live-1-mini) never entered the unknown-model computation and slipped past
the canary silently. The general models.drift.ts canary only detects
deprecation of already-referenced models, not the arrival of new ones — this
was the blind spot.
Extract the detection into a side-effect-free module (voice-models.ts) with a
broadened voice/audio family matcher (realtime|audio|live|transcribe|whisper|
voice|tts) and a knownVoiceModels allowlist. Both the live canary and a new
unit test drive the SAME detectVoiceModelDrift code path. The canary's critical
-vs-crash markers (NO_GA_REALTIME_MODELS= / UNKNOWN_REALTIME_MODELS=, added in
1e3e638) are preserved verbatim, so the drift collector still emits exit-2
critical drift rather than crashing.
Generalizes beyond gpt-live: any unseen voice/audio family is flagged the first
time it appears; chat/image/embedding models never become false positives.
commit: |
… false positives The broadened voice/audio matcher correctly widened coverage to catch new families like gpt-live, but it compared full model ids against a known-ID set. OpenAI ships dated snapshots of already-known families constantly, so the live Drift Tests run (28968203340) flagged 7 legitimate snapshots as critical drift (tts-1-1106, tts-1-hd-1106, gpt-audio-2025-08-28, gpt-4o-mini-transcribe-*, gpt-4o-mini-tts-*). Appending each new snapshot never converges — the daily job would go red every day and bury a real new-family signal. Fix: normalize each candidate id to a FAMILY KEY by stripping trailing dated snapshot (-YYYY-MM-DD) and build-tag (-NNN/-NNNN) suffixes, then compare the normalized family against a set of known FAMILIES. Dated snapshots of a known family collapse onto it and stay green; a genuinely new family (gpt-live, whose single-digit -1 is deliberately not stripped) still normalizes to an unknown family and is flagged. The NO_GA_REALTIME_MODELS= / UNKNOWN_REALTIME_MODELS= markers in ws-realtime.drift.ts are preserved.
jpr5
added a commit
that referenced
this pull request
Jul 15, 2026
… substring (#292) ## Problem The ws-realtime known-models drift canary (`src/__tests__/drift/ws-realtime.drift.ts`) originally filtered the `GET /v1/models` result with `models.filter((m) => m.includes("realtime"))`, so a NEW full-duplex voice **family** whose id lacks the `"realtime"` substring — e.g. `gpt-live-1` / `gpt-live-1-mini` — never entered the `unknown`-model computation and slipped past silently. The first fix broadened the family matcher to a general voice/audio vocabulary (`realtime | audio | live | transcribe | whisper | voice | tts`). That correctly widened coverage to catch new families — **but** it compared full model **ids** against a known-**id** set. OpenAI ships dated snapshots of already-known families constantly, so a LIVE dry-run of the drift suite on this branch (run **28968203340**) failed as critical drift on 7 legitimate snapshots: ``` tts-1-1106, tts-1-hd-1106, gpt-audio-2025-08-28, gpt-4o-mini-transcribe-2025-12-15, gpt-4o-mini-transcribe-2025-03-20, gpt-4o-mini-tts-2025-03-20, gpt-4o-mini-tts-2025-12-15 ``` These are just dated snapshots of families already conceptually known (`tts-1`, `tts-1-hd`, `gpt-audio`, `gpt-4o-mini-transcribe`, `gpt-4o-mini-tts`). Merged as-is, the daily scheduled drift job would go RED every day on false positives and bury a real new-family signal. Appending each new snapshot to a known-id set never converges. ## Fix — FAMILY NORMALIZATION (not whack-a-mole) `src/__tests__/drift/voice-models.ts`: - **`normalizeVoiceModelFamily(id)`** strips trailing version/snapshot suffixes from the END of the id, repeatedly: - dated snapshot `-YYYY-MM-DD` (`/-\d{4}-\d{2}-\d{2}$/`) - build/version tag `-NNN` / `-NNNN` (`/-\d{3,4}$/`) A single-digit trailing `-N` (as in `gpt-live-1`) is deliberately NOT stripped, so `gpt-live-1` normalizes to `gpt-live-1` — still an unknown family. - The known set migrated from ids to **families** (`knownVoiceModelFamilies`), built through `normalizeVoiceModelFamily` so it stays idempotent/in-lockstep. - `detectVoiceModelDrift` now compares each candidate's **normalized family** against the known-family set; `hasGA` likewise normalizes before matching `gaRealtimeModels` (also reduced to family keys). - The critical-vs-crash markers `NO_GA_REALTIME_MODELS=` / `UNKNOWN_REALTIME_MODELS=` in `ws-realtime.drift.ts` are **preserved verbatim**. Dated snapshots of a known family collapse onto it and stay green; a genuinely new family (`gpt-live`) still normalizes to an unknown family and is flagged. ## Red-Green Proof Drives the **real** `detectVoiceModelDrift` / `normalizeVoiceModelFamily` — the same functions the live canary invokes — against a representative `GET /v1/models` list containing the 7 real snapshots + `gpt-live-*`. ### RED (pre-normalization id-set matcher — CI failure reproduced locally) ``` UNKNOWN: tts-1-1106, tts-1-hd-1106, gpt-audio-2025-08-28, gpt-4o-mini-transcribe-2025-12-15, gpt-4o-mini-transcribe-2025-03-20, gpt-4o-mini-tts-2025-03-20, gpt-4o-mini-tts-2025-12-15, gpt-live-1, gpt-live-1-mini 7 real snapshots flagged (RED expects all 7): 7 -> tts-1-1106, tts-1-hd-1106, gpt-audio-2025-08-28, gpt-4o-mini-transcribe-2025-12-15, gpt-4o-mini-transcribe-2025-03-20, gpt-4o-mini-tts-2025-03-20, gpt-4o-mini-tts-2025-12-15 gpt-live-1 flagged: true gpt-live-1-mini flagged: true ``` All 7 real snapshots are falsely flagged (reproduces run 28968203340), and `gpt-live-1` is flagged. ### GREEN (family normalization — snapshots pass, new family still caught) ``` UNKNOWN: gpt-live-1, gpt-live-1-mini 7 real snapshots flagged (RED expects all 7): 0 -> gpt-live-1 flagged: true gpt-live-1-mini flagged: true ``` The 7 real snapshots are NOT flagged (they normalize to known families); `gpt-live-1` / `gpt-live-1-mini` ARE still flagged as a new family; `gpt-realtime-2.1` / `whisper-1` / `tts-1` stay green. Unit suite (`src/__tests__/ws-realtime-canary.test.ts`), including a regression guard on the 7 real ids and explicit normalization assertions: ``` ✓ src/__tests__/ws-realtime-canary.test.ts (7 tests) Test Files 1 passed (1) Tests 7 passed (7) ``` ## LIVE `Drift Tests` re-verification (the real gate) Dispatched the live workflow on this branch after the fix: - Run **28968741912** — `drift` job conclusion: **success** (also `agui-schema-drift`: success, `notify`: success). - `OPENAI_API_KEY` present in the drift step env, so the canary ran (not skipped); collector completed without crashing; `drift-report.json` `entries: []` — zero drift, the 7 previously-flagged snapshots no longer surface. No iteration was needed — the first live re-run after the fix was green. ## Pre-push gates (all pass) - `pnpm run format:check` — All matched files use Prettier code style - `pnpm run lint` (eslint) — clean - `pnpm run test` — **146 files, 4334 tests passed** (+2 new tests) - `pnpm run build` (tsdown) — Build complete
jpr5
added a commit
that referenced
this pull request
Jul 15, 2026
Adds scripts/drift-delta.ts: a pure, side-effect-free delta layer that reduces a base (main) report and head (PR) report to block/advisory/fixed keyed by provider+per-item-id. Routing is by key presence ONLY — DriftClass is annotation and never enters the block/advisory decision: - new-in-head (absent from base) -> block (diff-attributable) - present in both -> advisory (environmental/world drift) - base-only -> fixed (informational) isBaseReportReusable (O-2) guards cached-base reuse: non-empty entries + known-good conclusion + same-UTC-day. Includes the M-1 golden regression (#292): a real-drift/critical finding new-in-head MUST block regardless of class (the old class-routed rule would have greenlit it).
jpr5
added a commit
that referenced
this pull request
Jul 15, 2026
…e-present Plug the D6.3b delta step into the drift-live-pr job at the SEAM. The step loads drift-report-base.json + drift-report-head.json and runs computeDelta (scripts/drift-delta.ts) via a quoted-heredoc tsx invocation, matching the preflight step's proven pattern. Routing is by KEY PRESENCE only (#292 invariant), never by DriftClass: - block[] (new-in-head) → ::error:: annotation + exit 1 (required check fails) - advisory[] (base-present) → ::notice:: neutral annotation + summary, exit 0 - fixed[] → informational summary line only Also documents the fork→maintainer-dry-run fallback in-job: fork PRs run without secrets, so a maintainer runs the delta dry-run from a trusted checkout and records the block/advisory conclusion.
jpr5
added a commit
that referenced
this pull request
Jul 15, 2026
Adds scripts/drift-delta.ts: a pure, side-effect-free delta layer that reduces a base (main) report and head (PR) report to block/advisory/fixed keyed by provider+per-item-id. Routing is by key presence ONLY — DriftClass is annotation and never enters the block/advisory decision: - new-in-head (absent from base) -> block (diff-attributable) - present in both -> advisory (environmental/world drift) - base-only -> fixed (informational) isBaseReportReusable (O-2) guards cached-base reuse: non-empty entries + known-good conclusion + same-UTC-day. Includes the M-1 golden regression (#292): a real-drift/critical finding new-in-head MUST block regardless of class (the old class-routed rule would have greenlit it).
jpr5
added a commit
that referenced
this pull request
Jul 15, 2026
…e-present Plug the D6.3b delta step into the drift-live-pr job at the SEAM. The step loads drift-report-base.json + drift-report-head.json and runs computeDelta (scripts/drift-delta.ts) via a quoted-heredoc tsx invocation, matching the preflight step's proven pattern. Routing is by KEY PRESENCE only (#292 invariant), never by DriftClass: - block[] (new-in-head) → ::error:: annotation + exit 1 (required check fails) - advisory[] (base-present) → ::notice:: neutral annotation + summary, exit 0 - fixed[] → informational summary line only Also documents the fork→maintainer-dry-run fallback in-job: fork PRs run without secrets, so a maintainer runs the delta dry-run from a trusted checkout and records the block/advisory conclusion.
jpr5
added a commit
that referenced
this pull request
Jul 15, 2026
Implements the reviewed **aimock Drift-Detection Ironclad** hardening spec: https://app.notion.com/p/39d3aa38185281c582b1d819c2842518 Built via 4 gated waves (17 micro-tasks), each red-green against the real failure surface, per-wave reviewed to zero, plus a final full-branch integration CR. 21 files, ~+2238/-288. 224 drift tests green. ## What changed, by wave - **Wave A — collector taxonomy + primitive.** Pure `computeExitCode` (crit>0→2, quarantine>0→5, agUiSkipped→1, else 0); `collectDriftEntries` now *quarantines* unmapped/unparseable failures (capturing file:line) instead of throwing — the exit-1 crash that hid incident #294's root cause is gone. Report schema gains `DriftClass`/per-item `id`/`class`/`quarantine[]`. Shared `normalizeModelFamily(id, provider)` primitive (voice-models migrated onto it, behavior-identical). - **Wave B — exit-5 threading + W1 /models rewrite.** `drift-retry` + both workflows thread exit-5 (quarantine → run fails, no auto-fix abort). **Model identity no longer scrapes prose/source** (the incident root cause): enumerate live `/models` → normalize → subtract curated include/exclude → unclassified delta is the signal, with an LLM-triage seam. `models-scrape.drift.ts` retired. - **Wave C — classified alerting.** `InfraError` carries a structured HTTP `status`; Slack summary emits a closed `class` enum (real-drift / false-positive / infra-transient / stale-key / quarantine) with per-item id + file:line; key-freshness preflight fails loud on 401/403 (never a green skip); notify routes exit-5 → quarantine before the HTTP-drift fallback. - **Wave D — delta-gated PR check.** New `drift-live-pr` job (`pull_request`, path-filtered, no secrets on forks) runs base+head and **blocks only on failures the PR introduces (new-in-head), regardless of class** — a drift-code PR is judged on "did your change break the detector," not "is the world drifting today." Base-present failures are advisory. Fork PRs use a maintainer dry-run. ## Bugs the review process caught (not shipped) 1. **Anthropic `-YYYYMMDD` normalize gap** — dated Claude snapshots wouldn't collapse to their family → would false-positive (incident-2 class, untested provider). Fixed via the provider hook. 2. **Class-gating hole** — a first cut gated on collector-assigned class; a new-in-head `real-drift` would've been advisory → **would have re-greenlit the exact #292 incident**. Fixed to gate purely on new-in-head. 3. **NUL-byte** in `drift-delta.ts` rendered the module binary/unreviewable in diffs — caught only by the final full-branch CR. ## Follow-up before merge - **Keyed CI run required:** the live `/models` provider blocks were skipped in the impl env (no keys). A keyed CI run is the remaining real-surface validation. **DRAFT — awaiting review before merge to main.**
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The ws-realtime known-models drift canary (
src/__tests__/drift/ws-realtime.drift.ts) originally filtered theGET /v1/modelsresult withmodels.filter((m) => m.includes("realtime")), so a NEW full-duplex voice family whose id lacks the"realtime"substring — e.g.gpt-live-1/gpt-live-1-mini— never entered theunknown-model computation and slipped past silently.The first fix broadened the family matcher to a general voice/audio vocabulary (
realtime | audio | live | transcribe | whisper | voice | tts). That correctly widened coverage to catch new families — but it compared full model ids against a known-id set. OpenAI ships dated snapshots of already-known families constantly, so a LIVE dry-run of the drift suite on this branch (run 28968203340) failed as critical drift on 7 legitimate snapshots:These are just dated snapshots of families already conceptually known (
tts-1,tts-1-hd,gpt-audio,gpt-4o-mini-transcribe,gpt-4o-mini-tts). Merged as-is, the daily scheduled drift job would go RED every day on false positives and bury a real new-family signal. Appending each new snapshot to a known-id set never converges.Fix — FAMILY NORMALIZATION (not whack-a-mole)
src/__tests__/drift/voice-models.ts:normalizeVoiceModelFamily(id)strips trailing version/snapshot suffixes from the END of the id, repeatedly:-YYYY-MM-DD(/-\d{4}-\d{2}-\d{2}$/)-NNN/-NNNN(/-\d{3,4}$/)A single-digit trailing
-N(as ingpt-live-1) is deliberately NOT stripped, sogpt-live-1normalizes togpt-live-1— still an unknown family.knownVoiceModelFamilies), built throughnormalizeVoiceModelFamilyso it stays idempotent/in-lockstep.detectVoiceModelDriftnow compares each candidate's normalized family against the known-family set;hasGAlikewise normalizes before matchinggaRealtimeModels(also reduced to family keys).NO_GA_REALTIME_MODELS=/UNKNOWN_REALTIME_MODELS=inws-realtime.drift.tsare preserved verbatim.Dated snapshots of a known family collapse onto it and stay green; a genuinely new family (
gpt-live) still normalizes to an unknown family and is flagged.Red-Green Proof
Drives the real
detectVoiceModelDrift/normalizeVoiceModelFamily— the same functions the live canary invokes — against a representativeGET /v1/modelslist containing the 7 real snapshots +gpt-live-*.RED (pre-normalization id-set matcher — CI failure reproduced locally)
All 7 real snapshots are falsely flagged (reproduces run 28968203340), and
gpt-live-1is flagged.GREEN (family normalization — snapshots pass, new family still caught)
The 7 real snapshots are NOT flagged (they normalize to known families);
gpt-live-1/gpt-live-1-miniARE still flagged as a new family;gpt-realtime-2.1/whisper-1/tts-1stay green.Unit suite (
src/__tests__/ws-realtime-canary.test.ts), including a regression guard on the 7 real ids and explicit normalization assertions:LIVE
Drift Testsre-verification (the real gate)Dispatched the live workflow on this branch after the fix:
driftjob conclusion: success (alsoagui-schema-drift: success,notify: success).OPENAI_API_KEYpresent in the drift step env, so the canary ran (not skipped); collector completed without crashing;drift-report.jsonentries: []— zero drift, the 7 previously-flagged snapshots no longer surface.No iteration was needed — the first live re-run after the fix was green.
Pre-push gates (all pass)
pnpm run format:check— All matched files use Prettier code stylepnpm run lint(eslint) — cleanpnpm run test— 146 files, 4334 tests passed (+2 new tests)pnpm run build(tsdown) — Build complete