Update AssemblyAI skill: Voice Agent stored agents, LLM Gateway Claude 4 removal, /v2/upload gotcha, streaming 1011/billing - #16
Merged
Conversation
…y Claude 4 removal, /v2/upload binary gotcha, streaming 1011/billing - Voice Agents: document binding a session to a stored agent via `agent_id` in the first `session.update` (mutually exclusive with inline config), and add a new Voice Agents REST API (stored agents) section to api-reference.md (`POST/GET/PUT/DELETE /v1/agents` on agents.assemblyai.com). [spec #157] - LLM Gateway: `claude-opus-4-20250514` and `claude-sonnet-4-20250514` were removed from the available-models list (June 2026) — mark removed and add a Common Mistakes correction. [docs #133] - REST upload: warn that `/v2/upload` requires raw bytes (`--data-binary @file`); `-d`/JSON body returns a valid upload_url but fails later transcription with `Transcoding failed. File type application/json`. [docs #144] - Streaming: add close code `1011` (internal/connection error) and the inactivity-timeout closure under `3006`; add a session-based billing note (concurrent/dual-streamed sessions bill in parallel). [docs #126, #147] - Speech Understanding: note docs now lead with `speakers` (not `known_values`) for both name and role speaker identification; `known_values` still supported.
Universal-3.5 Pro is available on the pre-recorded/async side, not just streaming — opt in via speech_models: ["universal-3-5-pro"] on POST /v2/transcript (currently Preview). Documented at pre-recorded-audio/universal-3-5-pro.mdx. Adds it to the Pre-Recorded model table, notes contextual prompting + keyterms (up to 1,000), 18-language coverage with auto-fallback to Universal-2, and flags that the formal OpenAPI speech_models enum doesn't list it yet though the API accepts it.
…mentary keyterms) Per the Prompting and Keyterms guide (https://www.assemblyai.com/docs/streaming/prompting-and-keyterms), which applies to both streaming and async for Universal-3.5 Pro: - prompt is a contextual *description* of the audio (domain/scenario/detail), NOT formatting/behavioral instructions (those are ignored). - prompt and keyterms_prompt are COMPLEMENTARY (used together), not mutually exclusive. Fixed this across SKILL.md, streaming.md, api-reference.md, python-sdk.md, js-sdk.md. - Rewrote the SKILL.md Prompting section around U3.5 Pro: three specificity levels, start-with-nothing guidance, keyterms limits (100 streaming / 1,000 async), corrected ~1500 chars (was 'words'). - Dropped obsolete instruction-style guidance (Non-negotiable prefix, positive-phrasing, negative-prompt gotcha) and 'instruct via prompt' disfluencies clauses. - SDK prompting sections retitled to U3.5 Pro, use speech_models: ["universal-3-5-pro"] and show prompt + keyterms together (also drops the nonexistent aai.SpeechModel.universal_3_pro enum usage). Sync STT prompt left as-is (its spec defines it as an instruction prepended to the system prompt — a genuinely different endpoint).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated sync of the AssemblyAI skill against the latest
assemblyai-docsAPI spec (changes landed June 18–29, 2026, after the last skill update). Each change is tied to a verified docs/spec commit.Changes
Voice Agents — stored-agent binding via
agent_id(spec #157)session.updatecan now bind to a reusable stored agent by sending{"agent_id": "<id>"}as the only field insession, instead of inline config. The two modes are mutually exclusive — sending both raises a validation error.agent_idcan only be set beforesession.ready.references/api-reference.mddocumentingPOST/GET/PUT/DELETE /v1/agentsonagents.assemblyai.com, the create-body fields (requiredname/system_prompt/voice), and auth.SKILL.md.LLM Gateway — Claude 4 models removed (docs #133)
claude-opus-4-20250514andclaude-sonnet-4-20250514were removed from the LLM Gateway available-models list. Marked removed June 2026 inreferences/llm-gateway.mdand added a Common Mistakes correction (these IDs are common in training data).REST upload — binary-body gotcha (docs #144)
POST /v2/uploadrequires raw bytes (--data-binary @file). Using-d/--dataor a JSON body returns a validupload_urlbut later fails transcription withTranscoding failed. File type application/json. Added toreferences/api-reference.mdand a Common Mistakes row inSKILL.md.Streaming — close codes + billing (docs #126, #147)
1011(internal/connection-establishment error) and documented the inactivity-timeout closure under3006in the streaming Error Codes table.Speech Understanding —
speakersis now the primary form (docs #164)speakers(array of{name}/{role}objects) for both name and role identification; noted inreferences/speech-understanding.md.known_valuesremains fully supported in the spec.Verified but intentionally not changed
effort(low/medium) — described in prose but has no documented parameter syntax and is absent from the OpenAPI schema, so it's too speculative to add precisely.Pre-Recorded — Universal-3.5 Pro available for async
speech_models: ["universal-3-5-pro"]onPOST /v2/transcript. Documented atpre-recorded-audio/universal-3-5-pro.mdx(18 languages with auto-fallback to Universal-2, native code-switching, contextualprompt,keyterms_promptup to 1,000). Added to the Pre-Recorded model table inSKILL.md. (The formal OpenAPIspeech_modelsenum still lists onlyuniversal-3-pro/universal-2, but the async API accepts it.)Prompting reframed around Universal-3.5 Pro
Per the Prompting and Keyterms guide (applies to both streaming and async for Universal-3.5 Pro):
promptis a contextual description of the audio (domain → scenario → detail), not formatting/behavioral instructions (those are ignored).promptandkeyterms_promptare complementary (used together), not mutually exclusive — corrected acrossSKILL.md,streaming.md,api-reference.md,python-sdk.md,js-sdk.md.SKILL.mdPrompting section around Universal-3.5 Pro (three specificity levels, start-with-nothing guidance, keyterms limits: 100 streaming / 1,000 async, corrected~1500 chars). Dropped obsolete instruction-style guidance (Non-negotiable prefix, positive-phrasing, negative-prompt gotcha) and "instruct via prompt" disfluencies clauses. SDK sections retitled and now showprompt+keytermstogether withspeech_models: ["universal-3-5-pro"].promptleft unchanged — its spec defines it as an instruction prepended to the system prompt (a genuinely different endpoint).