Spike: voice orchestrator for s2code (GPT-Live/Realtime + s2code MCP tools) #337

Goal

Spike a voice orchestrator for s2code. You talk to it, and it steers and manages s2code sessions in every connected environment (local machine, dev box, Mac mini, and others) using only s2code's own MCP tools. No new server-side control surface.

Example asks: "What's running on the dev box?", "Tell the shuvcode thread to rerun tests", "Start a new thread in project X on the Mac and have it fix the failing build", "Interrupt that", "Read me the last result".

Rough design

• Voice loop: OpenAI GPT-Live (preferred) or the Realtime API as the speech-to-speech front end, with function calling.
• Tool bridge: a thin client that exposes s2code's MCP tools to the voice model as functions. Each call goes to the target environment's s2code MCP server over the existing authenticated MCP HTTP transport (reusing the existing MCP OAuth and environment links).
• Environment routing: use t3_environment_links / t3_environment_read to discover environments and pick a target. Speak the environment name back before any side-effecting call.
• Long-running work: fire the task and return right away (t3_thread_send, t3_thread_launch), then poll or wait (t3_thread_wait, t3_background_wait) and speak a short summary when it settles.
• Safety: destructive tools (delete, transfer, workflow stop) need an explicit spoken confirmation. Use a read-only mode by default for the first spike.

MCP tools it would use

• Discovery: t3_environment_links, t3_environment_read, t3_project_list, t3_project_read, t3_thread_list, t3_thread_search, t3_thread_read, t3_worktree_list, t3_worktree_status
• Steering: t3_thread_launch, t3_thread_send, t3_thread_interrupt, t3_thread_update, t3_thread_configure, t3_queue, t3_pending_request (answer approvals and user-input prompts by voice)
• Waiting: t3_thread_wait, t3_background_wait
• Later / guarded: t3_workflow_stop, t3_thread_transfer, t3_worktree_handoff, t3_project_create/clone, t3_environment_link/unlink, t3_preview_list/close

Speech options

1. Cloud S2S: GPT-Live or Realtime does speech-to-text, reasoning and text-to-speech in one loop. Lowest latency and least code.
2. Hybrid / fallback (Cap's preference for local voice): Parakeet (local speech-to-text) and Kokoro (local text-to-speech, OpenAI-compatible server) around a text model with tool calls. No audio leaves the house, and the API cost is text tokens only.
3. Mixed: Parakeet in, GPT-Live / a text model for reasoning, Kokoro out.

Funding note

Cap's ChatGPT Pro ($200 tier) plan should include about $200/month in OpenAI API credits (the new ChatGPT API-credits benefit), which covers GPT-Live and the Realtime API. Rough estimate: about 44 hours/month of GPT-Live with tool use (range about 30–60 h depending on backend usage; voice-only ceiling about 67 h). Pure Realtime speech-to-speech with tools and history is lower (about 10–35 h). Credits expire monthly, so unused hours are lost. That argues for GPT-Live for interactive use, with the local Parakeet/Kokoro path for always-on or long sessions.

Open questions

• GPT-Live vs. Realtime: which one has better tool-calling latency and reliability for multi-step MCP chains?
• Where does the orchestrator run: the desktop app, the server, or a standalone client? How does it hold auth for multiple environments at once?
• How to name and disambiguate environments, projects and threads by voice ("the shuvcode thread on the Mac")?
• What it speaks back vs. shows on screen for long outputs (diffs, logs).
• Barge-in and interrupt semantics: does saying "stop" interrupt the speech, the tool call, or the s2code thread?
• Approval UX for side-effecting and destructive calls.
• Do we need a lightweight summary tool server-side, or can the model read t3_thread_read output directly?

Spike success criteria

• Voice session lists environments and threads across at least 2 environments.
• By voice: read a thread's latest status, send a message to an existing thread, and launch a new thread in a named project.
• Interrupt a running thread by voice, and answer one pending approval by voice.
• Spoken summary when a waited-on thread settles.
• Median turn latency (end of speech to start of reply) measured for cloud S2S vs. the Parakeet+Kokoro hybrid.
• Cost per hour measured against the credit budget.
• Short write-up: recommendation and next steps.

Scope: spike / prototype only. No changes to existing MCP tool contracts.

— opened by Code Monkey (bot) for Cap