Run docgen against another repo; optional Grok/xAI provider

AGENTS.md

@@ -5,15 +5,15 @@
 Stable goals for this repository:
 
 1. **Embeddable generator** — **`pip install docgen`** (git URL or editable install from this repo), a consumer **bundle** (`docgen.yaml` + hints/narration), and shell/CI are enough to build and maintain narrated demos. **Do not vendor** this library into a product repo’s `src/`; pin via `requirements-docgen.txt` / `pipx` / `uv tool`. **No IDE assistant is required**; optional **`docgen wizard`** is a local web app only.
-2. **Hybrid config and prose** — **`docgen.yaml`** should stay maintainable: deterministic merges (**`yaml-generate`**, gap checks) plus **optional OpenAI** where it adds value (narration hints, declarative scene YAML). Prefer **Git-reviewed** changes over opaque single-shot generation.
-3. **Video stack** — Long-form demos pair **Markdown narration**, **OpenAI TTS**, **Whisper-style timestamps**, **Manim** visuals, **`compose`** (ffmpeg), **`concat`**, and **`validate`** (sync and narration lint). The CLI also supports **`pages`** for static preview sites.
+2. **Hybrid config and prose** — **`docgen.yaml`** should stay maintainable: deterministic merges (**`yaml-generate`**, gap checks) plus **optional LLM** (OpenAI or Grok) where it adds value (narration hints, declarative scene YAML). Prefer **Git-reviewed** changes over opaque single-shot generation.
+3. **Video stack** — Long-form demos pair **Markdown narration**, **TTS** (OpenAI or xAI), **Whisper-style timestamps**, **Manim** visuals, **`compose`** (ffmpeg), **`concat`**, and **`validate`** (sync and narration lint). The CLI also supports **`pages`** for static preview sites.
 4. **Stable contracts** — CLI, exit codes, and reusable workflows should stay predictable for downstream repos and automation.
 5. **Library, not app** — There is **no in-repo dogfood bundle**. Consumer projects (e.g. `course-builder`) are the integration test of record. The library must not import or special-case any consumer.
 6. **Tool-only generation** — Narration, merged **`docgen.yaml`**, compiled **`scenes.py`**, TTS audio, composed media, and other **generated** artifacts must come from **docgen** (CLI/library) and **committed wrapper scripts** that call it — not from hand-edited outputs passed off as sources. In a consumer bundle, prefer **`hints/*.md`** + **`yaml-generate`** over ad-hoc YAML surgery. Cursor rules: **`.cursor/rules/docgen-tools-only.mdc`**, **`.cursor/rules/no-asset-edits.mdc`**.
 
 ## Protected assets in a consumer bundle (Cursor must not edit)
 
-`docgen` + OpenAI are the **only** path that produces category **C** outputs — see **`.cursor/rules/no-asset-edits.mdc`**. Summary (paths relative to a consumer bundle, typically `docs/demos/`):
+`docgen` (OpenAI or Grok) is the **only** path that produces category **C** outputs — see **`.cursor/rules/no-asset-edits.mdc`**. Summary (paths relative to a consumer bundle, typically `docs/demos/`):
 
 - **Outputs (do not hand-edit):** `<bundle>/docgen.yaml` (as emitted by **`yaml-generate`**); `<bundle>/narration/*.md` (except `README.md`); `<bundle>/animations/scenes.py`, `timing.json`, `animations/specs/*.scene.yaml` (scene pipeline); `<bundle>/audio/*.mp3`; `<bundle>/images/*.png` (scene image assets from **`image-generate`**); `<bundle>/recordings/**` where applicable.
 - **Inputs (maintainer-owned):** `<bundle>/hints/**` with YAML front matter (`docgen.segment`, `docgen.wiring`); maintainer scripts under the bundle; `tests/**` fixtures inside this library; `<bundle>/narration/README.md`.
@@ -44,9 +44,9 @@ Commands registered on the **`docgen`** CLI include:
 - **`wizard`** — local web UI for narration/bootstrap workflows (focus files, **in-place narration revise**, per-segment **asset freshness** + **rebuild-from-here**, Vue **Benchmark** view, **Tool** tab to pip-upgrade docgen and pin `requirements-docgen.txt`).
 - **`gui`** — desktop window over the same Vue/Flask UI (`pip install 'docgen[gui]'` for pywebview). ``--smoke`` is a headless HTTP check. PyInstaller spec: ``packaging/docgen-gui.spec``. Frozen apps resolve templates/static/benchmark JSON via ``docgen.resources``.
 - **`freeze`** — ``docgen freeze`` builds the **`docgen-gui`** onedir (`pip install 'docgen[packaging]'`). Optional ``--smoke`` runs the binary headless. Do not run a full freeze in routine pytest; set ``DOCGEN_FREEZE_SMOKE=1`` for the optional test.
-- **`tts`** — text-to-speech for segment files.
-- **`timestamps`** — word/segment timing (`timing.json`). Default engine **`local`** aligns the known narration text against the mp3 offline (ffmpeg silencedetect, no API); **`--engine whisper`** keeps OpenAI whisper-1 transcription. Both emit the same Whisper-shaped blocks.
-- **`image-generate`** — render scene-spec **image elements** (`image:` + `prompt:` boxes) via the OpenAI Images API into the bundle (also runs for missing assets inside `generate-all`).
+- **`tts`** — text-to-speech for segment files (OpenAI or xAI `/v1/tts`).
+- **`timestamps`** — word/segment timing (`timing.json`). Default engine **`local`** aligns the known narration text against the mp3 offline (ffmpeg silencedetect, no API); **`--engine whisper`** uses OpenAI whisper-1 or xAI `/v1/stt` when `ai.provider` is grok. Both emit the same Whisper-shaped blocks.
+- **`image-generate`** — render scene-spec **image elements** (`image:` + `prompt:` boxes) via OpenAI Images or xAI Imagine into the bundle (also runs for missing assets inside `generate-all`).
 - **`manim`** — render Manim scenes declared in config.
 - **`compose`** — mux narration audio with visual sources via ffmpeg.
 - **`validate`** / **`validate --pre-push`** — drift, narration lint, Manim hints, **`timing_sync`**, **`story_end`** (last paced reveal vs audio end; hard fail), **`scene_assets`** (pre-render: stuck-board cadence, frame-budget overlaps, `MANIM_FONT` consistency, stale helpers / stale compiled class — hard fail; also a `generate-all` gate before Manim), **`av_sync`** (soft; prefers scene-spec labels as OCR anchors), **`subject_beat_coverage`** (declarative specs vs narration topic beats; hard fail when enabled), and related checks.
@@ -93,7 +93,11 @@ Tests should cover **CLI-visible behavior** and contracts that adopters rely on:
 - **Virtualenv:** the project is installed editable into **`/workspace/.venv`** (created by the startup update script). Shells do **not** auto-activate it — run `. /workspace/.venv/bin/activate` (or prefix the venv path) before `docgen`, `pytest`, or `ruff`. The `docgen` console script lives at `/workspace/.venv/bin/docgen`.
 - **System deps are pre-baked in the VM snapshot** (not the update script): `ffmpeg` + `tesseract-ocr` (validation/compose/OCR), plus `build-essential`, `python3-dev`, `libcairo2-dev`, `libpango1.0-dev`, `pkg-config` (needed to build the `manim` extra's `manimpango`/`pycairo` wheels). If a fresh VM ever lacks these, reinstall via apt before `pip install`.
 - **Standard commands** are in `README.md` / `pyproject.toml` / `.github/workflows/ci.yml`: lint `ruff check src/ tests/`; tests `pytest tests/ -v --tb=short`; **required** `docgen benchmark` (CI job `benchmark`). The CI unit job also exports `PYTHONPATH=src` (not needed locally because of the editable install, but harmless).
-- **OpenAI-gated vs offline commands:** `tts`, `timestamps --engine whisper`, `image-generate`, `narration-generate`, `scene-spec-generate`, and `yaml-generate --llm` call OpenAI and need `OPENAI_API_KEY` (integration tests auto-skip without it). Fully offline: `init`, `scene-compile`, `manim`, `compose`, `validate`, `lint`, `pages`, `concat`, `yaml-generate` (no `--llm`), `timestamps` (default `local` engine), and `benchmark`.
+- **OpenAI / Grok vs offline commands:** `tts`, `timestamps --engine whisper`, `image-generate`, `narration-generate`, `scene-spec-generate`, and `yaml-generate --llm` call a provider and need `OPENAI_API_KEY` or (`DOCGEN_AI_PROVIDER=grok` + `XAI_API_KEY`). Integration tests auto-skip without credentials. Fully offline: `init`, `scene-compile`, `manim`, `compose`, `validate`, `lint`, `pages`, `concat`, `yaml-generate` (no `--llm`), `timestamps` (default `local` engine), and `benchmark`.
+- **Generate against another repo (do not vendor):** this environment already has `docgen` on PATH. Point at a consumer checkout or clone URL — nothing is copied into that project's `src/`:
+  `docgen --repo /path/to/consumer generate-all`
+  `docgen --repo github.com/org/consumer init --defaults`
+  `--repo` finds `docs/demos/docgen.yaml`. Add the consumer as a Cloud **repository dependency** if you clone by GitHub URL (uses `GITHUB_TOKEN`). Set `XAI_API_KEY` or `OPENAI_API_KEY` as an environment secret for networked stages.
 - **`scene-compile` gotcha:** paced specs (`wait_word`) need a `timing.json` entry for that stem (`docgen timestamps` after TTS). Prefer `scene-compile --retime` after fresh timestamps; for a fully offline smoke render, author rows without wait indices only if you accept unpaced reveals.
 - **No in-repo dogfood bundle:** exercise the pipeline against a scratch bundle (`docgen init /tmp/<name> --defaults` in a throwaway git dir). Do not hand-edit consumer generated assets (see `.cursor/rules/no-asset-edits.mdc`).
 - **Wizard / desktop GUI:** `docgen wizard --port 8501` is the Flask UI (bundle optional for the Benchmark view). `docgen gui` / `docgen benchmark --gui` open the Vue benchmark view in a desktop window when `pywebview` is installed. `docgen gui --smoke` is the headless HTTP check (default pytest). `docgen freeze` builds the GUI onedir; do not run a full PyInstaller freeze in routine CI/pytest. Keep `packaging/docgen-gui.spec` and `docgen.gui.packaging` in sync.

README.md

@@ -1,8 +1,8 @@
 # docgen — documentation generator
 
 Reusable Python library and CLI for **narrated demo videos** built around **Manim**,
-**OpenAI TTS**, and **ffmpeg** composition. Aimed at long-form, scripted explainers
-that walk through how a system works.
+**TTS** (OpenAI or xAI Grok), and **ffmpeg** composition. Aimed at long-form,
+scripted explainers that walk through how a system works.
 
 ## Suite handbook (Courseforge)
 
@@ -31,12 +31,12 @@ If you still need the legacy behaviour, pin a pre-removal commit
 ## What docgen does today
 
 - **TTS narration** — generate MP3 audio from Markdown scripts via OpenAI
-  `gpt-4o-mini-tts`.
+  `gpt-4o-mini-tts`, or xAI `/v1/tts` when `ai.provider` is `grok`.
 - **Word-level timestamps without Whisper** — the default `local` engine aligns
   the known narration text against the TTS mp3 offline (ffmpeg `silencedetect`
-  + proportional interpolation); no API call or transcription. OpenAI
-  `whisper-1` remains available via `timestamps.engine: whisper` /
-  `docgen timestamps --engine whisper`. Both engines write the same
+  + proportional interpolation); no API call or transcription. Network
+  transcription (`timestamps.engine: whisper`) uses OpenAI `whisper-1` or xAI
+  `/v1/stt` when the provider is Grok. Both engines write the same
   `timing.json` shape.
 - **Manim animations (default: declarative scene specs)** — primary visual surface.
   Prefer **`animations/specs/*.scene.yaml`** via **`docgen scene-spec-generate`**
@@ -47,11 +47,12 @@ If you still need the legacy behaviour, pin a pre-removal commit
   map each paced label to a **`wait_word`** index. Hand-maintained custom Manim classes
   may still live in `animations/scenes.py` outside `BEGIN/END GENERATED SCENE` markers
   (use **`--skip-scene-retime`** to bypass the declarative stage).
-- **OpenAI image assets in Manim scenes** — a scene-spec box may be an **image
+- **Image assets in Manim scenes** — a scene-spec box may be an **image
   element** (`image: images/<name>.png` + `prompt:`); `docgen image-generate`
-  renders the prompt via the OpenAI Images API (default `gpt-image-1`) and the
-  compiled scene shows it with the `_image` helper (`ImageMobject`, `Group`
-  rows). `generate-all` fills in missing assets automatically.
+  renders the prompt via OpenAI Images (default `gpt-image-1`) or xAI Imagine
+  (`grok-imagine-image-2.0` when `ai.provider` is `grok`). The compiled scene
+  shows it with the `_image` helper. `generate-all` fills in missing assets
+  automatically.
 - **ffmpeg composition** — combine narration audio and Manim video into final
   segments, with a freeze-tail guard.
 - **Validation** — A/V drift, freeze ratio, OCR error scan, layout, narration lint,
@@ -65,8 +66,56 @@ If you still need the legacy behaviour, pin a pre-removal commit
   docs.
 
 **No IDE lock-in:** maintenance workflows are `docgen` CLI + YAML + shell/CI (and
-OpenAI where a command calls the API). The wizard is a local Flask app, not a
-plugin tied to one editor.
+an LLM/TTS provider where a command calls the API). The wizard is a local Flask
+app, not a plugin tied to one editor.
+
+### Run against another repo (do not vendor)
+
+Install docgen once (venv / pipx / this Cloud environment). Pass the **consumer**
+checkout or clone URL; nothing from this library is copied into that project's
+`src/`.
+
+```bash
+# Local checkout of the product repo
+docgen --repo /path/to/course-builder init --defaults
+docgen --repo /path/to/course-builder yaml-generate
+docgen --repo /path/to/course-builder generate-all
+
+# GitHub URL or org/repo shorthand (shallow clone into DOCGEN_REPO_CACHE)
+docgen --repo github.com/acme/course-builder generate-all
+# equivalent: DOCGEN_REPO=acme/course-builder docgen generate-all
+```
+
+`--repo` looks for `docs/demos/docgen.yaml` (then any other `docgen.yaml` under
+the checkout). `repo_root` in that yaml still points at the consumer so
+narration/scene prompts read *their* sources.
+
+### OpenAI or Grok (xAI)
+
+Default provider is **OpenAI** (`OPENAI_API_KEY`). To substitute **Grok** for
+chat, TTS, Whisper, and image calls:
+
+```yaml
+ai:
+  provider: grok          # openai | grok
+```
+
+```bash
+export DOCGEN_AI_PROVIDER=grok    # overrides yaml
+export XAI_API_KEY=xai-...
+docgen --repo /path/to/consumer generate-all
+```
+
+Existing OpenAI model names in YAML are remapped at call time (`gpt-4o` →
+`grok-4.6`, `gpt-image-1` → `grok-imagine-image-2.0`, TTS voice `coral` →
+`eve`). Chat and images use the OpenAI SDK at `https://api.x.ai/v1`. TTS and
+STT use xAI `POST /v1/tts` and `POST /v1/stt`. Keep `timestamps.engine: local`
+unless you specifically want network STT.
+
+A Cursor Cloud environment that already has ffmpeg / tesseract / Manim build
+deps can run the full pipeline: add `XAI_API_KEY` or `OPENAI_API_KEY` as an
+environment secret (and grant the consumer as a repository dependency if you
+clone by URL).
 
 ## Install (external tool — do not vendor into project `src/`)
 
@@ -115,6 +164,8 @@ CI also installs `ffmpeg` and `tesseract` via apt for unit tests — see `.githu
 
 ## Quick start (in a consumer repo)
 
+From **inside** the consumer bundle (library already on PATH):
+
 ```bash
 cd your-project/docs/demos          # bundle only — library is on PATH via pip
 pip install -r requirements-docgen.txt   # after docgen init, or use the pip line above
@@ -123,10 +174,18 @@ docgen generate-all        # TTS → timestamps → scene retime → Manim → c
 docgen validate --pre-push
 ```
 
+From a machine / Cloud environment that has **docgen installed** but is not the
+consumer repo:
+
+```bash
+docgen --repo /path/to/your-project generate-all
+```
+
 ## CLI commands
 
 | Command | Description |
 |---------|-------------|
+| `docgen --repo PATH_OR_URL …` | Target a consumer checkout or clone URL (also `DOCGEN_REPO`). Finds `docs/demos/docgen.yaml`. Does **not** copy this library into the consumer `src/` |
 | `docgen --version` | Show installed version + recommended `pip install` line (external tool) |
 | `docgen init [TARGET_DIR] [--defaults] [--segments-file FILE]` | Scaffold a bundle: `docgen.yaml`, `requirements-docgen.txt`, wrapper scripts, directories |
 | `docgen wizard [--port 8501]` | Local web GUI: focus files, **revise narration**, asset freshness / rebuild-from-here, Vue **Benchmark** view, and a **Tool** tab to upgrade the installed `docgen` package (pip) + rewrite `requirements-docgen.txt` |
@@ -169,14 +228,15 @@ your IDE or CI) is **not** replaced by the file. To make the file win, set
 environment, or **`DOCGEN_ENV_OVERRIDES=OPENAI_API_KEY,OTHER_KEY`** for specific
 keys only.
 
-When `OPENAI_API_KEY` is present in both the shell and `env_file`, docgen prints a
-one-line hint to stderr so a silent 401 from the wrong key is easier to diagnose.
+When `OPENAI_API_KEY` or `XAI_API_KEY` is present in both the shell and `env_file`,
+docgen prints a one-line hint to stderr so a silent 401 from the wrong key is
+easier to diagnose.
 
 ### Narration from source (owner hints)
 
 Under `narration_from_source` in `docgen.yaml`, the **project owner** lists
 optional `hints` (strings) that steer the model (audience, terminology, what to
-avoid). OpenAI generates the narration `.md` from your repo context
+avoid). The chat model generates the narration `.md` from your repo context
 (`context.paths` / `context.globs`, relative to `repo_root`) plus those hints; the
 result is what `docgen tts` reads. See `docgen.narrate_from_source`.
 
@@ -222,8 +282,12 @@ validation:
     prefer_scene_spec_labels: true  # OCR anchors from paced box labels when specs exist
     visual_types: [manim]    # only check types with on-screen text
 
+```yaml
+ai:
+  provider: openai           # openai | grok (xAI). Override with DOCGEN_AI_PROVIDER.
+
 timestamps:
-  engine: local              # local (default, offline) or whisper (OpenAI whisper-1)
+  engine: local              # local (default, offline) or whisper (OpenAI whisper-1 / xAI STT)
   silence_noise_db: -35.0    # ffmpeg silencedetect threshold for the local engine
   min_silence_sec: 0.3