Skip to content

feat(local): pass a JSON schema from run_model, starting with the quiz - #2032

Merged
MODSetter merged 3 commits into
MODSetter:devfrom
vintcessun:feat/1990-quiz-schema
Sep 29, 2026
Merged

MODSetter merged 3 commits into
MODSetter:devfrom
vintcessun:feat/1990-quiz-schema

Conversation

@vintcessun

@vintcessun vintcessun commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

What

The first of the per-format PRs for #1990: run_model() can pass a JSON schema, and the quiz is the first format to send one.

  • run_model() takes json_schema and passes it straight to generator.chat(). Both providers already accept it and already fall back, so nothing in them changes. The Generator protocol gains the parameter both implementations already take.
  • content/quiz/schema.py holds the quiz reply's schema beside its prompts. It uses only types, required and array bounds: options is bounded to exactly OPTIONS, since the builder drops any other count, and array bounds become a bounded repetition in json-schema-to-grammar.cpp at b11050. That the answer is one of the options cannot be said in a schema, so build() still checks it. OPTIONS moves into the schema module so the pipeline can import both without a cycle.
  • parse_json() is unchanged and still reads an unconstrained reply: from an endpoint that ignores response_format, or from the llama.cpp 400 retry.
  • Docs: the Constrained decoding section in selection.md says Studio passes schemas now, the Known-gaps line is narrowed to the formats still without one, and studio.md and proposals/agent/01-which-engine.md no longer say that nothing passes one.

Why

Small local models are the ones that drift off a JSON shape, and a quiz that drifts fails the job after the model has already spent the time. With a schema, llama.cpp masks every token that would break the shape. This implements decision 1 of docs/proposals/agent/04-workflows.md for the quiz.

Planned order for the remaining PRs, one format each, with its schema and a test that the schema reaches the model: flashcards → mind map → HTML → image → infographic → podcast outline and draft. The podcast goes last because its draft has the Repair loop. That last PR deletes the Known-gaps line and says Fixes #1990. Chat, and decisions 2–5 of 04-workflows.md, stay out of scope.

Refs #1990

How to test

cd surfsense_local/backend
uv run pytest -m unit
uv run pytest -m integration -k studio
uv run ruff check .
cd ../.. && python scripts/check_docs.py

New tests, each written failing first:

  • test_studio_generate.py::test_a_schema_reaches_llama_server, and ::test_a_format_that_asks_for_no_schema_sends_none for office and the summary.
  • test_studio_content.py::test_a_quiz_asks_the_model_for_its_schema, and ::test_the_quiz_schema_asks_for_what_the_builder_keeps.

Locally on Windows: unit 3053 passed, Studio integration 25 passed, and ruff check and check_docs.py are clean.

High-level PR Summary

This PR introduces JSON schema support for the quiz format in Studio's local model pipeline. The run_model() function now accepts a json_schema parameter that gets passed to the generator, enabling llama.cpp to constrain token generation to the specified schema. The quiz format is the first to use this capability, with its schema defined in a new schema.py module alongside the prompts. The schema uses only types, required fields, and array bounds to ensure small local models stay on track without drifting from the expected JSON structure. Documentation is updated to reflect that Studio now passes schemas, with the remaining formats (flashcards, mind map, HTML, image, infographic, podcast) still pending schema implementation.

⏱️ Estimated Review Time: 5-15 minutes

💡 Review Order Suggestion
Order File Path
1 surfsense_local/backend/worker/studio/content/quiz/schema.py
2 surfsense_local/backend/modules/llm/providers/protocols.py
3 surfsense_local/backend/worker/studio/shared/generate.py
4 surfsense_local/backend/worker/studio/content/quiz/pipeline.py
5 surfsense_local/backend/tests/unit/worker/test_studio_generate.py
6 surfsense_local/backend/tests/unit/worker/test_studio_content.py
7 docs/architecture/studio.md
8 docs/architecture/local-models/selection.md
9 docs/proposals/agent/01-which-engine.md

Need help? Join our Discord

@vercel

vercel Bot commented Sep 29, 2026

Copy link
Copy Markdown

@vintcessun is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants