Repository navigation
feat(local): pass a JSON schema from run_model, starting with the quiz - #2032
Merged
Merged
Conversation
vintcessun
requested review from
AnishSarkar22,
CREDO23 and
MODSetter
as code owners
September 29, 2026 11:29
|
@vintcessun is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
This was referenced Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The first of the per-format PRs for #1990:
run_model()can pass a JSON schema, and the quiz is the first format to send one.run_model()takesjson_schemaand passes it straight togenerator.chat(). Both providers already accept it and already fall back, so nothing in them changes. TheGeneratorprotocol gains the parameter both implementations already take.content/quiz/schema.pyholds the quiz reply's schema beside its prompts. It uses only types,requiredand array bounds:optionsis bounded to exactlyOPTIONS, since the builder drops any other count, and array bounds become a bounded repetition injson-schema-to-grammar.cppatb11050. That the answer is one of the options cannot be said in a schema, sobuild()still checks it.OPTIONSmoves into the schema module so the pipeline can import both without a cycle.parse_json()is unchanged and still reads an unconstrained reply: from an endpoint that ignoresresponse_format, or from the llama.cpp 400 retry.selection.mdsays Studio passes schemas now, the Known-gaps line is narrowed to the formats still without one, andstudio.mdandproposals/agent/01-which-engine.mdno longer say that nothing passes one.Why
Small local models are the ones that drift off a JSON shape, and a quiz that drifts fails the job after the model has already spent the time. With a schema, llama.cpp masks every token that would break the shape. This implements decision 1 of
docs/proposals/agent/04-workflows.mdfor the quiz.Planned order for the remaining PRs, one format each, with its schema and a test that the schema reaches the model: flashcards → mind map → HTML → image → infographic → podcast outline and draft. The podcast goes last because its draft has the Repair loop. That last PR deletes the Known-gaps line and says
Fixes #1990. Chat, and decisions 2–5 of04-workflows.md, stay out of scope.Refs #1990
How to test
New tests, each written failing first:
test_studio_generate.py::test_a_schema_reaches_llama_server, and::test_a_format_that_asks_for_no_schema_sends_nonefor office and the summary.test_studio_content.py::test_a_quiz_asks_the_model_for_its_schema, and::test_the_quiz_schema_asks_for_what_the_builder_keeps.Locally on Windows: unit 3053 passed, Studio integration 25 passed, and
ruff checkandcheck_docs.pyare clean.High-level PR Summary
This PR introduces JSON schema support for the quiz format in Studio's local model pipeline. The
run_model()function now accepts ajson_schemaparameter that gets passed to the generator, enabling llama.cpp to constrain token generation to the specified schema. The quiz format is the first to use this capability, with its schema defined in a newschema.pymodule alongside the prompts. The schema uses only types,requiredfields, and array bounds to ensure small local models stay on track without drifting from the expected JSON structure. Documentation is updated to reflect that Studio now passes schemas, with the remaining formats (flashcards, mind map, HTML, image, infographic, podcast) still pending schema implementation.⏱️ Estimated Review Time: 5-15 minutes
💡 Review Order Suggestion
surfsense_local/backend/worker/studio/content/quiz/schema.pysurfsense_local/backend/modules/llm/providers/protocols.pysurfsense_local/backend/worker/studio/shared/generate.pysurfsense_local/backend/worker/studio/content/quiz/pipeline.pysurfsense_local/backend/tests/unit/worker/test_studio_generate.pysurfsense_local/backend/tests/unit/worker/test_studio_content.pydocs/architecture/studio.mddocs/architecture/local-models/selection.mddocs/proposals/agent/01-which-engine.md