diff --git a/docs/architecture/local-models/selection.md b/docs/architecture/local-models/selection.md index 4f41538bb5..94aef63737 100644 --- a/docs/architecture/local-models/selection.md +++ b/docs/architecture/local-models/selection.md @@ -238,6 +238,13 @@ a `json_schema` request, which some templates, is retried once unconstrained. Chat prose is deliberately unconstrained. +Studio passes a schema through `run_model()` +([`generate.py`](../../../surfsense_local/backend/worker/studio/shared/generate.py)), +each format's beside its prompts, as the quiz's +([`schema.py`](../../../surfsense_local/backend/worker/studio/content/quiz/schema.py)). +A reply that arrives unconstrained, from an endpoint that ignores +`response_format` or from the 400 retry, is still read by `parse_json()`. + ## How it is tested [`surfsense_local/backend/tests/unit/llm/profile/`](../../../surfsense_local/backend/tests/unit/llm/profile/) @@ -252,6 +259,6 @@ over HTTP. - The tier fallback keys on the provider name, not on loopback: `Fingerprint.local` is `provider == "llamacpp"`, so a local endpoint reached through a connection falls to `capable` when nothing else is known; the decision is to key on `host_destination()`, which already computes loopback. - A remote listing row with no `hugging_face_id` always sets `vendor` (to `owned_by`, or to the id's prefix even when that is empty) and never reads the size in the name, so such a model is classified `frontier`: a `qwen3-4b` from a local endpoint whose listing carries no `hugging_face_id` gets frontier prompts. Featherless lists every model this way (`"owned_by": "Feather"`, no `hugging_face_id`), so every model there, Qwen3 0.6B included, gets frontier prompts. - Local fingerprints come from the filename only: `LlamaCppProvider` has no `inspect()`, so `from_llamacpp()`, which reads `general.parameter_count` from `/props`, is never called. -- No caller passes `json_schema`: the providers support constrained decoding, but no Studio format or chat call uses it, so format compliance still depends on the prompt. +- Only the quiz passes `json_schema`: mind map, flashcards, HTML, image, infographic and the podcast's outline and draft still ask for JSON in the prompt alone, so their format compliance depends on it. - Chat cannot send an image: `Message.content` is a `str`, so even a model with `vision` has no way to receive one. - Nothing measures whether three tiers are still needed; once constrained decoding carries format compliance, a tier would carry reasoning depth only, which plausibly collapses three tiers to two. diff --git a/docs/architecture/studio.md b/docs/architecture/studio.md index 5a61e19358..9390a1b744 100644 --- a/docs/architecture/studio.md +++ b/docs/architecture/studio.md @@ -53,7 +53,7 @@ Eight formats follow it: - **summary**: the model's markdown is the body, titled by its first `# ` heading. - **mindmap**: JSON nodes, at most 10 branches, become a markdown outline, an H1 over nested bullets, which Markmap draws on the client and which stays readable as text. - **flashcards**: JSON cards, at most 20, become a deck JSON file and a markdown body. -- **quiz**: JSON questions, at most 10, each kept only with exactly four options and an answer among them, become a quiz JSON file and a markdown body. +- **quiz**: JSON questions, written under a schema ([`schema.py`](../../surfsense_local/backend/worker/studio/content/quiz/schema.py)), at most 10, each kept only with exactly four options and an answer among them, become a quiz JSON file and a markdown body. - **html**: a JSON title and sections, at most 10. Every value is HTML-escaped into a fixed template, so the page cannot carry a script. - **podcast**: the model outlines the episode from the brief, then drafts it segment by segment, each reply capped at 12 tokens per word of the segment's target, and never under a planned 250-word segment's worth. Uncapped, Qwen3 1.7B looped on a 225-word segment until its 40,960-token window was full, and the JSON retry, which replays the failed reply, could not fit. The target is the outline's own guess: Qwen3 1.7B once gave a segment 20 words, wrote past them, and a 240-token cap ended both replies mid-JSON. The chosen audio model voices every line through audio.cpp's server, and the transcript is the body. - **image**: the model writes a title and an image prompt, and the image model paints the prompt. diff --git a/docs/proposals/agent/01-which-engine.md b/docs/proposals/agent/01-which-engine.md index 1f225b042b..9036686702 100644 --- a/docs/proposals/agent/01-which-engine.md +++ b/docs/proposals/agent/01-which-engine.md @@ -13,7 +13,7 @@ - Any GGUF model can be held to a JSON schema. At `b11050`, a request with `response_format` gets a grammar that allows the model's reasoning block first, then the JSON, optionally inside a `json` code fence ([`common/chat-auto-parser-generator.cpp`](https://github.com/ggml-org/llama.cpp/blob/b11050/common/chat-auto-parser-generator.cpp)). - In the same file, a request with both `response_format` and `tools` gets the schema parser and no tool-call parsing. A request uses one or the other. -- Both chat providers accept a `json_schema`, and no caller passes one yet ([selection](../../architecture/local-models/selection.md), Known gaps). +- Both chat providers accept a `json_schema`, and only Studio's quiz passes one yet ([selection](../../architecture/local-models/selection.md), Known gaps). ## Tool support for remote models diff --git a/surfsense_local/backend/modules/llm/providers/protocols.py b/surfsense_local/backend/modules/llm/providers/protocols.py index ae572f61eb..a28a704a98 100644 --- a/surfsense_local/backend/modules/llm/providers/protocols.py +++ b/surfsense_local/backend/modules/llm/providers/protocols.py @@ -41,6 +41,7 @@ def chat( max_tokens: int | None = None, temperature: float | None = None, reasoning: bool | None = None, + json_schema: dict | None = None, ) -> AsyncIterator[str]: """The answer text alone; a thinking model's trace is left out.""" ... @@ -53,6 +54,7 @@ def chat_deltas( max_tokens: int | None = None, temperature: float | None = None, reasoning: bool | None = None, + json_schema: dict | None = None, ) -> AsyncIterator[Delta]: """The reply as it streams, with the trace marked apart from the answer.""" ... diff --git a/surfsense_local/backend/tests/unit/worker/test_studio_content.py b/surfsense_local/backend/tests/unit/worker/test_studio_content.py index 3d03e2fd52..80d55cc4ec 100644 --- a/surfsense_local/backend/tests/unit/worker/test_studio_content.py +++ b/surfsense_local/backend/tests/unit/worker/test_studio_content.py @@ -1,6 +1,7 @@ """Content kinds: the model writes markdown or JSON, the worker renders it as-is.""" import json +from types import SimpleNamespace import pytest @@ -8,7 +9,9 @@ from worker.studio.content.flashcards import pipeline as flashcards from worker.studio.content.mindmap import pipeline as mindmap from worker.studio.content.quiz import pipeline as quiz +from worker.studio.content.quiz import schema as quiz_schema from worker.studio.content.summary import pipeline as summary +from worker.studio.shared import generate from worker.studio.shared.text import parse_json pytestmark = pytest.mark.unit @@ -49,6 +52,33 @@ def test_a_quiz_is_capped_where_its_prompt_says_it_is() -> None: assert len(json.loads(built.primary)["questions"]) == quiz.QUESTIONS +def test_a_quiz_asks_the_model_for_its_schema(monkeypatch: pytest.MonkeyPatch) -> None: + """The shape the builder reads is enforced while the model writes, so a + small model cannot drift off it; the prompt still says it in prose.""" + sent: list[dict | None] = [] + + def fake_run_model(*_args: object, json_schema: dict | None = None, **_kw: object): + sent.append(json_schema) + return '{"title": "T", "questions": []}' + + monkeypatch.setattr(generate, "run_model", fake_run_model) + model = SimpleNamespace(tier=Tier.COMPACT) + + quiz.render(model, [], None) + + assert sent == [quiz_schema.REPLY] + + +def test_the_quiz_schema_asks_for_what_the_builder_keeps() -> None: + """A question the builder would drop is one the grammar should not allow.""" + question = quiz_schema.REPLY["properties"]["questions"]["items"] + options = question["properties"]["options"] + + assert set(quiz_schema.REPLY["required"]) == {"title", "questions"} + assert set(question["required"]) == {"question", "options", "answer", "explanation"} + assert options["minItems"] == options["maxItems"] == quiz.OPTIONS + + def test_every_content_kind_is_asked_for_in_its_own_words_at_every_tier() -> None: """One prompt per kind and tier: a wording that suits a quiz never reaches a deck.""" asked = { diff --git a/surfsense_local/backend/tests/unit/worker/test_studio_generate.py b/surfsense_local/backend/tests/unit/worker/test_studio_generate.py index 61dc99c305..acf50f6a83 100644 --- a/surfsense_local/backend/tests/unit/worker/test_studio_generate.py +++ b/surfsense_local/backend/tests/unit/worker/test_studio_generate.py @@ -42,6 +42,29 @@ def test_a_cap_reaches_llama_server() -> None: assert fake.chat_bodies[-1]["max_tokens"] == 1200 +def test_a_schema_reaches_llama_server() -> None: + """A format that asks for JSON gets a grammar, not only a request in prose.""" + fake = FakeRouter(["qwen3"]) + provider = LlamaCppProvider("http://127.0.0.1:1234", transport=fake.transport()) + schema = {"type": "object", "properties": {"title": {"type": "string"}}} + + generate.run_model( + ResolvedGeneration(SELECTION, provider), "system", [], json_schema=schema + ) + + assert fake.chat_bodies[-1]["response_format"]["json_schema"]["schema"] == schema + + +def test_a_format_that_asks_for_no_schema_sends_none() -> None: + """Office writes Python and the summary writes markdown; neither is JSON.""" + fake = FakeRouter(["qwen3"]) + provider = LlamaCppProvider("http://127.0.0.1:1234", transport=fake.transport()) + + generate.run_model(ResolvedGeneration(SELECTION, provider), "system", []) + + assert "response_format" not in fake.chat_bodies[-1] + + class _ModelThatNeverFinishes: """Streams forever, or reads its prompt forever, until the caller hangs up.""" diff --git a/surfsense_local/backend/worker/studio/content/quiz/pipeline.py b/surfsense_local/backend/worker/studio/content/quiz/pipeline.py index 30087dbceb..857e9ff928 100644 --- a/surfsense_local/backend/worker/studio/content/quiz/pipeline.py +++ b/surfsense_local/backend/worker/studio/content/quiz/pipeline.py @@ -3,11 +3,11 @@ from modules.llm import prompting from modules.llm.profile import Tier from modules.llm.resolution import ResolvedGeneration +from worker.studio.content.quiz.schema import OPTIONS, REPLY from worker.studio.shared import generate from worker.studio.shared.artifact import Built, Source from worker.studio.shared.text import as_list, as_text, parse_json, slug -OPTIONS = 4 # The frontier prompt leaves the count to the material, so the ceiling is kept here. QUESTIONS = 10 @@ -15,9 +15,10 @@ def render( model: ResolvedGeneration, sources: list[Source], user_prompt: str | None ) -> Built: - return build( - generate.run_model(model, prompt(model.tier, user_prompt), sources), sources + raw = generate.run_model( + model, prompt(model.tier, user_prompt), sources, json_schema=REPLY ) + return build(raw, sources) def prompt(tier: Tier, user_prompt: str | None) -> str: diff --git a/surfsense_local/backend/worker/studio/content/quiz/schema.py b/surfsense_local/backend/worker/studio/content/quiz/schema.py new file mode 100644 index 0000000000..e96cbecbdd --- /dev/null +++ b/surfsense_local/backend/worker/studio/content/quiz/schema.py @@ -0,0 +1,36 @@ +"""The quiz reply's shape, enforced as a grammar while the model writes. + +Flat and plain on purpose: only types, `required` and array bounds, which +llama.cpp's json-schema-to-grammar turns into a bounded repetition at b11050. +That an answer is one of its options cannot be said here; the builder still +checks it. +""" + +# The prompts ask for exactly this many; the builder drops any other count. +OPTIONS = 4 + +REPLY = { + "type": "object", + "properties": { + "title": {"type": "string"}, + "questions": { + "type": "array", + "items": { + "type": "object", + "properties": { + "question": {"type": "string"}, + "options": { + "type": "array", + "items": {"type": "string"}, + "minItems": OPTIONS, + "maxItems": OPTIONS, + }, + "answer": {"type": "string"}, + "explanation": {"type": "string"}, + }, + "required": ["question", "options", "answer", "explanation"], + }, + }, + }, + "required": ["title", "questions"], +} diff --git a/surfsense_local/backend/worker/studio/shared/generate.py b/surfsense_local/backend/worker/studio/shared/generate.py index 9160017b56..b2d6841f1d 100644 --- a/surfsense_local/backend/worker/studio/shared/generate.py +++ b/surfsense_local/backend/worker/studio/shared/generate.py @@ -31,6 +31,7 @@ def run_model( *, repair: Repair | None = None, max_tokens: int | None = None, + json_schema: dict | None = None, ) -> str: """Send one system prompt plus the grounding to the chosen generation model. @@ -40,6 +41,9 @@ def run_model( A retry passes `repair`, which replays the failed reply as the model's own turn before asking for the correction. Without it an error naming a line points at a script the model was never shown, so it can only start over. + + A JSON format passes `json_schema`, which the runtime turns into a grammar. + An endpoint that ignores it still answers, and `parse_json` reads that. """ selected = model.selection messages = [ @@ -64,7 +68,11 @@ def run_model( reply = asyncio.run( _collect( model.generator.chat( - selected.name, messages, max_tokens=max_tokens, reasoning=False + selected.name, + messages, + max_tokens=max_tokens, + reasoning=False, + json_schema=json_schema, ) ) )