Repository navigation
Conversation
Every model may now choose the agent, so every build stages opencode and ripgrep: ENABLED_BY_DEFAULT is true, and SURFSENSE_LOCAL_OPENCODE_ENABLED=0 leaves them out of one build. The docs that said installers carry no agent now say they do, and packaging.md records the Windows stage's size (185 MB unpacked, 44 MB as xz) and that electron-builder signs opencode and rg with the app's inherited entitlements, whose allow-jit Bun needs.
Now that every installer carries the agent, a release that ships without it, or with an opencode that signing broke, would pass CI. Each runner now runs check-stage.mjs on the opencode folder its build packaged: both executables and their licences are there, and opencode and rg report their pinned versions. On macOS that runs the copy signed under the hardened runtime, and on a tag's Windows build the copy Azure signed. stage.mjs runs the same check before its swap.
free_port() binds port 0 and lets go, and seconds later opencode or uvicorn binds the number; a parallel run, or any outgoing connection, can take it in between. On the ladder that left opencode unserved and the thread opened as a chat. With SURFSENSE_LIVE_PORT_BASE set, the app's and opencode's ports come from a 100-port block below the Windows dynamic range; unset, any free port is used as before. Ported from slice/ladder-run.
…odels `python -m tests.live.sweep run` picks every OpenRouter model that takes tools, holds 32k tokens and is priced, one per model_key, leaves out the keys the 8-case ladder measured, and lists models from $3/M input as assumed to pass. Each model runs smoke, then the PDF brief and the board pack in lanes of their own; a smoke it fails stops it at 0 of 2, and a single failed case is run once more. Provider and harness faults get two retries and never count. A hard budget, a per-case dollar stop and time limit, a RAM guard and an append-only attempts.jsonl make it safe to leave running and to restart. It writes a summary per model and sweep-results.json for the capability list's generator, and `report` prints progress at any time. Classification, the RAM guard and the process handling are ported from slice/ladder-run's matrix runner.
With opencode on by default, any SURFSENSE_LOCAL_OPENCODE_ENABLED other than "1" turned it off: "true", "yes", and the empty string an unset CI variable gives all built an installer without the agent, and stage.mjs blamed "=0" as it removed the stage. Now unset or empty keeps the default, 1 and 0 decide, and anything else stops the build. The off message prints the value it read.
The pinned opencode-darwin-arm64 arrives Developer ID signed by Anomaly Innovations, with the hardened runtime, a secure timestamp and the five entitlements Bun asks for. osx-sign force-signed it with the app's inherited entitlements, which lack disable-executable-page-protection, and the release's --version smoke barely runs Bun's JIT. mac.signIgnore now leaves opencode's signature in place; rg arrives only ad-hoc signed and is still signed with the app. packaging.md also had Windows wrong: opencode.exe arrives Authenticode-signed by its vendor and rg.exe unsigned, and a tag's build replaces both with SurfSense's signature, so a dry run does not run an unsigned opencode.
Release CI ran check-stage.mjs on the packaged opencode only after electron-builder, and on a tag `--publish always` has by then uploaded every installer and stable*.yml to the draft release, so an opencode that signing broke reached the draft first. An afterSign hook, scripts/opencode/after-sign.mjs, now runs the same check once the app is signed, and on macOS notarized, and a failure stops electron-builder before any target is built. macOS and Windows call it on every build, signed or not; Linux signs nothing and skips it, so the post-build CI step stays as its check.
The screening sweep stops a model that fails smoke, so a false failure there costs it both scored cases. Ministral 14B answered "Monday, November 16, 2026", the note's date, and failed only because the check looked for "16 November". The check now takes either order; it still needs the search, the date and a citation of the note.
In the pilot the RAM guard held a 3-lane sweep to one or two cases at once, so an ETA at --lanes was three times too short. `report` now divides by the lane-minutes ended since the runner started over the minutes since, once ten minutes have passed. A re-run's note also names the case once.
…etry time-outs A 401 or 402 (a revoked key, no credits, "can only afford N tokens") reached the model-failure fallthrough, so once the balance ran low every smoke after it failed in seconds and its model was written as tested below 0/2, with nothing to stop the run. Such a run is now an `account` attempt: never counted, no try used, and the runner starts nothing new and says why in runner.log, status.json and `report`. The same command resumes. A 408 (as a status or a streamed code), httpx's ConnectTimeout, WriteTimeout, PoolTimeout and CloseError, and OpenRouter's "No endpoints found" are now provider faults, retried and never counted, so a Wi-Fi drop or a model with no provider up is not held against the model.
sweep-results.json listed every model under `models`, and the capability list's generator reads each row's counts as a measurement. An unresolved smoke (passed 0, run 0) became a studio_only row noted as a failed smoke, though the model never answered, and a pending row's null counts made the whole file fail validation. Only models with a verdict are under `models` now; unresolved, running and pending ones are under `unfinished`, which the generator ignores. The file also gives its `date`, the day the last counted run ended, for the rows' measured_on.
The screening sweep stops a model that fails smoke at 0 of 2, so the date check must not turn on spelling. The last pattern still missed "November 16th" (no word boundary between "16" and "th"), "the 16th of November", "Nov 16", "16 Nov", "2026-11-16", "16/11/2026" and lower case. It now takes those, and still refuses the old yard's 20 November and other days. The check is a function so a unit test can list the spellings.
…il on it
The board pack failed any answer without "assum", "I chose", "I went with"
and the like. In the screening pilot DeepSeek V4 Pro, an agent-level model
on the 8-case ladder, failed it twice that way: both answers made and named
a PDF and gave the reason ("ready to send as-is", with a Word draft
offered). With one re-run per split, wording alone made it "below". The
ladder branch already graded this rather than checked it; this ports only
that. The case still fails an answer that does not name the format, and
records `says_what_it_assumed` under result.json's new `metrics`.
A unit test drives the case on a scripted app with DeepSeek's answer.
The runner says one Ctrl-C stops new starts and waits for the cases in flight, but each case shared the runner's console, so the terminal's Ctrl-C reached every pytest child too. They died at once and were recorded as harness faults, each using up one of the case's two retries. A case now starts in its own process group (its own session elsewhere), so only the runner takes the Ctrl-C; a second one still kills the cases through kill_tree. The test runs a one-lane sweep in a hidden console of its own and presses Ctrl-C there while the case runs.
Nothing stopped a second `run` on the same --out while the first was alive, though resuming is "run the same command". The second runner took the first's live attempt folders for interrupted ones, charged their spend again, started the same cases a second time and, on Windows, crashed renaming a folder the first still held open. A runner now holds an OS lock on runner.lock for its whole run; a second one raises SweepBusyError naming the holder's pid, and the CLI exits with that message. The lock ends with its process, however it ends.
…the file Windows refuses to replace a file another process has open. The sweep's runner reads each in-flight run's spend.json, so a charge written at that moment raised PermissionError in the proxy's `finally`: it never reached the run's ledger, which the per-case stop and the runner's in-flight estimate both read. The same race with `report` reading sweep-results.json or status.json would have crashed the runner mid-sweep. Both now write through write_whole, which tries the replace again for up to a second.
… model A row was found only by its canonical key, so Ollama's qwen3.8:27b, a Q4_K_M .gguf or unsloth's GGUF repo of a tested model read as untested. A second, looser key, used only to find a row, also drops a path's folders, a file extension, quantisation tags, the instruct, it and chat tags, reads name:tag as name-tag and hyphenates a letter before a digit. Sizes and versions stay, so 27b never finds 14b and 3.8 never finds 3.6, and the list refuses two rows one looser key would find.
… flagships The tested list held only the ladder's 8-case rows. Two suites now add rows after them: openrouter-screen v1, two cases after a smoke check, at agent when both pass and studio_only otherwise with its counts; and assumed, an expensive flagship listed at agent without runs, 0 of 0, which capability_of reports with reason assumed instead of counts. scripts/catalog_rows.py writes them from the sweep's sweep-results.json, a model the list already names keeping its row by its key or by a server's spelling. write_capability_list.py keeps them when it rewrites the ladder's rows. The real results come later; a fixture stands in.
…y a score
Agentic ran only for a model measured at the agent's bar, or one the
user opted in with "Try the agent". Now every model may choose it, and
only a technical gate keeps one out: no opencode staged, a stated no on
tool calls, or a stated window under 32,768 tokens. A catalog that says
nothing, and a local runtime that cannot be read, let a model in.
new_chat_modes() says what a new chat on the selected model may be:
agentic_allowed and the gate that blocks it, the default mode (Agentic
for a model measured at an agent level on a remote host, Basic
otherwise), the reason the switch shows beside Agentic, and the mode
remembered under chat_mode in the selection's settings. The selection
read carries it as capability.modes, in place of agent_trial, whose
route is gone; an agent_trial left on reads as a remembered Agentic.
POST /workspaces/{id}/chat/threads takes mode. Agentic asked for and
gated is a 409 with the gate's code, and a 503 agent_unavailable when
opencode does not start; neither leaves a thread. A mode asked for is
remembered as the model's default. An agent thread's next turn is held
to the technical gates alone. The developer switch now starts every
model's chats in Agentic and lifts the window floor.
The composer still calls the old route until the next commit.
A new chat's composer has a Basic (Q&A) / Agentic switch beside "+". It starts at the selected model's default, follows the model when it changes, and sends the pick as the new thread's mode; a pick holds per model for the session, above the open workspace. Agentic says its score, the untested warning, the note on a local copy or that a flagship is assumed, beside the option and under the composer once picked; a gated Agentic is disabled with its reason. An open chat's switch shows the mode it keeps and offers the other only as "Start a new chat in <mode> mode". A refused Agentic chat is worded by its code. Settings drops "Try the agent": the chat model panel says the mode new chats start in and that each chat picks its own, and the level labels say how the model did in the Agentic tests. Strings in all ten locales; Basic, Agentic and Q&A join the glossary.
ENABLED_BY_DEFAULT was false until a model passed the agent test, so no installer staged opencode. Every chat model may now pick Agentic, so it is true: the release's "Stage opencode" step and pnpm dev stage opencode and ripgrep, and SURFSENSE_LOCAL_OPENCODE_ENABLED=0 still leaves them out. The docs that said the agent is off in installers say so no longer.
…ver its -chat variant The sweep writes null counts for a model it stopped short of and keeps a model a provider's errors left unresolved, and catalog_rows.py read every row's counts as a measurement: a pending row failed the whole file, and an unresolved smoke became a "Low Agentic score" row noted as a failed smoke. A model now gets a row only with a verdict (status done, or no status and every count), its level must agree with its counts, and it is dated by its own measured_on before the sweep's date and --date. The rest are printed as left out. Stripping -chat folds openai/gpt-5.2-chat into gpt-5.2, and OpenRouter lists the -chat one first, so the merge kept its 1-of-2 row and dropped gpt-5.2's own 2-of-2. Between two of the sweep's rows the shorter key, the model's own name, now wins, and the script says which row each one left out collided with.
…assed Settings showed an assumed row as "Passed SurfSense's Agentic tests.", then "Not run: ..." twice, once as evidence and once as the row's English note. It now says "Not run: SurfSense expects this expensive flagship model to pass." once, in place of the level's line, the evidence and the note. A local copy of an assumed model read as "Passed on its full-size version" beside Agentic, and as "measured through openrouter.ai" in Settings, for a model never run anywhere. Both now say it is assumed; the copy still starts in Basic.
The composer always sends a new thread's mode, and the API remembered any mode sent, so the first chat on every model stored its default at that moment as the user's choice. A model later screened as passing kept starting in Basic, and every model used under the developer switch kept starting in Agentic after it was turned off. ThreadCreate now takes `remember`, which the composer sends only for a mode picked in the switch; the mode is still always sent, so the chat opens as the switch shows. The remembered mode lived beside the selection and was cleared whenever the slot took another model, so switching away and back lost it. It is now one entry per provider, connection and name under `chat_modes`, kept across a change of model; an old agent_trial is kept as its model's Agentic.
Asking for Agentic can now be refused, by a gate with a 409 or, after up to a minute's wait for opencode, with a 503. The composer had already been cleared on send and was reset again on the way back to a new chat, so the user had to retype the question to try again or switch to Basic. A refused new chat now goes back with what was typed and its images, which the composer takes up again after its reset.
The RAM guard learnt its lane size from peaks but never went below the ladder's 1.6 GB, so a sweep told its cases peak near 1.2 GB still ran two or three at once on a 16 GB desktop. A lane size given below that is now the floor itself.
# Conflicts: # .github/workflows/release-local.yml # docs/architecture/agent.md # docs/architecture/packaging.md # docs/proposals/file-agent/04-runtime-and-packs.md # docs/proposals/file-agent/05-model-ladder-and-evals.md # docs/proposals/file-agent/06-product-shape.md # docs/proposals/file-agent/07-dev-setup.md # surfsense_local/electron/electron-builder.yml # surfsense_local/electron/scripts/opencode/enabled.mjs # surfsense_local/electron/scripts/opencode/enabled.test.mjs
The cases measure the agent, and a model off the tested list now starts new chats in Basic (Q&A), so a live run asks for Agentic.
The agent can take a minute to start while the model picker stays open, and the pick was written against whichever model the slot held once the thread opened. new_thread_mode now returns the model's entry with the mode, and the router remembers the pick under it.
The hand-off adds the sweep's screened and assumed rows to capabilities.json, and tests counted the list's rows, used gpt-4o-mini as the untested model, and took the sweep's own sizes for sizes the list does not name. They now count the ladder's rows, use made-up ids, and match against the ladder's rows alone.
…on's row catalog_rows kept every listed row by key, so a gpt-5.2-chat listed while gpt-5.2 had no verdict scored for gpt-5.2 for good, and an assumed flagship could never be replaced by a screening; the sweep also skipped assumed keys. The merge now keeps each model's strongest row (ladder, then screening, then assumption; the shorter key within a suite), a variant waits while its own model has no verdict, and the sweep chooses assumed flagships again. A below row whose cases hit the per-case dollar cap says so in its note.
Installers now ship the agent and new chats on measured models start Agentic, but attach and paste still took images there, and every turn with one came back as a 409 that Retry repeated. The composer turns both off in an Agentic chat and says why, and an image attached before a new chat was switched to Agentic comes back with the question, opening nothing.
A local model's tool calls and window are read only when a chat starts, so the switch kept offering Agentic after a 409 and every send asked again. The refused gate is now held per model for the session: Agentic is disabled with its reason and the next chat starts in Basic (Q&A).
…s Agentic Proposals 01, 03 and 07 and the roadmap still kept agent threads behind the developer switch or a tested list. The switch's own wording said it starts every model's new chats in Agentic, but a mode the user picked for a model still wins, and no opencode still keeps Agentic out. 07-dev-setup now says which models start Agentic and lists the composer's Basic (Q&A) as the likeliest reason a thread answers as a chat.
…ssume named flagships, and clear a lane's leftovers A reasoning flagship that spends more than a case's stop was counted as failing, so a model too dear to test started chats in Basic. Its first capped case now ends it as assumed to pass, like the dearest flagships, and --assume names more to take that way without running them. A case that died left its opencode holding the lane's database, and every later case in that lane failed before it started until six harness faults stopped the sweep. A lane's leftovers are now stopped before each case and after it.
They bound lanes 3 and 4's real ports, so they failed whenever a sweep ran beside them.
# Conflicts: # surfsense_local/backend/tests/live/sweep/test_model_list.py
…1 assumed flagships The sweep of 7 Oct 2026 ran every OpenRouter model that calls tools, holds 32,768 tokens and costs under $3/M of input on the smoke case, then the PDF brief and the board pack: 99 passed both and start chats in Agentic, 109 start in Basic (Q&A) with Agentic offered. Models from $3/M of input, the reasoning flagships whose output costs $10/M or more, and those whose case went past the $3 stop are assumed to pass. 11 models the providers kept rate-limiting stay untested. It cost $60.73.
OpenAI's models fill every field a tool lists, so a render with no document to continue and no template arrived with artifact_id 0 and template_source_id 0. The tool asked for source 0, refused it as not selected, and the model sent the same call until the turn ran out: in the OpenRouter sweep no OpenAI model rendered a document, and two Grok models hit the same refusal. Ids count from 1, so an optional *_id of 0 now names nothing.
…oken The settle helpers waited 20 ms, under the 32 ms the run store now waits between text notices, so a token was sometimes read before it was told and the tests failed under load.
…ad failed With an optional id of 0 left out, 18 OpenAI and Grok models ran the screening again: o3, GPT-4.1, GPT-5 mini, GPT-5.4 mini, GPT-5.1 Codex mini, GPT-5.6 Luna Pro, GPT-6 Luna and both Grok models now pass both cases and start chats in Agentic. 106 screened models are now agent, 102 below.
The list is written by its generators only, which keep the ladder's source; the sweep is described in model-capabilities.md.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configuration
⛔ Files ignored due to path filters (1)
📒 Files selected for processing (162)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Oct 9, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This release makes Agentic mode available to every chat model, widens the tested model list from 11 to 270, and ships the agent in installers.
What changes
Basic (Q&A) and Agentic, chosen per chat
chat_modes, and survives switching models.modeandremember..gguf,-instruct/-it/-chatand Ollamaname:tag. It never matches across sizes or versions.A wider tested list: 270 rows
tests/live/sweep, 7–8 Oct 2026) ran every OpenRouter model that calls tools, holds 32,768 tokens and costs under $3/M of input.agent) and 102 did not (studio_only).assumedrows, never run:scripts/catalog_rows.pyturns the sweep's results into rows. The board pack's "says what it assumed" check is now a grade, not a pass/fail.A fix found by the sweep: OpenAI's models fill every field a tool lists, so a render arrived with
artifact_id: 0andtemplate_source_id: 0. The tool refused source 0 and the model repeated the call, so no OpenAI model could make a document in Agentic. An optional*_idof 0 now counts as left out. Re-run on the fix, o3, GPT-4.1, GPT-5 mini, GPT-5.4 mini, GPT-5.1 Codex mini, GPT-5.6 Luna Pro, GPT-6 Luna and both Grok models pass.Installers ship the agent
ENABLED_BY_DEFAULT = true, so every build stages opencode 1.18.34 and ripgrep.SURFSENSE_LOCAL_OPENCODE_ENABLEDaccepts 1 or 0 and refuses anything else.afterSignhook runs it before any installer is made, and the release workflow smoke-tests it on all three runners.mac.signIgnore). It needs an entitlement the app's shared file lacks.Smaller fixes
Tests
dev_modand pass when run alone.test_probewas deselected; it fails ondev_modtoo.tests/integration/agentandtests/integration/chatpass (517, plus 13 re-run after a full disk).check_docs.pyare clean.secret.test.tsandopencode-leftovers.test.ts, already fail on Windows.tests/integration/llminstall-route tests fail on this machine ondev_modtoo. They can't findggml.dllin the worktree's backend folder, an environment issue.Before the release
Summary by CodeRabbit