Repository navigation
feat(plugins): remote MCP plugins, SurfSense's scraper plugins and the third-party registry - #2234
Merged
Merged
Conversation
A read-only tool runs without asking only on a plugin SurfSense runs, since annotations are the server's own claim; a third party's read-only tool asks until Always allow. Settings groups each plugin's tools as read-only and the rest, with one setting per group. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The client is the SDK's, pinned to one minor version and used only inside modules/plugins/remote/. It adds about 5 MB and brings OAuth and new protocol revisions; SurfSense's rules (deadlines, retries, never resending a call) sit around it. Each operation opens its own connection, since the SDK's tasks must close where they opened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mcp is pinned to 2.3.x for the plugin client and its test servers. jsonschema was already shipped through docling-core; the gateway now checks arguments with it, so it is declared directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration 0028 adds installed_plugins, plugin_credentials (encrypted), plugin_tools (with the server's order), plugin_calls (no foreign key to the plugin, so past calls survive a disconnect) and plugin_settings, one row for Restricted mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… through list_tools offers only ready tools under `<plugin>__<tool>` names: switched on, hosts allowed, other publishers' only once the user turns them on, colliding names kept distinct. call_tool refuses unknown tools and bad arguments with a sentence, asks for approval by the trust rules, records every call whole, and trims the model's copy at 20 KB. A later listing applies the rule for tools added later. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
McpSession lists every page of tools and calls them with progress and a deadline, through the SDK's client. Around it: the failing HTTP status kept (the SDK hides it), listing retried twice, a call never retried, and a 2025 server's forgotten session resent once. RemoteMcpSource puts it behind the gateway. Tested against the SDK's own server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…later Plugins reach Agentic threads through opencode first; the chat engine's router step is kept as a design for later, and ADR 0052 is deferred with it. `@` mentions stay for agent threads: the mention offers the tool that turn and the agent fills its inputs, so approval is as for any call. A user on a chat-only model has no plugins until the router step, and the screen says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SDK-built test server and its wrappers move to tests/mcp_test_server.py, beside office_stand_in.py, so route tests use it too; renamed so it is not mistaken for plugins/remote/mcp_server/. It gains the notes server and a wrapper that answers 401 without a token. The gateway's failing-source test raises a plain error, so the gateway's tests no longer import the remote client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…le the tables SurfSense's plugins are first-party and listed in the first-party list; every other publisher's are third-party and wait for "Allow third-party plugins", which replaces "Restricted mode" (`third_party_on`, source `first_party`). Migration 0028 settles on four tables: the sign-in folds into installed_plugins as an encrypted column, as provider_connections keeps a key; one plugin per server, its URL unique and stored in one spelling; plugin_settings is a one-row table like license_state. Tools stay rows, so each carries its switch and approval and tool search can index them later. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /plugins/custom checks the server's host before anything is sent, lists its tools and stores the plugin with its token encrypted; a server already connected answers 409, one needing a key 422, one out of reach 502. GET /plugins lists plugins and tools without credentials, PUT switches a tool or allows third-party plugins, DELETE disconnects and keeps past calls. An egress refusal now names every host still to allow (`hosts`), and a refused plugin host is recorded off so Settings > Network lists it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ists them The agent's opencode calls plugin servers with its own MCP client and keeps each call in its session, so SurfSense's call path and the plugin_calls table are removed. They return with the chat engine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ectly SurfSense becomes the controller, not a hop: each turn it gives opencode the ready servers and a rule per tool, and reads opencode's view of each server to explain a plugin it could not reach. The call path and plugin_calls move to the chat engine's later section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Off pauses a plugin: its sign-in and tool settings stay, and none of its tools reach a caller until it is switched back on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Servers are given only when opencode does not already hold them, rules are set on the thread's session, and denied tools are hidden from the model, so the open question is closed. Adds the plugin switch and, for later, taking access away mid-turn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before a turn, SurfSense re-lists ready plugins at most every five minutes, gives the thread's opencode their servers (keeping a held session), drops those no longer ready, and sets the session's allow/ask/deny rules. A plugin opencode cannot connect is named in an agent-plugins-unavailable frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
opencode's traffic leaves through a CONNECT proxy the API runs, which tunnels only to a ready plugin's host, checked on each connection, and refuses the rest. TLS stays end to end. A remote plugin must use HTTPS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Electron picks the proxy's port beside opencode's, gives it to the API, and sets opencode's proxy variables to it instead of the discard port. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reply shows "Couldn't reach Notion" or "Sign in to Notion again" from the agent-plugins-unavailable frame, in every language. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every refusal to connect a server carries a code the screen translates, GET /plugins/third-party reads the switch back, and a plugin's egress refusal says it is for a plugin, so the consent prompt words it as one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add an MCP server by URL with an optional key, pause or disconnect it, switch each tool and choose whether it asks, and allow third-party plugins behind a consent. The network prompt words a plugin's host as one. Translated into every language. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…vers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ard test The mode menu lists both modes for an open chat, the other marked "New chat"; the test still looked for the "Start a new chat in" item it replaced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d checks A step's whole result is read from its opencode session, and a plugin step's result is saved to Sources as a note the same call updates. Every listing of a plugin's server is a recorded check, with Check now on request, and a tool switched off on its own is marked added or changed until allowed. The check before a turn reads the stored check time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A plugin step reads "Notes: Search notes", opens to its whole result and offers Save to Sources. Each plugin card shows its last check with Check now, and marks tools that are new or changed. Translated into every language. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…0028 0028 has not shipped, so the plugin tables are created once with their check and tool-change columns, and 0029's raw ALTER TABLE goes. A database that ran the earlier 0028 is downgraded to 0027 once and upgraded again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An @<plugin>_<tool> in the message that names a ready tool gives it to the turn past the limit, with a note telling the agent the user asked for it. Notes go as separate synthetic parts, and reading a thread back drops every synthetic part from the user's words. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Through assistant-ui's composer trigger popover; a pick writes the tool's opencode name, @notes_count, into the message. Translated into every language. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/youtube/ is the backend's YouTube scraper behind two tools at /youtube/mcp, scrape and comments, licensed like Reddit, with progress per item from an on_item callback. youtube-transcript-api, fake-useragent and requests join the host for its subtitles. The backend's tests came with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/amazon/ is the backend's Amazon scraper behind one tool, scrape, at /amazon/mcp, licensed like Reddit, plain HTTP with the exit country following the domain. A block is an error item per input, as the backend's. The backend's tests came with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/walmart/ is the backend's Walmart scraper behind two tools at /walmart/mcp, scrape and reviews, licensed like Reddit, plain HTTP on US exits. A block is an error item per input, as the backend's. The backend's tests came with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/google_maps/ is the backend's Google Maps scraper behind two tools at /google-maps/mcp, scrape and reviews, licensed like Reddit; a sign-in wall is the tool's error. Its summary shows a review count only when known. The backend's tests came with it, those reading uncaptured responses skipped as there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… host The plugin host doc gains the four plugins, how each reaches its site and how a block reads; Instagram follows Google Search. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An expired key that was valid earlier in the day passed while Keygen was unreachable. Any refusal now ends the 24-hour grace. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/google_search/ is the backend's Google Search scraper behind one tool, scrape, at /google-search/mcp, licensed like Reddit. Google walling every address, with no SearXNG page to fall back on, is the tool's error; a SearXNG page says it is not Google's ranking. The captcha solvers move to proprietary/scraping/captcha/, async on httpx2, with 2captcha over HTTPS, set by SCRAPING_CAPTCHA_PROVIDER and SCRAPING_CAPTCHA_API_KEY. The shared warm-IP pool uses async Redis at REDIS_URL, and sticky IPs come from the proxy provider, so NodeMaven holds them too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Redis runs as a container that restarts with the platform. A copy of the host now starts without it and resubscribes when it returns, instead of failing to start; a Stop sent while Redis is down still ends the calls on the copy that took it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… container Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DataImpulse ignores a session id in the username and rotates the exit IP per request, measured live: five requests on one session went out on five IPs. Each port from 10000 to 20000 holds one IP, with or without a country, so a session now picks its port from its id. Reddit's warm-up, Amazon, Walmart and Google Search were all asking for sticky IPs and not getting them; Google Search paid a captcha solve on every page for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An agent wanting an article's text passed its URL as a query, and Google Search paid a render and a captcha solve to search for the URL. Another site's URL is now the tool's error, which its description warns of; Google Search URLs are still scraped as they are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/instagram/ is the backend's Instagram scraper behind two tools, scrape and details, at /instagram/mcp, licensed like Reddit. A keyword finds profiles through the Google Search plugin's scraper; Instagram refusing every address is the tool's error. Instagram's anonymous profile API now answers 401 on every address, so both tools fail on public profiles until they read the profile page instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/tiktok/ is the backend's TikTok scraper behind four tools, scrape, comments, trending and user_search, at /tiktok/mcp, licensed like Reddit. TikTok withholds profile feeds from a headless browser, so the image installs Xvfb and host/entrypoint.sh starts it when CRAWL_HEADED_XVFB_ENABLED is true, then execs the host so it still gets the container's signals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/indeed/ is the backend's Indeed scraper behind one tool, scrape, at /indeed/mcp, licensed like Reddit. Its Cloudflare-solving session is Scrapling's stealth Chromium, so the image needs no Camoufox, and each session now holds one sticky IP, since its clearance belongs to that IP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
proprietary/web_crawl/ is the backend's web crawler behind one tool, crawl, at /web-crawl/mcp, licensed like Reddit: pages read as markdown, or a site followed by its links. It refuses private addresses before any fetch, and its browser tiers await the async captcha solver. The first-party list renames the plugin from web-crawler to web-crawl, and Google Search's refusal of a page to read now points the agent at it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…host Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SurfSense signs in itself, at Connect, through the MCP SDK's OAuth client, and gives opencode only the bearer header: its own client lists tools with the same token, tokens stay encrypted, and the sign-in hosts get consent first. One fixed HTTPS redirect on the plugin host forwards to the app's loopback port, and an entry may carry a pre-registered client_id. A token is refreshed only as a turn starts, and a plugin is given to opencode again only when its token changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…container The docs no longer name a platform: the plugin host needs a container, Redis, object storage and a proxy in front that routes by path, and where it runs is the deployment's choice. The signed registry, its icons and the OAuth files are served by a small static container beside the plugin host, on the same hostname, so updating the list never restarts the plugins. CI commits the signature back to main, and the app refuses a list older than its cached copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d container plugins/registry/ holds the list publishers add to by pull request, starting with DeepWiki, and its icons. registry_check checks every entry and icon rule; on dev, CI signs the list with Ed25519 into published/, which the release pull request carries to main, so nothing pushes to main and the source list is never rewritten. A small Caddy container serves the signed list, its icons and SurfSense's OAuth client document and callback page on plugins.surfsense.com, apart from the plugin host so a listing never restarts the scrapers. CI does not contact listed servers: most need a sign-in, so a maintainer connects each listing on dev. plugins/README.md is the guide for publishers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…m with OAuth Once third-party plugins are allowed, the app fetches the signed registry at most every five minutes, verifies it against compiled keys, refuses an older list and entries claiming SurfSense or the license, and keeps the last good copy and its icons for offline use; an unpackaged app reads the repository. Settings > Plugins lists them under SurfSense's own, with Connect, and a removed plugin stops reaching the agent and says why. OAuth plugins, listed or added by URL, sign in at Connect through the MCP SDK's client: consent for the sign-in hosts, the browser, the callback, and tokens stored encrypted. A token is refreshed as a turn starts, giving opencode the new header; a refused refresh signs the plugin out, with Sign in again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@AnishSarkar22 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Important Review skippedToo many files! This PR contains 519 files, which is 219 over the limit of 300. To get a review, reduce the PR to 300 files or fewer by splitting it into smaller PRs or changing its base branch. Usage-priced reviews support at most 300 files. ⚙️ Run configuration
⛔ Files ignored due to path filters (4)
📒 Files selected for processing (519)
You can disable this status message by setting the
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
AnishSarkar22
changed the title
This was referenced Oct 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Plugins, built from the plugins proposal (
docs/proposals/plugins/README.md): a plugin is a remote MCP server whose tools the Agentic thread's agent can call. This PR adds the app side, SurfSense's own plugins, and the third-party registry.In the desktop app (
surfsense_local)modules/plugins/gateway/) is the one answer to which tools are ready and which ask first. It follows MCP annotations, as Claude's connectors do.@in the Agentic composer mentions a plugin. A turn says which plugins it could not use and why.plugins.surfsense.com. Refusals read as "Needs a license", "License expired" and the like.plugins.surfsense.comis a built-in destination, off by default.installed_plugins,plugin_toolsand the check failures.SurfSense's plugins (
plugins/remote/)/<id>/mcp:POST /cancel, passed between copies by Redis;host/.env.example.surfsense_backend, each behind MCP tools with progress: Google Search, Web Crawl, Reddit, YouTube, Instagram, TikTok, Google Maps, Amazon, Walmart and Indeed. What they share is inproprietary/scraping/:proprietary/license_check/) uses Keygen'svalidate-key, with a five-minute cache and a 24-hour grace that a refusal ends.plugins/remote/proprietary/is Business Source License 1.1 and changed by maintainers only, enforced in CI.The registry (
plugins/registry/)registry_checkchecks every entry and icon rule.dev, CI signs the list intopublished/, and the release pull request carries it tomain, so nothing pushes tomain.plugins/README.mdis the guide for publishers.Docs:
plugins.md,plugin-host.mdand the newplugin-registry.md;agent.mdandegress.md;Why
The product direction is local + API. Plugins let the agent use outside services, SurfSense's paid scrapers first, without building each one into the app. They are standard MCP servers, so any existing MCP server works unchanged. Remote plugins come first, hosted by their publishers; bundles that run on the user's machine come later. The hosted scraper API is being shut down, so the scrapers move out of
surfsense_backendinto their own servers, licensed through the SurfSense license.Fixes #
How to test
Automated
In the app
cd plugins/remote && uv run --project host --env-file host/.env python -m host. Copyhost/.env.examplefirst; the scrapers need a residential proxy, and Google Search a captcha key.cd surfsense_local/electron && SURFSENSE_LOCAL_PLUGIN_HOST_URL=http://127.0.0.1:8000 pnpm dev, then import a license.https://mcp.linear.app/mcp: the browser opens the sign-in, the row waits, then shows Connected with its tools.Before production
REGISTRY_SIGNING_KEYrepository secret;devifdevis protected;plugins.surfsense.com, with the registry's paths routed to its container;SURFSENSE_ACCOUNT_IDinplugins/remote/proprietary/license_check/keygen.py.🤖 Generated with Claude Code