diff --git a/plans/plugins/00-umbrella.md b/plans/plugins/00-umbrella.md new file mode 100644 index 0000000000..93c3c1752c --- /dev/null +++ b/plans/plugins/00-umbrella.md @@ -0,0 +1,55 @@ +# Plugins + +> A plugin is a folder in this repo. A third party adds one by pull request. The desktop app installs it on demand and runs it as its own process for one job. + +Shared contract: [`01-protocol.md`](01-protocol.md). Every stream below implements against that file. Do not invent a second wire format. + +## Workstreams + +Same letter can be picked up at the same time. A stream's own files are in order. + +| Stream | Files | Owns | Starts from | +|---|---|---|---| +| **SDK** | [`sdk/01-library-and-harness.md`](sdk/01-library-and-harness.md) | `plugins/sdk/`, `plugins/example/`, `plugins/README.md` | the protocol | +| **Runtime** | [`runtime/01-process.md`](runtime/01-process.md) then [`runtime/02-calls-and-results.md`](runtime/02-calls-and-results.md) | `surfsense_local/backend/modules/plugins/` runner and result import | the protocol. `02` reads the file `01`'s process leaves behind | +| **Catalog** | [`catalog/01-manifest-and-ci.md`](catalog/01-manifest-and-ci.md) then [`catalog/02-install-and-publish.md`](catalog/02-install-and-publish.md) | manifest checker, PR workflow, install, publish workflow | the protocol. `02` needs the manifest rules in `01` | +| **App** | [`app/01-api.md`](app/01-api.md) and [`app/02-screen.md`](app/02-screen.md) | plugin routes, the plugin screen | the route list in `01`. The screen can be built against that list before the routes exist | +| **Python** | [`python/01-interpreter.md`](python/01-interpreter.md) | the interpreter the packaged app spawns | nothing. Other streams use `uv`'s Python until this lands | + +**Demo:** SDK harness runs `plugins/example` on a laptop, and the API runs the same plugin as a subprocess and a `document` result shows up as a note. No GitHub release required for that demo. + +**Ship:** catalog publish workflow uploads a real tarball, the app installs it from the bundled catalog, a paid plugin stays locked without a license. + +## Locked decisions + +| Decision | Choice | +|---|---| +| Where source lives | `plugins//` in this repo. A third party contributes by pull request. | +| Id | Folder name. `^[a-z][a-z0-9-]{0,63}$`. Unique. Immutable once a version is published. `plugins/RESERVED` plus the prefix `surfsense-` are ours. `author` is a field, not part of the id. | +| Process | One subprocess per run. Not loaded into the API or the worker. | +| Talks to the app | Arguments, environment variables, and a results file. No socket and no request protocol. Stdout is logs. | +| Network | The plugin makes its own HTTP calls. `hosts` in `plugin.json` is copied into the catalog and shown. The app does not intercept those calls and Settings → Network does not apply to them. | +| Dependencies | `requirements.txt` with `==` pins. CI runs `pip install` into the tarball. The user's machine never runs `pip`. | +| SDK | `plugins/sdk/` in this repo. Not published to PyPI. The app ships that folder next to the plugin interpreter. A plugin tarball does not contain it. | +| License | `free` or `paid`. Community plugins are `free`. `paid` means one of ours, and it installs and runs only with an unexpired `individual` or `team` license. A trial does not unlock it. No price field. | +| Catalog | `catalog.json` is bundled in the installer, so the list is visible with no network. Refreshing it is a separate egress destination, off by default. The plugins themselves are not bundled. | +| Results | The plugin appends them to a file. After the process exits, `kind: "document"` becomes a note in the run's workspace. Any other kind is stored on the run and shown there. The platform does not require documents, and it does not define resume or dedup. | +| Secrets | Names in the manifest. Values go through `shared/secrets.py`. The app puts each one in an environment variable before spawn. | +| What we do not build | A sandbox, a process-count cap, WASM, `host/generate`, hot reload, pip or npm on the user's machine, inferring hosts from imports, plugin-authored HTML, payments. The host places an entry in the existing sidebar and draws its form. | + +## Caught while specifying + +These were not in the design conversation. They are decided here so a developer is not blocked on them. + +| Gap | Decision | +|---|---| +| The packaged app has no general Python. API and worker are PyInstaller binaries. | [`python/01-interpreter.md`](python/01-interpreter.md) ships a real interpreter. Plugins run on that, never inside the frozen binaries. | +| A result needs a shape the app can import. | One JSON object per line in the results file. `document` becomes a note. Anything else is kept on the run. | +| A document has to belong to a workspace. | A run is started in a workspace. Document results are notes in that workspace, then `ingest_document`. | +| The installer still downloads from GitHub. | That path may follow only `github.com` → `objects.githubusercontent.com`. The plugin's own HTTP is not this path. | +| A fetched `catalog.json` is the trust anchor for every sha256 in it. | The refresh URL is compiled into the app. A catalog entry whose tarball host is not `github.com` or `objects.githubusercontent.com` is rejected. | +| `documents.dedup_key` already exists. | Plugin results do not set it. A plugin that wants to skip work it already did uses its data directory. | + +## Out of scope + +Sandboxes, memory and process caps, charging by a community author, bundling plugin code in the installer, running a plugin inside the API process, Amazon or Reddit themselves. Those are plugins someone writes after this lands. The example plugin in the SDK stream is the one this work has to ship. diff --git a/plans/plugins/01-protocol.md b/plans/plugins/01-protocol.md new file mode 100644 index 0000000000..9aed5db373 --- /dev/null +++ b/plans/plugins/01-protocol.md @@ -0,0 +1,139 @@ +# Protocol + +The contract the SDK, the runtime, the catalog, and the app all implement. A change here is a change to every stream. + +Version is `1`. The handshake and `sdk` range are how a later version shows up. Do not add a field to a message without bumping it. + +## On disk + +``` +plugins// + plugin.json + main.py + requirements.txt # omitted when there are no dependencies + README.md + LICENSE + tests/ # optional + fixtures/ # optional +``` + +`site-packages/` is not in git. CI creates it inside the tarball. + +### `plugin.json` + +| Field | Rule | +|---|---| +| `id` | Equals the folder name. Pattern above. | +| `name` | Display string, 1–80 characters. | +| `author` | Display string. Not unique. | +| `version` | Semver `MAJOR.MINOR.PATCH`. | +| `sdk` | A PEP 440 specifier. The app's SDK version must satisfy it or the run does not start. | +| `license` | `free` or `paid`. | +| `hosts` | Array of hostnames, no scheme, no path, no port. Shown in the catalog. The app does not intercept the plugin's own connections. | +| `secrets` | Array of names. Values are never in this file. | +| `entries` | At least one. | + +An entry: + +| Field | Rule | +|---|---| +| `name` | `^[a-z][a-z0-9-]{0,63}$`, unique within the plugin. | +| `title` | Display string. | +| `inputs` | Array, may be empty. Each is `{ "name", "kind", "required" }`. `kind` is `string`, `number`, or `boolean`. | + +Unknown fields are ignored, so a future field does not break an older app. A missing required field, a bad id, or a second entry with the same name fails the manifest check. + +### Tarball + +`-.tar.gz`. One top-level directory `-/` containing the folder above plus `site-packages/` when CI installed dependencies. The `plugin.json` inside the tarball must deep-equal the catalog entry's manifest fields (`id`, `name`, `author`, `version`, `sdk`, `license`, `hosts`, `secrets`, `entries`). A mismatch rejects the install. The sha256 is of the gzip bytes. + +Extraction rejects any member whose path is absolute, contains `..`, or is a link that resolves outside the destination. The destination is `/plugins//.tmp-/`, then one rename to `/plugins///`. + +### Catalog + +One JSON file, `catalog.json`: + +```json +{ + "plugins": [ + { + "id": "example", + "name": "Example", + "author": "SurfSense", + "version": "1.0.0", + "sdk": ">=1,<2", + "license": "free", + "hosts": [], + "secrets": [], + "entries": [], + "url": "https://github.com/MODSetter/SurfSense/releases/download/plugin-example-1.0.0/example-1.0.0.tar.gz", + "sha256": "<64 hex>", + "yanked": false + } + ] +} +``` + +The app ships a copy taken at desktop build time. A refresh downloads this same file from a URL compiled into the app, and only from that URL. `yanked: true` stays in the file so an already-installed copy can be refused with a reason. + +## How a run starts + +There is no socket and no request protocol. The plugin does its own network calls, reads secrets from the environment, and appends results to a file. The app spawns it and reads that file after the process exits. + +``` + -m surfsense_plugin --inputs --results --data +``` + +| Argument | Meaning | +|---|---| +| `` | An entry name from the manifest. The app checks this before spawn. | +| `--inputs` | A JSON object. Keys are the entry's input names, values match the declared kinds. The app writes this file. | +| `--results` | An empty file the app creates. The plugin appends one JSON object per line. | +| `--data` | `/plugins//data`. Exists from install, survives upgrades, deleted on uninstall. | + +For each secret name in the manifest, the app sets `SURFSENSE_PLUGIN_SECRET_` to the stored value. Secret names match `^[a-z][a-z0-9_]{0,63}$` so they are environment variables. The app refuses to spawn when a declared secret has no stored value. + +Stdout and stderr are logs. The app keeps the last 16 KiB on the run. + +Exit 0 is success. Any other exit is `failed`. The app still imports every well-formed results line already in the file. + +Cancel: `SIGTERM`, then `SIGKILL` five seconds later. The run is `cancelled`. Lines already flushed stay. + +### Results file + +Each line is one JSON object, flushed before the next one: + +```json +{"kind":"document","body":{"title":"Show HN","content":"..."}} +``` + +| `kind` | `body` | What the app does after the process exits | +|---|---|---| +| `document` | `{ "title": str, "content": str }` | Insert a `NOTE` in the run's workspace, set `document_metadata` to `{ "pluginId", "entry", "runId" }`, enqueue `ingest_document`. Do not set `dedup_key`. | +| anything else | a JSON object | Insert a row on the run: `kind` + `body`. The screen shows it. | + +A line over 1 MiB, or a line that is not JSON, is skipped. The valid lines before and after it are still imported. A run that writes no lines is a success that produced nothing. `document` is how a source plugin puts something in the library. It is not required. + +The plugin's own HTTP is ordinary Python. The app does not see those calls, does not apply the `hosts` list to them, and does not apply Settings → Network to them. `hosts` is what the catalog shows. + +## SDK surface + +```python +import urllib.request +from surfsense_plugin import entry, result, secret + +@entry("search") +def search(query: str) -> None: + token = secret("token") + request = urllib.request.Request( + "https://hacker-news.firebaseio.com/v0/item/1.json", + headers={"Authorization": f"Bearer {token}"}, + ) + with urllib.request.urlopen(request) as response: + body = response.read().decode() + result("document", {"title": "Show HN", "content": body}) +``` + +`secret` reads the environment variable. A missing one raises. `result` appends one line to the results file and flushes it. There is no `fetch` in the SDK. + +The harness is `python -m surfsense_plugin.harness --input query=plugins`. It passes secrets from the process environment, runs the entry, and prints the results file. It does not intercept HTTP. diff --git a/plans/plugins/app/01-api.md b/plans/plugins/app/01-api.md new file mode 100644 index 0000000000..64bcbd4750 --- /dev/null +++ b/plans/plugins/app/01-api.md @@ -0,0 +1,37 @@ +# App — API + +> Owns: `surfsense_local/backend/modules/plugins/router.py`, `schemas.py`. +> Routes the screen in [`02-screen.md`](02-screen.md) calls. Runtime: [`../runtime/01-process.md`](../runtime/01-process.md). Install: [`../catalog/02-install-and-publish.md`](../catalog/02-install-and-publish.md). + +## Goal + +The screen can list plugins, install one, set a secret, and start a run, using the modules the other streams built. + +## Work + +| Method | Path | Behavior | +|---|---|---| +| `GET` | `/plugins` | The bundled catalog, or the refreshed one when `plugin_catalog` has been fetched. Each item includes `installed` version or null, `license` and whether it is unlocked, `hosts`, `yanked`. | +| `POST` | `/plugins/catalog/refresh` | Runs the refresh. Egress off → 403 with the destination name, same shape as the other egress denials. | +| `POST` | `/plugins/{id}/install` | The install function. 403 for egress, 402 for `license_required`, 409 for a bad hash or manifest. | +| `DELETE` | `/plugins/{id}` | Uninstall. | +| `PUT` | `/plugins/{id}/secrets/{name}` | Body `{ "value": str }`. Encrypts via `shared/secrets.py` under `plugin::`. `name` must be in the manifest. Never returns the value. | +| `GET` | `/plugins/{id}/secrets` | Names and a boolean `set`. Never values. | +| `POST` | `/workspaces/{id}/plugins/{plugin}/entries/{entry}/runs` | Body is the inputs object. 422 when a name or kind does not match the entry. 422 when a declared secret is unset, listing the names. 402 when locked. Inserts `plugin_runs` and enqueues `run_plugin`. | +| `GET` | `/workspaces/{id}/plugin-runs/{run}` | Status, error, log tail, and `plugin_results` rows. | +| `POST` | `/workspaces/{id}/plugin-runs/{run}/cancel` | Sets the cancel flag the task watches. | + +Emit `plugin.run.updated` on the existing events broker when a run status changes, carrying the run id, so the screen can invalidate. + +## Acceptance + +- List returns the example plugin from a fixture catalog with `installed: null`. +- Install, then list shows the version. +- Run with a missing required input returns 422 and does not insert a run. +- Run of a `paid` plugin with no license returns 402. +- A successful run of `plugins/example` returns a run id; polling the run reaches `succeeded`; a note exists in the workspace. +- `GET` of a secret returns `set: true` after `PUT` and does not contain the value. + +## Needs from + +Install, the runner task, license, egress, events. The route functions can be written and the 422 tests can land before the runner does, with the enqueue mocked. The success test waits for the runtime stream. diff --git a/plans/plugins/app/02-screen.md b/plans/plugins/app/02-screen.md new file mode 100644 index 0000000000..a812552f67 --- /dev/null +++ b/plans/plugins/app/02-screen.md @@ -0,0 +1,33 @@ +# App — screen + +> Owns: `surfsense_local/frontend/src/features/plugins/`. +> Calls: [`01-api.md`](01-api.md). + +## Goal + +Maya installs a plugin from a list, then uses it where she already works. An entry that returns documents is a sidebar action next to New chat, not a job she runs from an admin page. + +## Work + +- Installed entries are `SidebarNavAction` rows on the existing left sidebar, below Chats. The label is the entry title. Click opens a dialog built from the app's own inputs: text, number, checkbox. Submit starts the run in the current workspace. The dialog shows running, then succeeded, failed, or cancelled, with the error and the log tail, and a cancel button. +- A `document` result is not drawn in that dialog. It arrives in the sources list, which already lists notes. Other kinds are listed in the dialog as the kind and the JSON. +- Settings holds the catalog, not the run. Each row: name, author, version, the `hosts` list or the word "none", `free` or `paid`, and installed version when present. +- A `paid` row that is locked explains that a SurfSense license is required and links to the existing license settings. The install button is disabled. +- Install calls `POST /plugins/{id}/install`. Egress 403 shows the same Settings → Network explanation the other destinations use, naming `plugin_install`. +- Refresh calls `POST /plugins/catalog/refresh`. The same 403 names `plugin_catalog`. The list is already on screen from the bundled catalog before either destination is enabled. +- An installed plugin with a secret shows a field per name in Settings. Saving calls `PUT`. The field does not redisplay the value. The sidebar action stays disabled until every declared secret is set. +- Uninstall asks once, then `DELETE`. The sidebar row disappears. Notes already created stay in Sources. +- Freshness for the dialog is the existing SSE invalidate on `plugin.run.updated`, with a refetch interval while `running` as the fallback. + +Follow the frontend workflow. The dialog and the settings list use the components the app already has. A plugin does not ship a layout. + +## Acceptance + +- With both egress destinations off, the screen lists the bundled catalog and no request has left the machine. +- Install of `example` from a local fixture server adds a sidebar row. Running it with text `hi` from that row reaches `succeeded`, and Sources contains a note with content `hi`. +- A `paid` plugin with no license on disk cannot be installed from the screen. +- A failed run shows the error and the log tail. + +## Needs from + +[`01-api.md`](01-api.md). The screen can be built against those paths before the handlers exist. diff --git a/plans/plugins/catalog/01-manifest-and-ci.md b/plans/plugins/catalog/01-manifest-and-ci.md new file mode 100644 index 0000000000..dc75136414 --- /dev/null +++ b/plans/plugins/catalog/01-manifest-and-ci.md @@ -0,0 +1,33 @@ +# Catalog — manifest and pull-request checks + +> Owns: `surfsense_local/backend/modules/plugins/manifest.py`, `plugins/RESERVED`, `.github/workflows/plugin-check.yml`. +> Contract: [`../01-protocol.md`](../01-protocol.md). + +## Goal + +A bad plugin folder fails in CI before anyone reviews it. The checker is a library the install path calls too, so the rules exist once. + +## Work + +- `load_manifest(path) -> Manifest` and `check_tree(folder, reserved: set[str]) -> list[str]`. Empty list means the folder may be merged. The strings are the review comment. +- Rules, all of them, and no others: + - `plugin.json` matches the protocol table. + - `id` equals the folder name and is not in `reserved` and does not start with `surfsense-`. + - `version` is semver. When `origin/main` has the same id, the version is strictly greater. + - `license` is `free` unless the id is reserved. A reserved id may be `free` or `paid`. + - `requirements.txt`, if present, has every requirement pinned with `==` and no URL, no editable, no `--index-url`. + - `main.py` exists. `LICENSE` exists. No `site-packages/` directory in the tree. +- `plugins/RESERVED` is one id per line. `example` is not reserved. Add a line when we ship a paid plugin, in the same pull request as that plugin. +- Workflow `plugin-check.yml` on pull requests that touch `plugins/**`. It runs the checker on each changed `plugins//` and fails the job with the checker's lines. It does not install requirements and it does not start Electron. +- The checker does not read imports and does not fail on a socket call. + +## Acceptance + +- A fixture folder that is valid produces no errors. +- One test per rule above, each with the smallest folder that breaks that rule and the error string that names it. +- A second copy of `example`'s id in another folder is an error. +- A `requirements.txt` of `requests==2.32.3` is valid. `requests>=2` is not. + +## Needs from + +The protocol's manifest table. Nothing from the app. diff --git a/plans/plugins/catalog/02-install-and-publish.md b/plans/plugins/catalog/02-install-and-publish.md new file mode 100644 index 0000000000..639f7420fe --- /dev/null +++ b/plans/plugins/catalog/02-install-and-publish.md @@ -0,0 +1,35 @@ +# Catalog — install and publish + +> Owns: `modules/plugins/install.py`, `modules/plugins/catalog.py`, `.github/workflows/plugin-publish.yml`, the bundling of `catalog.json` in the desktop build. +> Contract: [`../01-protocol.md`](../01-protocol.md). Checker: [`01-manifest-and-ci.md`](01-manifest-and-ci.md). + +## Goal + +Merging a plugin publishes a tarball and a catalog entry. The app installs that tarball without running it, and without running `pip`. + +## Work + +- `installed_plugins` table: `id` primary key, `version`, `path`, `installed_at`. Hand-written migration. +- Publish workflow, on push to `main` that touches `plugins//**`, and not on the desktop release workflow. For each changed plugin: create a venv with the interpreter version from [`../python/01-interpreter.md`](../python/01-interpreter.md), `pip install -r requirements.txt --target site-packages` when the file exists, tar `-.tar.gz`, sha256 the gzip, upload it as an asset on the GitHub release `plugin--`. Rebuild `catalog.json` from every `plugins/*/plugin.json` plus the known asset URLs and digests, and upload it as the only asset of the release `plugin-catalog`, replacing the previous asset. +- Do not attach these assets to the `stable` desktop release. electron-updater reads `stable*.yml` from that release. +- Desktop build downloads `catalog.json` from `plugin-catalog` the way `fetch-llmfit.mjs` downloads a pinned asset, and electron-builder ships it in `extraResources`. A failed download fails the desktop build. The URL is a constant in that script and the same constant in the app. +- Install, given a catalog entry: require `plugin_install` egress enabled, otherwise raise the existing `EgressDeniedError`. Download the URL. Require the final host, after GitHub's redirect, to be `github.com` or `objects.githubusercontent.com`. Both are covered by the one destination `plugin_install`. Reject any other host before writing. Compare sha256. Compare the manifest inside the tarball to the catalog fields. Extract with the path rules in the protocol. Rename into place. Insert or update `installed_plugins`. Create `/plugins//data` if it is absent. Leave an older version directory in place. +- A yanked entry installs only when already installed, and a run of a yanked version fails with the reason `yanked`. +- `license: paid` consults `modules/license/verify.py`'s stored result. Missing, expired, `trial`, or `clock_untrusted` refuses the install and the run with `license_required`. `free` does not consult it. +- Refresh: download `catalog.json` from the compiled URL when `plugin_catalog` egress is enabled. Replace the in-memory catalog. The bundled file remains the fallback when the destination is off. +- Add `plugin_catalog` and `plugin_install` next to `OLLAMA_PULL` in `modules/egress/service.py`, and include them in `list_destinations`, so both appear in Settings → Network, off by default. +- Uninstall: delete `/plugins//` including `data` and every version, delete `installed_plugins`, delete secrets named `plugin::*`. Do not delete documents or `plugin_results`. + +## Acceptance + +- A tarball with a `../` member extracts nothing and leaves no directory behind. +- A sha256 mismatch leaves no directory behind. +- A manifest whose `hosts` differ from the catalog entry is rejected. +- A redirect to a host other than the two GitHub hosts is rejected and the body is not saved. +- Install with `plugin_install` disabled raises `EgressDeniedError` and does not open a connection. +- A `paid` entry with no license does not insert `installed_plugins`. The same entry with `license: free` does, with no license on disk. +- Uninstall removes the folder and the secret and leaves a note that the plugin created. + +## Needs from + +The manifest checker. The egress module and the license verifier, which exist. The publish workflow can land before the interpreter packaging does: it pins the same CPython minor version that workflow will ship. diff --git a/plans/plugins/python/01-interpreter.md b/plans/plugins/python/01-interpreter.md new file mode 100644 index 0000000000..968bb8b259 --- /dev/null +++ b/plans/plugins/python/01-interpreter.md @@ -0,0 +1,26 @@ +# Python — the interpreter a plugin runs on + +> Owns: `surfsense_local/electron/scripts/fetch-plugin-python.mjs`, the `extraResources` entry, `plugin_python()` in `modules/plugins/python.py`. + +## Goal + +The packaged app can spawn `python -m surfsense_plugin`. The PyInstaller binaries that are the API and the worker cannot do this: they are not a general interpreter, and a plugin is source. + +## Work + +- Pin CPython 3.12, the same minor the backend uses, from the python-build-standalone release. One asset per `linux-x64`, `darwin-arm64`, `darwin-x64`, `win32-x64`. Record the URL and sha256 in the fetch script, the way `fetch-llmfit.mjs` records llmfit. +- The script verifies the sha256, extracts, and stages atomically under `surfsense_local/electron/plugin-python/`. `electron-builder.yml` `extraResources` copies that directory to `plugin-python/`. +- `plugin_python() -> Path` returns the staged interpreter in the packaged app (`python` or `python.exe` on the staged path) and, when that path is absent, the interpreter running the API. Tests and `pnpm dev` hit the second branch and do not download anything. +- The same resources directory ships `plugins/sdk/` beside the interpreter. That is how `python -m surfsense_plugin` resolves. Do not publish the SDK to PyPI. +- The desktop build's `dist` script runs the fetch the same way it runs `fetch-llmfit.mjs`. A mismatch fails the build. +- Publish workflow in the catalog stream installs dependencies with this minor version. It does not need this script. It needs the version number, which lives in this file as `PYTHON_VERSION`. + +## Acceptance + +- `checkConfiguration()` fails when any of the four targets lacks a 64-hex sha256. That test runs with no network. +- On a developer machine, `plugin_python()` returns a path that runs `import sys; print(sys.version)` and reports 3.12 when the staged directory is present, and returns the current interpreter when it is absent. +- A packaged smoke, the kind `check-llmfit.mjs` already does for llmfit, runs the staged interpreter with `-c "import sys; assert sys.version_info[:2]==(3,12)"`. + +## Needs from + +Nothing. The runtime stream calls `plugin_python()` and works before this is packaged. diff --git a/plans/plugins/runtime/01-process.md b/plans/plugins/runtime/01-process.md new file mode 100644 index 0000000000..271926b0ab --- /dev/null +++ b/plans/plugins/runtime/01-process.md @@ -0,0 +1,30 @@ +# Runtime — process + +> Owns: `surfsense_local/backend/modules/plugins/runner.py`, `modules/plugins/tasks.py`, the `plugin_runs` table. +> Contract: [`../01-protocol.md`](../01-protocol.md). + +## Goal + +The worker starts a plugin process for one run and records how it ended. [`02-calls-and-results.md`](02-calls-and-results.md) reads the results file afterward. + +## Work + +- Hand-written migration for `plugin_runs`: `id`, `workspace_id`, `plugin_id`, `entry`, `inputs` JSON, `status` (`running` `succeeded` `failed` `cancelled`), `error`, `log_tail`, `started_at`, `finished_at`. +- `import_models()` and `import_tasks()` gain this module the same way the other modules do. A test in the existing registration test fails if they are missing. +- Huey task `run_plugin(run_id)`. The API enqueues it. The task spawns the interpreter from `plugin_python()` ([`../python/01-interpreter.md`](../python/01-interpreter.md) supplies the packaged path; until then the task accepts an explicit interpreter path and tests pass `sys.executable`). +- Arguments, as the protocol specifies: ` --inputs --results --data `. The task writes the inputs file and an empty results file. Working directory is the plugin directory. `PYTHONPATH` is the SDK directory shipped next to the interpreter, then `/site-packages` when that directory exists. The SDK is not installed from PyPI and is not inside the tarball. +- For each manifest secret, set `SURFSENSE_PLUGIN_SECRET_`. The API has already refused the run when one is missing. +- Capture stdout and stderr into a ring of 16 KiB on `log_tail`. +- Exit 0 sets `succeeded`. Any other exit sets `failed` with `error` `exit `. Either way, hand the results path to the import in [`02-calls-and-results.md`](02-calls-and-results.md). +- Cancel sets a flag the task watches. The task sends `SIGTERM`, waits five seconds, sends `SIGKILL`, sets `cancelled`. Do not add a process-count limit. +- The plugin is never imported into the API or the worker. The test that proves it spawns a process whose pid is not the worker's. + +## Acceptance + +- A run of `plugins/example` with input `hi` exits 0 and leaves one results line. The run status is `succeeded`. +- A plugin that exits 1: status `failed`, worker process still alive. +- Cancel mid-run: status `cancelled` within ten seconds, and the pid is gone. + +## Needs from + +The protocol. `plugins/example` from the SDK stream for the success test. A one-file fake plugin in this stream's tests is enough for handshake, cancel, and crash, so this stream does not wait on the SDK. diff --git a/plans/plugins/runtime/02-calls-and-results.md b/plans/plugins/runtime/02-calls-and-results.md new file mode 100644 index 0000000000..2e87f8a656 --- /dev/null +++ b/plans/plugins/runtime/02-calls-and-results.md @@ -0,0 +1,28 @@ +# Runtime — results + +> Owns: `surfsense_local/backend/modules/plugins/results.py`. +> Contract: [`../01-protocol.md`](../01-protocol.md). Process: [`01-process.md`](01-process.md). + +## Goal + +After the plugin process exits, the results file becomes notes and run rows. The plugin did its own HTTP and already had its secrets. This stream does not talk to it. + +## Work + +- Read the results path the process stream passes in. Import runs on `succeeded`, `failed`, and `cancelled`. A missing file imports nothing. +- Each line is one JSON object. A line over 1 MiB, or a line that is not JSON, is skipped and a line is appended to `log_tail`. The other lines are still imported. +- `kind: "document"` with `title` and `content`: insert a `Document` the same way `create_note` does (`NOTE`, `pending`), set `document_metadata` to `pluginId`, `entry`, `runId`, commit, call `ingest_document`. Do not set `dedup_key`. A second identical line inserts a second note. +- Any other kind: insert `plugin_results` (`run_id`, `kind`, `body` JSON). Hand-written migration. +- Do not open a connection to the plugin's hosts. Do not call `egress.require` for them. + +## Acceptance + +- A results file with one `document` line: a `NOTE` exists in the workspace and `ingest_document` was enqueued. +- Two identical `document` lines: two notes. +- A `kind` of `widget`: a `plugin_results` row, and no document. +- A file whose middle line is not JSON: the lines around it are imported, and `log_tail` mentions the skipped line. +- A plugin that performs HTTP during the run is not observed by this module. A test plugin can hit a local server; this module's code path does not connect to it. + +## Needs from + +[`01-process.md`](01-process.md) for the results path and the run row. The documents module, which already exists. diff --git a/plans/plugins/sdk/01-library-and-harness.md b/plans/plugins/sdk/01-library-and-harness.md new file mode 100644 index 0000000000..30e0ffc139 --- /dev/null +++ b/plans/plugins/sdk/01-library-and-harness.md @@ -0,0 +1,27 @@ +# SDK — library and harness + +> Owns: `plugins/sdk/`, `plugins/example/`, `plugins/README.md`. +> Contract: [`../01-protocol.md`](../01-protocol.md). + +## Goal + +An author runs a plugin on her machine, against the real protocol, without Electron and without the frozen app. + +## Work + +- Package `surfsense_plugin` in `plugins/sdk/`. `__main__` reads the arguments from the protocol, calls the matching `@entry` function, and exits 0 or 1. `entry`, `result`, and `secret` are the whole public surface. There is no `fetch`. No other module is imported by `plugins/example`. +- `secret` reads `SURFSENSE_PLUGIN_SECRET_`. `result` appends one JSON line to the results file and flushes it. +- `python -m surfsense_plugin.harness` as specified. It does not intercept HTTP. `SURFSENSE_PLUGIN_SECRET_` supplies a secret. +- `plugins/example/`. `license: free`, `hosts: []`, one entry `echo` with a string input `text`, no requirements. The entry sends one `document` result whose content is the input. This is the plugin the other streams' tests run. It is not a product plugin. +- `plugins/README.md` is the contributor guide: the folder layout, `plugin.json` fields, the four functions, the harness command, how a pull request is checked, and the fact that `requirements.txt` is installed by CI rather than on the user's machine. Point at `plugins/example` instead of pasting a second copy of it. + +## Acceptance + +- `python -m surfsense_plugin.harness plugins/example echo --input text=hi` prints one JSON line with `kind` `document` and content `hi`, and exits 0. +- A plugin whose entry raises exits non-zero and leaves any results lines it flushed before the exception. +- A unit test calls `result` twice and reads two JSON lines back, and calls `secret` with the variable set and with it missing. +- The example plugin's `main.py` imports only `surfsense_plugin`. + +## Needs from + +Nothing in the app. The runtime stream consumes this package later by spawning `python -m surfsense_plugin`. Until [`../python/01-interpreter.md`](../python/01-interpreter.md) lands, `python` is `uv run` from `plugins/sdk`.