[Plugins|Runtime] Run a plugin in its own process

.github/workflows/desktop-tests.yml

@@ -38,6 +38,8 @@ jobs:
           filters: |
             backend:
               - 'surfsense_local/backend/**'
+              # The backend starts plugins on this SDK, and its tests run real ones.
+              - 'plugins/core/sdk/**'
               - 'docs/contracts/**'
               - '.pre-commit-config.yaml'
               - '.github/workflows/desktop-tests.yml'

docs/architecture/data-model.md

@@ -12,7 +12,7 @@ Everything the desktop app knows lives in one SQLite file, `surfsense.db`, and i
 3. **Fix on paste.** Stale `new_*` prefixes, redundant columns and JSONB status blobs become the local shapes below.
 4. **No auth.** There is no users table, no memberships and no tokens.
 5. **SQLite.** One file, with JSON columns only where they earn their keep: document metadata, message content, artifact metadata.
-6. **Queue mechanics stay in `huey.db`.** User-visible job state is `documents.status`.
+6. **Queue mechanics stay in `huey.db`.** User-visible job state is `documents.status`, and `plugin_runs.status` for a plugin's run.
 
 The conventions every table follows are in [`shared/db.py`](../../surfsense_local/backend/shared/db.py):
 
@@ -136,12 +136,26 @@ Connections are in [`connections.md`](connections.md); selection and onboarding
 
 A destination is `host:<hostname>`: `host:huggingface.co` for model search and downloads, and one per remote host, shared by every connection to it; a loopback endpoint needs none. Rows under the earlier names `model_download`, `model_search` and `image_model_pull` are no longer read; revision `0012` renamed `ollama_pull` to `model_download` before that change. See [`license/app.md`](license/app.md) and [`egress.md`](egress.md).
 
+### `plugin_runs`
+
+One row per run of a plugin's action. What the run produced is not here: the plugin wrote it through the API while it ran ([the plugins proposal](../proposals/plugins/runtime/01-process.md)).
+
+| Column | Notes |
+|---|---|
+| `id`, `workspace_id` | foreign key to `workspaces`, cascading: a run goes with the workspace it was started in |
+| `plugin_id`, `version`, `action` | which installed version ran, and which of its actions |
+| `inputs` | JSON: what the user gave the action |
+| `status` | `queued` by default, then `running`, then `succeeded` or `failed`; `cancelled` is allowed but nothing sets it yet |
+| `error` | `exit <code>` when the plugin's process ended with anything but 0 |
+| `log_tail` | the last 16 KiB the plugin printed, stdout and stderr together |
+| `created_at`, `started_at`, `finished_at` | |
+
 ## On-disk layout
 
 ```text
 <data dir>/                       ~/.surfsense by default
 ├── surfsense.db
-├── huey.db                       two queues, ingest and studio, in one file
+├── huey.db                       three queues, ingest, studio and plugins, in one file
 └── data/
     └── workspaces/<workspace_id>/
         ├── documents/<document_id>/
@@ -161,6 +175,7 @@ erDiagram
   workspaces ||--o{ documents : contains
   workspaces ||--o{ chat_threads : contains
   workspaces ||--o{ artifacts : contains
+  workspaces ||--o{ plugin_runs : contains
   documents ||--o{ chunks : "split into"
   documents ||--o| artifacts : "body of"
   chunks ||--|| chunks_fts : "rowid, by trigger"
@@ -281,6 +296,20 @@ erDiagram
     bool enabled
     datetime last_call_at
   }
+  plugin_runs {
+    int id PK
+    int workspace_id FK
+    text plugin_id
+    text version
+    text action
+    json inputs
+    text status "queued, running, succeeded, failed, cancelled"
+    text error
+    text log_tail
+    datetime created_at
+    datetime started_at
+    datetime finished_at
+  }
 ```
 
 ## Revisions
@@ -304,6 +333,7 @@ erDiagram
 | `0015` | `0015_image_selection_by_build.py` | a local `image_gen` selection is renamed from the old list's name to its curated build's id (`stable-diffusion-1.5` to `v1-5-pruned_Q4_0`, and the two SDXL models); the map is frozen in the migration, and downgrading reverses it |
 | `0016` | `0016_local_audio_provider.py` | `selected_models` rebuilt so `audiocpp` may hold `audio_gen`, and only that, without a connection; downgrading drops an `audiocpp` selection |
 | `0019` | `0019_selection_settings.py` | `selected_models.settings`, a nullable JSON column added in place; its first entry is a server audio model's `voices` |
+| `0020` | `0020_plugin_runs.py` | `plugin_runs` |
 
 - Migrations run on every API start and are idempotent. Autogenerate is off: it renders a rename as a drop plus an add, which deletes a column's data silently, and `env.py` carries no `target_metadata`, so it cannot be used by accident.
 - SQLite cannot alter a CHECK constraint in place, so `0004`, `0009`, `0012` and `0013` copy `selected_models` into a new table.

docs/proposals/plugins/01-protocol.md

@@ -206,9 +206,11 @@ The newest published release is marked latest, so its `plugin-catalog.json` is t
 The app spawns a process, hands it context, and waits for it to exit. The plugin does its own HTTP — to the sources it scrapes, and to the app, which already serves its API on loopback. There is no socket of ours, no message framing, and no results file.
 
 ```
-<python> -m surfsense_plugin_sdk.run <plugin-dir> <action> --inputs <file> --data <dir>
+<python> -S -m surfsense_plugin_sdk.run <plugin-dir> <action> --inputs <file> --data <dir>
 ```
 
+`-S` hides the packages installed beside that Python. The packaged app's plugin Python has none, but in development it is the backend's, or the CLI's under `surfsense-plugins invoke`, and a plugin must not import their packages by accident. So a plugin imports the standard library, the SDK and its own `site-packages`, and nothing else, wherever it runs. `multiprocessing` passes this on to the processes it starts. A plugin that starts another Python itself, with `subprocess` and `sys.executable`, passes `-S` and adds its own `site-packages` there too.
+
 | Argument | Meaning |
 |---|---|
 | `<action>` | An action name from the manifest. The app checks this before spawn. |

docs/proposals/plugins/02-extending.md

@@ -63,7 +63,7 @@ Something the plugin should know before it starts: an id, a URL, a setting the u
 | File | Change |
 |---|---|
 | [`01-protocol.md`](01-protocol.md) | one row in the context table |
-| `modules/plugins/runner.py` | put it in the spawn environment |
+| `modules/plugins/runner/plugin_environment.py` | put it in the spawn environment |
 | `plugins/core/manifest/` | a rule, if the plugin has to declare it first |
 | `plugins/core/sdk/surfsense_plugin_sdk/<name>.py` | one accessor, its own file, exported from `__init__` |
 | `plugins/core/sdk/tests/unit/` | a plugin that reads it, and one that runs without it |

docs/proposals/plugins/cli/01-author-commands.md

@@ -34,7 +34,7 @@ An author goes from nothing to a plugin running in their own SurfSense in a few
   - Installs `requirements.txt` into `plugins/<id>/site-packages/` for this machine's platform through packaging's install step, `install_requirements(plugin, platform, into)`, so an author's machine installs what a release installs. It skips the install when `requirements.txt` has not changed since the last one. A dependency with no prebuilt wheel for this platform fails, naming the package and the platform.
   - Gives the plugin `plugins/<id>/dev-data/` as its data folder: git-ignored, kept between invokes, and never mixed with a copy installed in the app.
   - Says on stderr which action runs, in which workspace and at which address, then `Done.` or `Failed: exit <code>.`, so the plugin's own output stays the command's only output.
-  - Runs `python -m surfsense_plugin_sdk.run` on the CLI's own Python, with the environment the app's runner builds, streams the plugin's log to the terminal, and exits with its code. It sets no run id, so a note's provenance names no run. It does not intercept HTTP and does not stub the app.
+  - Runs `python -S -m surfsense_plugin_sdk.run` on the CLI's own Python, blind to the CLI's packages, with the environment the app's runner builds, streams the plugin's log to the terminal, and exits with its code. It sets no run id, so a note's provenance names no run. It does not intercept HTTP and does not stub the app.
   - Runs outside the app, so the app's egress consent does not apply. The SDK's `http` still refuses a host the manifest does not declare.
 - `plugins/example/`, a word counter. `access: free`, `hosts: []`, no secrets. One action, `count-words`, with a required string input `text` and an optional number input `top`, 10 when empty. It depends on `tabulate`, which is pure Python, so it packages into one `any` file. It adds a note titled "Word count #N" holding the text and a table of its `top` most frequent words, and counts N in `data()`. Its `main.py` imports only the SDK and its own package. The other streams' tests run it, and the guide points at it rather than pasting a second copy. It is not a product plugin.
 - `plugins/README.md` is the contributor guide. It opens with the journey above, then covers:

docs/proposals/plugins/python/01-interpreter.md

@@ -1,6 +1,6 @@
 # Python — the interpreter a plugin runs on
 
-> Owns: `surfsense_local/electron/scripts/fetch-plugin-python.mjs`, the `extraResources` entry, `plugin_python()` and `system_key()` in `modules/plugins/python.py`.
+> Owns: `surfsense_local/electron/scripts/fetch-plugin-python.mjs`, the `extraResources` entry, `plugin_python()` and `system_key()` in `modules/plugins/plugin_interpreter.py`.
 
 ## Goal
 

docs/proposals/plugins/runtime/01-process.md

@@ -1,6 +1,6 @@
 # Runtime — process
 
-> Owns: `surfsense_local/backend/modules/plugins/runner.py`, `modules/plugins/tasks.py`, the `plugin_runs` table, the `plugins` queue and its worker, and one change outside the module: deleting a workspace stops its runs first.
+> Owns: `surfsense_local/backend/modules/plugins/runner/`, `modules/plugins/tasks.py`, the `plugin_runs` table, the `plugins` queue and its worker, and one change outside the module: deleting a workspace stops its runs first.
 > Contract: [`../01-protocol.md`](../01-protocol.md).
 
 ## Goal
@@ -12,7 +12,7 @@ The worker starts a plugin process for one run, hands it its context, and record
 - Hand-written migration for `plugin_runs`: `id`, `workspace_id`, `plugin_id`, `version`, `action`, `inputs` JSON, `status` (`queued` `running` `succeeded` `failed` `cancelled`), `error`, `log_tail`, `created_at`, `started_at`, `finished_at`. `version` is the installed version the run uses, the one its release stamped, so install knows which directory is still open. `error` is the reason from the protocol's outcome table.
 - `import_models()` and `import_tasks()` gain this module the same way the other modules do. A test in the existing registration test fails if they are missing.
 - A `plugins` queue in `shared/queue.py`, drained by its own worker: one queue per consumer, as that file already does, so a plugin run never takes a slot from ingest or Studio. Plugin runs mostly wait on the network, as Studio jobs wait on a model, so the consumer runs four threads beside `STUDIO_WORKERS`; a fifth run waits as `queued`. That is the only limit on concurrent runs. Electron starts it as `worker-plugins` beside the other two (`WorkerQueue` in `electron/src/main/sidecars/python.ts`).
-- Huey task `run_plugin(run_id)`. The API enqueues it. It sets `running`, then spawns the interpreter from `plugin_python()` ([`../python/01-interpreter.md`](../python/01-interpreter.md) supplies the packaged path; until then the task accepts an explicit interpreter path and tests pass `sys.executable`).
+- Huey task `run_plugin(run_id)`. The API enqueues it. It sets `running`, then spawns the interpreter from `plugin_python()` ([`../python/01-interpreter.md`](../python/01-interpreter.md) supplies the packaged path; until then `plugin_python()` returns the interpreter running the worker, which is a real one in development and in tests).
 - Arguments, as the protocol specifies: `<plugin-dir> <action> --inputs <file> --data <dir>`. The task writes the inputs file in a directory of its own that it deletes when the run ends. Working directory is the plugin directory. `PYTHONPATH` is the SDK directory shipped next to the interpreter; the SDK adds `<plugin-dir>` and its `site-packages` itself.
 - The environment is built from scratch, never copied from the worker's, exactly as the protocol lists it: the `SURFSENSE_PLUGIN_*` context, one `SURFSENSE_PLUGIN_SECRET_<NAME>` per declared secret, decrypted only here, the operating-system allowlist, and the Python variables. A test spawns a plugin that prints its environment and fails on `SURFSENSE_LOCAL_SECRET` or any other `SURFSENSE_LOCAL_` name. The API has already refused the run when a declared secret has no value, or when a host in `hosts` has not been allowed.
 - The API URL is the worker's own: Electron passes `SURFSENSE_LOCAL_HOST` and `SURFSENSE_LOCAL_PORT` to this process already (`electron/src/main/sidecars/python.ts`), so build it from those rather than inventing a second source of truth.

docs/proposals/plugins/sdk/01-library.md

@@ -9,7 +9,7 @@ The SDK is the seam between every plugin and the app, so both of its sides are c
 
 ## Work
 
-- Package `surfsense_plugin_sdk` in `plugins/core/sdk/`, a project of its own, `plugins/core/sdk/pyproject.toml`, with no dependencies. `run/__main__.py`, run as `python -m surfsense_plugin_sdk.run`, reads the arguments from the protocol, puts the plugin's own directory and its `site-packages` on the path as the protocol says, calls the matching `@action` function, and exits 0 or 1.
+- Package `surfsense_plugin_sdk` in `plugins/core/sdk/`, a project of its own, `plugins/core/sdk/pyproject.toml`, with no dependencies. `run/__main__.py`, run as `python -S -m surfsense_plugin_sdk.run`, reads the arguments from the protocol, puts the plugin's own directory and its `site-packages` on the path as the protocol says, calls the matching `@action` function, and exits 0 or 1.
 - Strictly typed. Every public function is annotated, the package ships a `py.typed` marker, `__all__` lists exactly what is public, and nothing public is typed `Any` except `Response.json()`, whose shape comes from the source, not the SDK. Verbs return typed objects, such as `Document`, rather than dictionaries. pyright checks the SDK in strict mode, and the checks type-check every plugin against it ([`../release/02-pull-request-checks.md`](../release/02-pull-request-checks.md)).
 - Context accessors, one file each: `action`, `secret`, `data`. `action` passes the inputs to the function as keyword arguments. `secret` reads `SURFSENSE_PLUGIN_SECRET_<NAME>` and raises by name when it is undeclared or absent.
 - `http/`: `get`, `post` and `request` on the standard library, one file each for the host check, redirects, headers, the response and sending. It reads `hosts` from the plugin's own `manifest.json`, checks the host before every connection and every redirect, and raises `HostNotDeclared` naming the host. A redirect to another host drops `Authorization` and `Cookie`. Requests name the plugin in `User-Agent` unless the author sets one, since sources often refuse Python's own. Response headers are found however a name is written. One log line per request: method, host, status, or the reason it failed, and a failed connection raises `ConnectionError` naming the host. A default timeout per request, so a silent server fails the call rather than the whole run. Certificates are verified by the standard library against the system's certificate file, which the bundled Python finds on all three systems: `/etc/ssl` on Linux, `/private/etc/ssl` on macOS, the system store on Windows. The SDK depends on nothing.

plugins/README.md

@@ -77,7 +77,7 @@ def search(query: str) -> None:
 - `secret("token")` returns a secret the manifest declares, as the user entered it.
 - `data()` is your plugin's own folder (see below).
 - `document.add`, `list` and `update` work on the workspace the user ran the plugin in.
-- When something goes wrong, say why on stderr, or with `sys.exit("Hacker News refused your token. Update it in Settings.")`. The user reads it in the run's log.
+- When something goes wrong, say why on stderr, or with `sys.exit("Hacker News refused your token. Update it in Settings.")`. The user reads it in the run's log. Use `sys.exit`, not `exit`: the plugin's Python starts without the shortcuts meant for its interactive shell.
 
 ## Asking the user for things
 

plugins/core/cli/surfsense_plugin_cli/invoke/plugin/run_plugin.py

@@ -24,6 +24,8 @@ def run_plugin(
         finished = subprocess.run(
             [
                 sys.executable,
+                # Blind to the CLI's own packages, as the app's runner is to its own.
+                "-S",
                 "-m",
                 "surfsense_plugin_sdk.run",
                 str(folder),

plugins/core/cli/tests/invoke/test_running.py

@@ -56,3 +56,35 @@ def echo(text: str) -> None:
     )
 
     assert finished.returncode == 3
+
+
+def test_a_plugin_cannot_import_what_the_cli_has_installed(
+    cli, plugin, real_app
+) -> None:
+    """The app gives a plugin no packages but its own, so neither does invoke."""
+    plugin.write(
+        """
+import importlib.util
+
+from surfsense_plugin_sdk import action
+
+
+@action("echo")
+def echo(text: str) -> None:
+    print(importlib.util.find_spec("typer") is None)
+"""
+    )
+
+    finished = cli(
+        "invoke",
+        str(plugin.folder),
+        "echo",
+        "--input",
+        "text=hi",
+        "--api-url",
+        real_app.url,
+        "--workspace",
+        str(real_app.new_workspace()),
+    )
+
+    assert (finished.returncode, finished.stdout) == (0, "True\n")

plugins/core/sdk/surfsense_plugin_sdk/run/__main__.py

@@ -1,6 +1,6 @@
 """The command the app runs a plugin with, as the protocol fixes it.
 
-python -m surfsense_plugin_sdk.run <plugin-dir> <action> --inputs <file> --data <dir>
+python -S -m surfsense_plugin_sdk.run <plugin-dir> <action> --inputs <file> --data <dir>
 """
 
 import argparse

plugins/core/sdk/tests/conftest.py

@@ -87,6 +87,7 @@ def run(
         return subprocess.run(
             [
                 sys.executable,
+                "-S",
                 "-m",
                 "surfsense_plugin_sdk.run",
                 str(self.folder),

surfsense_local/backend/alembic/versions/0020_plugin_runs.py

@@ -0,0 +1,69 @@
+"""Record each run of a plugin's action.
+
+A run belongs to the workspace it was started in and goes when that workspace
+does. `cancelled` is among the statuses from the start: SQLite cannot alter a
+check constraint, so adding it later would mean rebuilding the table.
+
+Revision ID: 0020
+Revises: 0019
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+
+from alembic import op
+
+revision: str = "0020"
+down_revision: str | Sequence[str] | None = "0019"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    op.create_table(
+        "plugin_runs",
+        sa.Column("id", sa.Integer(), nullable=False),
+        sa.Column("workspace_id", sa.Integer(), nullable=False),
+        sa.Column("plugin_id", sa.String(), nullable=False),
+        sa.Column("version", sa.String(), nullable=False),
+        sa.Column("action", sa.String(), nullable=False),
+        sa.Column("inputs", sa.JSON(), nullable=False),
+        sa.Column(
+            "status",
+            sa.Enum(
+                "queued",
+                "running",
+                "succeeded",
+                "failed",
+                "cancelled",
+                name="pluginrunstatus",
+                native_enum=False,
+                create_constraint=True,
+            ),
+            nullable=False,
+        ),
+        sa.Column("error", sa.String(), nullable=True),
+        sa.Column("log_tail", sa.String(), nullable=True),
+        sa.Column(
+            "created_at",
+            sa.DateTime(),
+            server_default=sa.text("(CURRENT_TIMESTAMP)"),
+            nullable=False,
+        ),
+        sa.Column("started_at", sa.DateTime(), nullable=True),
+        sa.Column("finished_at", sa.DateTime(), nullable=True),
+        sa.ForeignKeyConstraint(
+            ["workspace_id"],
+            ["workspaces.id"],
+            name=op.f("fk_plugin_runs_workspace_id_workspaces"),
+            ondelete="CASCADE",
+        ),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_plugin_runs")),
+    )
+    op.create_index("plugin_runs_workspace", "plugin_runs", ["workspace_id"])
+
+
+def downgrade() -> None:
+    op.drop_index("plugin_runs_workspace", table_name="plugin_runs")
+    op.drop_table("plugin_runs")