The web dashboard and MCP server connect to a compose or bring-your-own store as least-privilege roles, not as its owner (#3914)

[erikdarlingdata]

Closes #3914.

Why

On a managed store, the web dashboard connects as viewer and the MCP server as mcp. On every other store, both hosts connected with postgres.connectionString. That is the collection login, which is the store owner. So a web session or an MCP token-holder on a compose or bring-your-own store had none of the role-level protections:

• the fail-closed secret-column carve (SMTP credentials, webhook URLs, the PagerDuty key, monitored servers' password blobs);
• the narrow single-table write grants;
• the statement_timeout backstop on composed queries.

The README said otherwise.

Erik's ruling on the issue:

1. The compose store provisions the roles itself.
2. Bring-your-own gets two optional login settings, and falls back to the owner with a startup warning.
3. The docs state the real behaviour.

What changes

The compose store provisions its own admin/viewer/mcp roles. It runs the managed batch, and the web and MCP hosts connect as viewer and mcp.

• How the service knows the store is its own. In a container, the store login must be the cluster's bootstrap superuser (oid 10, the role initdb created). The compose file's store image runs initdb --username $POSTGRES_USER with the service's login, so that is exactly the compose store. A superuser granted on someone else's cluster never is. Round 1 added a second condition: the cluster must hold no database besides the store's own, postgres and the templates (F2 below). This needs no new configuration, and an existing compose deployment gets the roles on its first start with the new image.

• Why not an env var or "any container". An adversarial design review showed both could re-key passwords that tools/provision-roles.sql set on a store the operator runs. That would break whatever logs in with them.

• Its own marker. The roles carry the comment darling-compose, not darling-managed. The script stamps darling-managed too, so this is what lets the service manage only the roles it created. A same-named role the service did not create is left alone. Provisioning is then refused and the warning gives the reason. Since round 1, each surface first tries its role with the credentials an earlier start wrote, and falls back to the owner only when there are none (F1 below).

• Credentials (the choice the task asked me to state). Service-generated passwords in the file: secret shape (Darling on Linux: Docker/compose distribution of the service + TimescaleDB store #1804): one plaintext file per role (pg-admin-credential, pg-viewer-credential, pg-mcp-credential).

  • Where. /var/lib/darling/credentials. The Dockerfile creates it at 0700; the service creates each file at 0600, then renames it into place.
  • When. Each file is written only after the batch commits.
  • Persistence. The new darling-credentials volume keeps them across container recreation. Without the volume they regenerate, and the marker makes re-asserting them safe.
  • What gets distrusted. A symlinked or group/other-readable file is treated as untrusted and regenerated. That is the Unix counterpart of the managed pre-plant check. Round 1 tightened this: the directory is checked too, and the file must be a regular file (F4, F6 below).
  • Why not compose secrets:. That would have meant three new operator-created secret files, which is new configuration. Also, compose fails up when a declared secret file is missing, so existing deployments would break.

• Readiness. The worker publishes an in-process verdict only after the batch commits, and a container host waits for it. Every collection-blocking stand-down settles the verdict, so a host never waits for one that is not coming. A refused or failed provisioning keeps each surface on its role with an earlier start's credentials, and falls back to the owner with the reason only when there are none, rather than taking down the only Linux UI (F1 below).

• Same batch, three differences (via a new ProvisioningTarget):

  • the owner and database are the login's own names, quoted, so a renamed POSTGRES_USER/POSTGRES_DB still works;
  • the marker is darling-compose;
  • the database-level REVOKE ALL … FROM PUBLIC is left out, because people read the compose store with roles of their own and that statement would take their CONNECT.

  The managed batch is byte-identical to before, pinned, apart from two comment words, the fix below, and round 1's F5 (the attribute list and the membership revoke, deliberately re-pinned).

• Reload. The The compose statement_timeout only reaches the live roles on a restart, unlike every other config_service knob #2918 reload re-asserts the compose store's statement_timeout, as on a managed store. It does so only when this process provisioned the roles.

Bring-your-own: postgres.webConnectionString and postgres.mcpConnectionString.

• Forms. Literal or env:/file: reference, like postgres.connectionString.
• When they are resolved. When the host starts, not at parse. So an unreadable reference keeps that one surface down with the reason logged instead of stopping collection. It never falls back to the owner.
• Unset. The surface connects as the owner, and the service logs, once per surface, the ruling's warning. It names the secret-column carve, the narrow write grants and the statement_timeout backstop.
• Owner login configured anyway. A configured login that is the owner gets the same warning.
• Managed mode. Either setting is a validation error there, like connectionString.

tools/provision-roles.sql now creates mcp too. It gets managed's exact grants and the same fail-closed column carve. This fixes a drift found on the way: the script's viewer never got the custom_alert_rules write that the managed one has (#3285), so pointing the web dashboard at it would have 42501'd every rule edit. A new ungated test holds the script's single-table grants, default privileges and role settings to BuildProvisioningSql's output. A live test runs managed provisioning over the script's roles and asserts that no privilege changes.

Fix to the managed batch (please look at this one): mcp gets ALTER DEFAULT PRIVILEGES … IN SCHEMA collect GRANT SELECT … TO mcp.

• The bug. The continuous aggregates are created by the TimescaleDB step after provisioning, in the same start. On a fresh store, every one was unreadable to the MCP tools until the next restart (42501: permission denied for view, reproduced red on the bundled runtime before the fix). A container runs for months on one start.
• Scope. SELECT only, and on collect only (no secrets). mcp still has no default write anywhere and no default read on config. The pin that forbade any mcp ADP is narrowed to exactly that.

Docs.

• README. The compose section, the postgres config table, the web and MCP identity paragraphs ("What a web token can reach", "The store-side identity is still…"), the Security & Least-Privilege Roles section, the troubleshooting paragraph, and the network-endpoints intro. The intro wrongly said MCP/web exposure is managed-only; a container honors it too.
• provision-roles.sql header. States the owner-identity behaviour.
• Text that said "a compose or bring-your-own store runs its web and MCP hosts as the owner." Corrected in the tool text, notes and comments (get_store_query_stats, StoreStatementStats, McpCommandDeadlines, DarlingStoreMetricsReader). The MCP description and the instructions got shorter, not longer.
• Compose and samples. docker-compose.yml gets the credentials volume, and its comments and the service darling.sample.json BYO block are updated.

Lite: there are no store roles in DuckDB, so there is no parity item.

Security review, round 1

Seven findings, each fixed as the coordinator decided; the commits are on this branch.

• F1 (Medium): one bad start no longer puts web and MCP on the owner for the whole process.
  • A start that does not provision the roles (a refusal, a failure, or a collector that stood down first) keeps each surface on its role. It uses the credential an earlier start wrote, trusted by the F4 checks.
  • The not-provisioned verdict carries the worker's credential directory. A stand-down uses the shipped one.
  • The store must accept that login before the host starts. If it refuses (the role was dropped or re-keyed), the surface stays down with the store's error and never becomes the owner.
  • Only a surface with no trusted credential falls back to the owner, and its warning gives both reasons.
  • The three cases log distinguishable lines: the worker's "provisioned" info line, the surface's "with the credential an earlier start provisioned" warning, and the owner warning.
  • The README fallback sentences match.
• F2 (Medium): the compose store must be a dedicated cluster. The facts read now lists every database except the store's own, postgres and the templates (by datistemplate, and template0/template1 by name). If there is any, provisioning is refused with a reason naming up to three and counting the rest (control characters print as ?).
• F3 (Low): an env: reference set to whitespace is an error, not a blank secret. The env branch uses IsNullOrWhiteSpace. In ResolveUnmanagedAsync, a set setting that resolves blank from any branch logs an Error and keeps the surface down.
• F4 (Low): the Unix trust check covers the directory, and each file must be a regular file.
  • Every start sets the directory back to 0700 before any file is read.
  • Nothing in the directory is read or written if it is a symbolic link, or if it still has group or other bits after the chmod.
  • I added one condition the finding did not ask for. If the directory had group or other WRITE bits before the chmod, none of its files are read that start, and the new ones replace them after the commit. Otherwise a 0600 file planted while the directory was open would pass the file check and have its password re-asserted on the role.
  • Each file must not be a symbolic link or a directory, must not be size 0 (which turns away devices, pipes and sockets before anything opens them), and must have no group or other bits.
  • One warning per start covers the directory.
  • A directory that cannot be created or checked (a read-only mount, say) is neither read nor written, and the roles are still provisioned with that start's passwords.
  • Residual: managed .NET cannot read a Unix file's owner, and no native interop was added. A 0600 file another user created passes the file check. What prevents that is the directory: the shipped darling-credentials named volume is seeded root:root 0700 by the Dockerfile. A bind mount whose host directory another host user owns or can write to is outside what the check can see. The same residual is stated in the code comment on UntrustedComposeCredentialReason and in the README.
• F5 (Low): an adopted role loses every attribute and membership the service never gave it.
  • Step 1b now reads LOGIN NOSUPERUSER NOCREATEROLE NOCREATEDB NOREPLICATION NOBYPASSRLS, and the password clause is unchanged.
  • A new section 11, after every grant, revokes every membership in which admin, viewer or mcp is the member. The batch grants none, which a new pin holds: every GRANT in it is a privilege ON something.
  • One deviation from the literal format('REVOKE %I FROM %I', ...), which was measured wrong. The statement adds GRANTED BY <recorded grantor> and CASCADE. On the bundled PostgreSQL 18, a superuser's plain REVOKE removes only a grant recorded as the bootstrap superuser's; another grantor's survives with a WARNING (mutation run: 1 membership left). RESTRICT fails the whole batch on a membership the member granted onward (mutation run: 2BP01: dependent privileges exist, provisioning failed).
  • The managed batch's text pins change deliberately.
• F6 (Low, Windows path): the compose credential writer is owner-only from creation.
  • A new DarlingFileSecurity.CreateHardenedFile creates the temp file with HardenFile's ACL at creation (FileSystemAclExtensions.Create, CreateNew). One ACL builder is now shared by both.
  • The directory is hardened when it is created.
  • The writer checks IsReadableByOrdinaryUsers and re-hardens once before it gives up and leaves the old file in place.
  • The read side distrusts a file that ordinary users can read.
• F7 (Low): the derived role login keeps the owner's stricter connection settings. It copies ChannelBinding, RequireAuth, CheckCertificateRevocation and SslNegotiation (all present in Npgsql 10.0.3). Options, the passfile and client certificates stay out.

Not changed: tools/provision-roles.sql does not get the F5 attribute list. The operator runs it on their own cluster, often as a managed-cloud admin that is not a true superuser, and switching off REPLICATION/BYPASSRLS there is untested.

Test plan

Security review, round 1 (after merging current dev, 5fb3750):

• Full Darling.Tests, with DARLING_TEST_PG on a fresh bundled-runtime cluster and DARLING_TEST_PGRUNTIME: 12,926 run, 3 failed, 15 skipped.
  • The 3 failures are the runtime version pins (DarlingPgRuntimeVersionPinTests x2, DarlingLiveStoreExtensionParityTests).
  • The only runtime zip on this machine (Darling/artifacts/pg-runtime.zip, dated Jul 26) is PostgreSQL 18.4 with TimescaleDB 2.28.1, and dev pins 18.6 and 2.30.1. These are the same 3 as the first run above, so they fail on the zip, not on this change.
  • The 15 skips are environment-gated: no symlink privilege on this box, no DARLING_TEST_PG_TARGET, and similar.
• Targeted and live classes: 510/510, 0 skipped. They cover DarlingStoreLogins, DarlingManagedRoles, DarlingSecretSource, DarlingFileSecurity, ComposeStoreRolesLive, DarlingSecuritySplitLive, ScramVerifierLive, NotificationRoutesRung, CustomAlertResolveGrantLive, DarlingComposeTests, ProvisionRoles*, ComposeStatementTimeout*, StartupCommandTimeout and ControlPlaneReload. DocCommentHygiene and the census/meta classes pass: 205/205.
• Each intermediate commit builds on its own with 0 warnings. The F5 and F3 commits pass their classes alone.
• F5, red first. With the decided literal format('REVOKE %I FROM %I', ...), the live compose test fails with 2BP01: dependent privileges exist, and provisioning fails. With CASCADE but no GRANTED BY, it fails with one membership left (the one granted by a non-bootstrap role). The shipped form passes.
• New ComposeStoreRolesLiveTests, on the bundled runtime:
  • F2: a cluster holding operator_app_3914 is refused with the exact reason. No role is created and no credential directory is made.
  • F5: CREATEROLE/CREATEDB on viewer, pg_read_all_data on mcp (the control shows it reads smtp_encrypted_password), pg_monitor on admin granted by a non-bootstrap role, and pg_signal_backend that mcp granted onward to an operator role are all gone after the next start. Passwords are unchanged, and mcp reading the secret column is 42501 again.
  • F1, through the real host entry (ResolveUnmanagedAsync, in a container):
    • after a refused start, and after a stand-down, web and MCP connect as viewer and mcp with the earlier start's credentials (15 s backstop, carve 42501), and log the "earlier start" warning, never the owner one;
    • viewer re-keyed by hand: the web surface returns no login and logs 28P01, and mcp still connects;
    • mcp's file deleted: that surface falls back to the owner, and the warning gives both reasons.
  • F6: a viewer credential file granted to BUILTIN\Users is distrusted, deleted and regenerated owner-only, and the role is re-keyed.
• New unit tests:
  • the F1 decision table: refusal, failure and stand-down, each with and without a trusted earlier credential, for both surfaces, plus precedence and the three warning texts;
  • the earlier-credential reader on real files: missing directory, missing file, trusted, Users-readable, empty, blank and a directory, with nothing deleted;
  • the Unix directory verdict: 8 before/after modes;
  • the F2 refusal texts, the 3-name cap, control characters, and the facts SQL;
  • F3 whitespace in DarlingSecretSource and through ResolveUnmanagedAsync (Error, no login);
  • the F7 stricter settings;
  • CreateHardenedFile: a folder with inheritable Users read makes an ordinary file readable, while the hardened create is protected from creation and refuses an existing path;
  • the batch revoking every membership and granting none.

Before round 1:

• Darling.Tests full suite, DARLING_TEST_PG on the rig (55504) and DARLING_TEST_PGRUNTIME: 12,761 run, 3 failed. All three are runtime version pins (DarlingPgRuntimeVersionPinTests x2, DarlingLiveStoreExtensionParityTests): the rig's runtime is TimescaleDB 2.28.1 and dev pins 2.30.1 since The store's TimescaleDB moves to 2.30.1 before the store opens, closing GHSA-hcfx-29v5-2rcw, with nothing ever reverted and its own alert (#3908) #3948, so they fail on the rig, not on this change. CI's fresh runtime covers them.

• After merging current dev (resume session): targeted classes 121/121; full Darling.Tests 12,781 run, 0 failed, 561 skipped (live-gated, no rig).

• New ComposeStoreRolesLiveTests (bundled runtime):

  • a foreign viewer is left alone and its password still logs in, and a non-bootstrap superuser is refused;
  • provisioning stamps darling-compose and writes owner-only files;
  • the real DarlingWebHostService and DarlingMcpHostService, started as in a container with DARLING_CONFIG, build their store pools as viewer and mcp (15 s statement_timeout; smtp_encrypted_password → 42501);
  • mcp reads a collect view created after provisioning, and an operator's own role keeps CONNECT;
  • a second start sends no password, and a lost credential file re-keys that role alone.

• BYO live test: the shipped script runs clean; the two settings resolve through the host entry (literal and file:) to viewer/mcp; an unreadable reference returns no login (not the owner); managed provisioning over the script's roles changes zero privileges.

• Red first: without the mcp collect default, the live test fails with 42501: permission denied for view zz_3914_after_provisioning.

• Mutation checks: pointing the web host's unmanaged branch back at config.Postgres.ConnectionString, and setting _composeStoreRolesProvisioned outside the provisioned branch, each fail their source pin.

• New DarlingStoreLoginsTests: the decision table, the warning text (the three losses, per surface), the derived login (drops Options/passfile/client cert), refusals, both batch targets, the reload gate, the settings, and the host and worker wiring by containment.

• Updated pins, deliberately:

  • StartupCommandTimeoutTests: the batch site moves to ProvisionRolesAsync, plus one bootstrap site for the pre-read (dev's 32 → 33), and the class prose no longer carries site counts that went stale;
  • ControlPlaneReloadDurabilityTests: the sentinel is read in the core, and the catch-hygiene sweep now covers DarlingStoreLogins.cs too;
  • DarlingManagedRolesTests: the mcp ADP pin is narrowed;
  • ProvisionRolesAclDriftTests: both read roles, plus a grant-parity test;
  • NotificationRoutesRungTests, StoreStatementStatsTests.

• Docker compose smoke on Linux (the shipped compose file, the image built from this branch, timescale/timescaledb:latest-pg17 with TimescaleDB 2.30.1). Store logins while the web dashboard and MCP served requests (pg_stat_activity, client backends):

  Stack	darling	viewer	mcp
  Before: dev's compose file with the published :nightly image	5	0	0 (the roles do not exist)
  After: this branch	2 (the worker)	1 (web)	1 (MCP)

  What else the after stack showed:

  • Both hosts log that they wait for the collector's verdict, then start after it.
  • The roles carry darling-compose and statement_timeout=15s, log_min_duration_statement=5000ms, log_parameter_max_length=0, and PUBLIC keeps CONNECT.
  • The credentials directory is 700 root (seeded from the image) and each file is 600 root.
  • A restart logs Role passwords: unchanged.
  • /api/views returns 200 as viewer, and get_store_query_stats (web and MCP) reports connected_as_owner: false.

• Upgrade path. The before stack (dev's compose file, no credentials volume, a store the nightly created), switched to this image, provisions on its first start and serves web and MCP as viewer and mcp. --force-recreate regenerates and re-asserts all three passwords and keeps working.

• postgres.mcpConnectionString pointed at the owner inside the container: the MCP host starts on it without waiting, and the "logs in as 'darling', the owner login" warning fires.

• Out-of-lane finds from the smoke, reproduced identically as the owner, so not caused by this change, and filed:

  • The fleet overview fails with 'Out of range of DateTime' until collection_health_hourly first materializes: a never-refreshed watermark is finite (4714 BC), not -infinity #3973: a never-refreshed aggregate's watermark is 4714 BC, which is finite, so get_fleet_overview and /api/fleet fail on a fresh store. Measured on 2.28.1 and 2.30.1.
  • A store with no readable log directory loses the hourly collector-cost flush: the failed store-log capture leaves the shared connection closed #3971: on a store with no readable log directory, the store-log capture's 58P01 makes Npgsql close the shared connection, and the collector-cost flush after it fails every hour.

Filed rather than fixed:

• Custom alert rules are never evaluated on the Linux compose store, which now has the viewer role the evaluator needs #3970, needs a design decision. Custom alert rules are never evaluated on the compose store, which now has the viewer role the evaluator needs. Enabling them would start firing rules users have already saved, and the ruling did not cover that.
• A store with no readable log directory loses the hourly collector-cost flush: the failed store-log capture leaves the shared connection closed #3971 and The fleet overview fails with 'Out of range of DateTime' until collection_health_hourly first materializes: a never-refreshed watermark is finite (4714 BC), not -infinity #3973, out of lane. See the smoke results above.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GdmA4ND1wLSqA91ax1m4xv