Repository navigation
Agent dispatch: alerts start an agent investigation, and opt-in agents act through MCP tools that check and log every action (supersedes #2138) #5632
Description
Claude posting for Erik Darling.
When an alert fires, Darling can hand it to an AI agent with a structured investigation brief. The agent investigates through Darling's MCP tools and reports back. As an opt-in per server and per action, the agent can also act. It can open a Jira ticket or a draft PR. It can force or unforce a Query Store plan, evict a plan, or kill an idle lead blocker. It can also tune alert thresholds, mute and unmute alerts, file bug reports, and propose new alerts for its environment. The agent acts through Darling's MCP action tools. Each tool checks the action, runs it with Darling's credential, verifies the result and reverts it if needed. Every call is logged for audit. The agent never holds a server credential. Darling ships no agent. Operators bring their own.
This issue supersedes #2138. Erik: "i think this supersedes 2138 and that can be closed now."
Erik added four design requirements for the agent route (sections 7.6 to 7.9):
- agents are going to need to be able to tune alert thresholds
- agents should be able to mute/unmute alerts
- agents should be able to file bug reports about issues
- agents should be able to request new alerts that fit situations in their environment
Erik on how agents act (2026-10-14):
yes, via mcp calls. they need to log their actions for audit.
Bring your own agent
Erik: "i'm fine providing a starter file, but packaging in an agent isn't something i think i can do presently. perhaps in the future that will become a thing."
- Darling ships no agent and no agent runner. Operators bring their own agent, from any vendor and on any model.
- Darling ships the contract: the request, the brief, the report and proposal tools, and the checks on every action.
- The contract does not depend on who the agent is. A packaged agent can come later, as one more client of the same contract. That is out of scope for this issue.
- Darling assumes the weakest agent that can connect: a new agent with no memory and no instructions file. Everything that it needs must come through the brief and the MCP tools (section 3).
- Darling trusts no agent. It gives no agent a server credential. An agent acts only by calling an MCP action tool, and the tool checks the action again before it runs.
- A starter instructions file (AGENTS.md) is an optional example in the docs. It is short, and it does not repeat the MCP server instructions. Nothing in Darling depends on it. An example of its content:
- Start from the brief. Then call
get_tool_guide. - Report through
submit_investigation_report. - Never act beyond the actions that the brief allows.
- Start from the brief. Then call
Why
- Agents already do this by hand. Two agent sessions poll
get_alert_historyon a timer. They investigate plan regressions, deadlocks and high CPU, and they report to a person. - The volume fits agents. One production store logged 858 analysis alerts in 21 days, about 40 a day. That is too many for a person to read. It is few enough for one agent per new alert.
- The loop finds real problems. PLAN_REGRESSION compares plans compiled for different inputs, and its advice to force the "best" plan makes the slow case slower #5630 came from it.
- PLAN_REGRESSION compares plans compiled for different inputs, and its advice to force the "best" plan makes the slow case slower #5630 also shows why actions need verification. The tool's own advice was "force the faster plan". In a scratch test, that force made two queries 4.4× and 10.6× slower.
What exists to build on
- The generic webhook channel can send
{{context_json}},{{incidents_json}}and{{dedup_key}}([QUESTION] Connection is broken: need PM restart ? #1506, [FEATURE] Generic webhook: a raw-JSON {{context_json}} token so automation consumers get the structured alert context instead of a flattened line #2302), with custom headers (Darling: store SQL login passwords on Linux, sealed to the service's key #5387). - Notification routes send alert families to different channels (Route alert families to different notification channels — self-monitor, reports, job failures, and performance pages have different audiences #3598).
- The MCP read tools an investigation needs:
get_alert_history,get_analysis_findings,get_analysis_facts,get_active_queries(withis_head_blocker),get_query_store_query_historyand the rest. - From [ON HOLD] [FEATURE] Auto force plan bot — detect bad Query Store plans (CPU-weighted), force with self-review, evict-first option #2138, all merged:
- CPU-weighted plan-regression scoring (#2138 Phase 0: CPU-primary plan-regression scoring + absolute-spend noise floor #2139).
- The parameter-sensitivity co-fire flag (#2138 gap 3: parameter-sensitivity co-fire flag + force-plan caution #2140).
structured_remediation, withFactRemediation.ForcePlanBlockersas the single policy gate (#2138: machine-first structured_remediation for MCP consumers #2146).- The per-server remediation credential (V113) and the journal's
actorcolumn (Add the remediation credential seam, the journal's actor and the evict-then-observe state machine (#2138 phase 1) #3170). - APC verdict facts in the force-plan surface (Force-plan remediation reports 'eligible, no blockers' for plans automatic plan correction already owns, has already failed to force, or has withdrawn — five for five on one incident #3652).
- Still closed on purpose:
IPlanForceExecutorhas no implementation, andPlanForceNoWritePathTestskeeps the write path shut.
Design
1. Agent routes
- A notification route gets a new channel type, agent.
- An agent route selects alerts by metric, not only by family. For example, it can send "Blocking Detected", "Deadlocks Detected", "High CPU" and "Analysis: queries" alerts to agents, and leave the others on Slack.
- A route sets the alert metrics, the servers, the autonomy level (section 6) and the limits.
2. Investigation requests
When an alert fires on an agent route, Darling writes an investigation request to the monitoring store. The request does not need a connected agent.
| Field | Content |
|---|---|
request_id |
a unique id |
dedup_key |
the alert's dedup key |
| server, alert metric, alert time | from the alert |
brief |
the investigation brief (section 3) |
state |
open, claimed, reported, closed or expired |
claimed_by, lease_until |
the agent that holds the request, and its lease |
report |
the agent's findings (section 5) |
-
One open request per server, alert family and time window. One incident can fire several alerts with one cause. For example, High CPU and Poison Wait fired together in one production incident. So Darling groups open requests:
- By server.
- By investigation family (below).
- By a time window, for example 30 minutes from the first alert.
A repeat or related alert joins the open request and increments its counter. It does not start a second agent. Alerts in two families start two agents.
-
Investigation families. Erik: "also one agent per alert family. blocking/deadlocking, query performance, etc." These families are finer than today's notification families (self-monitor, reports, agent-jobs and performance). A draft for Erik to confirm:
Family Alerts (draft) Blocking and deadlocks Blocking Detected, Blocking Wait Time, Deadlocks Detected, PostgreSQL Vacuum Horizon Blocked Query performance "Analysis:" findings on queries, Long-Running Query, Forced Plan Failing Resources High CPU, Poison Wait, tempdb Space, Version Store (PVS), Volume Free Space, Database File Growth Availability Database State, availability group alerts, PostgreSQL Wraparound Risk Jobs the agent-jobs family Self-monitor the self-monitor family -
Request limits per route. Darling cannot see what an agent spends. So each route has a limit on new requests per hour and per day. The viewer shows each limit and how much of it is used. At a limit, the alert still goes to its other channels, and Darling logs that it made no request.
-
Leases. An agent claims a request with a lease. The default is 15 minutes. If the lease ends without a report, the request opens again.
-
Expiry. A request that no agent claims expires after a set time. The default is 4 hours. The alert still goes to its other channels.
3. The investigation brief
The brief tells the agent where to start. Darling already composes the analysis story, so the brief is mostly a view of data that exists.
- The alert identity and its
context_json. - The analysis story, facts and drill-down of the finding, if the alert has one.
- The
structured_remediationverdict and its blockers. - Next calls: an ordered list of Darling MCP tool calls, with their arguments. For a plan regression, an example is
get_query_store_query_historyfor the query id, then the plan comparison. - Read rules: the server's read windows and read limits.
- Allowed actions: the actions that this route and server allow, and the autonomy level of each.
- Budget: a time limit and a tool-call limit.
Context for an agent that starts fresh
An agent that already knows the environment does well because it has history: earlier verdicts, false positives and corrections. A new agent has none of that. So Darling must supply it.
- The history of this alert. The brief attaches the earlier investigation reports for the same dedup key or analysis story, with their verdicts. A new agent then starts where the last one stopped.
- Recent changes on the server: installs, threshold changes, mutes and forced plans in the last N days.
- An environment profile. The operator writes it in the viewer, per server or per tag. Examples are the server's role, its business hours, its owners, its read limits, and patterns that are known to be noisy. Darling enforces the parts that are limits. The brief only states them.
- Investigation notes. An agent can attach a note to an analysis story, for example "this pattern is a known false positive, see PLAN_REGRESSION compares plans compiled for different inputs, and its advice to force the "best" plan makes the slow case slower #5630".
- A note reaches later briefs only after a person approves it.
- Each note shows its author, and it has an expiry.
- Without this review, one wrong note would mislead every later agent.
- A size budget. The brief has a fixed size, for example 4,000 to 8,000 tokens. It gives the facts and the verdict first. Then it gives the tool calls that fetch more.
- Product knowledge stays in the product. The MCP server instructions and
get_tool_guidealready explain the tools. Each alert type also gets an investigation playbook, and Darling keeps it current with each release. The starter file must not copy these.
4. How agents get requests
All three ways read the same request table.
- Pull (required). New MCP tools
list_investigation_requestsandclaim_investigation_request. Pull works when the agent cannot accept inbound connections. - MCP push (optional). Darling's MCP server sends a notification to subscribed agents when a request opens. The agent then claims the request as in (1).
- Webhook (optional). The generic webhook gets an
{{investigation_json}}token, for a runner that accepts a POST.
These tools write only to the monitoring store, like mute_analysis_finding.
5. The report
- The agent writes its report with a new MCP tool,
submit_investigation_report. - The report has:
- a summary.
- the root cause, or "not found".
- the evidence: tool calls and key numbers.
- a verdict on the alert: real, false positive, or needs a person.
- the actions it took or proposed (section 7).
- The tool enforces the shape.
submit_investigation_reportrefuses a report without a verdict, the evidence, and a root cause or "not found". So every agent produces a usable report, with or without an instructions file. - The viewer and MCP show the report next to the alert and the finding.
- A false-positive verdict with evidence is a candidate for a detector issue, like PLAN_REGRESSION compares plans compiled for different inputs, and its advice to force the "best" plan makes the slow case slower #5630.
- Feedback from a person. Erik: "a feedback mechanism is a good idea", "maybe not so broadly as right or wrong".
- A person can add feedback to a report in the viewer.
- The fields are open for design. Examples: the verdict was correct, the root cause was partly right, a step that the agent missed, and free text.
- Feedback feeds the practice requests (section 11), the investigation notes (section 3) and the false-positive counts in threshold proposals (7.6).
6. Autonomy levels
The level is set per route, per server and per action type. The default is 0.
| Level | Name | What the agent can do |
|---|---|---|
| 0 | Off | Nothing. Darling makes no request |
| 1 | Investigate | Read through MCP and report |
| 2 | Propose | Level 1, and propose actions. The action tool stores the proposal, and a person approves it in the viewer |
| 3 | Act | Level 2, and the agent runs an allowed action itself through an MCP action tool, with no approval step. The tool runs every check first |
- One switch turns off all agent actions on all servers at once. The viewer shows the switch. The service reads it again before every action.
- Lite can send Level 1 investigations through its generic webhook. Lite stays advise-only for actions, as decided on [ON HOLD] [FEATURE] Auto force plan bot — detect bad Query Store plans (CPU-weighted), force with self-review, evict-first option #2138.
7. Actions
There are three kinds of action.
- Outward actions (Jira tickets, PRs and bug reports) do not touch a monitored server. The agent runner does them with its own credentials. The agent puts the link in its report.
- Server actions (plan force, eviction and kill) change a monitored server. The agent runs them through Darling's MCP action tools. Each tool checks the action again, runs it with the server's remediation credential, verifies the result and reverts it if needed. The agent never gets that credential.
- Configuration actions (alert thresholds, mutes and new alert rules) change only the monitoring store. They use the MCP write tools that already exist, or new ones in the same class.
Each action call names the action, the target, the evidence, the reason and the agent. The action tools refuse a call that is missing a field. Darling journals every call and its result, with the reason. That includes refusals.
7.1 Open a Jira ticket
- The route sets the Jira site, project, issue type and labels.
- One ticket per dedup key or story. If an open ticket exists, the agent adds a comment to it.
- The ticket holds the report, the evidence and a link to the alert in the viewer.
7.2 Open a pull request
- The server or database config sets an allowlist of application repositories and the base branch.
- The PR is always a draft. The agent never merges.
- A typical PR adds an index, changes a query, or adds a statistics job. The PR body carries the evidence and any before and after measurements.
- The agent must not put customer data in a PR. Query text gets the same masking as stored statement text.
7.3 Force or unforce a Query Store plan (SQL Server)
This carries over Phase 2 of #2138.
- A target is eligible only when
structured_remediationsays so.ForcePlanBlockersstays the single policy gate. - New blockers from PLAN_REGRESSION compares plans compiled for different inputs, and its advice to force the "best" plan makes the slow case slower #5630:
- The statement carries
OPTION (RECOMPILE). - The best plan and the latest plan were compiled for different parameter values.
- The parameter-sensitivity flag is set. This one is already a blocker.
- The statement carries
- Evict first (optional). Darling evicts the plan and watches the next compile. It forces only if the new plan is not better.
- Self-review. After a force, Darling compares CPU per execution of the forced plan with the plan it replaced. The default windows are 15 minutes, 1 hour and 24 hours. If the force is not a net benefit, Darling unforces it and records it in the ledger.
- Ownership. Darling unforces only its own forces. It never changes a force that an operator or APC placed.
- Forcing failures. If
force_failure_countrises above 0, Darling reverts the force and reports it. - Limits: one force per query per cooldown (default 24 hours), and a daily maximum per server.
- Test: the PLAN_REGRESSION compares plans compiled for different inputs, and its advice to force the "best" plan makes the slow case slower #5630 reproduction becomes a test. Darling must refuse both of its "best" plans.
7.4 Evict a plan or recompile an object (SQL Server)
- Targeted only:
DBCC FREEPROCCACHE (plan_handle), orsp_recompileon one object. - Never a cache clear for the whole instance or database.
- "Recompile" here means that the next execution compiles again. The agent does not add an
OPTION (RECOMPILE)hint to code. A code change goes through a PR (7.2). - Self-review is the same as in 7.3. If the new plan is worse, Darling reports it. An eviction has no undo, so evictions have their own daily limit.
7.5 Kill a lead blocker
This is the action with the highest risk. It comes in the last phase, and it is off unless the route arms it.
Darling kills a session only when every condition holds. It checks them all again, on a live read, immediately before the kill.
| Condition | Default |
|---|---|
| The session is the head of a blocking chain. It blocks other sessions, and no session blocks it | required |
| The blocking has lasted at least N minutes | 5 minutes |
| It blocks at least M sessions, or the blocked sessions have waited at least W in total | 3 sessions, or 10 minutes |
The session is idle with an open transaction: sleeping with open_transaction_count > 0 on SQL Server, or idle in transaction on PostgreSQL |
required |
| Its last request ended at least I minutes ago | 2 minutes |
| The rollback is small: the log bytes that its transaction used are below a limit | 100 MB |
| It is a user session. It is not in KILLED/ROLLBACK, and it is not in a distributed transaction | required |
| Its login, program and database are not on the protected list. The default list holds backups, DBCC, index maintenance, replication and CDC agents, SQL Agent jobs and the monitoring service | required |
It is the session that the alert saw: the same session id, and the same login_time (SQL Server) or backend_start (PostgreSQL) |
required |
- This design never kills a head blocker that is running. A long rollback can hold locks longer than the blocker would. A person decides those cases.
- Before the kill, Darling records the session's login, host, program, open-transaction age and log bytes, and the hash of its last statement.
- After the kill, Darling checks that the chain cleared. It then reports the kill with that evidence.
- On PostgreSQL, the action is
pg_terminate_backend, run by a role withpg_signal_backend. The report also suggestsidle_in_transaction_session_timeoutas the lasting fix. - Limits: at most K kills per server per hour (default 1). If the same login and program need a second kill within 24 hours, Darling asks a person.
7.6 Tune alert thresholds
What exists today (from a code read):
- The built-in alert thresholds are one global row.
update_alert_settingschanges it, so a change applies to every server. - Custom alert rules can be scoped to a list of servers.
- There is no per-server override for a built-in threshold.
What this needs:
- Per-server and per-database threshold overrides for the built-in alerts. An agent can then tune one environment without changing the fleet.
- At Level 3, an agent changes only an override for its own servers. A change to the global row is always Level 2: a person approves it.
A tuning proposal holds:
- The alert, the scope, the current value and the proposed value.
- The evidence: how often the alert fired over the last N days, and how many of those alerts investigation reports judged false positives.
- A replay of the proposed value over the stored history (section 7.9).
Limits:
- One change moves a threshold by at most a set factor per 24 hours. The default is 2×.
- An agent cannot turn an alert off by tuning. That is a mute, and a mute has an expiry (7.7).
- Some alerts have a floor that agents cannot loosen past. The defaults are the self-monitor family, Deadlocks Detected, Database State and Volume Free Space.
Review:
- For 14 days after a change, Darling evaluates the old threshold in shadow. It records each alert that the old value would have raised.
- At the review date, Darling reports the effect: alerts before and after, and every shadow alert.
- If an investigation finds that a shadow alert was a real problem, Darling restores the old value and reports it.
7.7 Mute and unmute alerts
This builds on the mute rules that exist (create_mute_rule, update_mute_rule, set_mute_rule_enabled, delete_mute_rule and mute_analysis_finding). A muted alert is still logged. It is not delivered.
Rules for agent mutes:
- The scope must be narrow. A server and a metric are required. Today a mute rule with no fields mutes every alert on the whole fleet, so Darling refuses that shape from an agent.
- Every agent mute expires. The default is 24 hours. The maximum is 7 days at Level 3. A longer mute needs a person.
- Every agent mute has a reason and a link to the investigation report.
- A muted alert does not open an investigation request. Darling still counts its occurrences and shows them in the mute review.
- Ownership: an agent can unmute or change only the mutes that agents made. It can propose a change to a mute that a person made, but only a person can make that change.
- Agents cannot mute the self-monitor family, or any alert about the agent route itself.
7.8 File bug reports
When an investigation finds a problem in the monitoring tool itself, the agent files a bug report. Examples are a false positive from a detector, a collector error, wrong advice, or a slow MCP tool. #5630 is an example of such a report.
How it works:
- A new MCP tool,
submit_bug_report, stores a draft in the monitoring store. The draft holds:- The title and the component: collector, detector, alert or MCP tool.
- The evidence: tool calls and numbers.
- The repro steps, if there are any.
- The linked investigation report.
- Darling adds the build version, the schema version and the platform.
- The route sets the destination: the product's GitHub repository, the operator's Jira, or both.
- The agent runner files the report with its own credentials and stores the link on the draft.
Redaction (required):
- Darling makes a redacted copy of the draft. Server and database names become labels. Statement text is dropped or masked. Host names, IP addresses and cloud resource ids are removed.
- Only the redacted copy can be filed. A scan of the copy must pass before filing.
- A public tracker keeps everything, including edit history. So for a public repository, a person approves each report (Level 2). Level 3 is allowed only for a private tracker.
Deduplication and limits:
- Darling groups drafts with the same component and signature.
- Before filing, the runner searches open issues. If a match exists, it adds a comment instead of a new issue.
- A set number of reports per day per install. The default is 5.
7.9 Request new alerts
An agent can propose a new alert that fits its environment. Examples are "this database's log backups lag after the nightly load" or "this job fails at month end".
What exists today: custom alert rules, with list_custom_alert_templates, validate_custom_alert_rule, test_custom_alert_rule and create_custom_alert_rule.
What this needs:
- A replay tool,
backtest_custom_alert_rule.test_custom_alert_rulereads only the current value. A proposal must show how often the rule would have fired over the last N days, and on which servers. It must also show how often it would have fired together with an existing alert. - Shadow mode for new rules. A rule from an agent first runs in shadow for a trial period (default 7 days). Darling evaluates it and records each would-be alert, but delivers nothing.
- Promotion: after the trial, a person promotes the rule (Level 2). At Level 3, Darling promotes it when its fire rate stays below a set limit and it does not duplicate an existing alert.
- Scope: an agent rule names its servers. A fleet-wide rule always needs a person.
- Ownership: an agent can change or turn off only the rules that agents made. It never deletes a rule that a person made.
- If the rule language cannot express the alert, the agent files a feature request through 7.8.
8. What agents cannot change
At every level, an agent cannot:
- Change an agent route, an autonomy level or the global switch.
- Change a credential.
- Change, turn off or delete a notification route.
- Delete a mute rule, alert rule or plan force that a person made.
- Approve its own proposal.
These rules stop an agent from raising its own permissions, and from removing the alerts that report on it.
9. Credentials
- The monitoring credential stays read-only. This promise from [ON HOLD] [FEATURE] Auto force plan bot — detect bad Query Store plans (CPU-weighted), force with self-review, evict-first option #2138 does not change.
- Server actions use the per-server remediation credential (V113). It is optional, with no default and no fallback.
- Least privilege per action:
- Plan force and unforce: ALTER on the database.
- Plan eviction: ALTER SERVER STATE.
sp_recompileneeds ALTER on the object. - Kill on SQL Server: ALTER ANY CONNECTION.
- Kill on PostgreSQL: membership in
pg_signal_backend.
- ALTER ANY CONNECTION is a strong permission. The viewer shows the permission that each action needs when an operator arms it.
- Jira and git credentials stay in the agent runner, not in Darling.
10. Audit
- Every agent action is logged for audit. Erik: "they need to log their actions for audit."
- Darling writes the journal row itself, when an MCP action tool is called and again when it finishes. So the record does not depend on the agent.
- Each row holds the agent id, the action, the target, the reason, the evidence or report link, the state before, the result, and any revert.
- One journal holds every request, claim, report, action call, proposal, approval, refusal, execution, verification and revert.
- The viewer shows the actor, the reason and the report link for every threshold, mute and alert rule that an agent made. Each one has a revert button.
- Each row names the actor: the agent id, the person who approved, or the service.
- The viewer shows the journal per server, and MCP can read it.
11. Practice requests (optional)
Darling cannot vouch for an agent that it does not ship. So an operator can test their own agent first.
- Darling serves past alerts with known outcomes as practice requests. Examples are confirmed false positives and confirmed real problems from the investigation history.
- The agent investigates them at Level 1. The action tools run nothing for a practice request.
- Darling compares each verdict with the known outcome, and the viewer shows the score.
- An operator can use the score to decide whether to raise the agent's autonomy. Darling does not require it.
12. Data that agents receive (a setting)
Erik: "i think it's up to end users to manage this in a way that meets their needs. it's not a policy i want to set or enforce. perhaps this is a setting we provide."
- An agent can send what it reads to its model provider. That includes query text, plan XML, and login and host names.
- Darling sets no policy on this. The operator decides.
- Darling provides a setting that the MCP tools apply. Example choices: full detail (today's behaviour, and the default), statement literals masked, or metadata only.
- Until agents have their own identity (section 13), the setting applies to a whole MCP listener.
13. Agent identity
Erik: "this may not always be the case, though it is today. oidc is here, just not integrated with anything."
- Today, MCP has one bearer token per listener. So Darling cannot tell two agents apart. Any agent with the token can claim any request.
- The web dashboard already has OIDC sign-in, with claim-to-role mapping (OIDC sign-in for the web dashboard: give the web surface a per-user identity instead of one shared token #2550). OIDC is the likely path for agent identity too. The agent would pick up its identity from environment variables.
- With an identity, Darling can bind each agent to its routes, its autonomy levels, its request limits and the data setting.
- This is later work. Phase A works with one token per listener.
14. Untrusted text in monitored data (docs)
Erik: "this is something that users need to manage, not for me to set policy on. locally we can do these things, sure. we can set a good example in the docs, sure. but people are gonna people."
- Anyone who can run a query on a monitored server controls some of the text that reaches an agent. Examples are query text, program names and error messages. That text can try to steer the agent.
- Server actions are covered. Darling checks every one again before it acts (section 7).
- Outward actions run with the runner's credentials, so Darling cannot check them. The operator manages that risk.
- The docs describe the risk and show an example setup. For example: outward actions at Level 2, and a Jira or git credential that can do only what the route needs.
- Darling sets no policy on this.
15. Format numbers for the brief and the report
People write their own agents around the shape of the brief and the report. A later release can change that shape. Then an agent that was written for the old shape can fail with no clear sign.
- Every brief carries a format number. Every report and every proposal states the format number that the agent uses.
- When a release changes the shape, the format number goes up.
- Darling accepts the previous format for at least one more release.
- After that, Darling refuses a report or proposal in a format that it no longer supports. The error names the format that the agent sent and the formats that Darling accepts.
- The release notes list each format change.
What this changes in earlier decisions
- [ON HOLD] [FEATURE] Auto force plan bot — detect bad Query Store plans (CPU-weighted), force with self-review, evict-first option #2138, 2026-09-08: "plan forcing is never invocable from MCP. The remediation surface is the viewer, with a human in the loop." Level 2 keeps the person in the loop: the action tool only stores a proposal, and a person approves it in the viewer. Level 3 ends the first rule: an agent forces a plan by calling an MCP action tool.
- Level 3 removes the person. It is the opt-in that Erik asked for: "including an option to let agents act."
- The MCP rule "no MCP tool writes to a monitored server" ends for the action tools. They write to a monitored server only after approval at Level 2, or at Level 3. They do so only for actions that an operator armed on that server, and only with the server's remediation credential. Every other MCP tool keeps the rule. Erik, 2026-10-14: "yes, via mcp calls."
- [ON HOLD] [FEATURE] Auto force plan bot — detect bad Query Store plans (CPU-weighted), force with self-review, evict-first option #2138, 2026-09-19: the hold ends. This issue replaces [ON HOLD] [FEATURE] Auto force plan bot — detect bad Query Store plans (CPU-weighted), force with self-review, evict-first option #2138.
Phases
| Phase | Scope | Writes to a monitored server |
|---|---|---|
| A | Agent routes, requests grouped by server, family and window, request limits, the data setting, format numbers, the brief with its fresh-agent context, pull and claim, and reports. Level 1 only. A fake agent that runs only in CI, to test the contract. No agent ships | none |
| B | Feedback on reports. Outward actions: Jira tickets, draft PRs and bug reports. Configuration actions at Level 2: threshold overrides, mutes and new alert rules, with the replay tool and shadow mode. Practice requests | none |
| C | Server actions at Level 2, through MCP action tools: plan force and unforce, and eviction, with self-review and revert. This change implements IPlanForceExecutor and opens PlanForceNoWritePathTests |
yes, with approval |
| D | Level 3 for plan force, eviction and configuration actions | yes, within policy |
| E | Kill of an idle lead blocker: Level 2 first, then Level 3 | yes |
Acceptance tests
- A request survives when no agent is connected, and it expires as configured.
- Two alerts with one dedup key make one request.
- At Level 0 and Level 1, no write reaches a monitored server.
PlanForceNoWritePathTestsstays closed until Phase C. - The global switch stops an action that a person approved but Darling did not execute yet.
- An action tool call without a reason or evidence is refused, and the refusal is journaled.
- Every action tool call writes a journal row before it runs and another when it finishes, with the agent id, the reason and the result.
- No MCP tool other than the action tools writes to a monitored server.
- Both PLAN_REGRESSION compares plans compiled for different inputs, and its advice to force the "best" plan makes the slow case slower #5630 cases are refused as force targets.
- A force that is worse at its first review window is unforced, and the ledger records it.
- A kill proposal is refused when the session's
login_timechanged, when the session became active, or when its log bytes passed the limit. - A kill of a session on the protected list is refused.
- An agent mute without a server and a metric is refused. An agent mute without an expiry gets the default. An agent mute past the maximum is refused.
- An agent cannot change or delete a mute, alert rule or force that a person made. It cannot change an agent route, an autonomy level or the global switch.
- A threshold change past the step factor is refused. An agent change to the global row at Level 3 is refused.
- After a threshold change, the shadow evaluation of the old value records the alerts that the old value would have raised.
- A bug report whose redacted copy fails the scan is not filed.
- A new agent alert rule delivers nothing until it is promoted.
- A report without a verdict or evidence is refused.
- A brief for an alert with earlier reports attaches them, and it stays inside its size budget.
- An investigation note that a person has not approved does not reach a brief.
- The CI fake agent completes a request, claim and report cycle with no instructions file.
- Related alerts on one server, in one family and one window, make one request. Alerts in two families make two requests.
- A route at its hourly limit makes no new request, and the alert still goes to its other channels.
- With literals masked, the MCP tools return no statement literal.
- Feedback on a report reaches the practice-request set.
- A report in the previous format is accepted. A report in an unsupported format is refused, and the error names the format that was sent and the formats that are accepted.
Open questions for Erik
- The default thresholds in 7.3 and 7.5.
- Does Darling or the agent runner hold the Jira and git credentials? This draft puts them in the runner.
- Where does a person approve a Level 2 action: only in the viewer, or also with a Slack or Teams button?
- Can 7.5 ever kill a head blocker that is running? An example is a query past a runtime limit from an allowlisted program.
- Resolved: there is no reference runner. See "Bring your own agent".
- Where do bug reports go: this repository, a private tracker, or both? For this repository, does a person always approve?
- Do we build per-server threshold overrides for 7.6, or do agents tune only through custom rules?
- The maximum mute length at Level 3, and the threshold floors in 7.6.
- The investigation families, and the grouping window (section 2).
- The choices for the data setting (section 12).
Edited 2026-10-16 (Phase B shipped). Phase B landed on dev as #5926 (#5913), #5933 (#5916), #5934 (#5915), #5936 (#5917) and #5937 (#5918), after Erik's rulings of 2026-10-16 (plan v4, walked through and signed off). Where this body and the rulings differ, the rulings win:
- No approval step in the product. The Level table above says a Level 2 action "stores the proposal, and a person approves it in the viewer". That is superseded: when writes are on, the agent calls the write tools directly, every change is journaled with its reason and report link, and a person can undo it from the Agent changes list (web and Viewer). Phase A's agent notes keep their approval step, because an approved note feeds every later brief.
- The caps in this body were agent-drafted, not Erik's. The 2x per day step, the floors, the 7-day agent-mute maximum and the ownership rules were added when the issue was drafted with an agent's help and were never part of the feature as Erik meant it. Only three rules hold: (a) an agent turns an alert off only by a mute, never by disabling it (exclusion-list adds are blocked too; disabling or deleting a custom rule is allowed), (b) no muting the self-monitor family or the agent route's own alerts, (c) a reason and a report link on every agent change, journaled. Nothing else is capped.
- Credentials for Jira, git and the like belong to the organization, in the agent's runner. Darling records and dedups links only.
- Bug reports are not held for approval in the product; the customer's harness decides.
- Lite stays advise-only for actions.
- Per-server threshold overrides: agreed and built.
Readers of the sections above should treat their caps and approval wording as history, not as the spec.
Edited 2026-10-18 (Erik's rulings on the server actions). Erik answered the server-action questions on 2026-10-18. Where this body and these rulings differ, the rulings win. The read-only force pre-check get_plan_force_precheck is on dev (#6065).
- Remediation credential. Server actions use the per-server remediation login (V113). The store holds only the login name and an
env:orfile:reference to its password. The remediation slot does not accept an encrypted password. - Level 3 lets an agent change a monitored server. Level 2 changes only the monitoring store. A server action needs four things: the remediation login, that server's switch for that action, a Level 3 route, and
agent_actions_enabled.mcp.allowWritesdoes not bypass them. - The write code is in its own assembly,
PerformanceMonitor.Darling.Remediation. - No product limits on agent forces, unforces or kills. There is no cooldown, no daily cap and no regression floor. Each team puts its own rules in its agent instructions.
Darling/AGENTS.example.mdgets suggestions, not rules. This replaces the limits in section 7.3. - The force pre-check gives warnings, not refusals. The force tool blocks only a call that cannot do anything: the plan is not in that query's Query Store rows, or that plan is already forced. The execution count, the time since the last run and the row counts are shown as facts, with no threshold.
- An agent may replace or unforce any force, including a force that a person or automatic plan correction placed. The journal records the original plan and who placed it. So the change is auditable, and the undo can force that plan again. The unattended plan-force bot keeps its own rules and changes only its own forces.
- The review after a force. Darling checks at the first two Query Store collector passes after the force (about 10 minutes), then every hour, and stops at 24 hours. At each check, Darling opens a follow-up request with the evidence, and the agent decides. Darling does not unforce on its own. The brief asks the agent to get an adversarial second opinion from a sub-agent or a peer agent before it acts.
- Forced-plan state. A new collector read takes each database's forced plans from
sys.query_store_plan, so a force that never runs is visible. Each force is reported as working, not exercised, ignored, failing or removed. - Kill is an agent tool, with no product limits. In the same batch as the KILL, the tool checks one thing: the session still has the same session id and
login_time. The blocking capture adds the head blocker'sstatus,login_time,last_request_end_timeand transaction log bytes. - Kill on PostgreSQL is NOT built. The first kill PR covers SQL Server only. PostgreSQL kill stays open, and the kill PR must say so in its tool description, its CHANGELOG entry and a follow-up issue.
- Grouping. A new alert joins a request when it fires within 30 minutes of the request's latest alert, up to 4 hours after its first alert. Nothing else about grouping changes.
- Data setting. The three
mcp.dataDetailmodes stay. The brief says thatmaskLiteralshides compiled parameter values. Pseudonyms that stay the same across answers are metadataOnly: give a login or host the same pseudonym in every answer #6066. - Object recompile (
sp_recompile) is not built. Evicting one plan handle covers the cases seen so far.
Readers of section 7.3 should treat its limits and ownership rules as history, not as the spec.
Activity
erikdarlingdata commentedon Oct 9, 2026
Claude posting for Erik Darling.
Erik added four design requirements for the agent route. The issue body now carries them as sections 7.6 to 7.9.
What changed in the body:
backtest_custom_alert_rule. New agent rules run in shadow before they deliver.erikdarlingdata commentedon Oct 9, 2026
Claude posting for Erik Darling.
The body now says that Darling ships no agent. Operators bring their own. Erik: "i'm fine providing a starter file, but packaging in an agent isn't something i think i can do presently. perhaps in the future that will become a thing."
What changed in the body:
New section "Bring your own agent".
Section 3, "Context for an agent that starts fresh". The brief brings:
It has a size budget, and it points to tool calls for more.
Sections 5 and 7. The report and proposal tools refuse input with a missing field.
New section 11, practice requests (optional). Operators can score their own agent on past alerts with known outcomes.
Phase A has no reference runner now. A fake agent runs only in CI. Open question 5 is resolved.
New acceptance tests for the items above.
erikdarlingdata commentedon Oct 9, 2026
Claude posting for Erik Darling.
Erik ruled on six more design points. The body now carries them.
The phases, the acceptance tests and open questions 9 and 10 now include these points.
erikdarlingdata commentedon Oct 9, 2026
Claude posting for Erik Darling.
New section 15: format numbers for the brief and the report. Erik agreed: "oh that's fine, yes".
mcp.dataDetail setting, masker and host filter (#5632 Phase A, L5 dat…
FilterThroughHostAsync serves an mcp.dataDetail level; real-host mask…
Route agent members, investigation families and the category match le…
Route agent channel: load, save, viewer validation, WPF and web route…
Investigation store operations: open or join, claim, submit, list, sw…
Investigation retention: 365 days, brief cleared at 90 (#5632 Phase A…
Agent route columns join the viewer and mcp non-secret carves (#5632 …
Investigation brief composer, source reader and format numbers (#5632 …
Brief composer unit tests and the mcp.dataDetail README line (#5632 P…
39 remaining items
erikdarlingdata commentedon Oct 16, 2026
Claude posting for Erik Darling.
Two agent sessions investigate Darling alerts by hand today. Erik asked them what they would need from this issue's design. Most of their answers are already in this issue. The items below are not, so they are proposed as additions. A name search of dev found none of them, but that is not a full code check.
Proposed additions
joined: a new alert, or a higher severity, joined the open request.resolved: the alerts in the request cleared. This depends on Darling: an opt-in setting to send alert resolutions to Slack, Teams, email and webhooks #5825.expired: no agent claimed the request. The event then goes to a fallback route, so a missed request is not silent.Not adopted, because rulings in this issue decide them
Outside Darling
Covered by another issue
erikdarlingdata commentedon Oct 16, 2026
Claude posting for Erik Darling.
Phase C plan (draft for sign-off)
The nine additions from the 18:17Z comment, as a plan: a code check of dev at
3e68e0b42, the decisions, then the slices.1. What dev already has
WebhookAlertService.cs:2179,:3384-3478), sealed at rest (DarlingWebhookSecrets.cs:47). Missing: HMAC, timestamp and id headers, rotation.openedorjoined, only on the alert's own page (InvestigationWebhookToken.cs:42). The sweep journalsexpiredandclaims_exhausted(InvestigationRequestStore.cs:331-345) but delivers nothing. The long-poll wakes onOpenedonly. No severity column (InvestigationTables.cs:75-111), no event id. #5825 shipped resolution rows (DarlingAlertDeliverer.cs:437), so "resolved" has its input.context_json(V188). Blocking keeps chains and the report XML (AlertContextBuilders.cs:58); deadlock graphs already name objects and indexes (DeadlockGraphParser.cs:92,:284). High CPU holds only the reading, threshold, maintenance sessions and grade (AlertEngine.cs:1206-1240). The rest is read at claim. The store has most sources (readonly_reason,DarlingConfigHistoryReader.cs:264; compiled values,PlanInputComparison.cs); nothing freezes them. No "incomplete" marker.PgInvestigationBriefSourceReader.cs:87-97), not firings by time of day. The index exists (PgMigrations.cs:4104).AlertFingerprint.cs:45-79,IncidentGrouping.cs:475-500). Missing: frames in the key, 24-hour and 7-day counts, the gate, the digest, merging.next_callsadds every finding's tools, any family (InvestigationBriefComposer.cs:259-292). No plan-CPU rule, no hint field.InvestigationPracticeStore.cs:174). No deploy marker.jira,pull_request,bug_report(InvestigationReportLinks.cs:47); evidence is one array. Missing: family id, commit and release, verified versus inferred, queries run.server_nameis the display name (InvestigationToolCore.cs:466-472,DarlingWorker.cs:14571). No host name; the registry has both.2. Decisions
D1. Signature format.
webhook-id,webhook-timestamp,webhook-signature(HMAC-SHA256 overid.timestamp.body). Receivers have libraries in most languages.X-Darling-Signatureheader. Every receiver writes its own check.D2. Where the key lives.
darling.json. A leak signs for every receiver.D3. Rotation and replay.
D4. Which sends are signed.
D5. How events reach a runner.
once_keywrites each event once.D6. Event set and ids.
opened;joinedonly for a new metric or a higher severity (every join is noise);resolved;expiredandclaims_exhausted;evidence_ready(D9);follow_up(D16);deadlock_digest(D13).uuidper event row, sent aswebhook-id. B. theonce_keytext. Not unique across products.max_severitycolumn on the request.D7. When a request is "resolved".
SendResolutionsflag.D8. Fallback for an expired request.
fallback_route_idon the agent route; else the same route's human channels; else a self-monitor alert.D9. When evidence is frozen, and from what.
pending; when it lands,evidence_readyfires. A section never collected saysnot_collected.D10. Where evidence is stored, and how much.
jsonbcolumn on the request. A late section rewrites the whole value.mcp.dataDetailwhen served.D11. Recurrence window and cost.
D12. Deadlock signature basis.
D13. Gate and digest. A request opens for a new signature, or a 24-hour count over 3 times the 7-day daily average; the rest go to a
deadlock_digestevent every 4 hours.D14. Merging two signatures.
agent_actiongate: journaled, with a reason, undoable. Darling only.D15. Cross-family hints.
next_callsentry.D16. Re-wakes.
follow_up_at(30 days out at most, 20 pending per route at most). Darling opens a new request withfollow_up_of, as practice does.modify_date, added to the procedure-stats collector's read. Automatic, one join per collection. B. Query id turnover. No new read, noisy. C. A marker the operator's CI posts. Exact, needs CI work.D17. Report fields.
family_of, the first report's request id, checked to exist. B. Free text. Drifts.commitandreleaselink kinds. No migration.basison each evidence item, shown as two lists. B. Two arrays with a second set of checks.queries_runarray (20 items, 2,000 characters each), masked as statement text.D18. Server names.
display_nameandhost_nameon the request, the brief and the webhook token, read from the registry when served.server_namekeeps its meaning.D19. Lite.
3. Slices
At most four parts each; one Darling migration, in C1; Lite's schema step in L1. Signing and frozen evidence come first. Tests are behaviour tests, one per behaviour; no census, roster or source-text test (#5833,
.github/REVIEW-TRAPS.md).Wave 1
event_id; requestmax_severity,evidence_state,follow_up_of,follow_up_at,follow_up_condition;investigation_evidence; signature days and merges tables; reportfamily_of,queries_run; routesigning_key(sealed) andfallback_route_id; grants; retention. Tests: evidence clears with the brief at 90 days; signature days purge at 30; an open request keeps its evidence.Wave 2
pending, thenevidence_readyfires once.joined;resolvedwaits for every linked metric; an unclaimed request reaches the fallback route; a replayed event id is not delivered twice.family_ofnaming no request is refused; acommitlink is stored; an evidence item withoutbasisis refused;queries_runis masked atMetadataOnly.Wave 3
Lite (stacked PR, after Darling's)
The two items the 18:17Z comment did not adopt stay out, for the reasons given there.
Plan to be dialled in with Erik before sub-issues and lanes.
erikdarlingdata commentedon Oct 16, 2026
Claude posting for Erik Darling.
Erik agreed the Phase C plan as written on 2026-10-16; wave 1 lanes start now.
Phase C is filed as 16 sub-issues, in three Darling waves and a Lite stack. Each is small enough for one lane (at most four parts). There is one Darling migration for the phase, in C1 (V193 unless dev has moved; PR #5996 holds V192).
Wave 1 (no dependencies except C2 on C1)
Wave 2
family_of,commitandreleaselinks,basis,queries_run(needs Phase C store: the one migration, grants and retention (#5632 Phase C C1) #5997).Wave 3
agent_actiongate (needs Deadlock signatures and digest: signature, rollup, gate, digest (#5632 Phase C C9) #6005).Lite (stacked PR after Darling's)
Things the lanes must check first:
SendResolutionsis off (D7).Gaps found while re-checking the plan against dev, each handled in the slice named:
collect.procedure_stats.module_modify_date, because C13's deploy condition has no stored value to compare today.Pre-check: declare the shared table predicate before its first use, k…
Pre-check: roster the identity read and the unknown section name in t…
Count the force pre-check in the two census pins the lane missed (#5632)
An MCP read checks a Query Store plan before anyone forces it (#5632) (…
Update the grouping pins and Lite golden hashes for the sliding windo…
Update the Lite golden hashes for the sliding window text and the bri…
Forced-plans collector: one marker row per captured database, with SQ…
The pre-check's registry read also selects postgres_major_version, fo…
The grouping join's wall-clock test stays on the open time, widened t…
The force pre-check blocks only a force that cannot work, and warns a…
erikdarlingdata commentedon Oct 18, 2026
Claude posting for Erik Darling.
A request from two agent sessions that work real alerts: link a recurring alert to a known pattern. This is a note for later, not part of the current build order.
get_alert_history.Darling/AGENTS.example.md).Alerts keep joining a request while they keep firing, up to 4 hours (#…