Skip to content

Agent dispatch: alerts start an agent investigation, and opt-in agents act through MCP tools that check and log every action (supersedes #2138) #5632

Open

Description

@erikdarlingdata

Claude posting for Erik Darling.

When an alert fires, Darling can hand it to an AI agent with a structured investigation brief. The agent investigates through Darling's MCP tools and reports back. As an opt-in per server and per action, the agent can also act. It can open a Jira ticket or a draft PR. It can force or unforce a Query Store plan, evict a plan, or kill an idle lead blocker. It can also tune alert thresholds, mute and unmute alerts, file bug reports, and propose new alerts for its environment. The agent acts through Darling's MCP action tools. Each tool checks the action, runs it with Darling's credential, verifies the result and reverts it if needed. Every call is logged for audit. The agent never holds a server credential. Darling ships no agent. Operators bring their own.

This issue supersedes #2138. Erik: "i think this supersedes 2138 and that can be closed now."

Erik added four design requirements for the agent route (sections 7.6 to 7.9):

  1. agents are going to need to be able to tune alert thresholds
  2. agents should be able to mute/unmute alerts
  3. agents should be able to file bug reports about issues
  4. agents should be able to request new alerts that fit situations in their environment

Erik on how agents act (2026-10-14):

yes, via mcp calls. they need to log their actions for audit.

Bring your own agent

Erik: "i'm fine providing a starter file, but packaging in an agent isn't something i think i can do presently. perhaps in the future that will become a thing."

  • Darling ships no agent and no agent runner. Operators bring their own agent, from any vendor and on any model.
  • Darling ships the contract: the request, the brief, the report and proposal tools, and the checks on every action.
  • The contract does not depend on who the agent is. A packaged agent can come later, as one more client of the same contract. That is out of scope for this issue.
  • Darling assumes the weakest agent that can connect: a new agent with no memory and no instructions file. Everything that it needs must come through the brief and the MCP tools (section 3).
  • Darling trusts no agent. It gives no agent a server credential. An agent acts only by calling an MCP action tool, and the tool checks the action again before it runs.
  • A starter instructions file (AGENTS.md) is an optional example in the docs. It is short, and it does not repeat the MCP server instructions. Nothing in Darling depends on it. An example of its content:
    • Start from the brief. Then call get_tool_guide.
    • Report through submit_investigation_report.
    • Never act beyond the actions that the brief allows.

Why

What exists to build on

Design

1. Agent routes

  • A notification route gets a new channel type, agent.
  • An agent route selects alerts by metric, not only by family. For example, it can send "Blocking Detected", "Deadlocks Detected", "High CPU" and "Analysis: queries" alerts to agents, and leave the others on Slack.
  • A route sets the alert metrics, the servers, the autonomy level (section 6) and the limits.

2. Investigation requests

When an alert fires on an agent route, Darling writes an investigation request to the monitoring store. The request does not need a connected agent.

Field Content
request_id a unique id
dedup_key the alert's dedup key
server, alert metric, alert time from the alert
brief the investigation brief (section 3)
state open, claimed, reported, closed or expired
claimed_by, lease_until the agent that holds the request, and its lease
report the agent's findings (section 5)
  • One open request per server, alert family and time window. One incident can fire several alerts with one cause. For example, High CPU and Poison Wait fired together in one production incident. So Darling groups open requests:

    • By server.
    • By investigation family (below).
    • By a time window, for example 30 minutes from the first alert.

    A repeat or related alert joins the open request and increments its counter. It does not start a second agent. Alerts in two families start two agents.

  • Investigation families. Erik: "also one agent per alert family. blocking/deadlocking, query performance, etc." These families are finer than today's notification families (self-monitor, reports, agent-jobs and performance). A draft for Erik to confirm:

    Family Alerts (draft)
    Blocking and deadlocks Blocking Detected, Blocking Wait Time, Deadlocks Detected, PostgreSQL Vacuum Horizon Blocked
    Query performance "Analysis:" findings on queries, Long-Running Query, Forced Plan Failing
    Resources High CPU, Poison Wait, tempdb Space, Version Store (PVS), Volume Free Space, Database File Growth
    Availability Database State, availability group alerts, PostgreSQL Wraparound Risk
    Jobs the agent-jobs family
    Self-monitor the self-monitor family
  • Request limits per route. Darling cannot see what an agent spends. So each route has a limit on new requests per hour and per day. The viewer shows each limit and how much of it is used. At a limit, the alert still goes to its other channels, and Darling logs that it made no request.

  • Leases. An agent claims a request with a lease. The default is 15 minutes. If the lease ends without a report, the request opens again.

  • Expiry. A request that no agent claims expires after a set time. The default is 4 hours. The alert still goes to its other channels.

3. The investigation brief

The brief tells the agent where to start. Darling already composes the analysis story, so the brief is mostly a view of data that exists.

  • The alert identity and its context_json.
  • The analysis story, facts and drill-down of the finding, if the alert has one.
  • The structured_remediation verdict and its blockers.
  • Next calls: an ordered list of Darling MCP tool calls, with their arguments. For a plan regression, an example is get_query_store_query_history for the query id, then the plan comparison.
  • Read rules: the server's read windows and read limits.
  • Allowed actions: the actions that this route and server allow, and the autonomy level of each.
  • Budget: a time limit and a tool-call limit.

Context for an agent that starts fresh

An agent that already knows the environment does well because it has history: earlier verdicts, false positives and corrections. A new agent has none of that. So Darling must supply it.

  • The history of this alert. The brief attaches the earlier investigation reports for the same dedup key or analysis story, with their verdicts. A new agent then starts where the last one stopped.
  • Recent changes on the server: installs, threshold changes, mutes and forced plans in the last N days.
  • An environment profile. The operator writes it in the viewer, per server or per tag. Examples are the server's role, its business hours, its owners, its read limits, and patterns that are known to be noisy. Darling enforces the parts that are limits. The brief only states them.
  • Investigation notes. An agent can attach a note to an analysis story, for example "this pattern is a known false positive, see PLAN_REGRESSION compares plans compiled for different inputs, and its advice to force the "best" plan makes the slow case slower #5630".
    • A note reaches later briefs only after a person approves it.
    • Each note shows its author, and it has an expiry.
    • Without this review, one wrong note would mislead every later agent.
  • A size budget. The brief has a fixed size, for example 4,000 to 8,000 tokens. It gives the facts and the verdict first. Then it gives the tool calls that fetch more.
  • Product knowledge stays in the product. The MCP server instructions and get_tool_guide already explain the tools. Each alert type also gets an investigation playbook, and Darling keeps it current with each release. The starter file must not copy these.

4. How agents get requests

All three ways read the same request table.

  1. Pull (required). New MCP tools list_investigation_requests and claim_investigation_request. Pull works when the agent cannot accept inbound connections.
  2. MCP push (optional). Darling's MCP server sends a notification to subscribed agents when a request opens. The agent then claims the request as in (1).
  3. Webhook (optional). The generic webhook gets an {{investigation_json}} token, for a runner that accepts a POST.

These tools write only to the monitoring store, like mute_analysis_finding.

5. The report

  • The agent writes its report with a new MCP tool, submit_investigation_report.
  • The report has:
    • a summary.
    • the root cause, or "not found".
    • the evidence: tool calls and key numbers.
    • a verdict on the alert: real, false positive, or needs a person.
    • the actions it took or proposed (section 7).
  • The tool enforces the shape. submit_investigation_report refuses a report without a verdict, the evidence, and a root cause or "not found". So every agent produces a usable report, with or without an instructions file.
  • The viewer and MCP show the report next to the alert and the finding.
  • A false-positive verdict with evidence is a candidate for a detector issue, like PLAN_REGRESSION compares plans compiled for different inputs, and its advice to force the "best" plan makes the slow case slower #5630.
  • Feedback from a person. Erik: "a feedback mechanism is a good idea", "maybe not so broadly as right or wrong".
    • A person can add feedback to a report in the viewer.
    • The fields are open for design. Examples: the verdict was correct, the root cause was partly right, a step that the agent missed, and free text.
    • Feedback feeds the practice requests (section 11), the investigation notes (section 3) and the false-positive counts in threshold proposals (7.6).

6. Autonomy levels

The level is set per route, per server and per action type. The default is 0.

Level Name What the agent can do
0 Off Nothing. Darling makes no request
1 Investigate Read through MCP and report
2 Propose Level 1, and propose actions. The action tool stores the proposal, and a person approves it in the viewer
3 Act Level 2, and the agent runs an allowed action itself through an MCP action tool, with no approval step. The tool runs every check first

7. Actions

There are three kinds of action.

  • Outward actions (Jira tickets, PRs and bug reports) do not touch a monitored server. The agent runner does them with its own credentials. The agent puts the link in its report.
  • Server actions (plan force, eviction and kill) change a monitored server. The agent runs them through Darling's MCP action tools. Each tool checks the action again, runs it with the server's remediation credential, verifies the result and reverts it if needed. The agent never gets that credential.
  • Configuration actions (alert thresholds, mutes and new alert rules) change only the monitoring store. They use the MCP write tools that already exist, or new ones in the same class.

Each action call names the action, the target, the evidence, the reason and the agent. The action tools refuse a call that is missing a field. Darling journals every call and its result, with the reason. That includes refusals.

7.1 Open a Jira ticket

  • The route sets the Jira site, project, issue type and labels.
  • One ticket per dedup key or story. If an open ticket exists, the agent adds a comment to it.
  • The ticket holds the report, the evidence and a link to the alert in the viewer.

7.2 Open a pull request

  • The server or database config sets an allowlist of application repositories and the base branch.
  • The PR is always a draft. The agent never merges.
  • A typical PR adds an index, changes a query, or adds a statistics job. The PR body carries the evidence and any before and after measurements.
  • The agent must not put customer data in a PR. Query text gets the same masking as stored statement text.

7.3 Force or unforce a Query Store plan (SQL Server)

This carries over Phase 2 of #2138.

  • A target is eligible only when structured_remediation says so. ForcePlanBlockers stays the single policy gate.
  • New blockers from PLAN_REGRESSION compares plans compiled for different inputs, and its advice to force the "best" plan makes the slow case slower #5630:
    • The statement carries OPTION (RECOMPILE).
    • The best plan and the latest plan were compiled for different parameter values.
    • The parameter-sensitivity flag is set. This one is already a blocker.
  • Evict first (optional). Darling evicts the plan and watches the next compile. It forces only if the new plan is not better.
  • Self-review. After a force, Darling compares CPU per execution of the forced plan with the plan it replaced. The default windows are 15 minutes, 1 hour and 24 hours. If the force is not a net benefit, Darling unforces it and records it in the ledger.
  • Ownership. Darling unforces only its own forces. It never changes a force that an operator or APC placed.
  • Forcing failures. If force_failure_count rises above 0, Darling reverts the force and reports it.
  • Limits: one force per query per cooldown (default 24 hours), and a daily maximum per server.
  • Test: the PLAN_REGRESSION compares plans compiled for different inputs, and its advice to force the "best" plan makes the slow case slower #5630 reproduction becomes a test. Darling must refuse both of its "best" plans.

7.4 Evict a plan or recompile an object (SQL Server)

  • Targeted only: DBCC FREEPROCCACHE (plan_handle), or sp_recompile on one object.
  • Never a cache clear for the whole instance or database.
  • "Recompile" here means that the next execution compiles again. The agent does not add an OPTION (RECOMPILE) hint to code. A code change goes through a PR (7.2).
  • Self-review is the same as in 7.3. If the new plan is worse, Darling reports it. An eviction has no undo, so evictions have their own daily limit.

7.5 Kill a lead blocker

This is the action with the highest risk. It comes in the last phase, and it is off unless the route arms it.

Darling kills a session only when every condition holds. It checks them all again, on a live read, immediately before the kill.

Condition Default
The session is the head of a blocking chain. It blocks other sessions, and no session blocks it required
The blocking has lasted at least N minutes 5 minutes
It blocks at least M sessions, or the blocked sessions have waited at least W in total 3 sessions, or 10 minutes
The session is idle with an open transaction: sleeping with open_transaction_count > 0 on SQL Server, or idle in transaction on PostgreSQL required
Its last request ended at least I minutes ago 2 minutes
The rollback is small: the log bytes that its transaction used are below a limit 100 MB
It is a user session. It is not in KILLED/ROLLBACK, and it is not in a distributed transaction required
Its login, program and database are not on the protected list. The default list holds backups, DBCC, index maintenance, replication and CDC agents, SQL Agent jobs and the monitoring service required
It is the session that the alert saw: the same session id, and the same login_time (SQL Server) or backend_start (PostgreSQL) required
  • This design never kills a head blocker that is running. A long rollback can hold locks longer than the blocker would. A person decides those cases.
  • Before the kill, Darling records the session's login, host, program, open-transaction age and log bytes, and the hash of its last statement.
  • After the kill, Darling checks that the chain cleared. It then reports the kill with that evidence.
  • On PostgreSQL, the action is pg_terminate_backend, run by a role with pg_signal_backend. The report also suggests idle_in_transaction_session_timeout as the lasting fix.
  • Limits: at most K kills per server per hour (default 1). If the same login and program need a second kill within 24 hours, Darling asks a person.

7.6 Tune alert thresholds

What exists today (from a code read):

  • The built-in alert thresholds are one global row. update_alert_settings changes it, so a change applies to every server.
  • Custom alert rules can be scoped to a list of servers.
  • There is no per-server override for a built-in threshold.

What this needs:

  • Per-server and per-database threshold overrides for the built-in alerts. An agent can then tune one environment without changing the fleet.
  • At Level 3, an agent changes only an override for its own servers. A change to the global row is always Level 2: a person approves it.

A tuning proposal holds:

  • The alert, the scope, the current value and the proposed value.
  • The evidence: how often the alert fired over the last N days, and how many of those alerts investigation reports judged false positives.
  • A replay of the proposed value over the stored history (section 7.9).

Limits:

  • One change moves a threshold by at most a set factor per 24 hours. The default is 2×.
  • An agent cannot turn an alert off by tuning. That is a mute, and a mute has an expiry (7.7).
  • Some alerts have a floor that agents cannot loosen past. The defaults are the self-monitor family, Deadlocks Detected, Database State and Volume Free Space.

Review:

  • For 14 days after a change, Darling evaluates the old threshold in shadow. It records each alert that the old value would have raised.
  • At the review date, Darling reports the effect: alerts before and after, and every shadow alert.
  • If an investigation finds that a shadow alert was a real problem, Darling restores the old value and reports it.

7.7 Mute and unmute alerts

This builds on the mute rules that exist (create_mute_rule, update_mute_rule, set_mute_rule_enabled, delete_mute_rule and mute_analysis_finding). A muted alert is still logged. It is not delivered.

Rules for agent mutes:

  • The scope must be narrow. A server and a metric are required. Today a mute rule with no fields mutes every alert on the whole fleet, so Darling refuses that shape from an agent.
  • Every agent mute expires. The default is 24 hours. The maximum is 7 days at Level 3. A longer mute needs a person.
  • Every agent mute has a reason and a link to the investigation report.
  • A muted alert does not open an investigation request. Darling still counts its occurrences and shows them in the mute review.
  • Ownership: an agent can unmute or change only the mutes that agents made. It can propose a change to a mute that a person made, but only a person can make that change.
  • Agents cannot mute the self-monitor family, or any alert about the agent route itself.

7.8 File bug reports

When an investigation finds a problem in the monitoring tool itself, the agent files a bug report. Examples are a false positive from a detector, a collector error, wrong advice, or a slow MCP tool. #5630 is an example of such a report.

How it works:

  • A new MCP tool, submit_bug_report, stores a draft in the monitoring store. The draft holds:
    • The title and the component: collector, detector, alert or MCP tool.
    • The evidence: tool calls and numbers.
    • The repro steps, if there are any.
    • The linked investigation report.
  • Darling adds the build version, the schema version and the platform.
  • The route sets the destination: the product's GitHub repository, the operator's Jira, or both.
  • The agent runner files the report with its own credentials and stores the link on the draft.

Redaction (required):

  • Darling makes a redacted copy of the draft. Server and database names become labels. Statement text is dropped or masked. Host names, IP addresses and cloud resource ids are removed.
  • Only the redacted copy can be filed. A scan of the copy must pass before filing.
  • A public tracker keeps everything, including edit history. So for a public repository, a person approves each report (Level 2). Level 3 is allowed only for a private tracker.

Deduplication and limits:

  • Darling groups drafts with the same component and signature.
  • Before filing, the runner searches open issues. If a match exists, it adds a comment instead of a new issue.
  • A set number of reports per day per install. The default is 5.

7.9 Request new alerts

An agent can propose a new alert that fits its environment. Examples are "this database's log backups lag after the nightly load" or "this job fails at month end".

What exists today: custom alert rules, with list_custom_alert_templates, validate_custom_alert_rule, test_custom_alert_rule and create_custom_alert_rule.

What this needs:

  • A replay tool, backtest_custom_alert_rule. test_custom_alert_rule reads only the current value. A proposal must show how often the rule would have fired over the last N days, and on which servers. It must also show how often it would have fired together with an existing alert.
  • Shadow mode for new rules. A rule from an agent first runs in shadow for a trial period (default 7 days). Darling evaluates it and records each would-be alert, but delivers nothing.
  • Promotion: after the trial, a person promotes the rule (Level 2). At Level 3, Darling promotes it when its fire rate stays below a set limit and it does not duplicate an existing alert.
  • Scope: an agent rule names its servers. A fleet-wide rule always needs a person.
  • Ownership: an agent can change or turn off only the rules that agents made. It never deletes a rule that a person made.
  • If the rule language cannot express the alert, the agent files a feature request through 7.8.

8. What agents cannot change

At every level, an agent cannot:

  • Change an agent route, an autonomy level or the global switch.
  • Change a credential.
  • Change, turn off or delete a notification route.
  • Delete a mute rule, alert rule or plan force that a person made.
  • Approve its own proposal.

These rules stop an agent from raising its own permissions, and from removing the alerts that report on it.

9. Credentials

  • The monitoring credential stays read-only. This promise from [ON HOLD] [FEATURE] Auto force plan bot — detect bad Query Store plans (CPU-weighted), force with self-review, evict-first option #2138 does not change.
  • Server actions use the per-server remediation credential (V113). It is optional, with no default and no fallback.
  • Least privilege per action:
    • Plan force and unforce: ALTER on the database.
    • Plan eviction: ALTER SERVER STATE. sp_recompile needs ALTER on the object.
    • Kill on SQL Server: ALTER ANY CONNECTION.
    • Kill on PostgreSQL: membership in pg_signal_backend.
  • ALTER ANY CONNECTION is a strong permission. The viewer shows the permission that each action needs when an operator arms it.
  • Jira and git credentials stay in the agent runner, not in Darling.

10. Audit

  • Every agent action is logged for audit. Erik: "they need to log their actions for audit."
  • Darling writes the journal row itself, when an MCP action tool is called and again when it finishes. So the record does not depend on the agent.
  • Each row holds the agent id, the action, the target, the reason, the evidence or report link, the state before, the result, and any revert.
  • One journal holds every request, claim, report, action call, proposal, approval, refusal, execution, verification and revert.
  • The viewer shows the actor, the reason and the report link for every threshold, mute and alert rule that an agent made. Each one has a revert button.
  • Each row names the actor: the agent id, the person who approved, or the service.
  • The viewer shows the journal per server, and MCP can read it.

11. Practice requests (optional)

Darling cannot vouch for an agent that it does not ship. So an operator can test their own agent first.

  • Darling serves past alerts with known outcomes as practice requests. Examples are confirmed false positives and confirmed real problems from the investigation history.
  • The agent investigates them at Level 1. The action tools run nothing for a practice request.
  • Darling compares each verdict with the known outcome, and the viewer shows the score.
  • An operator can use the score to decide whether to raise the agent's autonomy. Darling does not require it.

12. Data that agents receive (a setting)

Erik: "i think it's up to end users to manage this in a way that meets their needs. it's not a policy i want to set or enforce. perhaps this is a setting we provide."

  • An agent can send what it reads to its model provider. That includes query text, plan XML, and login and host names.
  • Darling sets no policy on this. The operator decides.
  • Darling provides a setting that the MCP tools apply. Example choices: full detail (today's behaviour, and the default), statement literals masked, or metadata only.
  • Until agents have their own identity (section 13), the setting applies to a whole MCP listener.

13. Agent identity

Erik: "this may not always be the case, though it is today. oidc is here, just not integrated with anything."

  • Today, MCP has one bearer token per listener. So Darling cannot tell two agents apart. Any agent with the token can claim any request.
  • The web dashboard already has OIDC sign-in, with claim-to-role mapping (OIDC sign-in for the web dashboard: give the web surface a per-user identity instead of one shared token #2550). OIDC is the likely path for agent identity too. The agent would pick up its identity from environment variables.
  • With an identity, Darling can bind each agent to its routes, its autonomy levels, its request limits and the data setting.
  • This is later work. Phase A works with one token per listener.

14. Untrusted text in monitored data (docs)

Erik: "this is something that users need to manage, not for me to set policy on. locally we can do these things, sure. we can set a good example in the docs, sure. but people are gonna people."

  • Anyone who can run a query on a monitored server controls some of the text that reaches an agent. Examples are query text, program names and error messages. That text can try to steer the agent.
  • Server actions are covered. Darling checks every one again before it acts (section 7).
  • Outward actions run with the runner's credentials, so Darling cannot check them. The operator manages that risk.
  • The docs describe the risk and show an example setup. For example: outward actions at Level 2, and a Jira or git credential that can do only what the route needs.
  • Darling sets no policy on this.

15. Format numbers for the brief and the report

People write their own agents around the shape of the brief and the report. A later release can change that shape. Then an agent that was written for the old shape can fail with no clear sign.

  • Every brief carries a format number. Every report and every proposal states the format number that the agent uses.
  • When a release changes the shape, the format number goes up.
  • Darling accepts the previous format for at least one more release.
  • After that, Darling refuses a report or proposal in a format that it no longer supports. The error names the format that the agent sent and the formats that Darling accepts.
  • The release notes list each format change.

What this changes in earlier decisions

Phases

Phase Scope Writes to a monitored server
A Agent routes, requests grouped by server, family and window, request limits, the data setting, format numbers, the brief with its fresh-agent context, pull and claim, and reports. Level 1 only. A fake agent that runs only in CI, to test the contract. No agent ships none
B Feedback on reports. Outward actions: Jira tickets, draft PRs and bug reports. Configuration actions at Level 2: threshold overrides, mutes and new alert rules, with the replay tool and shadow mode. Practice requests none
C Server actions at Level 2, through MCP action tools: plan force and unforce, and eviction, with self-review and revert. This change implements IPlanForceExecutor and opens PlanForceNoWritePathTests yes, with approval
D Level 3 for plan force, eviction and configuration actions yes, within policy
E Kill of an idle lead blocker: Level 2 first, then Level 3 yes

Acceptance tests

  • A request survives when no agent is connected, and it expires as configured.
  • Two alerts with one dedup key make one request.
  • At Level 0 and Level 1, no write reaches a monitored server. PlanForceNoWritePathTests stays closed until Phase C.
  • The global switch stops an action that a person approved but Darling did not execute yet.
  • An action tool call without a reason or evidence is refused, and the refusal is journaled.
  • Every action tool call writes a journal row before it runs and another when it finishes, with the agent id, the reason and the result.
  • No MCP tool other than the action tools writes to a monitored server.
  • Both PLAN_REGRESSION compares plans compiled for different inputs, and its advice to force the "best" plan makes the slow case slower #5630 cases are refused as force targets.
  • A force that is worse at its first review window is unforced, and the ledger records it.
  • A kill proposal is refused when the session's login_time changed, when the session became active, or when its log bytes passed the limit.
  • A kill of a session on the protected list is refused.
  • An agent mute without a server and a metric is refused. An agent mute without an expiry gets the default. An agent mute past the maximum is refused.
  • An agent cannot change or delete a mute, alert rule or force that a person made. It cannot change an agent route, an autonomy level or the global switch.
  • A threshold change past the step factor is refused. An agent change to the global row at Level 3 is refused.
  • After a threshold change, the shadow evaluation of the old value records the alerts that the old value would have raised.
  • A bug report whose redacted copy fails the scan is not filed.
  • A new agent alert rule delivers nothing until it is promoted.
  • A report without a verdict or evidence is refused.
  • A brief for an alert with earlier reports attaches them, and it stays inside its size budget.
  • An investigation note that a person has not approved does not reach a brief.
  • The CI fake agent completes a request, claim and report cycle with no instructions file.
  • Related alerts on one server, in one family and one window, make one request. Alerts in two families make two requests.
  • A route at its hourly limit makes no new request, and the alert still goes to its other channels.
  • With literals masked, the MCP tools return no statement literal.
  • Feedback on a report reaches the practice-request set.
  • A report in the previous format is accepted. A report in an unsupported format is refused, and the error names the format that was sent and the formats that are accepted.

Open questions for Erik

  1. The default thresholds in 7.3 and 7.5.
  2. Does Darling or the agent runner hold the Jira and git credentials? This draft puts them in the runner.
  3. Where does a person approve a Level 2 action: only in the viewer, or also with a Slack or Teams button?
  4. Can 7.5 ever kill a head blocker that is running? An example is a query past a runtime limit from an allowlisted program.
  5. Resolved: there is no reference runner. See "Bring your own agent".
  6. Where do bug reports go: this repository, a private tracker, or both? For this repository, does a person always approve?
  7. Do we build per-server threshold overrides for 7.6, or do agents tune only through custom rules?
  8. The maximum mute length at Level 3, and the threshold floors in 7.6.
  9. The investigation families, and the grouping window (section 2).
  10. The choices for the data setting (section 12).

Edited 2026-10-16 (Phase B shipped). Phase B landed on dev as #5926 (#5913), #5933 (#5916), #5934 (#5915), #5936 (#5917) and #5937 (#5918), after Erik's rulings of 2026-10-16 (plan v4, walked through and signed off). Where this body and the rulings differ, the rulings win:

  • No approval step in the product. The Level table above says a Level 2 action "stores the proposal, and a person approves it in the viewer". That is superseded: when writes are on, the agent calls the write tools directly, every change is journaled with its reason and report link, and a person can undo it from the Agent changes list (web and Viewer). Phase A's agent notes keep their approval step, because an approved note feeds every later brief.
  • The caps in this body were agent-drafted, not Erik's. The 2x per day step, the floors, the 7-day agent-mute maximum and the ownership rules were added when the issue was drafted with an agent's help and were never part of the feature as Erik meant it. Only three rules hold: (a) an agent turns an alert off only by a mute, never by disabling it (exclusion-list adds are blocked too; disabling or deleting a custom rule is allowed), (b) no muting the self-monitor family or the agent route's own alerts, (c) a reason and a report link on every agent change, journaled. Nothing else is capped.
  • Credentials for Jira, git and the like belong to the organization, in the agent's runner. Darling records and dedups links only.
  • Bug reports are not held for approval in the product; the customer's harness decides.
  • Lite stays advise-only for actions.
  • Per-server threshold overrides: agreed and built.

Readers of the sections above should treat their caps and approval wording as history, not as the spec.


Edited 2026-10-18 (Erik's rulings on the server actions). Erik answered the server-action questions on 2026-10-18. Where this body and these rulings differ, the rulings win. The read-only force pre-check get_plan_force_precheck is on dev (#6065).

  • Remediation credential. Server actions use the per-server remediation login (V113). The store holds only the login name and an env: or file: reference to its password. The remediation slot does not accept an encrypted password.
  • Level 3 lets an agent change a monitored server. Level 2 changes only the monitoring store. A server action needs four things: the remediation login, that server's switch for that action, a Level 3 route, and agent_actions_enabled. mcp.allowWrites does not bypass them.
  • The write code is in its own assembly, PerformanceMonitor.Darling.Remediation.
  • No product limits on agent forces, unforces or kills. There is no cooldown, no daily cap and no regression floor. Each team puts its own rules in its agent instructions. Darling/AGENTS.example.md gets suggestions, not rules. This replaces the limits in section 7.3.
  • The force pre-check gives warnings, not refusals. The force tool blocks only a call that cannot do anything: the plan is not in that query's Query Store rows, or that plan is already forced. The execution count, the time since the last run and the row counts are shown as facts, with no threshold.
  • An agent may replace or unforce any force, including a force that a person or automatic plan correction placed. The journal records the original plan and who placed it. So the change is auditable, and the undo can force that plan again. The unattended plan-force bot keeps its own rules and changes only its own forces.
  • The review after a force. Darling checks at the first two Query Store collector passes after the force (about 10 minutes), then every hour, and stops at 24 hours. At each check, Darling opens a follow-up request with the evidence, and the agent decides. Darling does not unforce on its own. The brief asks the agent to get an adversarial second opinion from a sub-agent or a peer agent before it acts.
  • Forced-plan state. A new collector read takes each database's forced plans from sys.query_store_plan, so a force that never runs is visible. Each force is reported as working, not exercised, ignored, failing or removed.
  • Kill is an agent tool, with no product limits. In the same batch as the KILL, the tool checks one thing: the session still has the same session id and login_time. The blocking capture adds the head blocker's status, login_time, last_request_end_time and transaction log bytes.
  • Kill on PostgreSQL is NOT built. The first kill PR covers SQL Server only. PostgreSQL kill stays open, and the kill PR must say so in its tool description, its CHANGELOG entry and a follow-up issue.
  • Grouping. A new alert joins a request when it fires within 30 minutes of the request's latest alert, up to 4 hours after its first alert. Nothing else about grouping changes.
  • Data setting. The three mcp.dataDetail modes stay. The brief says that maskLiterals hides compiled parameter values. Pseudonyms that stay the same across answers are metadataOnly: give a login or host the same pseudonym in every answer #6066.
  • Object recompile (sp_recompile) is not built. Evicting one plan handle covers the cases seen so far.

Readers of section 7.3 should treat its limits and ownership rules as history, not as the spec.

Activity

  1. erikdarlingdata commented on Oct 9, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    Erik added four design requirements for the agent route. The issue body now carries them as sections 7.6 to 7.9.

    1. agents are going to need to be able to tune alert thresholds
    2. agents should be able to mute/unmute alerts
    3. agents should be able to file bug reports about issues
    4. agents should be able to request new alerts that fit situations in their environment

    What changed in the body:

    • Section 7 now has a third kind of action: configuration actions. They change only the monitoring store.
    • 7.6 Tune alert thresholds. It needs per-server threshold overrides, because the built-in thresholds are one global row today. It has a step limit, floors, and a shadow evaluation of the old value for 14 days.
    • 7.7 Mute and unmute. Agent mutes must name a server and a metric, and they always expire. Agents change only the mutes that agents made.
    • 7.8 File bug reports. Only a redacted copy is filed, and a scan must pass first. A person approves each report for a public repository.
    • 7.9 Request new alerts. It needs a new replay tool, backtest_custom_alert_rule. New agent rules run in shadow before they deliver.
    • Section 8 (new) lists what agents can never change, at any level. Examples are agent routes, autonomy levels, the global switch, credentials, notification routes, and anything a person made.
    • Phases B and D, the acceptance tests and the open questions (6 to 8) now include these items.
  2. erikdarlingdata commented on Oct 9, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    The body now says that Darling ships no agent. Operators bring their own. Erik: "i'm fine providing a starter file, but packaging in an agent isn't something i think i can do presently. perhaps in the future that will become a thing."

    What changed in the body:

    • New section "Bring your own agent".

      • Darling ships the contract, not an agent.
      • It assumes the weakest agent that can connect, and it trusts no agent.
      • A starter AGENTS.md is an optional example in the docs. Nothing depends on it.
      • A packaged agent stays possible later, as one more client of the same contract.
    • Section 3, "Context for an agent that starts fresh". The brief brings:

      • The earlier reports for the same alert.
      • Recent changes on the server.
      • An environment profile that the operator writes.
      • Investigation notes that a person approved.

      It has a size budget, and it points to tool calls for more.

    • Sections 5 and 7. The report and proposal tools refuse input with a missing field.

    • New section 11, practice requests (optional). Operators can score their own agent on past alerts with known outcomes.

    • Phase A has no reference runner now. A fake agent runs only in CI. Open question 5 is resolved.

    • New acceptance tests for the items above.

  3. erikdarlingdata commented on Oct 9, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    Erik ruled on six more design points. The body now carries them.

    Point Erik's ruling Where in the body
    One agent per incident Yes, and "one agent per alert family. blocking/deadlocking, query performance, etc." Section 2: requests grouped by server, family and window, with a draft family table
    Data that agents receive "it's up to end users to manage this … perhaps this is a setting we provide" Section 12: a setting, with full detail as the default. No policy
    Untrusted text in monitored data "something that users need to manage … we can set a good example in the docs" Section 14: docs and an example setup. No policy
    Agent identity One token today. "oidc is here, just not integrated with anything" Section 13: OIDC is the likely path, later work
    Feedback on reports "a feedback mechanism is a good idea", "maybe not so broadly as right or wrong" Section 5: feedback from a person, with the fields open for design
    Cost "sure, that makes sense" Section 2: request limits per route, per hour and per day

    The phases, the acceptance tests and open questions 9 and 10 now include these points.

  4. erikdarlingdata commented on Oct 9, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    New section 15: format numbers for the brief and the report. Erik agreed: "oh that's fine, yes".

    • The problem: a release that changes the shape of the brief or the report can break agents that people wrote for the old shape.
    • The fix: each brief, report and proposal carries a format number. Darling accepts the previous format for at least one more release. After that, it refuses with an error that names the formats.
    • Also updated: Phase A and the acceptance tests include this.
  5. 39 remaining items

  6. erikdarlingdata commented on Oct 16, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    Two agent sessions investigate Darling alerts by hand today. Erik asked them what they would need from this issue's design. Most of their answers are already in this issue. The items below are not, so they are proposed as additions. A name search of dev found none of them, but that is not a full code check.

    Proposed additions

    1. Signed webhooks.
      • The generic webhook signs its body with HMAC-SHA256, and it sends a timestamp header.
      • The receiver can then reject a forged or a replayed call.
      • This matters because a request can lead to a server action.
    2. More request events than "opened".
    3. Evidence frozen at alert time. The DMVs change before an agent looks. So the brief keeps a snapshot from the alert time:
      • High CPU:
        • the top statements by CPU in the breach window, with their plans
        • active requests grouped by host and program, with the highest concurrency seen
        • a plan-change flag for each top statement
        • the Query Store state of each database, with the reason when it is read-only
        • the SQL Agent jobs that ran in the window
        • the top waits and any blocking chain
      • Deadlocks: the parsed graph, with object and index names resolved at capture time. Temporary objects are gone before an agent can look them up.
      • Blocking: the head blocker's statement and its offset. Today the alert often shows only the procedure header.
      • Plan regressions:
        • each target's last execution time, and whether it still runs in the window. A target that stopped running after a code change is not worth an investigation.
        • the compiled parameter values of the good plan and the bad plan
      • When a collection has not landed yet, the brief says that the evidence is not complete. Darling sends a second event when it lands.
    4. Recurrence. The brief says on how many of the last 7 days the same server and family fired within 30 minutes of this time of day, with the dates. Most repeat alerts are daily jobs that an agent has already ruled on.
    5. Deadlock signatures, and a digest.
      • Darling builds a stable signature key for each deadlock from its frames and its lock resources. Today the dedup key is per graph, so two deadlocks of one pattern get two keys.
      • The request carries the signature's counts for this window, the last 24 hours and the last 7 days.
      • A request opens only for a new signature, or for a known one whose 24-hour count is more than 3 times its daily average over 7 days.
      • All other deadlocks go into one digest every 4 hours, with counts by signature.
      • An agent can merge two signatures into one family through MCP, because line numbers move when a procedure changes.
    6. Hints between families.
      • A High CPU request carries a hint for the query performance family when its top statement has a new plan, and that plan uses at least 5 times the CPU per execution of the best plan in 7 days. A new plan alone is not enough, because nightly index maintenance makes new plans.
      • A plan-regression request whose top target kept its plan and ran more often carries the opposite hint.
    7. Re-wakes from the report.
      • A report can set a follow-up time. Darling wakes the agent again at that time, for example to check that a forced plan held through the next busy window.
      • A report can also set a condition, for example "after the next deploy that changes this module". Darling then re-wakes the agent with the counts before and after.
      • That needs a deploy marker per server and database. Darling could detect one from the module's modify date, or from query id turnover.
    8. Report fields.
      • a family id that links all reports for one pattern
      • a fix record: the ticket, the PR, the merge commit and the release
      • separate lists of verified facts and inferred facts
      • the queries that the agent ran on the monitored server during the investigation
    9. Both server names in the request: the display name and the host name. Alert rows use one or the other, and agents lose time matching them.

    Not adopted, because rulings in this issue decide them

    • Forcing plans with the agent's own credential, through a stored procedure. Agents act through Darling's MCP action tools, as section 7.3 describes. Erik, 2026-10-14: "yes, via mcp calls."
    • Known patterns that an agent writes with no review. Investigation notes keep their approval step, as the 2026-10-16 rulings say. An approved note can carry a disposition (watch, fix in flight, declined by the owner) and an expiry. A deadlock note can match a signature key from item 5.

    Outside Darling

    • Database logins, and the credentials for the ticket system, git, feature flags and APM, stay with the organization, in the agent's runner. This follows the 2026-10-16 rulings.
    • The agents' own instruction files stay with the operator. The optional starter file in "Bring your own agent" does not change.

    Covered by another issue

  7. erikdarlingdata commented on Oct 16, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    Phase C plan (draft for sign-off)

    The nine additions from the 18:17Z comment, as a plan: a code check of dev at 3e68e0b42, the decisions, then the slices.

    1. What dev already has

    # Addition State Where, and what is missing
    1 Signed webhooks Partly Static operator headers (a bearer token), sent only to the channel's own origin (WebhookAlertService.cs:2179, :3384-3478), sealed at rest (DarlingWebhookSecrets.cs:47). Missing: HMAC, timestamp and id headers, rotation.
    2 Events past "opened" Partly The webhook token says opened or joined, only on the alert's own page (InvestigationWebhookToken.cs:42). The sweep journals expired and claims_exhausted (InvestigationRequestStore.cs:331-345) but delivers nothing. The long-poll wakes on Opened only. No severity column (InvestigationTables.cs:75-111), no event id. #5825 shipped resolution rows (DarlingAlertDeliverer.cs:437), so "resolved" has its input.
    3 Evidence frozen at alert time Partly Link rows keep the alert's values and context_json (V188). Blocking keeps chains and the report XML (AlertContextBuilders.cs:58); deadlock graphs already name objects and indexes (DeadlockGraphParser.cs:92, :284). High CPU holds only the reading, threshold, maintenance sessions and grade (AlertEngine.cs:1206-1240). The rest is read at claim. The store has most sources (readonly_reason, DarlingConfigHistoryReader.cs:264; compiled values, PlanInputComparison.cs); nothing freezes them. No "incomplete" marker.
    4 Recurrence Absent The brief reads earlier reports (PgInvestigationBriefSourceReader.cs:87-97), not firings by time of day. The index exists (PgMigrations.cs:4104).
    5 Deadlock signatures, digest Partly The key is not per graph: it hashes server, type and the object set (AlertFingerprint.cs:45-79, IncidentGrouping.cs:475-500). Missing: frames in the key, 24-hour and 7-day counts, the gate, the digest, merging.
    6 Hints between families Partly next_calls adds every finding's tools, any family (InvestigationBriefComposer.cs:259-292). No plan-CPU rule, no hint field.
    7 Re-wakes Absent Nearest pattern: practice inserts a request naming its source (InvestigationPracticeStore.cs:174). No deploy marker.
    8 Report fields Partly Links jira, pull_request, bug_report (InvestigationReportLinks.cs:47); evidence is one array. Missing: family id, commit and release, verified versus inferred, queries run.
    9 Both server names Partly server_name is the display name (InvestigationToolCore.cs:466-472, DarlingWorker.cs:14571). No host name; the registry has both.

    2. Decisions

    D1. Signature format.

    • A. Standard Webhooks: webhook-id, webhook-timestamp, webhook-signature (HMAC-SHA256 over id.timestamp.body). Receivers have libraries in most languages.
    • B. Our own X-Darling-Signature header. Every receiver writes its own check.
    • Recommend A. The id header doubles as D6's event id.

    D2. Where the key lives.

    • A. One sealed key per generic channel and per route with its own URL, in the existing sealed-value path. One more sealed field.
    • B. One service-wide key in darling.json. A leak signs for every receiver.
    • Recommend A. Darling generates it and shows it once.

    D3. Rotation and replay.

    • A. Two keys live for a 7-day overlap, both signatures sent. Receivers keep working while they update.
    • B. One key; rotation breaks receivers until they update.
    • Replay: a 5-minute timestamp window and the id as dedupe key, enforced by the receiver (docs and the starter sample).
    • Recommend A, with the 5-minute window.

    D4. Which sends are signed.

    • A. Generic webhook only, on once a key exists. Teams, Slack and PagerDuty have their own auth.
    • B. Required on every generic send. Forces a key on every operator.
    • Recommend A.

    D5. How events reach a runner.

    • A. The journal is the event log; a signed POST to the agent route and the MCP long-poll both read it.
    • B. MCP only. No push for a webhook runner.
    • Recommend A. The journal's once_key writes each event once.

    D6. Event set and ids.

    • opened; joined only for a new metric or a higher severity (every join is noise); resolved; expired and claims_exhausted; evidence_ready (D9); follow_up (D16); deadlock_digest (D13).
    • Ids: A. a uuid per event row, sent as webhook-id. B. the once_key text. Not unique across products.
    • Recommend this set and A, with a max_severity column on the request.

    D7. When a request is "resolved".

    • A. Every metric linked to it has a resolution row after its last alert. Exact.
    • B. The first metric resolved. Fires early on a request that grew.
    • Recommend A. It reads resolution rows, not the route's SendResolutions flag.

    D8. Fallback for an expired request.

    • A. A fallback_route_id on the agent route; else the same route's human channels; else a self-monitor alert.
    • B. Always a self-monitor alert. One more page type for every operator.
    • Recommend A.

    D9. When evidence is frozen, and from what.

    • A. In the send path. Delays every agent-routed page.
    • B. Right after the open, in the worker, from the store's collected rows only. No new reads on the monitored server.
    • C. At claim (today). Rows may have aged out of the window.
    • Recommend B. A section not yet collected is pending; when it lands, evidence_ready fires. A section never collected says not_collected.

    D10. Where evidence is stored, and how much.

    • A. A child table, one row per section (CPU top statements, active requests by host and program, Query Store state, jobs, waits and blocking, deadlock graph, head blocker, regression targets), 32 KB each like report evidence. Plans by hash, not XML.
    • B. One jsonb column on the request. A late section rewrites the whole value.
    • Recommend A. Cleared with the brief at 90 days; masked at mcp.dataDetail when served.

    D11. Recurrence window and cost.

    • A. The request's linked metrics, 7 days, 30 minutes either side of the first alert's time of day; days and dates; muted rows apart. One index seek per metric.
    • B. Every metric in the family. Needs the family map in SQL, and more rows.
    • Recommend A. Read at compose time: past alert rows do not change.

    D12. Deadlock signature basis.

    • A. Today's object set. Two patterns on one table share a key.
    • B. Object set plus procedure names plus lock modes, no line numbers or offsets.
    • C. B plus statement offsets. Splits on every edit.
    • Recommend B. D14 merges what B still splits.

    D13. Gate and digest. A request opens for a new signature, or a 24-hour count over 3 times the 7-day daily average; the rest go to a deadlock_digest event every 4 hours.

    • A. The gate applies to agent requests only; human pages are unchanged.
    • B. The gate also holds the human page. A paging change hidden in an agent feature.
    • Counts: a daily rollup (server, signature, day, count) written by the alert pass, not 7 days of graph XML reread per alert.
    • Recommend A, with the rollup.

    D14. Merging two signatures.

    • A. An MCP action through the Phase B agent_action gate: journaled, with a reason, undoable. Darling only.
    • B. Operator only, in the viewer.
    • Recommend A.

    D15. Cross-family hints.

    • A. A hint field and a next_calls entry.
    • B. Open a second request in the other family. Doubles agent work.
    • Recommend A, with the proposed 5-times rule and its reverse.

    D16. Re-wakes.

    • Time: a report sets follow_up_at (30 days out at most, 20 pending per route at most). Darling opens a new request with follow_up_of, as practice does.
    • Deploy condition: A. module modify_date, added to the procedure-stats collector's read. Automatic, one join per collection. B. Query id turnover. No new read, noisy. C. A marker the operator's CI posts. Exact, needs CI work.
    • Recommend time, and A, with counts before and after from procedure stats.

    D17. Report fields.

    • Family id: A. family_of, the first report's request id, checked to exist. B. Free text. Drifts.
    • Fix record: commit and release link kinds. No migration.
    • Verified and inferred: A. a required basis on each evidence item, shown as two lists. B. Two arrays with a second set of checks.
    • Queries run: a queries_run array (20 items, 2,000 characters each), masked as statement text.
    • Recommend A, the two kinds, A, and the array.

    D18. Server names.

    • A. display_name and host_name on the request, the brief and the webhook token, read from the registry when served.
    • B. Stored at open. Goes stale on a rename.
    • Recommend A. server_name keeps its meaning.

    D19. Lite.

    • A. Lite gets 1 to 4, 6, 8, 9 and time follow-ups, in a stacked PR after Darling's (10-13 both-products ruling). Not the D14 action.
    • B. Darling only.
    • Recommend A.

    3. Slices

    At most four parts each; one Darling migration, in C1; Lite's schema step in L1. Signing and frozen evidence come first. Tests are behaviour tests, one per behaviour; no census, roster or source-text test (#5833, .github/REVIEW-TRAPS.md).

    Wave 1

    • C1 Phase C store. The one migration: journal event_id; request max_severity, evidence_state, follow_up_of, follow_up_at, follow_up_condition; investigation_evidence; signature days and merges tables; report family_of, queries_run; route signing_key (sealed) and fallback_route_id; grants; retention. Tests: evidence clears with the brief at 90 days; signature days purge at 30; an open request keeps its evidence.
    • C2 Signed webhooks (needs C1). Sender headers, key generation and rotation, key screens in the viewer and web, receiver docs with a sample check. Tests: a receiver recomputing the HMAC accepts a send; one changed byte fails; both keys verify during rotation; the sample rejects a 6-minute-old timestamp; Teams and Slack sends carry no signing headers; a route URL is signed with the route's key.
    • C3 Server names (no dependency). Tests: a renamed display name shows on the next read; the host name matches the registry.
    • C4 Recurrence (no dependency). Tests: a 02:10 alert counts a 01:45 fire and not a 02:41 fire; muted fires count apart; no history says zero.

    Wave 2

    • C5 Evidence capture and High CPU (needs C1). Capture step, section writer, CPU sections, brief section. Tests: the top statements in the breach window are frozen; a later collection does not change them; a missing section is pending, then evidence_ready fires once.
    • C6 Deadlock, blocking and regression evidence (needs C5). Tests: a temporary-object deadlock keeps its names after the graph row ages out; the head blocker's statement comes from its offset, not the procedure header; a target that stopped running is marked; good and bad compiled values both appear.
    • C7 Request events (needs C1, C2). Event writer, resolved detector, fallback delivery, MCP cursor and wake. Tests: a same-severity join sends nothing; a higher one sends joined; resolved waits for every linked metric; an unclaimed request reaches the fallback route; a replayed event id is not delivered twice.
    • C8 Report fields (needs C1). Tests: family_of naming no request is refused; a commit link is stored; an evidence item without basis is refused; queries_run is masked at MetadataOnly.

    Wave 3

    • C9 Deadlock signatures and digest (needs C1, C7). Signature, rollup, gate, digest. Tests: a procedure edit keeps the signature; a new signature opens a request; a 3-times spike opens one; the rest arrive in one digest per 4 hours; human pages are unchanged.
    • C10 Signature merge action (needs C9, the Phase B gate). Tests: merged signatures share counts; undo splits them; a Level 1 route is refused.
    • C11 Cross-family hints (needs C5). Tests: a 5-times plan adds the hint; a 2-times new plan does not; the reverse hint appears on a regression request.
    • C12 Time follow-ups (needs C1, C7, C8). Tests: a follow-up opens on time with the earlier report in its brief; the 21st pending one is refused; one past 30 days is refused.
    • C13 Deploy-condition follow-ups (needs C12). Collector join, marker detection, before and after counts. Tests: a module change re-wakes once; an unchanged module does not.

    Lite (stacked PR, after Darling's)

    • L1 Lite schema step, events and evidence over DuckDB (needs C5 to C7).
    • L2 Lite recurrence, names and report fields (needs C3, C4, C8).
    • L3 Lite time follow-ups and signing-key storage (needs C2, C12).

    The two items the 18:17Z comment did not adopt stay out, for the reasons given there.

    Plan to be dialled in with Erik before sub-issues and lanes.

  8. erikdarlingdata commented on Oct 16, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    Erik agreed the Phase C plan as written on 2026-10-16; wave 1 lanes start now.

    Phase C is filed as 16 sub-issues, in three Darling waves and a Lite stack. Each is small enough for one lane (at most four parts). There is one Darling migration for the phase, in C1 (V193 unless dev has moved; PR #5996 holds V192).

    Wave 1 (no dependencies except C2 on C1)

    Wave 2

    Wave 3

    Lite (stacked PR after Darling's)

    Things the lanes must check first:

    Gaps found while re-checking the plan against dev, each handled in the slice named:

    • C1 also adds collect.procedure_stats.module_modify_date, because C13's deploy condition has no stored value to compare today.
    • C1 puts the signing key and previous-key columns on the notification settings row as well as on routes (D2 says per generic channel and per route).
    • D19 gives Lite hints, but no L slice names them; C11 builds the rule in the shared composer so Lite gets it with L1's evidence.
  9. erikdarlingdata commented on Oct 18, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    A request from two agent sessions that work real alerts: link a recurring alert to a known pattern. This is a note for later, not part of the current build order.

    • Recurrence: how often this alert fired at this time of day (#5632 Phase C C4) #6000 already tells an investigation brief how often an alert fired at this time of day.
    • The agents also want "this is the Nth day in a row at this time", linked to an entry in a known-patterns file. A known nightly job is one example.
    • They also want it outside investigation briefs, for example in get_alert_history.
    • The known-patterns file would sit beside the agent instructions this issue describes (Darling/AGENTS.example.md).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions