Skip to content

[FEATURE] Generic webhook: a raw-JSON {{context_json}} token so automation consumers get the structured alert context instead of a flattened line #2302

Closed

Description

@gotqn

Which component(s) does this affect?

  • Lite
  • Darling
  • Documentation

Problem Statement

I consume alerts in an Azure Logic App that fans out to six Teams channels by group and creates/dedups Azure DevOps work items per [#1140] incident fingerprint, setting each ticket's occurrence count from [#2216]'s Total Occurrences.

Everything automation needs already exists structured inside the product. AlertContext.Incidents carries DedupKey, TotalOccurrences, IncidentStartedUtc, OccurrenceCount, InvolvedObjects and DetailFields, and metricName / serverName / currentValue / thresholdValue / severity / timestamp are passed as discrete arguments into every Build*Payload.

Each channel then flattens that structure, and each one loses a different half:

Channel Scalars Incident structure
Teams / Slack only inside the human-readable summary survives, but as name/value strings spread across display sections
Generic discrete — exactly right RenderContextForTemplate joins the whole context into one `"
PagerDuty discrete keyed dictionary, but only incidents[0].DedupKey survives, as DerivePagerDutyDedupKey's own comment notes

So an automation consumer has to choose between discrete scalars and incident structure - no channel offers both. I chose Teams and pay for it: of roughly 49 actions in my workflow, about 31 exist only to undo the flattening:

Two of those are guesses about presentation that fail silently rather than loudly:

  • the server is derived as last(split(summary, ' on ')), which yields the wrong server for any server or database whose name contains " on " - and that wrong value then poisons ticket dedup
  • incident sections are detected with contains(string(section.facts), 'Dedup Key')

Parsing {{context}} back into structure is not a viable workaround: the delimiters are " | " and ": ", and Victim SQL routinely contains ": " and can contain " | ".

Proposed Solution

Add one raw-JSON token to the generic channel:

{{context_json}} - the AlertContext the alert already carries, serialized (incidents included); {} when there is no context.

That single token is the minimal viable version; everything else I need is derivable from it. The change is additive and small:

  1. Extend the s_genericPlaceholders alternation and the values dictionary in BuildGenericPayload.
  2. The one subtlety: a raw-JSON token must bypass EscapeForJson. The six existing tokens are string values inside quotes; this one is a JSON value substituted unquoted. A small explicit "raw" set keeps the single-pass MatchEvaluator exactly as it is.
  3. DefaultGenericBodyTemplate stays byte-identical, so no existing user sees a change - and since unknown {{foo}} tokens are already left literal (per the existing comment), older templates keep working untouched.

Optional, only if they are cheap:

  • {{incidents_json}} - just the incidents array, for consumers that want nothing else.
  • {{dedup_key}} - the same value DerivePagerDutyDedupKey already computes, including its stable metric+server fallback when an alert has no incident. That fallback key exists in the code today but is never exposed to any consumer, which is why level/threshold alerts (High CPU, TempDB Space, Collection Stopped, Store Job Over Cadence) force me into a title-matching heuristic for dedup.

Deliberately not requested: an alert "group" or "category" field. Mapping a metric to a routing group is my policy, not the product's.

Use Case

My generic body template becomes, with context unquoted:

{
  "metric": "{{metric}}",
  "server": "{{server}}",
  "severity": "{{severity}}",
  "value": "{{value}}",
  "threshold": "{{threshold}}",
  "timestamp": "{{timestamp}}",
  "context": {{context_json}}
}

Worked example - a Deadlocks Detected alert carrying two distinct fingerprints, which is exactly what I test with.

Today: I flatten facts across all sections into one array, then split tickets by filtering sections whose serialized facts contain the literal 'Dedup Key', special-cased to deadlocks — because deadlock incidents are self-contained (AlertIncidentRenderer.BuildItem with includeDetailFields: true) while other fingerprinted alerts append a separate bare Incident item beside their data item. Then, per ticket, I run a Filter + Compose pair per fact to recover Dedup Key, Database, Occurrences, Total Occurrences, Incident Since and the timestamp.

With {{context_json}}: I iterate context.incidents[] directly, read dedup_key / total_occurrences / incident_started_utc as properties, and the deadlock special case disappears - per-incident tickets then work uniformly for blocking, PVS, volume free space and long-running query too.

That removes about 28 of my 49 actions, including a nested per-fact loop costing one action execution per fact (~22 executions for a two-incident deadlock alert before any real work happens - billed per execution on Consumption). It also removes both silent-failure guesses above, since the server arrives in the payload and incidents are an array rather than something I sniff for.

Alternatives Considered

  1. Keep scraping the Teams payload (status quo). Works, but ~60% of the workflow is presentation-parsing, and it re-breaks whenever alert wording changes. A new alert type silently routes to my catch-all group instead of failing visibly.
  2. Parse {{context}} from the generic channel. Not safe - delimiter collision with Victim SQL, as above.
  3. Enable Teams and Generic together and correlate them. Two HTTP posts landing as two independent workflow runs with no shared correlation id. Strictly worse.
  4. A whole new "structured JSON" channel. More work for you and another UI section, when the generic channel already has an operator-editable Body (JSON) box and a token engine. A token is the smaller change, which is why I am asking for it this way.
  5. Read the store or MCP instead of the webhook. Inverts push into poll and loses the [[BUG] Email/webhook cooldown is keyed on (server, metric), not the #1140 fingerprint - distinct concurrent deadlock/blocking incidents are silently dropped (tray shows, Teams doesn't) #1154] per-fingerprint delivery cooldown semantics my ticket updates are built on.

Additional Context

No response

Activity

  1. erikdarlingdata commented on Aug 17, 2026

    @erikdarlingdata
    Owner

    Taking this. Shape as proposed: {{context_json}} on the generic channel via an explicit raw-token set that bypasses EscapeForJson, DefaultGenericBodyTemplate byte-identical, plus the two cheap optionals — {{incidents_json}} and {{dedup_key}} including the metric+server fallback DerivePagerDutyDedupKey already computes but never exposes. PR to follow.

  2. erikdarlingdata commented on Aug 17, 2026

    @erikdarlingdata
    Owner

    Shipped in #2310 (2a50ce9): {{context_json}}, {{incidents_json}} and {{dedup_key}} on the generic channel, exactly as proposed — raw tokens via an explicit set that bypasses EscapeForJson, single-pass MatchEvaluator untouched, DefaultGenericBodyTemplate byte-identical (pinned), unknown tokens still literal. Both SKUs, since the channel is the shared Notifications project.

    Two review-driven strengthenings beyond the proposal: (1) context_json serializes a REDACTED copy — code-block bodies carry the standard see-email/in-app hint and the typed Remediation payload is stripped, so copy-paste T-SQL follows the same never-on-a-webhook rule as every other channel (the heading and IsCodeBlock flag survive, so your automation still learns a remediation exists); (2) Save-time validation and the Test button render the raw tokens against a quote-bearing stand-in context, so the mis-quoting mistake — "context": "{{context_json}}" — fails where you can see it instead of on the first real deadlock alert.

    One deliberate divergence from the issue's sketch: the JSON shape is the PascalCase persisted-ContextJson projection (Incidents[].DedupKey / TotalOccurrences / IncidentStartedUtc), not snake_case — one canonical shape whether your Logic App reads the webhook or the alert-history row, round-trip-pinned through the same serializer. {{dedup_key}} carries the identical key the PagerDuty channel derives, including the serverId+metric fallback for level/threshold alerts, so cross-channel ticket correlation works out of the box.

    Lands on the next nightly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions