Skip to content

Analysis pages with long compound stories die on Slack with invalid_attachments: #3493's size discipline covers the prose, not the details that carry the finding #3612

Closed

Description

@erikdarlingdata

Summary

Analysis notification pages with long compound stories are lost on the Slack channel: the delivery fails with HTTP 400: invalid_attachments, the row persists in config_alert_log with the error, and the page — the product's highest-severity compound finding — never reaches anyone. Three instances on one production store class across two builds (Sep 17 00:52Z and Sep 18 01:21Z on .444; Sep 18 15:42Z on 3.8.0), all five-fact stories rooted in PLAN_REGRESSION (PLAN_REGRESSION→CPU_SPIKE→PARAMETER_SENSITIVITY→QUERY_SPILLS→TEMPDB_USAGE, severity 1.75, confidence 0.80); a three-fact analysis page delivered in the same hour on the twin store. #3493's size-bounded section splitting fixed the digest path and did not touch this one.

Mechanism (VERIFIED at source; which text object breaches is the lane's first read)

The Slack payload is one attachment whose blocks carry everything, so invalid_attachments covers any block-level violation inside it. WebhookAlertService.BuildSlackPayload applies the 3,000-character text-object limit only to prose (AddSlackProseSections, #3493). The context.Details path, where analysis findings carry their substance, has no size discipline at all:

  • a detail with a Body renders as one section with *{Heading}*\n{Body} — unsplit, uncapped. For an analysis finding this is the Advice detail (Investigation: …\n\nRemediation: …), composed per root fact key from the finding's facts (FactAdvice.Compose → the value-stated composer for the root; frozen into StoryText);
  • field-bearing details chunk into sections of 10 fields (AddSlackFieldSections) but each field's text is uncapped against Slack's 2,000-character field limit;
  • the block total is never checked against the 50-block message limit (the tail-side cap noted on Split oversized Slack prose across section blocks, inside the block budget, with stated omission (#3493) #3494 and left as a convention).

The failing stories share a root whose composed advice is long, and they are the longest chains; the delivered story had a different root. Fact count is the visible correlate; the breach is most plausibly the Advice Body crossing 3,000 characters. Discriminator, from the store, no Slack needed: the failed rows persist their context_json; length() of each detail's Body and each field value for the three failed rows against the one delivered row names the limit crossed. If every text object is under its limit, the breach is the block count.

Fix

Extend the #3493 contract to every text object and to the block total in the Details path: a Body splits across sections inside the block budget exactly as prose does (same helper, same stated omission when the budget runs out — "N characters omitted; the full text is in the alert's detail"); field values truncate at the field limit with the omission stated; the message states when details were dropped to stay inside 50 blocks, and which. One helper, three call sites, the existing SlackTextObjectLimit / SlackMessageBlockLimit constants. Pin with a five-fact PLAN_REGRESSION fixture built from a real persisted context_json (the failed rows are the fixture), asserting every text object and the block total sit inside the limits and that the omission is stated rather than silent. Email and Teams render the same details through their own paths — check whether Teams has an equivalent limit and whether email needs nothing.

Interim exposure, stated plainly: every analysis page whose advice body crosses the limit dies on Slack; the finding persists in analysis_findings and the alert log, so nothing is lost but the push — and the push is the point. On this fleet that is ~3 pages in two days, all top-severity.

Evidence: monitoring seat's tick record 20260918T164232Z (three failed rows with send_error, the same-hour delivered control); source as cited. Related: #3493 (digest path), #3494 (the tail-side note), #3466 (the family).

Activity

  1. erikdarlingdata commented on Sep 18, 2026

    @erikdarlingdata
    OwnerAuthor

    Mechanism corrected by the store's own record before any code moved — the discriminator named in the issue ran, and it refutes the leading hypothesis. Over the full two-day census of analysis pages on the store (15 pages, 3 failed — exactly the known set, no hidden victims): the three failed rows carry a maximum detail Body of 2,105–2,122 characters, all under Slack's 3,000, while a page that DELIVERED carries a 2,835-character body, larger than any failure. Text length is not the breach. The clean separator is detail count: the failures have 19, 20 and 21 details; the largest delivered page has 18. The boundary sits exactly between 18 and 19. At the Details path's roughly two blocks per detail (a divider plus a section, or a divider plus field-sections) on top of the fixed head and tail blocks, 19 details crosses Slack's 50-block ceiling, and Slack reports that as invalid_attachments. So the block budget with stated omission — the aside on #3494 — is the fix, and the per-text-object caps are hygiene this failure mode never touched (real bodies reach 2,835, so they are worth capping anyway). Two consequences for the design: budget on blocks, not facts — a four-fact story with verbose composers can produce 19 details as easily as a five-fact one; and the fixture that pins this is a 19-to-21-detail context, not a long body. Lane re-aimed accordingly. Credit: the monitoring seat's config_alert_log read.

  2. erikdarlingdata commented on Sep 18, 2026

    @erikdarlingdata
    OwnerAuthor

    Closed by #3618 (merged to dev).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions