Skip to content

feat(messaging): refuse a transactional email the send checks would skip - #556

Closed
Silthus wants to merge 19 commits into
feat/email-send-code-formfrom
feat/email-send-admission-checks
Closed

Silthus wants to merge 19 commits into
feat/email-send-code-formfrom
feat/email-send-admission-checks

Conversation

@Silthus

@Silthus Silthus commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Moved here from #121850, closed on 2026-10-25 so this fork stack is the only copy. It opens upstream again after review.

Problem

  • A transactional send that PostHog accepts with 202 can still be skipped later in the worker: a suppressed or invalid recipient, a paused workflow, a suspended project, or an unverified sender. The caller sees success and the email never leaves. Contract #120921 promises "no silent skips": those checks answer before the 202, with why and fix.
  • An email edited in PostHog between the 202 and the worker run went out with the new content, so the caller could not tell which version they sent.
  • Map: Silthus/posthog#461, slice 07 of strand Silthus/posthog#535.

Impact

Pays into Grow Workflows through billed workflows_emails actions per paying org (not measured yet as a monthly count). In scope: Messaging, email API sends. Expected move: small, up: a caller who learns why a send failed fixes it instead of churning. Run cost: one suppression lookup and one cached MX lookup per send, before the queue.

Merge Danger

Warning

🟡 Merge risk: medium · two-way door: no migration and no queued-format change, so a revert restores the old behavior (merge-risk rubric)

  • cdp-api now refuses a send that it queued before. A wrong check would refuse good email. Every refusal names its code, and the checks are the ones the worker already skipped on.
  • The worker loads a stored revision for a transactional email edited after its 202. A wrong lookup would send the wrong content. A missing revision, or a step with secret inputs, falls back to the live version and counts cdp_hogflow_pinned_version_missing_total.
  • Sends only reach these paths behind the workflows-transactional-email-api flag. It stays off until every stack PR has deployed on Django, cdp-api, the hogflow worker and the email worker (#121272 has the reason). Before that, an old Django would read a new cdp-api's refusal as a retryable 503.

Changes

Kind Files Lines
Production 17 +346 −60
Tests 6 +379 −37
Total (this layer) 23 +725 −97

What a caller now sees

  • Both send routes answer 400 before queueing when a send check fails. The body carries code, detail, attr, why, fix and docs_url. Nothing is queued, and the idempotency key is released, so a fixed retry with the same key goes through.
code When attr
invalid_recipient The address is malformed, or its domain has no mail servers to
recipient_suppressed An earlier email to the address bounced or was marked as spam to
template_paused Email sending is paused for the email's workflow none
sending_suspended Email sending is suspended for the project none
sender_not_verified The step's sender is missing, not an email channel, or not verified from
sender_not_allowed An explicit from names an address no verified sender covers from
  • sender_not_verified keeps its meaning for a project with no verified sender at all. An explicit from off the verified domains is now sender_not_allowed, as the contract names it. It was sender_not_verified on #121272.
  • detail says what the check found, without run-log wording. The worker's run log writes "Skipping send." before the same detail.
  • The OpenAPI schema now documents the 429 body of the existing throttling. It tells the two cases apart: throttled (rate limit, retry) and quota_limited (monthly quota, do not retry).
  • A replayed idempotency key still returns the first 202 before any check runs.
  • A transactional email runs the workflow version it was accepted at. An edit in PostHog after the 202 applies to the next send, not to one already queued.
  • Two live settings still apply to a queued email: a step marked sensitive after the 202 stays sensitive, and a step with secret inputs runs the live version.

How it works

  • cdp-api's send route runs EmailSendPolicy.check(candidate) after the idempotency reserve and before the quota check. It is the same policy the worker runs at delivery, so the two cannot drift.
  • The policy gains an address check that only admission passes: EmailValidationService.findUndeliverable, the syntax and MX check the worker already used to skip a recipient. It fails open when DNS errors.
  • Admission uses the same EmailValidationService instance as the hog flow executor, so both read one MX cache.
  • Django maps a cdp-api 400 with a send-check code to Rejected(code, detail, why, fix) and renders the full error body.
  • The worker loads the accepted revision's actions, edges and trigger for a run whose own event is $transactional_email and whose flowVersion is older. So a trigger edit after the 202 cannot unpin it. The lookup is cached per flow and version.
  • The pinned version takes the live sensitive flag of each email step, and the flag only turns on.
  • When the revision is missing, or a step holds secret inputs that revisions do not store, the worker logs, counts cdp_hogflow_pinned_version_missing_total, and runs the live version.
  • The worker hides person properties from a run because of the run's own $transactional_email event, also when it refreshes the person. master decided this from the live trigger, so a trigger edit after the 202 could change it.

Mechanical: getSkipReason now delegates to findUndeliverable; emailPauseMetadata is extracted for reuse.

How did you test this code?

Test rationale: each case below failed before its fix and passes after it.

Test seams:

  • cdp-api's send route, with the queue faked and DNS faked per domain: one case per policy code. Each asserts the 400 body, the outcome counter, and that nothing was queued.
    • A keyed send to a paused workflow answers 400 template_paused. After the pause is lifted, the same key and body answer 202 and queue one email. It catches a refusal that keeps the key.
    • A refusal's detail does not start with "Skipping send".
    • A code send's own sender decides the sender check over the workflow's sender, both ways. A verified override of an unverified workflow sender queues. An unverified override of a verified one answers sender_not_verified and queues nothing.
  • Both Django send routes at the HTTP boundary, with cdp-api faked at its HTTP client: each send-check code maps to the full error body, 429 tells throttled from quota_limited, and an explicit from off the verified domains answers sender_not_allowed.
  • The hog flow worker's loadHogFlows: a transactional email edited after its 202 runs the accepted version, also when the edit changes the trigger to schedule. A step turned sensitive after the 202 stays sensitive. A step with secret inputs runs the live version. An event workflow runs its live version.

Red before green:

  • The pin test fails with the worker files swapped back to the parent commit (Expected - 1, Received + 1: the edited exit reason) and passes on HEAD.
  • With the Django send-check commit's production files reverted, 9 cases in test_transactional_email.py fail. They pass on HEAD.

Local gate at the stack top 987b927630e, which holds this head unchanged, all green: CDP jest 3690 (1 skipped), cdp-api serial 288, pytest 1764, repo-wide mypy, nodejs tsc, and the rest of the gate. The pre-push preflight ran in strict mode with 0 failures.

Live run at 987b927630e (Django, cdp-api and the email worker on a migrated clone of the local test database, delivering to maildev), case 05: a code send with a cc the project suppresses answered 400 recipient_suppressed with attr: "cc", why and fix. The job count stayed the same, and after 15 seconds maildev held nothing.

Earlier live run at 1e18413cd1c, before the review fixes: a suppressed to
before: cyclotron jobs 0, email_status rows 0, maildev 0
POST send/email/ to a suppressed recipient
HTTP 400 {"code":"recipient_suppressed","attr":"to","why":"An earlier email to bounced@example.com bounced or was marked as spam, ...","fix":"Send to another address, or remove the address from the suppression list ..."}
after 15s: cyclotron jobs 0, email_status rows 0, maildev 0

Cross-version (base b20afdb4465, head ca8b9f00c38, real cdp-api route, worker and Django handling in each tree):

Crossing Result
Base cdp-api's 202, this head's Django Pass: read as Queued.
This head's cdp-api refusal, base Django Fails behind the flag: base Django maps the 400 to Unavailable, so the caller gets a 503 without why and fix.
A template send edited after its 202, base cdp-api and this head's worker Pass: the worker runs the accepted revision.
The same, this head's cdp-api and a base worker Pass: the base worker runs the live version, as before this PR.

Gaps:

  • DNS is faked in the tests. A real MX lookup is the code the worker already runs.
  • No live run edits a workflow after its 202. The worker tests and the cross-version harness show the pin.
  • The cross-version run could not show person masking across versions, because its person lookup was mocked.
  • The base-built cross-version values are kept as fixtures in proof storage, not committed as tests.
  • Prod shape: gap, accepted until read-only production access arrives on 2026-11-02, for revision availability and the stored step configs the pin meets.

Release status

  • This change is behind a feature flag and is not available to users

Docs update

None. The transactional email API docs page is written after the stack lands.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5 (claude-opus-5-5)

Question Answer Who Where
Where do the checks run? In cdp-api, after the idempotency reserve and before the quota check, so a replay returns first and a refused send uses no quota 🤖 Agent pending approval
What runs when the accepted revision is missing? The live version, with a warning log and a counter, over refusing the send 🤖 Agent pending approval
Is the address check in the shared policy? Yes, passed in by admission only, because delivery already skips those recipients earlier 🤖 Agent pending approval

Open: the rollout decision on #121272: keep the deploy order plus the flag, or give code sends their own cdp-api path so an old cdp-api answers 404 and Django answers 503 (➡️ the current plan is the deploy order plus the flag).

Gap: the cross-version base-built values sit in proof storage, not in committed tests.

Review (GPT-6.1 Sol, round 1 over the whole stack, then a final read of the fixes):

Finding Response
invalid_recipient names the recipient address Kept: the contract has invalid_recipient name the recipient so a caller can skip it, as master's recipient_suppressed does.
Final read No change in this layer since round 1. No findings.

The stack's QA swarm then reviewed this layer. Round 1 fixed the pin (decided from the run's own event, trigger from the revision, sensitive kept, live version for secret inputs), the refusal detail wording, and three smaller items. Round 2 fixed person masking on a pinned run and two copy nits. Final read 1 and the rebase check in final read 3 found nothing.

Involvement: strand conductor relays answered, one choice from Michael on replay scope, no grilling rounds. Reviews: GPT-6.1 Sol round 1 over the whole stack and a final read of the fixes. Stack QA swarm: design pass (GPT-6.1 Sol high); round 1 and round 2 (review and team lanes GPT-6.1 Sol high, fresh eyes Claude Opus 5.5 high); final read 1 and final read 3 over the rebase (GPT-6.1 Sol high, clean); fixers Claude Opus 5.5 high; cross-version proof (Claude Opus 5.5); live run at the stack top.

  • Skills invoked: /tdd, /improving-drf-endpoints, /writing-tests, /writing-code-comments, /writing-pr-descriptions.
  • No duplicate: no open PR adds admission checks to the transactional email route.
  • Public artifact: all fixtures are invented and use example.com and .invalid domains.

🤖 Generated with Claude Code

@Silthus
Silthus force-pushed the feat/email-send-code-form branch from 5067eed to f55e16e Compare October 23, 2026 21:24
@Silthus
Silthus force-pushed the feat/email-send-admission-checks branch from 1af2c08 to 5cec76f Compare October 23, 2026 21:24
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

🤖 CI report

⚠️ Comment density — 4% of added code lines are comments (29 of 678)

This section warns when comments are more than 3% of the code lines a PR adds, and alerts above 6%. Before agent-assisted PRs, the typical share was about 2%. Only full-line comments count. Docstrings, generated files, snapshots, migrations, and workflow files are left out.

Comments that restate the code, record how the change came about, or narrate the next line add noise for the next reader. Keep the comments that explain a reason the code cannot show, and remove the rest. See .agents/skills/writing-code-comments/SKILL.md for the house rules.

Files with the most added comment lines:

File Comment lines Added lines
nodejs/src/cdp/services/hogflows/hogflow-manager.service.ts 6 61
nodejs/src/cdp/services/hogflows/transactional-email.ts 5 31
nodejs/src/cdp/services/messaging/email-send-policy.ts 5 45
nodejs/src/cdp/consumers/cdp-cyclotron-worker-hogflow.consumer.ts 4 43
nodejs/src/cdp/cdp-api.serial.test.ts 3 180
nodejs/src/cdp/consumers/cdp-cyclotron-worker-hogflow.test.ts 1 111
nodejs/src/cdp/services/messaging/email-validation.service.ts 1 23
products/messaging/backend/facade/contracts.py 1 6

This check does not block merging. It updates on every push and clears when the share drops.

Silthus and others added 19 commits October 23, 2026 23:02
The transactional email route now runs the email send policy before it
queues, so a suppressed recipient, a paused workflow, a suspended project
or an unusable sender answers 400 with its code, why and fix instead of a
202 that delivery later skips.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ive it

The send route now checks that the recipient's address is well formed and
that its domain has mail servers, and answers 400 `invalid_recipient`
with why and fix. Delivery keeps its own check, which skips the send when
a domain stops taking mail after the 202.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A send check that fails in the CDP service now reaches the caller with
its code, why, fix and docs link, instead of a 503. Every refused send
carries the docs link for its code. A `from` address that no verified
sender can send from answers `sender_not_allowed`, so callers can tell it
from a project with no verified sender at all.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker reloads a workflow's live steps when it dequeues a job, so a
template edit after the 202 changed an email that was already queued. A
transactional email now runs the steps of the version it was accepted
at, read from the workflow's revisions. Other workflows keep picking up
live edits. When no revision holds that version, it sends the live one
and counts it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A queued transactional email is now recognized by its own event, and
runs the trigger of its accepted version too, so a later trigger edit
cannot unpin it. A step marked sensitive after the send was accepted
stays sensitive, and a version whose step holds secret inputs runs live,
because revisions do not store secrets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An admission refusal's detail read "Skipping send: ...", although the
send was refused and never queued. The policy and the address check now
state what they found, and the delivery log adds "Skipping send:".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant