Skip to content

feat(messaging): send each email once per idempotency key on send/email - #121867

Closed
Silthus wants to merge 85 commits into
PostHog:masterfrom
Silthus:feat/email-send-route-idempotency
Closed

Silthus wants to merge 85 commits into
PostHog:masterfrom
Silthus:feat/email-send-route-idempotency

Conversation

@Silthus

@Silthus Silthus commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Problem

  • A caller of POST /api/projects/@current/messaging/send/email/ cannot retry safely. The route reads no Idempotency-Key, so a retry after a timeout can send the email twice. Contract #120921 promises the key on both send routes.
  • Map: Silthus/posthog#461, slice 06 of strand Silthus/posthog#535.

Stacked on #121850, on #121272, on #121084, on #121083, on #118924. This fork PR targets master, so the diff here includes all five. This layer alone: feat/email-send-admission-checks...feat/email-send-route-idempotency.

Impact

Pays into Grow Workflows through billed workflows_emails actions per paying org (not measured yet as a monthly count). In scope: Messaging, email API sends. Expected move: small, up: a caller can retry a send without sending it twice, which the API needs to replace another email provider. Run cost: n/a, the key check already runs on the template route.

Merge Danger

Warning

🟡 Merge risk: medium · two-way door: no migration, and a revert stops the route from sending keys (merge-risk rubric)

  • The fingerprint and the moved active check change public retry semantics at a real-email boundary. A wrong fingerprint would replay a different send, or refuse a real retry with 422.
  • The workers now reload a saved workflow after commit, for every workflow. Before, a reload inside the transaction cached the old row, so a pause did not stop new sends until the cache refreshed.
  • What limits it: the flag gates the route, the tests change the recipient, the content, the route and the named sender, and the template route keeps the fingerprint it has today.

Changes

Kind Files Lines
Production 10 +128 −50
Tests 6 +134 −44
Generated types and snapshots 3 +12 −3
Total (this layer) 19 +274 −97

What a caller now sees

  • send/email/ reads the Idempotency-Key header, on both the template form and the rendered-content form.
  • A retry with the same key and the same request returns the first 202, with the same message_id and replayed: true. No second email goes out.
  • The same key with another request answers 422 idempotency_key_reused. Another request means another recipient, other data, other rendered content, sender, reply-to or capture choice, or the other send route.
  • The same key while the first request is still in progress answers 409 idempotency_key_in_flight.
  • A retry replays after its workflow was paused, and after the project's default sender changed when the caller named no sender. A workflow that isn't active answers 400 transactional_email_paused after the key lookup.
  • A deleted template, a local template whose content changed, or a project with no default sender still answers 404, 409 or 400 on a retry.

How it works

  • Django passes the key and route: "send_email" to cdp-api. The template route (messaging_templates/{id}/send/) sends no route.
  • cdp-api binds each key to transactionalSendFingerprint: the workflow, to, distinct_id and variables, plus the whole send and the route when either is present. A sender Django chose as the default (from.isDefault) is left out.
  • cdp-api checks that the workflow is active after the key lookup. Django checks only that it exists.
  • A workflow save publishes its reload to the workers after commit, as a delete already does. So a pause stops new sends once the reload arrives. A failed publish does not fail the committed save or skip the later after-commit steps.
  • A template-route send has neither, so it keeps the fingerprint it has on master. A retry across the deploy still matches the key it reserved.
  • The reserve, replay and release are the store master already has (scope transactional-email). Slice 05 of the map needs no PR of its own for that reason.

Mechanical: IDEMPOTENCY_KEY_PARAMETER moved to the shared module, so both routes document the header; the regenerated OpenAPI types and MCP tool schema.

How did you test this code?

Test rationale: each case below failed before its fix and passes after it.

Test seams:

  • cdp-api's send route with the queue faked: a key reused with another recipient, other code content, or on the other route answers 422 idempotency_key_reused and queues once.
  • Both Django send routes at the HTTP boundary, with cdp-api faked at its HTTP client: each way of naming the email (URL id, template.key, template.id, template_key) passes the key, reports a replay, and sends route only from send/email/.
  • The workflow model's save: a pause publishes no reload before commit.

Local gate at the stack top 987b927630e, which holds this head unchanged, all green: cdp-api serial 288, CDP jest 3690 (1 skipped), pytest 1764, MCP 5035, repo-wide mypy, nodejs tsc, and the rest of the gate. The pre-push preflight ran in strict mode with 0 failures.

Cross-version: owed, because the cdp-api request gains route and the key's fingerprint changes. Checked with this layer's base and head side by side, before the review fixes. The fixes did not touch the fingerprint, and the run did not repeat at this head:

Producer Consumer Result
New Django, route: "send_email" Old cdp-api Parses. The schema drops route, so the key is bound as on master. Until cdp-api deploys, a key reused on the other route replays instead of answering 422.
Old Django, no route New cdp-api Parses. A template-route fingerprint equals master's, so a key reserved before the deploy still matches. Old Django sends no key from send/email/.
New Django, from.isDefault Older cdp-api from this stack Parses. The schema drops isDefault, so the default sender stays in the fingerprint. An inactive workflow answers 503 instead of 400. Nothing is sent.
Django from this stack without isDefault New cdp-api Parses. The fingerprint keeps the sender, so a retry through new Django answers 422, not a replay. The flag stays off until both sides deploy, so no such key exists.
Cross-version run output
base cdp-api fingerprint, template route   760a7d467338675ced1b496a2d3757d2e4e8617c1f6e22211d02e259fe4aa307
HEAD cdp-api fingerprint, template route   760a7d467338675ced1b496a2d3757d2e4e8617c1f6e22211d02e259fe4aa307 EQUAL
HEAD cdp-api fingerprint, send/email route 2fd6a14aa7deb8f420f93296c4311f9a0f693f8934499c1d19df20bf4b5efb06 DIFFERENT
base cdp-api parses a HEAD Django payload with route: true | route kept: false
HEAD cdp-api parses a base Django payload without route: true

Live run at 987b927630e (Django, cdp-api and the email worker on a migrated clone of the local test database, delivering to maildev):

Case Request Answer
03 A code send with an Idempotency-Key, sent twice Both 202 with the same message_id; the second says replayed: true. Maildev held 1 email.
04 The same key with another cc 422 idempotency_key_reused, with docs_url. Still 1 email.

Gaps:

Release status

  • This change is behind a feature flag and is not available to users

Docs update

None. The transactional email API docs page is written after the stack lands.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5 (claude-opus-5-5)

Question Answer Who Where
Which code does a reused key answer? idempotency_key_reused, the name master already answers, over the contract's earlier invalid_idempotent_request 👤 Human pending approval
Does the idempotent enqueue need its own PR? No: master's cdp-api already reserves, replays and releases through the shared store 🤖 Agent pending approval
Is the route part of the fingerprint? Yes, so a key reused on the other route answers 422 🤖 Agent pending approval
Does a template-route key keep its fingerprint? Yes, over adding the new parts everywhere, so a retry across the deploy matches 🤖 Agent pending approval
Which retries replay after something changed? A paused workflow and a changed default sender; a deleted or changed template still refuses ("Fix common cases") 👤 Human pending approval

Open: should a reused key with a changed template.hash replay or answer 422? It replays today, because the fingerprint does not include the template version. Neither answer sends twice (➡️ keep the replay, and say in the Idempotency-Key help that the key binds the recipient, data and send choices, not the template version).

Open: the rollout decision on #121272: keep the deploy order plus the flag, or give code sends their own cdp-api path (➡️ the current plan is the deploy order plus the flag).

Review (GPT-6.1 Sol, round 1 over the whole stack, then a final read of the fixes):

Finding Response
Replay depends on template, workflow and sender checks that can change Fixed in part, as Michael chose: a paused workflow and a changed default sender replay. Tests: replay after a pause, a changed default sender replays, a changed named sender answers 422.
Final read: a default-sender key fingerprints differently across a mixed deploy Rejected, recorded as a rollout precondition. Only pods from this stack can mix, and the flag stays off until this layer has deployed on both sides. The worst case is a refused retry, never a second send.

The stack's QA swarm then reviewed this layer. Round 1 moved the workflow reload after commit and raised this grade to medium. Round 2 made that reload robust to a failed publish and fixed two comments. Final read 1 and the rebase check in final read 3 found nothing. Follow-up: the MCP tool workflows-send-email cannot send an Idempotency-Key yet.

  • Rollout precondition: the workflows-transactional-email-api flag stays off until tracking code v3 minting is on and every stack PR, this one included, has deployed on Django, cdp-api, the hogflow worker and the email worker.

Involvement: strand conductor relays answered, one choice from Michael on replay scope, no grilling rounds. Reviews: GPT-6.1 Sol round 1 over the whole stack and a final read of the fixes. Stack QA swarm: design pass (GPT-6.1 Sol high); round 1 (review and team lanes GPT-6.1 Sol high, fresh eyes Claude Opus 5.5 high); round 2 on the fixes (review lane GPT-6.1 Sol high, fresh eyes Claude Opus 5.5 high); final read 1 and final read 3 over the rebase (GPT-6.1 Sol high, clean); fixers Claude Opus 5.5 high; live run at the stack top.

  • Skills invoked: /tdd, /improving-drf-endpoints, /writing-tests, /writing-code-comments, /writing-pr-descriptions.
  • No duplicate: no open PR adds the idempotency key to send/email/.
  • Public artifact: all fixtures are invented and use example.com domains.

🤖 Generated with Claude Code

@trunk-io

trunk-io Bot commented Oct 22, 2026

Copy link
Copy Markdown

Merging to master in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

Silthus and others added 3 commits October 23, 2026 15:12
A message template gets a code key unique per team, an owner, a source and
a content hash, so code can upload and lock a template by key.

Part 1 of 3 of PostHog#118924.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lates

Code uploads an email template by key, which locks it against edits in
PostHog. Giving a template a key adopts it. A send names the template by
key or id and carries a content hash, so a stale upload is refused.

Part 2 of 3 of PostHog#118924.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Silthus and others added 3 commits October 23, 2026 18:05
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus
Silthus force-pushed the feat/email-send-route-idempotency branch from 9a2b0e1 to 0398172 Compare October 23, 2026 21:24
Silthus and others added 5 commits October 23, 2026 21:37
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The template library marks a code-managed template with its source and
key, shows it read-only, and offers a release back to PostHog.

Part 3 of 3 of PostHog#118924.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ield

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Silthus and others added 20 commits October 23, 2026 23:02
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A queued transactional email is now recognized by its own event, and
runs the trigger of its accepted version too, so a later trigger edit
cannot unpin it. A step marked sensitive after the send was accepted
stays sensitive, and a version whose step holds secret inputs runs live,
because revisions do not store secrets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An admission refusal's detail read "Skipping send: ...", although the
send was refused and never queued. The policy and the address check now
state what they found, and the delivery log adds "Skipping send:".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…der changes

cdp-api checks that the workflow is active after it looks up the idempotency key, and answers
transactional_email_paused with why and fix. Django keeps only the check that the workflow exists.
The fingerprint leaves out the project's default sender, which the caller didn't name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker that reloads inside the save's transaction caches the old row,
so a pause did not stop new transactional sends until the cache aged out.
The post_save publish now waits for commit, as the delete publish does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h fails

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus
Silthus force-pushed the feat/email-send-route-idempotency branch from 0398172 to ba5d72e Compare October 24, 2026 17:28
@Silthus

Silthus commented Oct 25, 2026

Copy link
Copy Markdown
Contributor Author

Note

🤖 Claude Opus 5.5 responding on behalf of Michael

Two decisions for this PR:

  • A reused Idempotency-Key with a different request answers idempotency_key_reused, the name master already uses. The contract's earlier invalid_idempotent_request is dropped, and Transactional email SDK contract for posthog-python and posthog-node #120921 and the SDKs follow this name.
  • A retry under the same key replays after a paused workflow or a changed default sender. A deleted or changed template still refuses. A changed template.hash alone replays the first send, and the docs should say so before the flag turns on.

Slice 05 gets no PR of its own: the reserve, replay and release here use the store master already has.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature/agent-infra Feature Tag: Agent infra

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant