Skip to content

feat(messaging): send attachments with a code send - #121852

Closed
Silthus wants to merge 116 commits into
PostHog:masterfrom
Silthus:feat/email-send-attachments
Closed

Silthus wants to merge 116 commits into
PostHog:masterfrom
Silthus:feat/email-send-attachments

Conversation

@Silthus

@Silthus Silthus commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Problem

  • A team moving from Resend keeps every call site that attaches a file on Resend, such as a calendar invite or a PDF receipt. PostHog's code-form send cannot attach anything, so the migration never finishes (Silthus/posthog#543).
  • Map: Silthus/posthog#461, slice 2 of strand Silthus/posthog#543. Contract: #120921. An amendment waits for approval before it is posted there.

Stacked on #121851 (cc, bcc, headers and tags), on #121272 and its stack. This fork PR targets master, so the diff here includes the layers below. This layer alone: feat/email-send-fields...feat/email-send-attachments.

Impact

Pays into Grow Workflows through billed workflows_emails actions from orgs that send by code (not measured yet: needs send attribution, #120628, merged and an insight of code sends per org). In scope: Messaging, email API sends. Expected move: small, up. Run cost: SES's attachment charge per GB sent, small at a 512 KB cap. The same queue and worker otherwise.

Merge Danger

Warning

🟡 Merge risk: medium · two-way door (merge-risk rubric)

  • The SES send that every workflow email takes gains an Attachments list on its simple message. A wrong mapping would change a file's name, type or inline placement.
  • A send without attachments builds the same SES message as before. The existing SES tests cover it unchanged, and the send route sits behind workflows-transactional-email-api with no SDK sending attachments yet.
  • Rollout precondition: the flag stays off until tracking code v3 minting is on and every stack PR has deployed on Django, cdp-api, the hogflow worker and the email worker (#121272 has the reason). An older cdp-api or worker drops the attachments and still answers 202, as a rollback after the flag turns on would.

Changes

Kind Files Lines
Production 11 +241 −6
Tests 7 +275 −1
Generated types and snapshots 6 +157 −0
Total (this layer) 24 +673 −7

What a caller can now do

  • The code form takes attachments: up to 10 files, 512 KB in total after decoding. Each has filename, base64 content, an optional content_type and an optional content_id.
  • A content_id shows the file inline, where the HTML references it as cid:<content_id>.
  • template.id refuses attachments, like the fields of the slice below.
  • The route refuses, naming the field and the fix: content that is not base64 or is empty, more than 10 files or 512 KB, a file type SES blocks (such as .exe, also as the whole name), a filename with a path or line break, a content_type that is not a MIME type, and a content_id with < or >. No error repeats the content.
  • content_type and content_id take at most 78 characters, the SES limit, so a longer value answers 400 instead of failing at SES after the 202.
  • An attachment key the route does not take, such as contentType, answers 400 invalid_request on attachments. Before, the route dropped it, and an invite could lose its method=REQUEST.
  • A refused attachment carries a why that says what attachments PostHog takes.
  • The same Idempotency-Key with another attachment answers 422 idempotency_key_reused.

How it works

  • Django decodes the content to check its size, and hands it to cdp-api base64-encoded in send.attachments.
  • The worker puts the attachments on the email step's hog function and checks them again with the same limits, so an attachment from any path fails closed.
  • SES: a send with attachments keeps the simple SES message and lists the files in Content.Simple.Attachments. A file with a content_id goes inline, any other as an attachment.
  • The headers, copies and reply-to stay where every other send puts them, so a send without attachments builds the same SES message as before.
  • An earlier revision built one raw MIME message for SES. Native SES attachments replace it, so the raw path and helpers/raw-mime.ts are gone.
  • Maildev gets the attachments through nodemailer, with file and URL access off.
  • Lifecycle rows list each file by name, type and size, never its content, and mark the row attachments_omitted.
  • A rerun skips a row marked attachments_omitted with a warning, because it can't rebuild the send. The other rows of the page still rerun.

Why the 512 KB cap: the content rides inside the queued workflow run, through the workflow queue and then the email queue, so it counts against the queue's message size. SES itself takes up to 40 MB.

Not in this PR: the SDK parameter (lanes Silthus/posthog#536 and Silthus/posthog#537).

How did you test this code?

Test seams:

  • The send serializer, without a database, for every attachment limit.
  • The send route at the HTTP boundary, with cdp-api faked at its HTTP client.
  • cdp-api's send route with the queue faked, for the worker's size cap.
  • EmailService against a faked SES client, for the SES attachments.
  • The hog flow executor delivering to real maildev, from state.send to the delivered file.
  • The lifecycle row producer.

Test rationale: what each new test catches:

  • Thirteen refusals at the serializer, one per limit, each asserting the content never appears in the error.
  • The route hands cdp-api the attachments with type and content id.
  • cdp-api refuses a send over 512 KB.
  • cdp-api answers 422 idempotency_key_reused for a key reused with another attachment.
  • A blocked file type's why says what attachments PostHog takes.
  • SES gets the simple message with the caller's and PostHog's headers in order, the reply-to, and Content.Simple.Attachments: the invite as an attachment with its type, and the logo inline by its content id.
  • A run's attachments reach maildev. This test failed with the wiring from state.send to the email step removed.
  • A lifecycle row keeps { filename, contentType, size }, drops the content and carries attachments_omitted.
  • A rerun page skips the code send whose attachments were not stored and reruns the rest.
  • The serializer refuses a 79-character content_type or content_id and takes 78, refuses a file named .exe, and refuses an unknown attachment key.

Each new test failed without its production change. Local gate at the stack top 987b927630e, which holds this head unchanged, all green: pytest 1764, CDP jest 3690 (1 skipped), cdp-api serial 288, MCP 5035, repo-wide mypy, nodejs tsc, and the rest of the gate. The pre-push preflight ran in strict mode with 0 failures.

Live run at 987b927630e (Django, cdp-api and the email worker on a migrated clone of the local test database, delivering to maildev), case 01: a code send with cc copy@example.com and invite.ics (text/calendar; method=REQUEST) answered 202. Maildev got one email to jane@example.com with the cc and the attachment invite.ics as text/calendar. Maildev receives through nodemailer, not SES, so the live run does not reach the SES message. The EmailService test covers its attachments.

Earlier live run on 9d7a26704b5, before the review fix and before native SES attachments: inline image and refusals
Send Answer Delivered
Rendered content with cc, bcc, headers and a tag, plus invite.ics (text/calendar; method=REQUEST) and an inline logo.png (content_id: logo) 202 invite.ics as text/calendar; method=REQUEST; name=invite.ics, and the logo as Content-ID: <logo>
Attachment setup.exe 400 attr: attachments__0__filename, "Mailbox providers block .exe files. Send a link to the file instead."
One attachment of 512 KB + 1 byte 400 attr: attachments, "Attachments can hold 512KB in total. Send a link to larger files."

Cross-version (base 6a82b47b1b2, head b41402aca63, real Django body, cdp-api route, workers and result store in each tree):

Crossing Result
Base Django, cdp-api or store into this head Pass. A base-stored row keeps the content, and this head reruns it with the attachments.
Flag-off handoff, both ways Pass. Attachments travel only in state.send.
This head's code send, base cdp-api or base worker Fails behind the flag. Base strips the attachments and answers 202; the invite arrives without its .ics.
This head's stored row, base rerun Fails behind the flag. Base ignores attachments_omitted and sends again without the files, or a worker from this head fails the step.
This head's code send recorded by a base pod Base keeps up to 512 KB of attachment content in the ClickHouse snapshot.

Not checked: real SES. A sandbox SES account was not available to this lane, so delivery of SES attachments, open and click tracking on a send with attachments, and how mail clients render the invite, are unproven. The queue size of a full 512 KB send is unverified. The base-built cross-version values are kept as fixtures in proof storage, not committed as tests. Prod shape: gap, accepted until read-only production access arrives on 2026-11-02.

Release status

  • This change is behind a feature flag and is not available to users

Docs update

None. No doc under docs/ covers the send route.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5 (claude-opus-5-5)

  • Built as one lane of a map conductor run for Michael, test-first, on the raw MIME spike from spike/email-attachments. The design pass replaced the spike's raw message with native SES attachments.
  • Skills: /tdd, /improving-drf-endpoints, /writing-tests, /writing-pr-descriptions, impact, merge-risk. .agents/security.md was read before the attachment handling.
  • Review: one GPT-6.1 Sol round found a medium issue, now fixed. Headers named key and value turned the raw message's header map into one malformed header.
  • Decisions: attachments go to SES as native Content.Simple.Attachments, over a raw MIME message, so every send keeps one SES message shape. Names that end in a dot or a space get the extension check too.
  • The stack's QA swarm then reviewed this layer. Round 1 fixed the rerun of a send whose attachments were not stored, the SES length limits on content_type and content_id, and a file named only .exe. Round 2 found nothing to fix. The rebase check in final read 3 found nothing.
  • Open: the rollout decision on #121272: keep the deploy order plus the flag, or give code sends their own cdp-api path (➡️ the current plan is the deploy order plus the flag).
  • Involvement: one GPT-6.1 Sol round in the lane. Stack QA swarm: design pass (GPT-6.1 Sol high); round 1 (review and team lanes GPT-6.1 Sol high, fresh eyes Claude Opus 5.5 high); round 2 on the fixes (review lane GPT-6.1 Sol high, fresh eyes Claude Opus 5.5 high); final read 3 over the rebase (GPT-6.1 Sol high, clean); fixers Claude Opus 5.5 high; cross-version proof (Claude Opus 5.5); live run at the stack top.

🤖 Generated with Claude Code

@trunk-io

trunk-io Bot commented Oct 22, 2026

Copy link
Copy Markdown

Merging to master in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

Silthus and others added 3 commits October 23, 2026 15:12
A message template gets a code key unique per team, an owner, a source and
a content hash, so code can upload and lock a template by key.

Part 1 of 3 of PostHog#118924.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lates

Code uploads an email template by key, which locks it against edits in
PostHog. Giving a template a key adopts it. A send names the template by
key or id and carries a content hash, so a stale upload is refused.

Part 2 of 3 of PostHog#118924.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Silthus and others added 3 commits October 23, 2026 18:05
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus
Silthus force-pushed the feat/email-send-attachments branch from 6095d91 to 320f9a6 Compare October 23, 2026 21:24
Silthus and others added 10 commits October 23, 2026 21:37
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The template library marks a code-managed template with its source and
key, shows it read-only, and offers a release back to PostHog.

Part 3 of 3 of PostHog#118924.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ield

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Silthus and others added 24 commits October 23, 2026 23:02
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A keyed retry that listed the same headers in another order changed the
idempotency fingerprint, so it got a 422 instead of a replay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The unknown-field check moves beside the other send field rules, so the
tag serializer refuses an unknown key by name as the root body does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A code send can attach up to 10 files, 512KB in total: base64 content, a
filename, an optional content type and an optional content id to show the
file inline. The send route refuses blocked file types, paths and line
breaks in names, and unsafe MIME parameters, without repeating the content.

The worker checks the attachments again, sends them through SES as one raw
MIME message that keeps PostHog's tracking and unsubscribe headers, and
attaches them in maildev. Lifecycle snapshots list each file without its
content.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nodemailer read a header map with `key` and `value` names as one header,
so a send with attachments and headers by those names went out with a
malformed header and without PostHog's tracking and Auto-Submitted
headers. Pass the prepared headers as a list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A refused attachment answered with the why of a field PostHog doesn't
know. It now says what attachments PostHog takes. The rendered send's
expected payload also carries the default sender flag the code form added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A file named `.exe` has no suffix to the path parser, so it passed the
blocked file type check. The extension is now the text after the last dot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An attachment with `contentType` or `cid` got a 202 and went out without
them. Attachments now refuse an unknown key by name, as tags do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ters

The email provider takes at most 78 characters for each, so a longer one
got a 202 and then failed at send.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A lifecycle snapshot lists attachments without their content, and a
rerun restored that send as is, so its lifecycle row threw and failed the
whole page. The snapshot now marks the omission, and the rerun skips that
row with a warning, as it does for a sensitive email.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t caps

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature/agent-infra Feature Tag: Agent infra

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant