Skip to content

feat(cdp): add the email attachments table to cyclotron - #127995

Merged
trunk-io[bot] merged 66 commits into
masterfrom
email-v2/13b-cdp-email-attachments-table
Nov 4, 2026
Merged

trunk-io[bot] merged 66 commits into
masterfrom
email-v2/13b-cdp-email-attachments-table

Conversation

@Silthus

@Silthus Silthus commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Transactional email attachments (#127986) need a cyclotron table before the service code that uses it deploys. Migrations deploy apart from services, so the table lands first in its own PR.

Changes

  • Adds cyclotron_email_attachments: one row per file, keyed by the email job's id.
  • The file bytes stay out of cyclotron_jobs.state, so lifecycle snapshots in Kafka and ClickHouse never carry them.
  • Indexes on job_id for the worker and created_at for the janitor's sweep.
  • job_id has no foreign key, because the janitor deletes finished jobs on its own schedule. A row outlives its job only until the sweep.
  • position keeps the files in the order the caller sent them.
  • sha256 and size_bytes let the worker refuse a file that changed or went missing, and let the janitor report the table size without reading the bytes.
  • The janitor's sweep deletes rows older than 7 days, so a send whose job outlives that fails closed with attachments_missing.

Note

Merge danger: low, two-way door. A new, empty table that nothing reads until #127986 deploys.

How did you test this code?

Test rationale: no new test. The cdp test suites in #127986 run against this table.

  • Live run: rust/bin/migrate-entry cyclotron-node --fresh applies every cyclotron-node migration, this one last, on a scratch database.
  • CI: personhog's Kafka tests fail in two Rust test batches, independent of this table. They failed on master too until chore(cohorts): recover test readers from kafka transport errors #127882, which has merged. This branch picks up the fix with its next master merge.
Migration run excerpt
Applied 20261016000001/migrate add batch resolver parent index (8.060173ms)
Applied 20261103000001/migrate add email attachments (16.102981ms)
All requested migrations completed successfully

Indexes:
    "cyclotron_email_attachments_pkey" PRIMARY KEY, btree (id)
    "idx_cyclotron_email_attachments_created_at" btree (created_at)
    "idx_cyclotron_email_attachments_job_id" btree (job_id)

The full log lives in private proof storage.

Release status

  • No feature flag controls this change

Docs update

  • None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5 (claude-opus-5-5)

Skills: /writing-pr-descriptions

More context for AI agents

🤖 Generated with Claude Code

Silthus and others added 9 commits October 31, 2026 08:21
…il route

A code send posts its accepted content to transactional_email/send. The route
runs the same reserve, replay, refusal and enqueue steps as a template send.
The accepted send binds the idempotency fingerprint, and a miss on a new
workflow reads it from the primary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…flow

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ode-send

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ode-send

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A send with captureEngagement false still put the recipient's distinct_id
in its SES header and pixel/redirect tracking codes. Minting still emits
the v2 payload, which has no capture choice, so the SES delivery, bounce,
open and click callbacks for that send captured engagement events.

Every reader skips engagement capture for a code without a distinct_id,
so leaving it out honors the choice on every deployed reader now. The v3
payload and its minting switch stay as they are. The send-time capture
and the message assets row keep their distinct_id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The capture-off fix also changed the pixel and redirect codes that dev
and test mint, but only the SES header path had a test. The tracking URL
test now runs per capture choice and fails if those codes keep the
distinct_id of a capture-off send.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A code send accepts up to 10 cc and 10 bcc addresses. The route checks
every recipient before the 202: a malformed, suppressed or unreachable
copy refuses the whole send with the code `to` would get. The accepted
send now refuses unknown fields, so a newer caller never loses one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A capture-off send already mints its v2 codes without a distinct_id, and
every reader skips engagement capture without one. The v3 capture-choice
payload was never minted, so remove its marker, parser branch, rollout
constant and the captureEngagement fields that only fed it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus Silthus self-assigned this Nov 3, 2026
A queued job for a deleted workflow read the primary on every batch,
one writer query per missing id. Remember ids the primary confirmed
missing for 60 seconds (bounded to 10,000 ids, cleared by a reload),
and read a batch's replica misses from the primary in one query while
keeping the in-flight dedupe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🤖 CI report

✅ Trunk lane — non-backend lane

This PR is assigned to the non-backend lane. It does not run backend Python tests and may merge in parallel with PRs in other lanes.

Silthus and others added 16 commits November 3, 2026 09:46
A workflow the replica did not have yet was read from both databases on
every send and worker batch until replication caught up. Keep what the
primary returned, hit or miss, for 60 seconds (bounded to 10,000 ids,
cleared by a reload naming the id), so later lookups read neither
database; after expiry the replica is read first again.

The send-route and worker tests now hide the row from the replica at the
query boundary, so they cover a replica miss followed by a primary hit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A primary read that started before a reload could resolve after it and
store the old state for the next 60 seconds, so a workflow disabled or
paused during that read kept sending. A reload now drops the in-flight
read, and a read that a reload overtook returns its rows to its own
callers but stores nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	nodejs/src/cdp/services/hogflows/actions/hog_function.ts
…' into email-v2/13b-cdp-email-attachments-table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A newer Django can send a field this cdp-api does not deliver, such as cc,
bcc or attachments. Before this, the field was dropped, the send went out
without it, and the caller saw a 202. Now the send answers 400
invalid_request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and tracking

The outer send object refused unknown fields, but content, from and
tracking still dropped theirs, so a newer caller could get a 202 for an
email sent without them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	nodejs/src/cdp/services/hogflows/transactional-email.ts
Silthus and others added 10 commits November 3, 2026 20:57
…' into email-v2/13b-cdp-email-attachments-table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	nodejs/src/cdp/services/hogflows/hogflow-executor.service.test.ts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…' into email-v2/13b-cdp-email-attachments-table
The code-key and capture tests sent real email through the shared maildev,
which races email.service.test.ts clearing the same inbox when both files
run in one CI shard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The template route now requires a token without a kind claim, and the code
send route keeps requiring kind code_send, so neither token works on the
other route.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Silthus and others added 2 commits November 4, 2026 07:30
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Silthus added a commit that referenced this pull request Nov 4, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus
Silthus marked this pull request as ready for review November 4, 2026 10:12
@parameterai

parameterai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Risk: No findings

This PR adds one idempotent migration creating the cyclotron_email_attachments Postgres side table (job-keyed file rows with team_id, order position, sha256, and byte content) plus job_id and created_at indexes. It is schema-only: no code in the repo reads or writes the table yet, so it introduces no reachable request path, and tenant scoping is present via team_id. The only risk worth tracking is that nothing enforces a size cap at the database level — that must land with the service code in #127986.

Sentinel reviewed b6e32d3 · Review settings

@Silthus
Silthus requested a review from a team November 4, 2026 14:07
Silthus and others added 2 commits November 4, 2026 14:39
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tachments-table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from email-v2/13-cdp-code-send-copies to master November 4, 2026 17:23
@trunk-io

trunk-io Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

😎 Merged successfully - details.

…ail-attachments-table

# Conflicts:
#	nodejs/src/cdp/cdp-api.serial.test.ts
@Silthus

Silthus commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/trunk merge

@trunk-io
trunk-io Bot merged commit 0f60135 into master Nov 4, 2026
212 checks passed
@trunk-io
trunk-io Bot deleted the email-v2/13b-cdp-email-attachments-table branch November 4, 2026 20:15
@deployment-status-posthog

deployment-status-posthog Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-11-04 20:45 UTC Run
prod-us ✅ Deployed 2026-11-04 21:17 UTC Run
prod-eu ✅ Deployed 2026-11-04 21:17 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants