[mattdevdba]

Issue #, if available: #664 (RFC), towards #417

Description of changes:

Problem

DynamoDB keeps only the current state of each v2-executions row, so the history behind team metrics (time in each status, gate wait, rewinds, attempts, throughput) is lost as soon as a row changes. #417 needs per-intent drill-down and metrics that can be recomputed when a definition changes. The EMF metrics in #570 are aggregated and have no history to recompute from.

Solution

An optional reporting lake, off by default behind reporting_enabled. This is PR 2 of the split proposed in #664: execution state only. The graph feed follows #621, and dashboards and permissions would be a separate proposal.

• Feed. A Kinesis Data Streams for DynamoDB destination on v2-executions, with microsecond timestamps, feeds Firehose. A transform Lambda writes one JSON line per change to S3 under executions/record_type=<sort-key prefix>/dt=<yyyy-MM-dd>/. The destination is a separate resource, so the table definition doesn't change.
• Left out. Agent output transcripts (OUTPUT#), agent sign-in bookkeeping, GSI keys, and on the execution header the prompt, starter email, credential binding and MCP server maps.
• Backfill. A resumable Lambda copies existing rows in as SNAPSHOT changes. The docs include the invoke loop.
• Query layer.
  • A Glue changes table with partition projection, so there's no crawler to run.
  • An Athena workgroup with a scan limit.
  • Views: executions, stages, gates, approvals, steering, sensor_runs, units, unit_prs, metrics, graph_reads, intent_progress, decision_log, execution_status_history, stage_state_history.
  • Terraform applies the views from terraform/modules/reporting/views/*.sql through an aws_lambda_invocation of a small views Lambda, which reruns whenever a SQL file changes. This is my suggested answer to open question 4 in the RFC.
• Operations.
  • A reader IAM policy (output) for BI tools.
  • Freshness and transform-error alarms, which notify reporting_alarm_actions if set.
  • The bucket can only be force-destroyed when deletion_protection is off, because its history can't be rebuilt from the table.
• Layout.
  • lambda/reporting: three entry points built separately. record-types.json is shared by the transform and the Glue partition projection.
  • terraform/modules/reporting.
  • Root terraform/reporting.tf: variables, the module with count, and outputs.
  • Docs: docs/using-the-platform/reporting.md (setup, data model, counting rules), plus short notes in setup.md and data-model.md.

Testing

• npx vitest run --project=reporting: 21 tests.
  • Row building against the shared v2-process-keys.js builders, including the redactions.
  • The Firehose transform: Ok, Dropped and ProcessingFailed in one batch.
  • The backfill against DynamoDB Local, including resuming from startKey.
  • Statement splitting and Athena execution for the views, including a check of every shipped view file.
• New terraform/tests/reporting.tftest.hcl (mock providers), added to the Terraform workflow. It covers the streaming destination, shard and on-demand modes, deletion protection, the projection range (explicit and default), the record-type projection, and the views invocation input.
• terraform fmt -check -recursive, terraform validate -no-tests with the CI local-backend override, npm run lint, oxfmt --check, npm run dep:check (no new findings), secretlint, npm run sdk:check, and uv run zensical build.
• We run an equivalent lake (the same views, in Python) in our own deployment. This JS port hasn't been deployed yet. In particular, the views Lambda's Glue and Athena permissions for CREATE OR REPLACE VIEW haven't been checked against a live account.

Impact

None unless reporting_enabled is set; no other code path changes. When it's set, the stack adds a Kinesis stream, a Firehose stream, three Lambdas, an S3 bucket, a Glue database and an Athena workgroup. The docs link to pricing for each.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.