Remove legacy flow compilation before queue identity and routing #647

Summary

Remove the legacy migration-based flow compiler before implementing #650 and #651.

Worker startup becomes the only supported deployment path for flow definitions. The same cleanup removes the now-useless FlowWorkerConfig.compilation option, including its skip and remote data-loss modes.

This is the first implementation stage of #653. It gives queue identity and private per-step queues one compiler, one startup contract, and one set of fixtures from the start.

Why remove the complete path

The legacy workflow has four coupled surfaces:

pgflow compile
  -> ControlPlane
  -> compileFlow()
  -> generated flow-definition migration

Keeping it for new deployment metadata would require a second implementation of queue-mode and route validation. Removing only ControlPlane would break pgflow compile. Remove the complete legacy unit instead.

Scheduling

#647 legacy compiler removal
  -> #650 queue identity
  -> #651 private per-step queues
  -> downstream validation
  -> stable queue-routing release

Complete this cleanup before #650 starts. #650 and #651 then extend only the final startup compiler and never need a temporary dual-path support matrix.

If downstream validation later stops #651, keep #647 as an independent simplification and publish neither queue change.

Implementation order

1. Replace compileFlow()-based integration setup with startup compilation or direct fixtures while the old helpers still exist.
2. Make startup compilation mandatory and move it before worker-function tracking and worker-row insertion.
3. Remove the allow_data_loss SQL argument and edge-worker call together, then generate the migration and database types.
4. Delete compileFlow(), ControlPlane, pgflow compile, their exports, fixtures, tests, and package targets after callers are gone.
5. Remove current documentation and navigation, add migration guidance and the Changeset, then run the complete package and website checks.

Remove

pkgs/cli

• Delete pgflow compile, its command registration, unit tests, and E2E test.
• Stop creating supabase/functions/pgflow/ during installation and remove its packaged fixture and install tests.
• Remove the compile-only serve:functions:e2e and e2e:compile targets from pkgs/cli/project.json; keep the remaining install E2E target.
• Remove compile-command output, options, README sections, and troubleshooting links.

pkgs/edge-worker

• Delete ControlPlane, its public export, server code, test Edge Function, config entry, fixtures, unit tests, and E2E tests.
• Remove the entire FlowWorkerConfig.compilation option and its internal plumbing. Its only special values were the legacy skip and remote data-loss paths.
• Every flow worker compiles or verifies its definition during startup; remote startup never authorizes destructive recompilation.
• Complete compilation or verification before track_worker_function() or worker-row insertion. A failed startup creates or changes no worker registration. Existing enabled HTTP registrations remain eligible for ensure_workers() retries; process workers remain externally supervised.
• Update startup-compilation tests and public configuration types without changing the existing local recompile path. Runtime code no longer reads a compilation property; JavaScript callers that still pass one cannot change startup behavior.

pkgs/dsl and dependent tests

• Delete the public SQL-text compileFlow() implementation and export.
• Delete the unused public TypeScript comparator compareFlowShapes(), its private comparison helpers, ShapeComparisonResult, and comparator-only tests. Keep extractFlowShape() and its types.
• Remove SQL-string compiler tests. Move any still-relevant DSL behavior assertions to flow-shape or startup SQL tests rather than losing coverage.
• Replace compileFlow() setup in client and edge-worker integration helpers with startup compilation or direct test fixtures.

pkgs/core

• Remove allow_data_loss from the schema source for ensure_flow_compiled(). Keep local-mode detection and local destructive recompilation unchanged.
• Generate the migration from pkgs/core/schemas/; do not edit historical migrations.
• Update pgTAP coverage and regenerate pkgs/core/src/database-types.ts.

Documentation, examples, and navigation

• Remove the CLI, edge-worker, and demo supabase/functions/pgflow/ fixtures plus matching config.toml entries.
• Redirect current manual-compilation, Compile API, and ControlPlane pages to startup compilation or the migration notice.
• Remove migration-based flow compilation from current examples, package READMEs, ARCHITECTURE_GUIDE.md, website navigation, reference indexes, and generated indexes. Keep legacy URLs as redirects to startup compilation or the migration notice.
• Update startup-compilation, worker-configuration, versioning, and first-deployment guides for mandatory startup compilation.
• Keep historical changelogs and release-news pages intact; current docs identify those APIs as removed.

Behavior after removal

missing concrete flow  -> compile during worker startup
matching flow          -> verify and start
local plain-flow mismatch -> keep the current automatic local recompilation behavior
remote mismatch        -> fail without deleting data; deploy a new concrete slug

#651 must later extend this contract so local StepQueuedFlow mode or route mismatches use the same automatic local recompilation behavior.

#647 adds no new worker-drain, generation, or restart protocol.

A structural or semantic production change uses a new concrete slug:

const greetUserV2 = new Flow({ slug: 'greetUserV2' })

Startup shape verification cannot detect handler-code or TypeScript contract changes. The versioning guide must continue to require a new slug for those changes.

Preserve

• Preserve current plain-flow local same-slug hot reload and destructive recompilation behavior.
• Leave the startup compiler ready for Persist physical queue identity for flow tasks #650's queue identity and Add private per-step queues with typed step workers #651's queue-mode and route-map extensions.
• Preserve the existing SQL startup compiler, flow-shape extraction, and concrete-slug advisory lock.
• Preserve create_flow() and add_step() as database building blocks used by startup compilation.
• Preserve existing generated migrations as ordinary historical SQL in user repositories.
• Preserve the production enable-fence guidance from Document the worker-function enable fence for production deployments #654.

Migration requirements

PostgreSQL keeps overloads with different argument lists. The migration must explicitly remove the production data-loss entry point:

DROP FUNCTION pgflow.ensure_flow_compiled(text, jsonb, boolean);

Then create or keep the intended signature without allow_data_loss, regenerate database types, and check that no callable overload accepts production data-loss authorization.

This intentionally ends startup compatibility with old workers that call the three-argument function. Do not retain a compatibility overload. The stable release therefore uses a fenced upgrade, not a rolling old/new worker upgrade:

record worker_functions.enabled
  -> disable affected HTTP functions
  -> deprecate and drain old workers
  -> apply the pgflow database migration
  -> deploy workers built with the new packages
  -> restore the exact recorded enabled values
  -> check new worker heartbeats

Process deployments stop their workers and external supervisors before the database migration, then update and restart them afterward.

The migration notice must tell existing users to:

1. Use Document the worker-function enable fence for production deployments #654's fence for HTTP workers, or stop process workers and their supervisors.
2. Upgrade the fixed pgflow package set and apply its database migration as one coordinated deployment.
3. Remove local supabase/functions/pgflow/, matching function configuration, and any deployed remote pgflow ControlPlane function.
4. Remove pgflow compile from scripts and CI.
5. Remove the compilation property from worker configuration, including false, {}, and { allowDataLoss: true } forms.
6. Make each worker import its complete flow definition.
7. Keep existing generated migration files as immutable migration history.

Release and checks

• Add a Changeset that lists every removed public API. The repository's fixed package group versions together.
• Let the Changesets release PR collect Remove legacy flow compilation before queue identity and routing #647, Persist physical queue identity for flow tasks #650, and Add private per-step queues with typed step workers #651. Do not merge that release PR before queue validation, rollout documentation, and final checks finish.
• Run the normal Core migration workflow: schema tests, Atlas migration generation, migration verification, type generation, and pgTAP.
• Run DSL, CLI, client, and edge-worker build/type/unit/integration checks after removing compiler-dependent fixtures.
• Run CLI install E2E, edge-worker E2E and export smoke checks, and the website build/link checks.
• Check the packed CLI and published package exports so removed commands and exports do not survive in dist.

Acceptance criteria

• This cleanup finishes before Persist physical queue identity for flow tasks #650 queue-identity implementation starts.
• ControlPlane code, exports, templates, demo functions, tests, and current docs are removed.
• pgflow compile, its tests, options, installation output, and current docs are removed.
• Public SQL-text compileFlow(), compareFlowShapes(), and their now-unused types and helpers are removed without losing startup-shape coverage.
• Client and edge-worker tests no longer use compileFlow() as database setup.
• Worker startup is the only supported owner of flow compilation and verification.
• Compilation or verification finishes before worker-function tracking or worker-row insertion; failure leaves registration state unchanged.
• Existing enabled HTTP registrations may retry after failure, while process-worker retries remain the external supervisor's responsibility.
• FlowWorkerConfig.compilation is absent from public types and runtime branching; passing the removed property cannot skip verification or authorize remote deletion.
• The three-argument ensure_flow_compiled(text, jsonb, boolean) overload is absent after migration.
• Production mismatches fail without deleting data.
• Plain-flow local recompilation tests remain green without new lifecycle coordination.
• The final release documentation states that old workers cannot start against the new database signature and gives the fenced upgrade sequence.
• Current versioning docs require a new concrete slug for structural and undetectable semantic changes.
• Existing-user migration instructions cover local and deployed ControlPlane functions, configuration, scripts, worker imports, package coordination, and historical migrations.
• A Changeset and release notes identify every removed public surface.
• Packed artifacts, package exports, CLI help, current docs, and website navigation contain no legacy compiler surface.
• Any stable release containing Persist physical queue identity for flow tasks #650 and Add private per-step queues with typed step workers #651 contains no migration-based flow compiler.

Out of scope

• Worker drain or quiescence redesign.
• Worker restart-state or worker_functions.enabled changes.
• Startup-failure or shape-mismatch persistence.
• Stable aliases.
• Per-step queue implementation.
• Custom or shared queues.
• Changes to flow-shape semantics.