ci: run PostgreSQL tests in isolated lane

[TheSentinel454]

Why

PR #6660 introduced four PostgreSQL-backed persistence tests and a focused CI selector, but an exact test-name list does not automatically cover future database tests. Broad ignored-test execution also exposed shared-schema races and ambiguity between desired-state and migration-applied schema expectations.

What

• Establish a discoverable convention: PostgreSQL unit modules use postgres_tests, PostgreSQL integration binaries use a postgres_ prefix, and unrelated external-infrastructure tests use an external_infra_ prefix.
• Add a dedicated nextest PostgreSQL profile and archive-backed CI job covering the relevant crates without enumerating test names.
• Create a run-scoped desired-state source database and a unique PostgreSQL database per test process for parallel-safe isolation, with cleanup on success, failure, or interruption.
• Route destructive migration and migration-parity tests to clean template0 databases while desired-state tests clone the desired-state database.
• Document discovery, schema modes, required role privileges, and a bounded portable local runner.
• Keep the existing infrastructure-free unit-test jobs unchanged.

This is intentionally limited to test and CI harness behavior. It does not move production database code, change persistence semantics, or implement issue #20's broader shared test-utility refactor.

Risk

Low production risk because all changes are confined to tests, CI configuration, documentation, and test harness scripts.

Remaining operational risks:

• Cleanup retries dropdb --force five times across roughly four seconds. Exhaustion warns with the database name but deliberately does not mask the test result; individual PostgreSQL diagnostics are suppressed.
• The seven-package boundary is duplicated between the runner and CI archive and must remain synchronized if PostgreSQL tests move to a new crate.
• Portable SHA fallback branches were exercised on Linux; no macOS Blox workstation was needed for this Linux CI artifact.
• GitHub's unchanged generic Unit Tests job currently fails while cold-linking buzz-voice because sherpa-onnx-c-api is absent. The branch changes no voice/build/toolchain inputs; the exact parent previously passed that job, and Blox passes the same infrastructure-free suite (1,450/1,450). A prior-head one-job retry and the final-head run both reproduced the hosted-runner failure.

Verification

Author Blox workstation 2020088:

• cargo fmt --all -- --check
• just clippy
• just test-unit: 1,450 passed; PostgreSQL tests remain skipped in fast jobs
• PR Centralize replaceable event persistence #6660 focused tests: 4/4 passed through the final runner
• Full lane: 282/282 passed in 10.087s with a non-superuser role limited to CREATEDB, CREATEROLE, and pg_read_all_stats
  • 272 desired-state tests
  • 10 migration-applied tests
  • 6 explicitly filtered external-infrastructure tests
• Cleanup fault injection: a deliberately underprivileged 282-test run produced the expected 277 passes and 5 failures, and the post-run catalog audit found zero lane databases after retry cleanup. The restored successful run also left zero lane databases.

GitHub exact final head 92c231e48f299ea2af23817763f5ae80ac013d68:

• PostgreSQL Tests: 282/282 passed across 10 binaries in 98.886s; 1,240 skipped, including 6 via the profile filter
• The shared relay/PostgreSQL archive predecessor completed successfully
• Unit Tests: unrelated sherpa-onnx-c-api native-link failure described above
• Independent exact-head review on separate Blox workstation 2022762: no substantive findings after all initial findings were addressed. The reviewer independently verified cleanup retry behavior, exit-status preservation, shell portability, syntax, and a clean exact-head worktree.

References

• Stacked on Centralize replaceable event persistence #6660 at exact head 561de54be4d9c2b622b7c2aa5b61bc3068f47e2a
• buzz-db: shared test infrastructure (scratch DB, community seeding) TheSentinel454/buzz#20
• buzz-db: shared test infrastructure (scratch DB, community seeding) TheSentinel454/buzz#20 (comment)

Generated with Codex.

Update — August 24, 2026 review follow-up

• Centralized the repeated PostgreSQL test URL resolution in crate-local, test-only helpers without introducing the broader shared utility refactor from issue Add just goose for one-command agent setup + fix API token auth #20.
• Restored descriptive hybrid/Redis test function names. Structural external_infra_*_tests modules now own exclusion, and the nextest filter only recognizes module path segments.
• Added a three-second source guard that scans every Rust file and fails CI when an ignored PostgreSQL test would be omitted or an external-infrastructure test would be included. Fixture tests cover accepted modules/binaries and both failure modes.
• Removed the two implementation-plan documents.

Performance profiling on Blox workstation 2027352 (same prebuilt archive and cargo-nextest 0.9.143 for every comparison):

• Current per-test database model, 8 workers: 10.15–10.40s across three runs. The 283 createdb calls and 284 dropdb calls consumed 28.85–29.46 aggregate seconds, about 39–40% of aggregate test-process duration.
• Reusing one database per worker without cleanup: 6.55–6.84s, about 35% faster, but all three runs failed because global matcher-queue tests inherited a quiesced community from an earlier test; the failing test varied with scheduling.
• Reusing workers with TRUNCATE … RESTART IDENTITY CASCADE: all 282 tests passed, but total time regressed to 10.47–10.67s. The 265 truncate calls consumed 24.34–25.45 aggregate seconds, so truncation merely replaced most clone/drop cost.
• Raising concurrency to 16 workers reduced one isolated-database run to 8.79s, but 24 workers exposed a cluster-global pg_stat_activity race. The profile remains at the proven-safe 8 workers.

Decision: retain unique per-test databases. Harness cleanup uses dropdb --if-exists --force; it drops the database rather than truncating tables or deleting rows.

Verification at local branch head 60086c1f6ee34a193c342255a5e8e6293b60e988:

• Discovery inventory: exactly 282 intended tests; all 6 hybrid/Redis tests present and structurally excluded.
• PR Centralize replaceable event persistence #6660 focused persistence tests: 4/4 passed.
• Full PostgreSQL lane: 282/282 passed three times; 1,240 skipped each run; wall time 9.72–9.96s without profiling shims.
• Non-ignored tests with a desired-state database: 1,234/1,234 passed; one mesh-demo timeout from the first run passed immediately in isolation.
• cargo fmt --all -- --check and all-target/all-feature clippy for buzz-db, buzz-deletion, and buzz-relay passed.

Final independent re-review at exact head 42097c0136ab3fa8f0efe5720ae829c04575b9b6 on separate Blox workstation 2028455: no substantive residual findings. The reviewer independently exercised ordinary strings, zero/one/three-hash raw strings, line/block comments, and string-contained lookalikes, then reran the full 353-file scan, shell syntax, Python AST parsing, and diff check.

Independent review found that the source guard's initial regular expression could miss raw-string ignore reasons and treat a commented attribute as real. Final head 42097c0136ab3fa8f0efe5720ae829c04575b9b6 parses valid ordinary/raw Rust string literals only at attributes found in comment-sanitized source; regression fixtures cover both cases. The fixture suite, Python compilation, shell syntax, diff check, and full 353-file repository scan pass on Blox.

Update — August 25, 2026 restack

PR #6660 merged, so this follow-up was rebased from its former exact parent onto current origin/main (8d2d0ff5ad42733e9949442c4b6358d0ba87f9a8). The final candidate head is 05dcc2ab28948e3ab79bb44839a69f2ba44648a2; the PR no longer carries #6660's pre-squash history.

Fresh exact-head verification on Blox workstation 2028572:

• Discovery guard passed across 353 Rust files; inventory remained exactly 282 intended PostgreSQL tests.
• PR Centralize replaceable event persistence #6660 focused persistence tests: 4/4 passed.
• Full PostgreSQL lane: 282/282 passed three times; 1,243 skipped each run; wall time 9.70–9.87s.
• cargo fmt --all -- --check and all-target/all-feature clippy for buzz-db, buzz-deletion, and buzz-relay passed.

Update — August 25, 2026 final hosted verification

• GitHub PostgreSQL Tests: 282/282 passed in 53.787s; 1,243 skipped.
• The final guard scanned all 353 Rust files successfully before the hosted lane ran.
• The shared relay/archive prerequisite and infrastructure-free Unit Tests job both passed at exact final head 42097c0136ab3fa8f0efe5720ae829c04575b9b6.

[TheSentinel454]

Checks are failing, please look into that.

[TheSentinel454]

🤖 Follow-up on the earlier “Checks are failing” review note: I fixed the workflow environment scoping and the PostgreSQL schema/isolation issues exposed by the broad lane, then reran the exact pushed head. The PostgreSQL job passed 282/282 tests, and the complete GitHub CI workflow is green: https://github.com/block/buzz/actions/runs/32867749069. An independent exact-head Blox review also found no substantive issues.

[github-actions]

🔐 Codex Security Review

Status: review required for the current range.

The current range is 571c1902d0ca55cfd4ccf6b91eeb731909cc10be...f97dc09cf989f8fb955284d271ac6be9e9b37ed9.
A new review must complete for this exact range. When manual authorization
is required, a Block organization member must comment exactly
@buzz-security-review f97dc09cf989f8fb955284d271ac6be9e9b37ed9 to authorize a new review.
Any previous review applies only to its recorded range.

[wpfleger96]

🤖 I combined three independent source/runtime passes at 8ab6bba4882e48c738e38b9e3b1b9a7ae193c20c. The new lane covers the intended PostgreSQL tests and the hosted run passed 364/364, but there are three blocking reliability/compatibility gaps to address before approval:

1. IMPORTANT — the lane flakes under its configured cold parallel load. A clean PostgreSQL 16/Redis run of the exact workflow command discovered all 364 tests but finished 363/364 because pool_observability could not create its size-one pool during the initial eight-way clone/connect burst (PoolTimedOut). That test passed alone and the warmed full lane then passed 364/364, which rules out a deterministic product failure but demonstrates nondeterministic CI failure. Remove the startup/connect race or shape/reduce concurrency, then repeat the cold full lane.

2. IMPORTANT — the documented local guard fails with supported macOS tooling. scripts/check-postgres-test-discovery.py imports Python 3.11-only tomllib, while Hermit does not pin Python and the scripts invoke unversioned python3. Following the documented command on macOS with Python 3.9.6 fails with ModuleNotFoundError: tomllib; it passes under Python 3.12. Pin Python 3.11+ in the repo toolchain or use a compatible TOML parser.

3. IMPORTANT — ordinary out-of-line Rust modules are falsely rejected. The discovery parser associates only #[path = "..."] mod postgres_tests; declarations with their files. A conventional mod postgres_tests; plus src/postgres_tests.rs layout is compiled and discovered by Rust/nextest, but the guard rejects its ignored PostgreSQL tests. Add standard Rust module-path resolution and a regression fixture.

The package finder otherwise returns the expected seven packages; the wrapper schema-mode checks pass; and the removed bespoke PostgreSQL selectors are represented in the successful hosted lane. This review applies only to the named head. Since the branch also needs a rebase, rerun the guard fixtures and a cold full PostgreSQL lane at the rebased tip before requesting approval.

[TheSentinel454]

🤖 Thanks for the thorough review — I pushed fixes for all three findings in 877e3e8d32 and 0e429bc72f.

• The observability test now gives the pool’s initial cold connection a 1s acquire budget. The timeout behavior under test is unchanged; only setup headroom changed. I restarted PostgreSQL immediately before the full eight-worker lane twice, and both cold runs passed all 367 tests (12.279s and 12.295s test time).
• Discovery no longer requires Python 3.11’s tomllib. It uses tomllib when available and a narrow Cargo package-name parser otherwise. The full discovery suite passed under actual Python 3.9.21 and with a no-tomllib regression fixture.
• The checker now resolves ordinary Rust out-of-line modules (mod postgres_tests; with either postgres_tests.rs or postgres_tests/mod.rs) as well as explicit #[path] modules, with a conventional nested-module fixture.

The GitHub checks are running on the new exact head 0e429bc72fe2147a378bc6a6981ccc917b4c2259.

[TheSentinel454]

🤖 Final update: the PR is now rebased onto current main (bc006f6708) at head 8218718738.

All review findings are addressed, including the conventional nested-module single-file case, Python 3.9/no-tomllib support, and cold PostgreSQL pool startup. While validating the final head, one slow GitHub runner exposed that forced checkpoints were consuming most of the five-minute job budget. The disposable CI PostgreSQL service now disables durability-only settings while preserving database-per-test isolation; the lane still drops isolated databases rather than truncating or deleting shared rows.

Final evidence:

• GitHub PostgreSQL job: 367/367 passed across 11 binaries in 52.114s (1m47s for the whole job): https://github.com/block/buzz/actions/runs/33425094347/job/99602824024
• Blox cold lane: 367/367 passed; durability benchmark improved 12.225s → 8.501s.
• Latest-main inventory stayed at 367; the four new base commits added ordinary unit tests, not PostgreSQL tests.
• Exact-head independent review on a separate Blox workstation found no substantive issues.
• The complete GitHub workflow is green after one unrelated desktop smoke retry.

When you have a chance, please re-review the updated head so the stale changes-requested state can be cleared.

[wpfleger96]

The three previously reported findings are addressed: the pool setup now has cold-start headroom, discovery supports Python 3.9 without tomllib, and conventional nested out-of-line modules have regression coverage. Exact-head CI is green, including 367/367 PostgreSQL tests.

IMPORTANT — migration-module tests do not receive the promised empty database. scripts/postgres-test-wrapper.sh:52-57 matches migration::postgres_tests::*, but nextest supplies the full test path, for example runtime::migration::postgres_tests::run_migrations_applies_consolidated_initial_schema_on_fresh_database. Bash case patterns match the whole string, so this falls through to schema_mode=desired and clones the desired-state template rather than template0. I reproduced that exact name through the wrapper; it emitted BUZZ_TEST_SCHEMA_MODE=desired and --template=desired_template. The current migration tests mask the defect by resetting public themselves, but a future test following crates/buzz-db/TESTING.md can begin from the wrong schema or have a helper skip migration setup.

Make the pattern prefix-aware and add a wrapper regression using the realistic fully qualified name (for example runtime::migration::postgres_tests::...) that asserts migration mode and template0. This is a correctness blocker because the lane's documented schema-isolation contract does not currently hold.

[TheSentinel454]

🤖 Addressed the latest migration-mode selector finding in 23b157eb68.

• The wrapper now normalizes NEXTEST_TEST_NAME with a leading :: and matches migration::postgres_tests at any module depth, while preserving root/nested migration_schema_ handling and ordinary desired-state behavior.
• scripts/test-postgres-test-wrapper.sh now includes the realistic fully-qualified runtime::migration::postgres_tests::run_migrations_applies_consolidated_initial_schema_on_fresh_database regression and asserts both migration mode and template0.
• The regression failed before the wrapper change and passed afterward. The real focused migration test passed through the isolation harness, and the full Blox lane passed 367/367 across 11 binaries in 12.180s. Discovery, formatting, workspace clippy, Bash syntax, diff checks, and zero-leftover-database cleanup also passed.
• An independent review on a separate Blox workstation at exact head 23b157eb68fe8c114c85997c58bf8b57f212a890 found no substantive issues and additionally verified false-positive behavior.

GitHub CI is running on the new head. Please re-review this revision when the checks complete.

[TheSentinel454]

🤖 Rebased and reverified at exact head f0453714773463770f422ff14930f2662fd590be.

• The branch was rebased onto the live main tip 2f3dd850db3afe27e56f18cbcd3548eabdd9b9c2. The CI conflict from the new workflow-provenance coverage is resolved by keeping its 21 non-ignored workflow_sink tests in backend integration while the four ignored PostgreSQL cases join the isolated PostgreSQL lane automatically, without duplicate database execution.
• The latest migration-selector finding remains fixed: fully qualified nested runtime::migration::postgres_tests::* names use template0, with both wrapper and real focused-test regressions passing.
• Exact-head Blox verification passed the discovery/wrapper guards, actionlint, formatting, workspace/all-target Clippy, the 21/21 pure workflow selector, and the complete PostgreSQL lane: 367/367 across 11 binaries in 11.993s, with zero leftover test databases.
• The actual GitHub PostgreSQL job passed 367/367 in 51.604s (1.826s schema setup; 66s for the test step): https://github.com/block/buzz/actions/runs/33438959584/job/99649638685
• A fresh independent review on separate Blox workstation buzz-tornquist-pr-6730-review-aug28 at the exact head found no substantive issues and independently confirmed the 21 pure + 4 PostgreSQL workflow-test split.

All inline review threads are resolved. The remaining changes-requested state is the stale review body for the migration selector that is now covered by the regression above; please re-review this head when convenient.

main advanced again after the exact-head review/CI run with desktop-only changes (5673c3307e, cb3144999b). They add no PostgreSQL tests, merge cleanly with this PR, and the PR remains mergeable, so I did not invalidate the completed exact-head evidence with another force-push.

[TheSentinel454]

🤖 Fresh review/update at exact head f97dc09cf989f8fb955284d271ac6be9e9b37ed9:

• Rebased cleanly onto current main at 571c1902d0ca55cfd4ccf6b91eeb731909cc10be. The 24 PR commits are patch-equivalent after the rebase.
• Main added 13 ignored PostgreSQL migration tests. They are automatically discovered by the existing structural convention, increasing the lane from 367 to 380 tests without adding names or packages to an exact selector.
• Exact-head Blox verification passed 380/380 tests across 11 binaries in 13.876s, with 0.617s schema setup and zero leftover buzz_nt_* databases. The focused new migration test also confirmed the wrapper starts nested runtime::migration::postgres_tests::* cases from template0.
• The actual GitHub PostgreSQL job passed 380/380 in 59.795s with 1.956s schema setup; the whole job took 1m41s: https://github.com/block/buzz/actions/runs/33506733417/job/99859921502
• Formatting, workspace/all-target Clippy, 21/21 infrastructure-free workflow tests, discovery and wrapper regressions, Bash/Python syntax, actionlint, and diff checks all passed.
• I reviewed the transferred rebased diff locally without touching the dirty checkout, then repeated the source/harness review on the separate exact-head Blox review workstation. No new substantive findings; all inline threads remain resolved.

The remaining changes-requested state is the stale review body for the migration selector, which is covered by the realistic wrapper regression and the focused migration test above. Please re-review this head when convenient.

[wpfleger96]

The prior migration-routing blocker is fixed. Prefixing the nextest name with :: and matching *::migration::postgres_tests::* correctly covers both root and fully qualified nested paths while preserving segment boundaries; migration_schema_* remains prefix-aware. The regression fixture now uses the real runtime::migration::postgres_tests::run_migrations_applies_consolidated_initial_schema_on_fresh_database path and asserts both BUZZ_TEST_SCHEMA_MODE=migration and template0.

I also reviewed the rebase range against the previously reviewed tip. The material differences are upstream conflict resolutions in the CI selectors/test-module classification plus this focused wrapper fix; I found no new CRITICAL or IMPORTANT issue. Exact-head CI is green: discovery scanned 392 Rust files, wrapper schema-mode checks passed, and the isolated PostgreSQL lane ran 380/380 tests successfully, including the fully qualified migration suite and cold pool-observability case.