CI's throwaway PostgreSQL runs max_worker_processes = 8, so TimescaleDB background jobs mostly cannot launch — the gated-live suite tests a configuration the product deliberately avoids #1888

Found while root-causing the #1862 CI failure. Not fixed there, because raising the limit makes background policy jobs launch RELIABLY, which is a behaviour change for the whole gated-live suite and wants its own verification pass.

The gap

The product sizes worker processes deliberately. DarlingManagedPostgres.BuildConfAppend's v2 worker block derives them from the live hypertable count:

timescaledb.max_background_workers = HypertableCount + 2
max_worker_processes               = 3 + (HypertableCount + 2) + 8

and its comment records exactly why, from a live capture: PostgreSQL's default max_worker_processes = 8 "cannot launch TimescaleDB's 26 per-hypertable compression policy jobs — the postmaster logs 'failed to launch job ... failed to start a background worker' storms and most policy runs fail (caught live: 21 failures vs 7 successes in timescaledb_information.job_stats)".

CI's throwaway cluster does not use BuildConfAppend. The Initialize and start throwaway PostgreSQL step in build.yml (and the mirror in nightly.yml) appends only three settings:

shared_preload_libraries = 'timescaledb'
port = 5541
listen_addresses = '127.0.0.1'

So the gated-live suite runs against max_worker_processes = 8 (PostgreSQL's default) with timescaledb.max_background_workers = 16 (TimescaleDB's default) — the exact under-provisioned shape the product exists to avoid.

Why it matters

The suite is testing a configuration no customer runs. Every TimescaleDB background policy — compression, retention, continuous-aggregate refresh — is launched by a background worker. On CI those launches routinely fail outright, so a large part of what the product does in the field is exercised only by the tests' explicit foreground run_job calls, never by the scheduler that actually drives it in production.

And it makes failures intermittent instead of reproducible. Whether a given job launches depends on a free slot at that instant, so a test that races the scheduler passes or fails by luck of slot availability. That is precisely how #1862's compression flake behaved: it failed twice on CI in two unrecognizably different ways, and could not be reproduced locally at all until max_worker_processes was raised on the same rig, after which the background run fired every time.

Measured on this repo's pg-runtime (PostgreSQL 18.4 + TimescaleDB 2.28.1), same cluster, only that setting changed:

What to do

Mirror the product's sizing in build.yml and nightly.yml so the suite runs the configuration the product ships. The reason this is not a one-line drive-by: it will make background jobs launch reliably for the first time, so any other test that has been quietly relying on the scheduler being unable to run is going to start failing. #1862 fixed the three compression tests that had that dependency (they now park the policy job before eligible chunks exist); whether others exist is unknown, and finding out is the actual work here.

A cheaper interim option is to leave CI as-is and treat the under-provisioning as deliberate, but then it should be written down in the workflow, because right now it reads as an oversight and the divergence from BuildConfAppend is invisible.