Skip to content

feat(relay): make Postgres pool size configurable, default 50 - #3191

Merged
tlongwell-block merged 1 commit into
mainfrom
eva/db-pool-size-50
Jul 28, 2026
Merged

feat(relay): make Postgres pool size configurable, default 50#3191
tlongwell-block merged 1 commit into
mainfrom
eva/db-pool-size-50

Conversation

@tlongwell-block

Copy link
Copy Markdown
Collaborator

Summary

  • Raise the relay's Postgres pool cap from the buzz-db default of 20 to 50 per pool, and expose BUZZ_DB_POOL_SIZE for per-deploy tuning
  • Applies to the writer pool and, when READ_DATABASE_URL is set, the reader pool; zero/unparsable values fall back to the default
  • The buzz-db library default is unchanged — only the relay opts into the larger cap

Why

During the 2026-07-27 18:40–19:05Z traffic burst on bb-public, per-pod PG pools pinned at 20 fleet-wide and ~380 requests failed on the 3s acquire timeout — membership checks, channel access lookups, and historical queries returning errors to users. The database was nowhere near a limit: Aurora (db.r8g.8xlarge, ~5,000 max connections) sat at 19% CPU, 201 connections (~4% of capacity), commit latency flat at 0.01ms.

The 20-connection default was sized for "four relay pods against PG max_connections=100" (the comment in buzz-db says exactly that). Production now runs 12–15 pods against Aurora — the per-pod cap is the binding constraint, not the DB.

Budget at the new default: 15 pods × (50 writer + 50 reader + 5 audit) ≈ 1,575 potential connections, ~30% of Aurora's ceiling — and actual usage stays demand-driven (min_connections stays 2, connections only open under load).

Same shape as #2521 (BUZZ_REDIS_POOL_SIZE), which fixed the identical class of ceiling on the Redis side.

Testing

  • cargo test -p buzz-relay: 762 passed, 1 failed — the lone red is api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo, the known pre-existing flake; it fails identically on clean main at the same SHA (verified via git stash / rerun)
  • New test db_pool_size_env_override_and_invalid_fallback covers override, zero, and unparsable fallback
  • defaults_are_valid extended to pin the new default
  • cargo clippy -p buzz-relay --all-targets -- -D warnings and cargo fmt --check clean

The buzz-db default of 20 connections per pool was sized months ago
for a handful of relay pods against PG max_connections=100. Production
now runs 12+ pods against Aurora (~5,000 connections), where that cap
is the binding constraint: during the 2026-07-27 18:40Z traffic burst,
per-pod pools pinned at 20 fleet-wide and ~380 requests failed on the
3s acquire timeout while the database sat at 19% CPU and ~4% of its
connection capacity.

Raise the relay's default to 50 and expose BUZZ_DB_POOL_SIZE for
per-deploy tuning. The cap applies to the writer pool and, when
READ_DATABASE_URL is set, the reader pool. Zero or unparsable values
fall back to the default. The buzz-db library default is unchanged.

At 15 pods x (50 writer + 50 reader + 5 audit) = ~1,575 potential
connections, ~30% of Aurora's ceiling; actual usage stays demand-driven
(min_connections=2).

Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
@tlongwell-block
tlongwell-block requested a review from a team as a code owner July 27, 2026 19:42
@tlongwell-block
tlongwell-block merged commit 2ce2d71 into main Jul 28, 2026
32 checks passed
@tlongwell-block
tlongwell-block deleted the eva/db-pool-size-50 branch July 28, 2026 13:17
tlongwell-block pushed a commit that referenced this pull request Jul 28, 2026
* origin/main:
  feat(relay): make Postgres pool size configurable, default 50 (#3191)

Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>

# Conflicts:
#	crates/buzz-relay/src/main.rs
tlongwell-block pushed a commit that referenced this pull request Jul 28, 2026
* origin/main: (70 commits)
  feat(relay): make Postgres pool size configurable, default 50 (#3191)
  Publish symbol-bearing debug relay images (#3250)
  feat(tracing): add datastore tracing plumbing (#2760)
  fix(buzz-acp): accept id-keyed config options when resolving model switch (#2795)
  fix(desktop): probe legacy Goose install dir on Windows (#3248)
  refactor(desktop): extract install command execution into install_exec (#3251)
  Polish composer activity layout and transitions (#3151)
  feat(invites): add use-limited invite links (#3141)
  fix(node): bump Buzz-supplied Node runtimes past OpenClaw's >=24.15.0 floor (#3218)
  fix(desktop): preserve thread anchor through layout reflow (#3212)
  feat(search): parse from:/in:/after:/before: and pass them in the filter (#2871)
  fix(desktop): fetch join policies through native networking (#2862)
  fix(desktop): republish agent identity records when a persona rename propagates (#2607)
  fix(desktop): keep project Inbox previews compact (#3193)
  Inbox refactor (#2045)
  Fix composer selection formatting and drop overlay (#3172)
  Refine pending message status (#3153)
  feat(admin): show reported message content in report detail (#3149)
  fix(desktop): recover full local storage on startup (#3182)
  Replace mobile reconnect banners with skeleton shimmer (#3143)
  ...

Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
cameronhotchkies pushed a commit to cameronhotchkies/buzz that referenced this pull request Jul 28, 2026
…y-to-env

* origin/main: (22 commits)
  Polish mobile message and search layouts (block#3121)
  Add mobile message image galleries (block#3312)
  chore(release): release Buzz Desktop version 0.5.0 (block#3213)
  feat(relay): make Postgres pool size configurable, default 50 (block#3191)
  Publish symbol-bearing debug relay images (block#3250)
  feat(tracing): add datastore tracing plumbing (block#2760)
  fix(buzz-acp): accept id-keyed config options when resolving model switch (block#2795)
  fix(desktop): probe legacy Goose install dir on Windows (block#3248)
  refactor(desktop): extract install command execution into install_exec (block#3251)
  Polish composer activity layout and transitions (block#3151)
  feat(invites): add use-limited invite links (block#3141)
  fix(node): bump Buzz-supplied Node runtimes past OpenClaw's >=24.15.0 floor (block#3218)
  fix(desktop): preserve thread anchor through layout reflow (block#3212)
  feat(search): parse from:/in:/after:/before: and pass them in the filter (block#2871)
  fix(desktop): fetch join policies through native networking (block#2862)
  fix(desktop): republish agent identity records when a persona rename propagates (block#2607)
  fix(desktop): keep project Inbox previews compact (block#3193)
  Inbox refactor (block#2045)
  Fix composer selection formatting and drop overlay (block#3172)
  Refine pending message status (block#3153)
  ...

Signed-off-by: npub1x4hk035p3p9q39a3fcrd2fe30lpkrhr5dwe0cqzzjphxyyh8m0gsq4vqap <356f67c681884a0897b14e06d527317fc361dc746bb2fc0042906e6212e7dbd1@buzz.block.builderlab.xyz>
Co-authored-by: npub1x4hk035p3p9q39a3fcrd2fe30lpkrhr5dwe0cqzzjphxyyh8m0gsq4vqap <356f67c681884a0897b14e06d527317fc361dc746bb2fc0042906e6212e7dbd1@buzz.block.builderlab.xyz>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants