feat(supervisor): add opt-in dequeue backpressure

[nicktrn]

The supervisor can now pause dequeuing - and freeze consumer-pool scale-up - when a backpressure signal says the cluster can't place more work, then ramp dequeuing back up gradually once it clears. The signal is a verdict published to a Redis key by a cluster-side component; the supervisor reads it on a short refresh and gates preDequeue on it.

Off by default (TRIGGER_DEQUEUE_BACKPRESSURE_ENABLED). Everything fails open: a missing, stale, or unreadable verdict never pins the brake, and the hot-path read is a synchronous cached lookup with no I/O. The scale-up freeze leaves scale-down untouched, and on release the resume is ramped so a deep queue isn't hammered all at once.

Dry-run is on by default (TRIGGER_DEQUEUE_BACKPRESSURE_DRY_RUN): even once enabled it only logs what it would have done, and surfaces the computed state through metrics, until explicitly set to act. Prometheus: supervisor_backpressure_engaged, _dry_run, _skipped_dequeues_total.

Refs TRI-5354

[changeset-bot]

🦋 Changeset detected

Latest commit: e67644b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 25 packages

Name	Type
@trigger.dev/core	Patch
@trigger.dev/build	Patch
trigger.dev	Patch
@trigger.dev/plugins	Patch
@trigger.dev/python	Patch
@trigger.dev/redis-worker	Patch
@trigger.dev/schema-to-json	Patch
@trigger.dev/sdk	Patch
@internal/cache	Patch
@internal/clickhouse	Patch
@internal/llm-model-catalog	Patch
@trigger.dev/rbac	Patch
@internal/redis	Patch
@internal/replication	Patch
@internal/run-engine	Patch
@internal/schedule-engine	Patch
@internal/testcontainers	Patch
@internal/tracing	Patch
@internal/tsql	Patch
@internal/zod-worker	Patch
@internal/sdk-compat-tests	Patch
@trigger.dev/react-hooks	Patch
@trigger.dev/rsc	Patch
@trigger.dev/database	Patch
@trigger.dev/otlp-importer	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

This pull request implements a Redis-backed dequeue backpressure system for the supervisor. When enabled, the supervisor periodically reads a backpressure verdict from Redis and uses it to pause consumer scale-up and probabilistically skip dequeue attempts. The system includes verdict timestamps and staleness handling, a post-release ramp that gradually resumes dequeueing, a dry-run mode, Prometheus metrics, environment configuration, supervisor wiring to start/stop the monitor and Redis client, and a new consumer-pool shouldPauseScaling hook to freeze scale-up while allowing scale-down. The feature is opt-in and disabled by default.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	❓ Inconclusive	The description provides detailed context about the feature's behavior, configuration options, and key guarantees, but does not follow the template's checklist and testing sections.	Consider filling in the template sections: explicitly confirm the checklist items (contributing guide, PR title convention, code testing), provide a Testing section describing test steps, and ensure the Changelog section is clearly populated.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main feature being added: an opt-in dequeue backpressure mechanism for the supervisor.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/supervisor-dequeue-backpressure-tri-5354

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[pkg-pr-new]

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@c3bc59b

trigger.dev

npm i https://pkg.pr.new/trigger.dev@c3bc59b

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@c3bc59b

@trigger.dev/plugins

npm i https://pkg.pr.new/@trigger.dev/plugins@c3bc59b

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@c3bc59b

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@c3bc59b

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@c3bc59b

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@c3bc59b

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@c3bc59b

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@c3bc59b

commit: c3bc59b

[coderabbitai]

🧹 Nitpick comments (1)

apps/supervisor/package.json (1)

21-21: Confirm ioredis security status for ioredis@^5.3.2

There are no published CVEs or GitHub security advisories specifically for ioredis v5.3.2; dependency scanners may still flag indirect/transitive issues. Consider running your package manager’s audit and upgrading to the latest ioredis release to pick up dependency fixes.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 90871fd0-a183-489a-9c7d-01e39644bc6c

📥 Commits

Reviewing files that changed from the base of the PR and between 44842f9 and b73683c.

⛔ Files ignored due to path filters (1)

• pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml

📒 Files selected for processing (4)

• apps/supervisor/package.json
• apps/supervisor/src/backpressure/redisBackpressureSignalSource.test.ts
• apps/supervisor/src/backpressure/redisBackpressureSignalSource.ts
• apps/supervisor/src/index.ts

🚧 Files skipped from review as they are similar to previous changes (3)

• apps/supervisor/src/backpressure/redisBackpressureSignalSource.test.ts
• apps/supervisor/src/index.ts
• apps/supervisor/src/backpressure/redisBackpressureSignalSource.ts

📜 Review details

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (30)

• GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 8)
• GitHub Check: sdk-compat / Node.js 20.20 (ubuntu-latest)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 8)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 8)
• GitHub Check: internal / 🧪 Unit Tests: Internal (2, 8)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 8)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 8)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 8)
• GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - pnpm)
• GitHub Check: internal / 🧪 Unit Tests: Internal (8, 8)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 8)
• GitHub Check: webapp / 🧪 Unit Tests: Webapp (4, 8)
• GitHub Check: sdk-compat / Node.js 22.12 (ubuntu-latest)
• GitHub Check: internal / 🧪 Unit Tests: Internal (5, 8)
• GitHub Check: internal / 🧪 Unit Tests: Internal (3, 8)
• GitHub Check: internal / 🧪 Unit Tests: Internal (1, 8)
• GitHub Check: internal / 🧪 Unit Tests: Internal (7, 8)
• GitHub Check: internal / 🧪 Unit Tests: Internal (6, 8)
• GitHub Check: internal / 🧪 Unit Tests: Internal (4, 8)
• GitHub Check: sdk-compat / Cloudflare Workers
• GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
• GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
• GitHub Check: sdk-compat / Deno Runtime
• GitHub Check: typecheck / typecheck
• GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - npm)
• GitHub Check: sdk-compat / Bun Runtime
• GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp
• GitHub Check: packages / 🧪 Unit Tests: Packages (1, 1)
• GitHub Check: Analyze (javascript-typescript)
• GitHub Check: Build and publish previews

🧰 Additional context used

📓 Path-based instructions (1)

**/*.{js,ts,tsx,jsx,css,json,md}

📄 CodeRabbit inference engine (AGENTS.md)

Use Prettier for code formatting and run pnpm run format before committing

Files:

• apps/supervisor/package.json

🧠 Learnings (4)

📓 Common learnings

Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: packages/redis-worker/CLAUDE.md:0-0
Timestamp: 2026-03-02T12:43:43.173Z
Learning: Applies to packages/redis-worker/**/*@(job|queue|worker|background).{ts,tsx} : Use trigger.dev/redis-worker for all new background job implementations, replacing graphile-worker and zodworker

Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3625
File: apps/webapp/app/services/taskMetadataCache.server.ts:270-291
Timestamp: 2026-05-15T08:05:57.683Z
Learning: In the triggerdotdev/trigger.dev codebase, `populateByCurrentWorker()` in `apps/webapp/app/services/taskMetadataCache.server.ts` intentionally logs and swallows Redis errors rather than rethrowing. The design rationale: rethrowing would propagate into `ChangeCurrentDeploymentService.call` and break deploy promotion when Redis is briefly unavailable; the 24h `TASK_META_CACHE_CURRENT_ENV_TTL_SECONDS` TTL acts as the self-healing window for cache drift, and next-promotion overwrites the env key sooner in practice. A compensating DEL on failure is also not a win because if Redis is unreachable the DEL fails identically, and Lua scripts are atomic so a partial write is impossible. Do not flag this log+swallow pattern as a bug in future reviews.

Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: packages/redis-worker/CLAUDE.md:0-0
Timestamp: 2026-03-02T12:43:43.173Z
Learning: Applies to packages/redis-worker/**/redis-worker/src/queue.ts : Job queue abstraction should be Redis-backed in src/queue.ts

Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 3114
File: apps/supervisor/src/index.ts:83-98
Timestamp: 2026-03-27T18:11:57.032Z
Learning: In `apps/supervisor/src/index.ts`, `RESOURCE_MONITOR_ENABLED` (env var in `apps/supervisor/src/env.ts`) defaults to `false`. As a result, the local `ResourceMonitor`-based `maxResources`/`skipDequeue` gating in `preDequeue` is inactive in compute mode deployments. Do not flag local resource monitor usage in compute mode as a live bug; it has no practical impact unless `RESOURCE_MONITOR_ENABLED` is explicitly set to `true`.

Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3754
File: apps/webapp/app/v3/mollifierStaleSweepWorker.server.ts:30-32
Timestamp: 2026-06-01T12:05:44.112Z
Learning: In the triggerdotdev/trigger.dev codebase, the mollifier stale-entry sweep (`initMollifierStaleSweepWorker` in `apps/webapp/app/v3/mollifierStaleSweepWorker.server.ts`) intentionally runs per-webapp instance without a distributed lease in its initial implementation. All Redis ops (cursor, counts hash, reconcile) are individually atomic and produce correct shared state even with multiple concurrent sweepers. The known limitation is that OpenTelemetry metric output (`recordStaleEntry`, `reportStaleEntrySnapshot`) multiplies by N webapp instances, mis-calibrating alert thresholds by a factor of N. A SETNX-based per-tick lease (SET NX PX on the sweep's existing Redis) is the planned follow-up fix. Until then, alert thresholds should be scaled accordingly. Do not re-raise this as a blocking correctness bug — it is a documented metric-scaling limitation with a tracked follow-up.

Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3558
File: internal-packages/run-engine/src/run-queue/index.ts:420-424
Timestamp: 2026-05-12T06:43:12.346Z
Learning: In the triggerdotdev/trigger.dev codebase (`internal-packages/run-engine/src/run-queue/index.ts`), the established convention in `RunQueue` read-path methods (e.g., `lengthOfQueue`, `lengthOfQueues`, `currentConcurrencyOfQueues`) is to **fail open** on transient Redis pipeline errors: pipeline result errors (`baseErr`, `ctrErr`, etc.) are coerced to `0` rather than surfaced or re-thrown. This is intentional — the project treats Redis command errors the same as missing keys for these counter reads. Do not flag this pattern as a bug or suggest throwing/propagating these errors in future reviews.

Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3754
File: apps/webapp/app/env.server.ts:1104-1129
Timestamp: 2026-06-01T11:37:12.623Z
Learning: In triggerdotdev/trigger.dev (apps/webapp/app/env.server.ts), new background/periodic worker feature flags should hard-default to "0" (explicitly opt-in) rather than inheriting a parent feature flag (e.g., TRIGGER_MOLLIFIER_ENABLED). Inheriting a parent flag causes the new worker to auto-start on upgrade for any deployment that already has the parent flag enabled, turning on unexpected background load without an explicit rollout step. Each new worker component should require its own explicit opt-in via its own env var (e.g., TRIGGER_MOLLIFIER_STALE_SWEEP_ENABLED defaults to "0", not to process.env.TRIGGER_MOLLIFIER_ENABLED ?? "0").

Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3625
File: apps/webapp/app/services/taskMetadataCache.server.ts:72-84
Timestamp: 2026-05-15T08:05:54.659Z
Learning: In `triggerdotdev/trigger.dev`, the `task-meta:*` Redis keyspace in `apps/webapp/app/services/taskMetadataCache.server.ts` is fully self-owned: `RedisTaskMetadataCache` is the sole writer and sole reader of this keyspace. Do not flag the `decode()` function (which casts parsed JSON to `EncodedEntry` and wraps in try/catch) for missing Zod schema validation. The existing `JSON.parse` + `try/catch` → `null` fallback is intentional; a `null` return triggers a safe PG fallback and cache back-fill. Adding Zod validation on every `HGET` was explicitly rejected as unnecessary CPU overhead on the hot path with no real safety benefit given the single-writer contract.

Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: apps/supervisor/CLAUDE.md:0-0
Timestamp: 2026-03-02T12:42:47.652Z
Learning: SupervisorSession should manage the dequeue loop with EWMA-based dynamic scaling

📚 Learning: 2026-03-02T12:43:43.173Z

Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: packages/redis-worker/CLAUDE.md:0-0
Timestamp: 2026-03-02T12:43:43.173Z
Learning: Applies to packages/redis-worker/**/redis-worker/**/*.{test,spec}.{ts,tsx} : Use testcontainers for Redis in test files for redis-worker

Applied to files:

• apps/supervisor/package.json

📚 Learning: 2026-04-27T16:40:37.692Z

Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 3456
File: apps/webapp/package.json:230-230
Timestamp: 2026-04-27T16:40:37.692Z
Learning: In `apps/webapp/remix.config.js` (Remix 2.x, CJS server build via `serverModuleFormat: "cjs"`), ESM-only npm packages must be added to the `serverDependenciesToBundle` array so esbuild inlines them rather than emitting a `require()` call. The `engines` field allows Node >=18.19.0 || >=20.6.0, so `require(esm)` (Node 20.19+) cannot be relied upon. Packages already listed include p-limit, p-map, axios, and (as of PR `#3456`) uuid. When upgrading a dependency that drops CJS support, always check the post-build artifact for `require("<package>")` and add it to `serverDependenciesToBundle` if present.

Applied to files:

• apps/supervisor/package.json

📚 Learning: 2026-06-02T21:20:56.997Z

Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-06-02T21:20:56.997Z
Learning: Install dependencies with `pnpm i` (requires pnpm `10.33.2` and Node.js `20.20.2`)

Applied to files:

• apps/supervisor/package.json

🔇 Additional comments (1)

apps/supervisor/package.json (1)

21-21: Confirm ioredis version range is consistent across the monorepo

The monorepo’s other package.json files already use the same ioredis version range ("ioredis": "^5.3.2"), including:

• apps/webapp/package.json
• internal-packages/redis/package.json
• internal-packages/testcontainers/package.json