Skip to content

feat(host-service): generic worker_threads pool — git status + commit-files off the event loop - #5750

Merged
Kitenite merged 5 commits into
mainfrom
feat-host-service-worker-pool
Jul 19, 2026
Merged

Kitenite merged 5 commits into
mainfrom
feat-host-service-worker-pool

Conversation

@Kitenite

@Kitenite Kitenite commented Jul 17, 2026 •

Copy link
Copy Markdown
Member

Links

Summary

  • Adds a generic worker_threads pool inside host-service (packages/host-service/src/workers/) and moves git.getStatus + git.getCommitFiles compute onto it. The worker spawns the git subprocesses itself, so stdout draining AND parsing leave the event loop that also relays terminal I/O and serves all tRPC/WS for the org.
  • v1 had exactly this offload (git-task-worker); v2 lost it. Stress tests showed background getStatus queueing 3.7–14.6s under 8-workspace churn — this PR is the infrastructure that makes raising the limiter's concurrency safe (deferred, see Follow-ups).

How It Works

  • Runner: WorkerTaskRunner ported from v1 desktop (v1 keeps its copy; it retires with v1) with two changes: idle reaping (workers terminated after 30s idle — host-service is per-org and long-lived, a 3-org machine must not hold 12 warm workers) and demand-sized spawning (v1 spawned the full pool on any task).
  • Static registry: task modules export defineWorkerTask<In, Out> defs, namespaced git/*; host-worker.ts imports them into a build-time registry. Callers pass the same def to HostWorkerPool.run() for end-to-end typing. Adding a domain = one task module + one import.
  • Inline fallback: missing bundle → run handlers on the main thread (pre-PR behavior) with one warning; worker crash → task retried inline, 3 crashes/60s opens a circuit to inline-only. Pool infrastructure can never fail a caller.
  • Credential env: createGitEnvResolver split out of createGitFactory — the env is resolved in-process (needs the credential provider) and crosses to the worker as plain data; the worker builds its own SimpleGit.
  • Bundling (side-by-side with host-service.js, same pattern as pty-daemon): electron-vite input via apps/desktop/src/main/host-worker/index.ts, second Bun.build in packages/host-service/build.ts, copy in packages/cli/scripts/build-dist.ts. Resolution: SUPERSET_HOST_WORKER_SCRIPT_PATH → side-by-side → workspace dist → inline fallback.

Manual QA Checklist

  • Dev app E2E: git.getStatus on the superset repo through the real app returns a correct snapshot in worker mode (435ms; zero [host-worker-pool] fallback warnings in the host-service log; dist/main/host-worker.js emitted)
  • Standalone bundle: bun run build:host emits dist/host-worker.js; plain-node smoke (new Worker('./dist/host-worker.js')) round-trips the task protocol (ESM entry works under node)

Testing

  • src/workers/host-worker-pool.test.ts (6 tests): worker↔inline parity on a real fixture repo (modified/staged/untracked buckets), unknown-task error propagation, idle reaping, missing-bundle inline fallback, crash → inline retry → circuit open. Mutation-verified: disabling reaping fails the reap test; removing the inline retry fails the crash test.
  • src/workers/no-native-worker-imports.test.ts: worker graph stays free of native addons and process singletons (db/events/daemon); transitive regressions fail the host-worker bundle build.
  • Full host-service suite (904 tests), bun run typecheck (host-service + desktop), bun run lint clean.

Design Decisions

  • Copy the v1 runner instead of extracting a shared package: v1 is sunset; its copy dies with it (repo v1/v2 duplication policy).
  • Handlers ship in both the main bundle and the worker bundle: the inline fallback requires them in-process anyway; the git helpers were already in the main bundle.
  • Pool concurrency mirrors the limiter's min(4, cpus−1): the limiter stays the only queue for status refreshes; pooled tasks never meaningfully queue behind it.

Known Limitations

  • scheduleBaseRefFetch's dedupe state is per-thread now — N workers can, worst case, schedule up to N base-ref fetches for the same worktree within one freshness window. Bounded and rare; noting rather than complicating the payload.

Follow-ups

  • Raise gitStatusRefreshLimiter concurrency now that snapshots don't cost loop time (measure first)
  • GitWatcher rescan/filtering offload — the actual loop-stall driver under many-workspace churn (per the measured findings)
  • getDiff / listCommits / getBranchSyncStatus tenants

Open in Stage

Summary by cubic

Introduces a generic worker_threads pool for packages/host-service and moves git.getStatus and git.getCommitFiles off the event loop to cut stalls and improve responsiveness. Addresses SUPER-1544 by restoring v1’s offload pattern, adding commit-file task deduping ahead of a small cap, and hardening the inline fallback (sync-throw safe, coalesced inline crash-retry).

  • New Features

    • Added WorkerTaskRunner pool with demand-based spawning and 30s idle reaping; concurrency defaults to min(4, cpus−1)+2.
    • Inline fallback and crash handling: worker crashes retry inline; coalesced crashes count once; after 3 crashes/60s open an inline-only circuit and reject the queued backlog; inline path mirrors timeout/abort semantics and is sync-throw safe; re-drains after postMessage failures so bad payloads don’t wedge a slot.
    • Typed task registry and worker entry for git/* tasks; createGitEnvResolver resolves credential env in-process and passes plain data to the worker.
    • Switched tRPC git.getStatusSnapshot and git.getCommitFiles to the pool with 15s timeouts; commit-file tasks capped at 2 and deduped by key outside the cap so identical requests share one slot and one run.
  • Dependencies

    • Built and ship host-worker.js side-by-side with host-service.js:
      • apps/desktop adds src/main/host-worker/index.ts to electron-vite.
      • packages/host-service/build.ts emits host-worker.js (ESM); packages/cli/scripts/build-dist.ts copies it next to host-service.js.
    • Added exports entry ./host-worker in packages/host-service/package.json and resolution order SUPERSET_HOST_WORKER_SCRIPT_PATH → side-by-side → workspace dist → inline fallback.

Written for commit c6ecda6. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • New Features
    • Git status and commit-file operations now run in the background for improved app responsiveness.
    • Added support for efficient handling of repeated Git requests.
  • Bug Fixes
    • Improved resilience when background processing is unavailable or interrupted by automatically retrying operations through a fallback path.
    • Added safeguards for timeouts, cancellation, and worker failures.
  • Chores
    • Updated desktop and distribution packaging to include the required background-processing support.

@coderabbitai

coderabbitai Bot commented Jul 17, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: cbe74c6a-b623-4d23-bf13-4ca45825e9c5

📥 Commits

Reviewing files that changed from the base of the PR and between 8034da9 and c6ecda6.

📒 Files selected for processing (2)
  • packages/host-service/src/workers/host-worker-pool.test.ts
  • packages/host-service/test/integration/no-native-worker-imports.test.ts

📝 Walkthrough

Walkthrough

Adds a worker-thread runtime for host-service Git operations, including typed task contracts, pooling, retries, fallbacks, crash handling, packaging, and integration tests. Git status and commit-file router procedures now dispatch work through the host worker pool.

Changes

Host worker Git execution

Layer / File(s) Summary
Worker contracts and Git tasks
packages/host-service/src/workers/worker-task-protocol.ts, packages/host-service/src/workers/define-worker-task.ts, packages/host-service/src/workers/tasks/git.ts
Defines typed task messages, serialized errors, task declarations, and Git status/commit-file handlers.
Worker task runtime
packages/host-service/src/workers/WorkerTaskRunner.ts
Adds worker spawning, queue strategies, coalescing, latest-wins handling, timeouts, aborts, crash reporting, idle reaping, and disposal.
Host worker dispatch and fallback
packages/host-service/src/workers/host-worker.ts, packages/host-service/src/workers/host-worker-pool.ts
Registers worker tasks and provides script resolution, inline execution, crash retries, circuit breaking, and pooled task access.
Git router integration
packages/host-service/src/runtime/git/*, packages/host-service/src/trpc/router/git/git.ts
Extracts Git environment resolution and routes status and commit-file operations through worker tasks with concurrency and request deduplication.
Worker bundle packaging
packages/host-service/package.json, packages/host-service/build.ts, apps/desktop/..., packages/cli/scripts/build-dist.ts
Exports, bundles, emits, and stages host-worker.js alongside the host-service bundle.
Worker pool validation
packages/host-service/src/workers/host-worker-pool.test.ts, packages/host-service/src/workers/test-fixtures/*, packages/host-service/test/integration/*
Tests worker parity, fallback behavior, lifecycle handling, crash recovery, queue draining, payload failures, and worker import restrictions.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant GitRouter
  participant HostWorkerPool
  participant WorkerTaskRunner
  participant HostWorker
  participant SimpleGit
  GitRouter->>HostWorkerPool: Submit Git task and gitEnv
  HostWorkerPool->>WorkerTaskRunner: Queue task with timeout and dedupe options
  WorkerTaskRunner->>HostWorker: Send task payload
  HostWorker->>SimpleGit: Execute Git operation
  SimpleGit-->>HostWorker: Return Git result
  HostWorker-->>WorkerTaskRunner: Send success or serialized error
  WorkerTaskRunner-->>HostWorkerPool: Resolve or reject task
  HostWorkerPool-->>GitRouter: Return Git result or inline fallback result
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 16.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is a clear conventional commit that matches the main change: adding a host-service worker_threads pool for git work.
Description check ✅ Passed The description covers the change, implementation, testing, QA, limitations, and follow-ups; only the template’s exact checklist section is missing.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat-host-service-worker-pool

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ghost

ghost commented Jul 17, 2026 •

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Jul 17, 2026 •

Copy link
Copy Markdown
Contributor

🚀 Preview Deployment

🔗 Preview Links

Service Status Link
Neon Database (Neon) ✅ View Branch
Vercel API (Vercel) ✅ Open Preview
Vercel Web (Vercel) ✅ Open Preview
Vercel Marketing (Vercel) ✅ Open Preview
Vercel Admin (Vercel) ✅ Open Preview
Vercel Docs (Vercel) ✅ Open Preview

Preview updates automatically with new commits

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (2)
packages/host-service/src/workers/no-native-worker-imports.test.ts (1)

11-47: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Move this cross-cutting invariant test to the integration-test directory.

This scans the entire worker subtree rather than testing one co-located source file. Move it under packages/host-service/test/integration/ and update WORKERS_DIR to resolve packages/host-service/src/workers.

Based on learnings, cross-cutting host-service invariant scans belong under packages/host-service/test/integration/.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/host-service/src/workers/no-native-worker-imports.test.ts` around
lines 11 - 47, Move the worker graph purity test containing `FORBIDDEN`, `walk`,
and the `worker graph purity` suite into
`packages/host-service/test/integration/`. Update `WORKERS_DIR` to resolve the
host-service `src/workers` directory from its new location, while preserving the
existing scan patterns and assertions.

Source: Learnings

packages/host-service/src/workers/host-worker-pool.test.ts (1)

79-90: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Avoid a real 600ms wait in this test.

Use a fake clock or event-loop-based synchronization with a zero/minimal configured idle timeout. The current wall-clock sleep slows the suite and remains scheduling-sensitive.

Based on learnings, Superset test code should avoid real wall-clock timers.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/host-service/src/workers/host-worker-pool.test.ts` around lines 79 -
90, Update the “idle workers are reaped after idleTimeoutMs” test to avoid the
real 600ms sleep and scheduling-sensitive assertion. Configure a zero or minimal
idle timeout in makePool, then use fake-clock advancement or event-loop
synchronization to trigger reaping deterministically while preserving the
assertions that one worker is created and eventually reduced to zero.

Source: Learnings

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/host-service/src/workers/host-worker-pool.ts`:
- Around line 141-148: Update the circuit-opening logic in the host worker pool
to set inlineOnly and route future calls inline without immediately disposing
this.runner. Defer runner retirement and disposal until all outstanding calls on
the existing runner have settled, so queued and active tasks can complete
normally rather than receiving WorkerTaskAbortedError.
- Around line 110-123: Update the inline fallback paths in the host worker pool,
including the no-runner branch and the WorkerTaskError crash branch, to use a
shared inline executor instead of calling def.handler directly. Have that
executor honor WorkerTaskOptions timeoutMs and abort signal semantics so
already-aborted requests do not start and inline execution times out
consistently with runner execution.

In `@packages/host-service/src/workers/WorkerTaskRunner.ts`:
- Around line 289-295: Update the task dispatch flow in WorkerTaskRunner around
the WorkerTaskRequestMessage and slot.worker.postMessage call to catch
synchronous postMessage failures. On failure, reject and clean up the active
task, release the worker slot, and continue draining the queue without allowing
the exception to escape the event handler.

---

Nitpick comments:
In `@packages/host-service/src/workers/host-worker-pool.test.ts`:
- Around line 79-90: Update the “idle workers are reaped after idleTimeoutMs”
test to avoid the real 600ms sleep and scheduling-sensitive assertion. Configure
a zero or minimal idle timeout in makePool, then use fake-clock advancement or
event-loop synchronization to trigger reaping deterministically while preserving
the assertions that one worker is created and eventually reduced to zero.

In `@packages/host-service/src/workers/no-native-worker-imports.test.ts`:
- Around line 11-47: Move the worker graph purity test containing `FORBIDDEN`,
`walk`, and the `worker graph purity` suite into
`packages/host-service/test/integration/`. Update `WORKERS_DIR` to resolve the
host-service `src/workers` directory from its new location, while preserving the
existing scan patterns and assertions.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4bf7fcc7-293d-4d3f-b6e2-9d0014785dde

📥 Commits

Reviewing files that changed from the base of the PR and between 7a4e275 and d1922cd.

📒 Files selected for processing (17)
  • apps/desktop/electron.vite.config.ts
  • apps/desktop/src/main/host-worker/index.ts
  • packages/cli/scripts/build-dist.ts
  • packages/host-service/build.ts
  • packages/host-service/package.json
  • packages/host-service/src/runtime/git/git.ts
  • packages/host-service/src/runtime/git/index.ts
  • packages/host-service/src/trpc/router/git/git.ts
  • packages/host-service/src/workers/WorkerTaskRunner.ts
  • packages/host-service/src/workers/define-worker-task.ts
  • packages/host-service/src/workers/host-worker-pool.test.ts
  • packages/host-service/src/workers/host-worker-pool.ts
  • packages/host-service/src/workers/host-worker.ts
  • packages/host-service/src/workers/no-native-worker-imports.test.ts
  • packages/host-service/src/workers/tasks/git.ts
  • packages/host-service/src/workers/test-fixtures/crashing-worker.ts
  • packages/host-service/src/workers/worker-task-protocol.ts

@greptile-apps

greptile-apps Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR introduces a generic worker_threads pool inside host-service and moves git.getStatus and git.getCommitFiles off the main event loop. Credential env resolution is split from createGitFactory so credentials can be serialized to plain data and sent to the worker, which builds its own SimpleGit and runs git subprocesses entirely off-thread.

  • WorkerTaskRunner (ported from v1 desktop): demand-sized spawning, idle reaping after 30 s, fifo/coalesce/latest-wins queue strategies, timeout + abort, and full slot lifecycle guarding.
  • HostWorkerPool: lazy singleton wrapping the runner; script resolution mirrors the pty-daemon pattern; missing bundle or crash-loop degrades gracefully to inline execution rather than failing callers.
  • Bundling: host-worker.ts added as a second electron-vite entry and a second Bun.build output, emitted side-by-side with host-service.js in both the desktop and standalone CLI distributions.

Confidence Score: 4/5

Safe to merge with awareness of the crash-accounting issue; callers always get correct results via inline fallback, but the pool can be permanently switched to inline-only faster than intended when concurrent coalesced requests meet a crashing worker.

The crash timestamp counter in recordCrash() is incremented once per pool.run() caller, not once per actual worker crash. When the coalesce strategy is active, every concurrent coalesced caller independently catches the WorkerCrashedError and calls recordCrash(), potentially opening the circuit from a single actual crash and permanently disabling the worker offload for the process lifetime.

packages/host-service/src/workers/host-worker-pool.ts — specifically the run() / recordCrash() interaction under coalesce concurrency

Important Files Changed

Filename Overview
packages/host-service/src/workers/host-worker-pool.ts Core pool class with crash-circuit logic; the coalesce+crash timestamp inflation bug means a single worker crash can permanently disable the pool under concurrent coalesced requests
packages/host-service/src/workers/WorkerTaskRunner.ts Solid port of v1 runner with idle reaping; coalesce/latest-wins/abort handling all guarded correctly; exit-code-0 is treated as crash (minor)
packages/host-service/src/workers/tasks/git.ts Clean task definitions; handlers build their own SimpleGit with the env received from the main thread; no native/singleton imports
packages/host-service/src/workers/host-worker.ts Minimal dispatch loop over a build-time registry; duplicate-type guard on startup; error serialization on uncaught handler errors
packages/host-service/src/trpc/router/git/git.ts Routes getStatus and getCommitFiles through the pool; credentials resolved in-process and passed as plain env; dedupeKey construction is correct
packages/host-service/src/runtime/git/git.ts Clean extraction of createGitEnvResolver from createGitFactory; semantically equivalent to original, enables env-as-plain-data crossing to worker thread
packages/host-service/src/workers/host-worker-pool.test.ts Good coverage: parity test on real fixture repo, idle reap, missing-bundle fallback, crash to inline retry to circuit open; coalesce+crash path not tested
packages/host-service/src/workers/worker-task-protocol.ts Well-typed wire protocol; discriminated union for ok/error responses; serializeWorkerError handles non-Error throws correctly
packages/host-service/src/workers/define-worker-task.ts Simple identity wrapper for worker task definitions; provides end-to-end generic typing between caller and handler
packages/host-service/build.ts Adds second Bun.build entry for host-worker.ts; exits on failure; mirrors existing host-service.js build pattern
packages/cli/scripts/build-dist.ts Adds cpSync of host-worker.js alongside host-service.js; consistent with existing pty-daemon side-by-side copy pattern
apps/desktop/electron.vite.config.ts Adds host-worker as a second electron-vite main-process entry alongside pty-daemon; mirrors that pattern exactly
packages/host-service/src/workers/no-native-worker-imports.test.ts Grep-based guard for direct native/singleton imports in src/workers/**; correctly relies on bundle build for transitive regressions

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant tRPC as tRPC Router
    participant Pool as HostWorkerPool
    participant Runner as WorkerTaskRunner
    participant Worker as Worker Thread
    participant Git as git subprocess

    tRPC->>tRPC: createGitEnvResolver resolves credentials in-process
    tRPC->>Pool: pool.run(gitStatusSnapshotTask, input)
    Pool->>Runner: runner.runTask(taskType, payload)
    Note over Runner: coalesce returns inFlight promise if same dedupeKey
    Runner->>Worker: postMessage task request
    Worker->>Git: SimpleGit with env from payload
    Git-->>Worker: parsed result
    Worker-->>Runner: postMessage result
    Runner-->>Pool: resolve
    Pool-->>tRPC: GitStatusSnapshot

    alt Worker crash
        Worker-->>Runner: exit event
        Runner-->>Pool: reject WorkerCrashedError
        Pool->>Pool: recordCrash
        Pool->>Pool: def.handler inline retry
        Pool-->>tRPC: result via inline fallback
    end

    alt Crash budget exceeded
        Pool->>Pool: "inlineOnly = true permanently"
        Note over Pool: all future tasks run inline
    end
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant tRPC as tRPC Router
    participant Pool as HostWorkerPool
    participant Runner as WorkerTaskRunner
    participant Worker as Worker Thread
    participant Git as git subprocess

    tRPC->>tRPC: createGitEnvResolver resolves credentials in-process
    tRPC->>Pool: pool.run(gitStatusSnapshotTask, input)
    Pool->>Runner: runner.runTask(taskType, payload)
    Note over Runner: coalesce returns inFlight promise if same dedupeKey
    Runner->>Worker: postMessage task request
    Worker->>Git: SimpleGit with env from payload
    Git-->>Worker: parsed result
    Worker-->>Runner: postMessage result
    Runner-->>Pool: resolve
    Pool-->>tRPC: GitStatusSnapshot

    alt Worker crash
        Worker-->>Runner: exit event
        Runner-->>Pool: reject WorkerCrashedError
        Pool->>Pool: recordCrash
        Pool->>Pool: def.handler inline retry
        Pool-->>tRPC: result via inline fallback
    end

    alt Crash budget exceeded
        Pool->>Pool: "inlineOnly = true permanently"
        Note over Pool: all future tasks run inline
    end
Loading
Prompt To Fix All With AI
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
packages/host-service/src/workers/host-worker-pool.ts:105-127
**Coalesced callers each record a crash on one worker failure**

When `strategy === "coalesce"` is used (e.g. `gitCommitFilesTask`), `runner.runTask()` returns the same shared `taskPromise` to every waiting caller. When the worker crashes, that promise rejects with `WorkerCrashedError` and _every_ caller's independent `pool.run()` catch-block fires, each calling `recordCrash()`. A single worker crash can therefore push N timestamps into `crashTimestamps` — one per concurrent coalesced caller. With `CRASH_BUDGET = 3`, three simultaneous `getCommitFiles` tRPC requests for the same `workspaceId:commitHash` pair hitting a single crashing worker opens the circuit immediately and permanently disables the worker pool for the rest of the process lifetime.

### Issue 2 of 2
packages/host-service/src/workers/WorkerTaskRunner.ts:246-258
**Graceful exit (code 0) while running a task is treated as a crash**

The `exit` handler always calls `handleWorkerFailure` with `WORKER_CRASH_ERROR_NAME`, even for exit code 0. If a future task handler or library somehow causes a clean `process.exit(0)` mid-task, the active task gets a `WorkerCrashedError`, `HostWorkerPool.recordCrash()` fires, and the crash budget is consumed. The `error` event guard only fires for uncaught exceptions, not for intentional exits. Checking `if (code !== 0 || slot.activeTaskId)` before treating the exit as a crash would narrow the crash-accounting to genuinely unexpected exits.

Reviews (1): Last reviewed commit: "feat(host-service): generic worker_threa..." | Re-trigger Greptile

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 17 files

You’re at about 93% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Architecture diagram
sequenceDiagram
    participant Client as tRPC Client
    participant TRPC as tRPC Router
    participant Pool as HostWorkerPool
    participant Runner as WorkerTaskRunner
    participant Worker as worker_threads
    participant Git as SimpleGit (off-thread)
    participant Inline as Inline Fallback Handler
    participant Cred as Credential Provider

    Note over Client,Cred: NEW: git.getStatusSnapshot & git.getCommitFiles via worker pool

    Client->>TRPC: git.getStatusSnapshot(input)
    TRPC->>Cred: resolveGitTaskEnv(worktreePath)
    Cred-->>TRPC: credential env data

    TRPC->>Pool: run(gitStatusSnapshotTask, { worktreePath, gitEnv })

    alt Bundle found & circuit open
        Pool->>Runner: runTask("git/getStatusSnapshot", payload)
        Runner->>Runner: ensureWorkerCapacity()
        Runner->>Worker: postMessage({ kind: "task", taskType, payload })
        Worker->>Worker: look up handler in registry
        Worker->>Git: createUserSimpleGit().env(gitEnv)
        Git->>Git: git status / diff (off event loop)
        Git-->>Worker: result
        Worker-->>Runner: postMessage({ kind: "result", ok: true, result })
        Runner-->>Pool: resolved
        Pool-->>TRPC: status snapshot
    else Worker crashes
        Worker->>Worker: exit/error
        Runner-->>Pool: WorkerCrashedError
        Pool->>Pool: recordCrash()
        Pool->>Inline: def.handler(input) [retry inline once]
        alt Crash budget exhausted
            Pool->>Pool: inlineOnly = true, dispose runner
        end
        Inline-->>Pool: result
        Pool-->>TRPC: status snapshot
    else Bundle not found
        Pool->>Pool: inlineOnly = true
        Pool->>Inline: def.handler(input)
        Inline-->>Pool: result
        Pool-->>TRPC: status snapshot
    end

    TRPC-->>Client: status snapshot

    Note over Client,CRED: NEW: Idle worker reaping (30s)

    Runner->>Runner: scheduleIdleReaping()
    alt Worker idle for 30s
        Runner->>Runner: clear idle timer
        Runner->>Worker: terminate()
    end

    Note over Client,CRED: NEW: Coalesce deduplication for getCommitFiles

    Client->>TRPC: git.getCommitFiles(input)
    TRPC->>Pool: run(gitCommitFilesTask, ..., { dedupeKey, strategy: "coalesce" })
    alt In-flight request with same dedupeKey
        Pool->>Pool: return existing promise
        Pool-->>TRPC: coalesced result
    else No in-flight request
        Pool->>Worker: execute normally
        Worker-->>Pool: result
        Pool-->>TRPC: result
    end
    TRPC-->>Client: commit files
Loading

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

@Kitenite

Copy link
Copy Markdown
Member Author

CDP stress re-run: 8-workspace × 20k-file churn, before vs after this PR

Same harness as the pre-pool measurement (200 tracked-file appends/500ms per worktree × 8, background getStatus pollers on all 8 every 2s, direct HTTP/WS instrumentation, loop probe = trivial-route RTT @100Ms). Worker mode confirmed for the whole run (zero [host-worker-pool] fallback warnings).

churn-8 + pollers main (pre-pool) this PR
event-loop probe p95 / max 27.9 / 183 ms 5.2 / 17.3 ms
cheap tRPC RTT p50 / p95 4.1 / 127 ms 1.0 / 3.8 ms
terminal echo RTT p50 / p95 5.7 / 19.9 ms 4.8 / 13.1 ms
background poll latency p50 / p95 / max 3.7 / 10.6 / 14.6 s 3.1 / 9.3 / 11.3 s
host-service CPU p50 / max 29 / 52% 35 / 57%

Takeaways:

  • The pool delivers what it targets: loop availability under churn improved ~5–30× (probe max 183→17ms, query p95 127→3.8ms). The loop no longer pays snapshot parse costs.
  • Background poll latency barely moved — as predicted, it's gitStatusRefreshLimiter queueing at concurrency 4, which is exactly the follow-up this PR unblocks.
  • Not apples-to-apples on absolute getStatus wall times (fixture warmth and diff-accumulation timing differ between runs; snapshots now also run 4-way truly parallel, contending on disk). A controlled per-snapshot latency measurement should accompany the limiter-concurrency raise.

Idle baseline on this branch for reference: loop probe p95 1.9ms, query p50 0.7ms, echo p50 2.0ms.

…ut/abort, crash dedupe, no-abort circuit, postMessage cleanup
@Kitenite

Copy link
Copy Markdown
Member Author

Review findings addressed in 4ccc47d

Finding (reviewer) Resolution
Circuit-open disposes the runner and aborts its in-flight tasks (CodeRabbit, cubic P1) Fixed — the runner is detached, not disposed: outstanding tasks settle on their own merits, idle reaping cleans up the threads, and pool.dispose() still disposes detached runners.
One worker crash recorded once per coalesced caller (greptile P1) Fixed — coalesced callers share one error instance; recordCrash dedupes by error identity (WeakSet). Regression test uses 3 coalesced callers = exactly the old budget.
Inline fallback ignores timeoutMs/abort (CodeRabbit, cubic) Fixed — runInline mirrors worker semantics: pre-aborted → immediate abort error, timeoutMs races the handler (handler isn't cancellable, same as a worker task whose promise settles first).
postMessage throw wedges the slot until timeout (CodeRabbit, cubic P2) Fixed — caught in drainQueue: the task rejects with a descriptive error, the slot is freed immediately. Test proves the next worker round-trip is prompt.
getCommitFiles can occupy all workers ahead of getStatus (cubic P2) Mitigated — pool concurrency now carries +2 headroom over the limiter's min(4, cpus−1); idle reaping keeps the extra threads free when unused. A real priority lane is noted as a follow-up alongside the limiter-concurrency raise.
Exit code 0 mid-task counted as crash (greptile P2) Skipped intentionally — a worker never legitimately process.exit(0)s while holding a task; if it does, the task genuinely died and inline retry + crash accounting is the correct response.

All 9 pool tests pass (3 new regression tests, each verified to fail against the pre-fix code); typecheck + lint clean.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 3 files (changes from recent commits).

You’re at about 94% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

…ed backlog at circuit-open, inline timeout parity, commit-files cap

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files (changes from recent commits).

You’re at about 99% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

…ash-retry, dedupe commit-files ahead of the cap
@Kitenite
Kitenite merged commit 727af9d into main Jul 19, 2026
12 of 13 checks passed

This branch was previously deployed

1 inactive deployment
preview — c6ecda61 Deployed Jul 19, 2026 by Kitenite via Deploy Marketing #6634
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant