meta: broker-safe global artifact repository feature burndown

[zackees]

?## Context

Issue #601 started as the managed zccache sidecar install gate, then uncovered a broader broker/global-cache design problem: fbuild has large shared artifacts that must remain in one authoritative per-user artifact repository, not private stores scoped by whichever daemon/backend version the running-process broker selects.

This issue is the parent/meta burndown for that feature area. The original investigation remains on #601; implementation slices and completion state should roll up here.

Sub-Issue Map

• Parent investigation: fix: gate managed zccache sidecar install and surface blocking status to clients #601
• Original broker/cache strategy concern: fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)
• Sub-issue split: fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)
• A: daemon identity and broker resolution: fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)
• B: global artifact repository ownership: fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)
• C: download/install concurrency and client blocking: fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)
• D: compatibility, migration, and recovery policy: fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)
• Synthesis: fix: gate managed zccache sidecar install and surface blocking status to clients #601 (comment)

Goal

Make fbuild broker adoption safe for large shared artifacts by enforcing a clear local daemon/cache identity model, preserving the global artifact repository, and adding install/download gates plus client-visible wait status.

The desired invariant is:

Local fbuild has one authoritative shared artifact repository per daemon identity/cache root. Broker backend version negotiation must not create private package/toolchain/framework/sidecar stores.

Feature Burndown

1. Define daemon/cache identity

• Specify the local daemon identity tuple, likely (user, dev/prod mode, canonical FBUILD_CACHE_DIR or default cache root, trust domain).
• Document that daemon/backend version is not a cache-owner dimension.
• Decide policy for parallel daemon versions against the same cache root:
  • strict refusal / single active daemon, or
  • allowed only when an explicit cache-schema compatibility check passes.
• Keep CI EXPLICIT_INSTANCE "ci-trusted" as an intentional trust-domain exception.

2. Align broker resolution with that identity

• Ensure local SHARED_BROKER resolution converges on the daemon identity above.
• Prevent broker negotiation from routing clients to a backend whose cache root differs from the legacy direct HTTP daemon identity.
• Decide and implement replace/upgrade/refuse behavior when a newer fbuild asks for an existing cache-root identity.
• Treat broker CacheRuntime as daemon provenance only, not as an artifact repository root.
• Add tests that same FBUILD_CACHE_DIR / dev-mode inputs publish the same CacheData across broker-negotiated daemon versions.

3. Preserve global artifact repository ownership

• Document fbuild-owned global roots for package/toolchain/framework archives, managed sidecars, cache data, indexes, temp, logs, locks, runtime, and config.
• Clarify which roots may be shared across daemon versions and which are per-runtime/provenance only.
• Audit code paths that could accidentally introduce broker-instance or daemon-version path segments into artifact storage.
• Add regression tests around fbuild_paths::get_cache_root() / FBUILD_CACHE_DIR and CacheManifest output.

4. Add shared install/download gates

• Add a process-shared managed zccache install lock before network fetch begins.
• Add package/toolchain/framework install gates below project-lock level so different projects do not race on the same global package root.
• Re-check final install state while holding each install lock before downloading.
• Prefer unique staging dirs plus atomic final rename for all large installs, matching the safer managed zccache pattern.
• Recover cleanly from stale _staging dirs, partial sidecar installs, interrupted downloads, and dead installers.

5. Surface blocked-client status

• Expose structured wait/install state: lock name, dependency/package name, version, phase, active installer vs waiter, and human message.
• Broadcast phase changes over /ws/status.
• Include dependency wait/install state in /health, /api/daemon/info, or a dedicated status endpoint.
• Emit periodic NDJSON/status events for streaming builds while waiting on project/package/sidecar locks.
• Make non-streaming clients avoid request timeout when intentionally waiting behind another install.

6. Compatibility and recovery guardrails

• Keep broker version-block/refusal fatal instead of falling through into an incompatible daemon.
• Add cache schema/version compatibility checks before allowing multiple fbuild versions to share a repository.
• Guard disk-cache migration ID immutability.
• Add stale dead-PID lease recovery tests.
• Add interrupted install recovery tests for managed zccache and package/toolchain/framework installs.
• Add CI explicit-instance manifest tests so ci-trusted isolation stays intentional and local shared stays shared.

Acceptance Criteria

• Local fbuild has a documented daemon/cache identity contract.
• Broker-negotiated local daemons cannot silently fork package/toolchain/framework/sidecar artifacts into private versioned stores.
• Same cache-root identity resolves to one compatible daemon policy: reuse, upgrade/replace, or clear refusal.
• Concurrent fbuild clients/projects do not duplicate large downloads for the same package/toolchain/framework/sidecar.
• Blocked clients receive clear status and do not appear hung while waiting on another fbuild operation.
• Recovery from stale locks, partial installs, and interrupted downloads is covered by tests.
• CI trust-domain isolation remains explicitly tested and separate from local shared-cache behavior.

Related

• fix: gate managed zccache sidecar install and surface blocking status to clients #601 tracks the original managed zccache install-gate investigation and broker/cache discussion.
• chore: remove zccache crate linkage and use the managed sidecar for cache APIs #600 removed compile-time zccache crate linkage; this tracker is about runtime broker/cache/install policy.

[zackees]

Merged implementation slice #604 (feat: define broker cache identity) into main.

Landed in this slice:

• local daemon/cache identity contract and broker cache-ownership labels
• cache schema/version diagnostics for future compatibility checks
• CI ci-trusted explicit-instance manifest helpers
• managed zccache and package install gates before network download/extract
• dependency install status propagation through daemon info, websocket/status snapshots, streaming build status events, and CLI status output
• non-streaming build/API wait behavior that avoids client-side total timeout while blocked behind intentional install contention
• CLI/Python broker daemon cache identity/schema verification before reuse

This advances #603 but does not close it. Remaining parent-burndown work still includes final resolver policy for parallel daemon versions, deeper cache-schema compatibility rules, broader interrupted-install/dead-lease recovery coverage, and any follow-on tests/docs needed to mark the meta issue complete.

[zackees]

Merged recovery/compatibility guardrail slice #607 into main.

Landed in this slice:

• locked existing disk-cache migration ids/order as an append-only compatibility prefix
• added a dead-PID lease reaping regression test that proves stale lease rows are removed and entries.pinned is recalculated

This advances the #603 recovery guardrails, specifically migration ID immutability and stale dead-PID lease recovery coverage. It does not close #603; remaining parent-burndown work still includes final parallel-daemon resolver policy, deeper cache schema compatibility behavior, interrupted install recovery coverage, and any remaining docs/tests needed for the full broker-safe global artifact repository feature.

[zackees]

Progress update: opened #609 for the interrupted-install recovery guardrail.

This adds regression coverage that a recorded package/toolchain/framework-style install directory without .install_complete is treated as incomplete during disk-cache reconciliation: the directory is removed and the cache index no longer reports it as installed.

Validation run:

• cargo test -p fbuild-packages disk_cache::gc --lib
• cargo fmt --all --check
• git diff --check