A transient fault poisons a promise-cache entry for the life of the process — one blip and a partition can never be provisioned again #1369

The sanctioned promise-cache recipe caches failures forever. Eight live sites hold a pool.Run(...) (or a hand-rolled ReplaySubject) in a field that is never evicted, so one transient fault poisons the entry for the life of the process — the pod must be restarted before that partition / container / connection ever works again.

This is the fourth appearance of this exact class in a week, which is what makes it a defect in the recipe, not in eight authors:

• MeshNodeStreamCache's per-path entry cache — fixed by 2776a0c (EvictFaultedEntry), whose comment already names the error class: "an Npgsql connect failure … replayed forever until a manual recycle."
• MeshNodeStreamCache._queries — CompileWatcher faults with Postgres timeout during source re-establishment query #1316, fixed by fix: a faulted synced query is evicted, not replayed forever (#1316) #1367: one slow Postgres handshake made nodetype-sources:Edu/Module permanently unreadable, and the resulting null snapshot classified as a gating PreWarmStatus.CompileError that blocked a rollout.

Why it latches

IoPoolExtensions.Run pumps the leaf into a ReplaySubject<T> and hands back subject.AsObservable(). A ReplaySubject latches terminals, OnError included — so once the cached observable faults, every later subscriber is replayed that same exception, immediately, forever. Replay(1).AutoConnect(1) / .RefCount() behave identically: the connectable is backed by one subject instance that has already terminated.

Nothing in the recipe removes the entry, so nothing ever re-attempts.

The worst instance

PostgreSqlPartitionStorageProvider.EnsurePartitionProvisioned:

return _provisioned.GetOrAdd(def.Schema!, _ =>
    _ioPool.Run(ct => EnsureSchemaAsync(def, ct)).Select(_ => Unit.Default));

One transient DDL failure — a connect blip, a lock_timeout, a momentary permission problem — permanently blocks provisioning for that partition. EnsurePartitionProvisioned is the documented gate every write to a not-yet-provisioned partition composes on (EnsurePartitionProvisioned(p).SelectMany(_ => write…)), so from that moment every write to the partition fails 42P01 undefined_table, for the process lifetime, with no self-heal. A partition that is silently and permanently broken until someone restarts the pod.

Every site

Already correct, for contrast (three different hand-rolled spellings of the same idea — more evidence the recipe should own it): OpenGraphPreviewService, ContentService.collections, GitHubAppTokenService. The first two evict by key rather than pair-exact, so a failing subscriber can evict a healthy replacement.

What the fix must be

In the recipe, not at eight call sites — otherwise a ninth arrives with the next one-shot someone writes. ControlledIoPooling.md currently documents pool.Run(...) into a ConcurrentDictionary as the way to build a one-shot; the documented way must not poison on failure.

Semantics:

• Cache success, evict failure. A retry is a new attempt, never a replay.
• Not a retry loop, timer, or poller. Eviction only means "the next caller who asks will try again"; the cache never re-attempts on its own. (The 2026-06-08 outage was a retry watchdog amplifying a mishandled error into a resubscribe storm.)
• The caller still sees the error. Evicting must not swallow the fault — the subscriber that hit it gets it; the cache just stops serving it to everyone else.
• Pair-exact eviction. Several subscribers can be attached when the fault arrives; removing by key alone can drop a healthy replacement.
• In-flight entries are never evicted — eviction is triggered by the terminal OnError, so concurrent callers keep sharing one attempt.
• Attach via Do, not a bookkeeping Subscribe: subscribing would be the AutoConnect(1) first subscriber and would connect every discarded chain.