hotfix(citable): quorum guard against mostly-draft aggregator cache - #631
Merged
Conversation
- Aggregator throws AllBenchmarksDraftError when <50% of live specs produce a live bench (was: only when 0/N live). Prevents a mixed set like 2/26 live being cached for 60s while /api/stat returns live for the same slugs. - Bumped all-benchmarks cache key v13 to v14 to flush the current poisoned snapshot on prod. - Warm /api/citable, /api/llm-context, /api/mcp explicitly on prod deploy so the aggregator cache starts hot. - Added tests for the new quorum threshold.
Flotapponnier
added a commit
that referenced
this pull request
Jun 22, 2026
…HF Space description (#654) Two fixes bundled because both must land for the next prod-deploy to succeed. loadBenchmark fell back to loadAllBenchmarks (strict) when the per-slug cache miss happened. The strict aggregator throws AllBenchmarksDraftError when the quorum guard from #631 trips, which crashes next build during any Prom blackout. Both fallback sites now use loadAllBenchmarksSafe, which substitutes draft placeholders for the same blackout. Behavior at runtime is unchanged when Prom is healthy. HF Space short_description was 86 chars, HF rejects anything > 60. Shortened to 46.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problème
Sur prod, 24 des 26 benches renvoient
status=insufficientvia/api/citablealors que/api/stat/<slug>renvoiestatus=liveavec leader et value pour les mêmes slugs.Vérifié juste maintenant (12:58 UTC) sur openchainbench.com:
Cause
Les deux chemins de chargement appellent
loadBenchmarkUnfilteredCached(slug), mais le cache aggrégateurall-benchmarks-v13est indépendant du cache per-benchbench-unfiltered-v10.Au premier hit
/api/citableaprès un cold start, l'aggrégateur fan-out 26 requêtes Prom parallèles. Sous burst, plusieurs throw →draftPlaceholderForSpec→ mixed set 2 live / 24 placeholders. Le throwAllBenchmarksDraftErroractuel ne se déclenche QUE si literalement 0 bench est live, donc 2/26 passe → set caché 60s. Les hits suivants/api/statpopulent le per-bench cache avec live, mais l'aggregator cache reste sur le mixed set.Fix
aggregateBenchmarksthrow désormais quand moins de la moitié des specs déclarées live (en YAML) produisent un bench live dans le run courant. Floor de 4 pour ne pas tripper en test fixtures.all-benchmarks-v13→v14pour flush le snapshot poisoned actuel.prod-deploy.ymlwarm explicitement/api/citable,/api/llm-context,/api/mcpaprès deploy, pour que l'aggregator cache parte chaud.Tests
Ajouté coverage:
AllBenchmarksDraftErrorTest plan
prod-deploy.ymlcurl https://openchainbench.com/api/citable | jq '[.benchmarks[] | select(.status=="live")] | length'retourne ≥ 13 (sur 26)200 /api/citabledans le workflow summary