fix(ci): shallow-clone benchmark uploads with 6-month retention - #24806
Merged
Merged
Conversation
The 'Upload benchmarks' steps used github-action-benchmark, which does a full clone of AztecProtocol/benchmark-page-data per upload. That repo has grown to 3.9GB of append-only gh-pages history, making each upload take ~8 minutes and every upload slower than the last. Replace the action with ci3/upload_benchmarks, which writes the same window.BENCHMARK_DATA format but: - appends via a shallow, blob-filtered, sparse clone of just the target data dir (~4s / 89MB today vs ~8min / 3.9GB) - in --maintain mode (bench/next uploads only) prunes data dirs with no update in 6 months and squashes history older than 6 months into a single root, so the repo can no longer grow without bound. Squashing uses a blob-less clone and rewrites only commit objects (trees and blobs are content-addressed and reused), rate-limited to monthly via a stamp file and guarded by force-with-lease against concurrent uploads. ci3/upload_benchmarks_test covers append, squash+prune, idempotence, new-dir initialization, and MAX_ITEMS trimming against a local fixture repo.
ludamad
approved these changes
Jul 21, 2026
charlielye
added this pull request to the merge queue
Jul 22, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to no response for status checks
Jul 22, 2026
…paths The merge-queue run exposed three failures: - pushing rewritten history from a partial clone lazy-fetches promisor objects one at a time (93 minutes, then the connection dropped) -> the squash is removed from the job entirely; run it out-of-band from a full clone when needed - the datapoint entry exceeded Linux's 128KB per-argv-element cap (jq: Argument list too long) -> large inputs now reach jq as files - append_and_push runs inside an if, which suppresses errexit, so the jq failure was swallowed and a truncated data.js was pushed as a success -> every critical step now aborts the attempt explicitly
Remove the retention/prune machinery entirely: the shallow depth=1 append is unaffected by repo history size, so maintenance is not needed to fix the upload time and any history rewriting is better done out-of-band.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The "Upload benchmarks" CI step takes ~8 minutes. Root cause:
benchmark-action/github-action-benchmarkdoes a full clone ofAztecProtocol/benchmark-page-data(no depth/filter/single-branch) to append one datapoint. That repo's gh-pages history is ~4GB.Changes
ci3/upload_benchmarksreplaces the action at all three upload sites inci3.yml. It writes the identicalwindow.BENCHMARK_DATAformat (verified against the live data), so the dashboards are unaffected. It is deliberately append-only — no retention or history maintenance:--depth=1 --filter=blob:none --sparseclone scoped to the target data dir: measured against the live repo, ~4.5s / 89MB versus ~8min / 4GB. Because the clone is depth=1, this stays fast no matter how large history grows. Push conflicts with concurrent uploads are handled by retrying from a fresh clone.Argument list too long).|| return 1): the retry loop'sifsuppressesset -einside the function, and a swallowed mid-append failure would otherwise commit and push a truncateddata.jsas a "success".ci3/upload_benchmarks_testis a self-contained fixture-repo test (run it directly) covering append, new-dir initialization,MAX_ITEMStrimming, >128KB entries, and loud failure on an unreachable repo.An earlier revision of this PR also squashed/pruned old history in-job ("retention maintenance"). That was removed: a merge-queue run showed that pushing rewritten history from a partial clone lazy-fetches promisor objects one at a time (93 minutes, then the connection dropped), and measurement shows ~96% of the repo's weight is the recent six months of large
data.jsversions anyway — old-history squashing barely shrinks it and is unnecessary for upload speed.Notes
alert-thresholdcomparison. It was configured withcomment-on-alert: false/fail-on-alert: falseat all three sites, so it had no observable effect.BENCH_BRANCHand writes tobench/data.jsat the repo top level (updated as recently as today). The script keeps writing there rather than silently changing the destination.bench/next/data.jsis ~70MB at the tip because every datapoint embeds the full metric set (~700KB), which is also why the data repo grows ~25MB/day and the dashboard ships 70MB to every browser. Narrowing the schema is where the real size win is.