Problem
Both TPC correctness jobs key their dataset cache on the workflow file:
key: tpch-${{ hashFiles('.github/workflows/pr_build_linux.yml') }} # :705
key: tpcds-${{ hashFiles('.github/workflows/pr_build_linux.yml') }} # :779
Any edit to pr_build_linux.yml rotates the key and regenerates the SF=1 dataset; that file changed 9× in 7 days. Today gh api actions/caches shows 10 TPC entries / ~2.6 GiB, including entries written from throwaway PR refs that later runs cannot restore — they only evict the shared LRU budget that protects the ~4.1 GiB Linux-cargo-debug cache, whose eviction costs a ~26-minute cold build.
Change
4a — pin the generators first. The key cannot be stable while the inputs are not: GenTPCHData.scala:125 clones databricks/tpch-dbgen HEAD (only one file is SHA-pinned), and the tpcds-kit checkout has no ref: (pr_build_linux.yml:785-790).
4b — stable key + restore-keys + main-only save:
- key on the pinned generator revisions + Spark version, not the workflow file;
restore-keys: tpch-sf1- / tpcds-sf1-;
- save only from
main, consistent with the existing large-cache policy.
Expected gain
~3.5 min per avoided regeneration, and it removes ~2.6 GiB of cache churn. Direct ≈10 min/72 h; the real value is avoiding the cold builds the churn causes.
Verification
gh api actions/caches: TPC entries fall from 10 to ~2; no Cache not found for TPC across consecutive non-database runs.
- A deliberate generator-revision bump still rotates the key.
- TPC jobs still green on a PR and on the queue.
Risk / rollback
Stale dataset reuse if the generators are unpinned — hence 4a first. Revert the keys; one regeneration restores the old behaviour.
Problem
Both TPC correctness jobs key their dataset cache on the workflow file:
Any edit to
pr_build_linux.ymlrotates the key and regenerates the SF=1 dataset; that file changed 9× in 7 days. Todaygh api actions/cachesshows 10 TPC entries / ~2.6 GiB, including entries written from throwaway PR refs that later runs cannot restore — they only evict the shared LRU budget that protects the ~4.1 GiBLinux-cargo-debugcache, whose eviction costs a ~26-minute cold build.Change
4a — pin the generators first. The key cannot be stable while the inputs are not:
GenTPCHData.scala:125clonesdatabricks/tpch-dbgenHEAD (only one file is SHA-pinned), and thetpcds-kitcheckout has noref:(pr_build_linux.yml:785-790).4b — stable key + restore-keys + main-only save:
restore-keys: tpch-sf1-/tpcds-sf1-;main, consistent with the existing large-cache policy.Expected gain
~3.5 min per avoided regeneration, and it removes ~2.6 GiB of cache churn. Direct ≈10 min/72 h; the real value is avoiding the cold builds the churn causes.
Verification
gh api actions/caches: TPC entries fall from 10 to ~2; noCache not foundfor TPC across consecutive non-database runs.Risk / rollback
Stale dataset reuse if the generators are unpinned — hence 4a first. Revert the keys; one regeneration restores the old behaviour.