Skip to content

ci: key the TPC dataset caches on pinned generators #6102

Description

@ErikBPF

Problem

Both TPC correctness jobs key their dataset cache on the workflow file:

key: tpch-${{ hashFiles('.github/workflows/pr_build_linux.yml') }}   # :705
key: tpcds-${{ hashFiles('.github/workflows/pr_build_linux.yml') }}  # :779

Any edit to pr_build_linux.yml rotates the key and regenerates the SF=1 dataset; that file changed 9× in 7 days. Today gh api actions/caches shows 10 TPC entries / ~2.6 GiB, including entries written from throwaway PR refs that later runs cannot restore — they only evict the shared LRU budget that protects the ~4.1 GiB Linux-cargo-debug cache, whose eviction costs a ~26-minute cold build.

Change

4a — pin the generators first. The key cannot be stable while the inputs are not: GenTPCHData.scala:125 clones databricks/tpch-dbgen HEAD (only one file is SHA-pinned), and the tpcds-kit checkout has no ref: (pr_build_linux.yml:785-790).

4b — stable key + restore-keys + main-only save:

  • key on the pinned generator revisions + Spark version, not the workflow file;
  • restore-keys: tpch-sf1- / tpcds-sf1-;
  • save only from main, consistent with the existing large-cache policy.

Expected gain

~3.5 min per avoided regeneration, and it removes ~2.6 GiB of cache churn. Direct ≈10 min/72 h; the real value is avoiding the cold builds the churn causes.

Verification

  • gh api actions/caches: TPC entries fall from 10 to ~2; no Cache not found for TPC across consecutive non-database runs.
  • A deliberate generator-revision bump still rotates the key.
  • TPC jobs still green on a PR and on the queue.

Risk / rollback

Stale dataset reuse if the generators are unpinned — hence 4a first. Revert the keys; one regeneration restores the old behaviour.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ciCI/CD, GitHub Actions, build toolingenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions