Repository navigation
ci: stop pulling Docker Hub anonymously — drop the unused pgvector pre-pull, add a digest-pinned GHCR mirror #6400
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # Docker Hub images CI uses, mirrored BY DIGEST to ghcr.io/systemorph/dockerhub/<name>:<tag>. | ||
| # | ||
| # Why (Doc/Architecture/DockerHubInCi): GitHub-hosted runners pull from Docker Hub ANONYMOUSLY, | ||
| # and the anonymous limit is per egress IP, shared with every other tenant on that runner pool. | ||
| # Measured 2026-10-09: `toomanyrequests` red every test shard and the Chart Gate of core PR runs | ||
| # (#6381 among them). A CI step that needs one of these images resolves it through | ||
| # .github/scripts/dockerhub-mirror.sh `ref <name>:<tag>` and logs in to ghcr.io with the job's | ||
| # GITHUB_TOKEN (`packages: read`) — no stored credential, so it works on forks and on Dependabot. | ||
| # | ||
| # One line per image: <docker hub name>:<tag>@sha256:<digest>. The digest is the pin; the mirror | ||
| # copies exactly those bytes (`docker buildx imagetools create`, read back and compared), and | ||
| # `ref` hands the consumer `ghcr.io/systemorph/dockerhub/<name>:<tag>@<digest>`. Adding or moving | ||
| # an entry is a main merge: dockerhub-mirror.yml syncs on that push, BEFORE any pull request | ||
| # depends on the new line (a consumer of an un-synced line is red, naming it). | ||
| # | ||
| # Resolve a digest without spending the pull limit (a manifest HEAD is not counted): | ||
| # tok=$(curl -fsS "https://auth.docker.io/token?service=registry.docker.io&scope=repository:<name>:pull" | jq -r .token) | ||
| # curl -fsSI -H "Authorization: Bearer $tok" -H "Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json" \ | ||
| # https://registry-1.docker.io/v2/<name>/manifests/<tag> | grep -i docker-content-digest | ||
|
|
||
| # The registry's token server — chart-gate `Bundle registry scripts (executed)`; the same digest | ||
| # deploy/helm/values.registry.example.yaml pins (the script refuses a mismatch). | ||
| cesanta/docker_auth:1.14.0@sha256:98e0307e0d2dcbafe10098814a97d248143b78ca478693f78c3c0d67fe776ab4 | ||
|
|
||
| # The Loki binary that validates the rendered config — chart-gate `Observability values`; the tag | ||
| # is the appVersion of the pinned loki-stack chart (deploy/aks/scripts/check-observability-values.sh). | ||
| grafana/loki:2.9.3@sha256:eb92f1a439171542fd718f929fad38c917b3cad15ec830ba4742e2ba5ab03313 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| #!/usr/bin/env bash | ||
| # | ||
| # dockerhub-mirror.sh — the route by which a CI job reaches a Docker Hub RUNTIME image it declares in | ||
| # .github/dockerhub-mirror.list (Doc/Architecture/DockerHubInCi). Base layers of images core BUILDS | ||
| # (`FROM` lines in path-filtered workflows) are out of its scope and listed on that page. | ||
| # | ||
| # dockerhub-mirror.sh ref <name>:<tag> print ghcr.io/systemorph/dockerhub/<name>:<tag>@<digest> | ||
| # for a line of .github/dockerhub-mirror.list; RED for an | ||
| # image the list does not declare | ||
| # dockerhub-mirror.sh sync copy every listed digest to the mirror (packages: write), | ||
| # read each one back and compare; RED on any mismatch | ||
| # dockerhub-mirror.sh --self-test the parsing and refusal arms, no network | ||
| # | ||
| # 🚨 No retries and no fallback to docker.io. A consumer that cannot reach the mirror is red, naming | ||
| # the line — a silent fall back to the anonymous pull is exactly the rate-limited path this replaces. | ||
| # For a declared image, `sync` is the only thing that touches Docker Hub, once per new digest, from a | ||
| # main/scheduled run. | ||
| set -euo pipefail | ||
|
|
||
| SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| LIST="${DOCKERHUB_MIRROR_LIST:-$SELF_DIR/../dockerhub-mirror.list}" | ||
| MIRROR="${DOCKERHUB_MIRROR_PREFIX:-ghcr.io/systemorph/dockerhub}" | ||
|
|
||
| # Emits "<name>:<tag> <digest>" per declared line; RED on a malformed one, so a typo can never | ||
| # read as "not declared", and RED on a second line for the same <name>:<tag> — otherwise `sync` | ||
| # would mirror both digests while `ref` hands consumers whichever came first. | ||
| entries() { | ||
| [ -f "$LIST" ] || { echo "::error::dockerhub-mirror: no list at $LIST" >&2; return 1; } | ||
| local line n=0 seen=" " | ||
| while IFS= read -r line || [ -n "$line" ]; do | ||
| n=$((n + 1)) | ||
| line="${line%%#*}"; line="$(printf '%s' "$line" | tr -d '[:space:]')" | ||
| [ -n "$line" ] || continue | ||
| if [[ ! "$line" =~ ^([a-z0-9]+([._-][a-z0-9]+)*(/[a-z0-9]+([._-][a-z0-9]+)*)*):([A-Za-z0-9_][A-Za-z0-9._-]{0,127})@(sha256:[0-9a-f]{64})$ ]]; then | ||
| echo "::error::dockerhub-mirror: $LIST line $n is not <name>:<tag>@sha256:<64 hex>: '$line'" >&2 | ||
| return 1 | ||
| fi | ||
| if [[ "$seen" == *" ${BASH_REMATCH[1]}:${BASH_REMATCH[5]} "* ]]; then | ||
| echo "::error::dockerhub-mirror: $LIST line $n declares ${BASH_REMATCH[1]}:${BASH_REMATCH[5]} a second time — one digest per <name>:<tag>; move the existing line instead of appending" >&2 | ||
| return 1 | ||
| fi | ||
| seen="$seen${BASH_REMATCH[1]}:${BASH_REMATCH[5]} " | ||
| printf '%s:%s %s\n' "${BASH_REMATCH[1]}" "${BASH_REMATCH[5]}" "${BASH_REMATCH[6]}" | ||
| done < "$LIST" | ||
| } | ||
|
|
||
| ref() { | ||
| local want="$1" all name digest | ||
| all="$(entries)" || return 1 | ||
| while read -r name digest; do | ||
| if [ "$name" = "$want" ]; then printf '%s/%s@%s\n' "$MIRROR" "$name" "$digest"; return 0; fi | ||
| done <<< "$all" | ||
| echo "::error::dockerhub-mirror: '$want' is not declared in .github/dockerhub-mirror.list — CI pulls no Docker Hub image anonymously. Add <name>:<tag>@sha256:<digest> there (the file says how to resolve the digest without spending the pull limit), merge it, and let dockerhub-mirror.yml sync it before a pull request depends on it." >&2 | ||
| return 1 | ||
| } | ||
|
|
||
| sync() { | ||
| local all name digest target got rc=0 | ||
| all="$(entries)" || return 1 | ||
| [ -n "$all" ] || { echo "::error::dockerhub-mirror: the list declares no image — a sync over nothing has proven nothing" >&2; return 1; } | ||
| while read -r name digest; do | ||
| target="$MIRROR/$name" | ||
| if got="$(docker buildx imagetools inspect "$target@$digest" --format '{{json .Manifest.Digest}}' 2>/dev/null)" && [ "${got//\"/}" = "$digest" ]; then | ||
| echo "$name: already mirrored at $digest" | ||
| else | ||
| echo "$name: copying docker.io/$name@$digest → $target" | ||
| if ! docker buildx imagetools create --tag "$target" "docker.io/${name%:*}@$digest"; then | ||
| echo "::error::dockerhub-mirror: could not copy docker.io/${name%:*}@$digest to $target" >&2 | ||
| rc=1; continue | ||
| fi | ||
| fi | ||
| # stderr stays in the log: an unreadable copy must say WHY it read as nothing. | ||
| if ! got="$(docker buildx imagetools inspect "$target" --format '{{json .Manifest.Digest}}')"; then got=""; fi | ||
| got="${got//\"/}" | ||
| if [ "$got" != "$digest" ]; then | ||
| echo "::error::dockerhub-mirror: $target reads back as '${got:-nothing}', not the declared $digest — a consumer pinned to the declared digest would not find it" >&2 | ||
| rc=1 | ||
| else | ||
| echo "$name: $target = $digest (read back)" | ||
| fi | ||
| done <<< "$all" | ||
| return "$rc" | ||
| } | ||
|
|
||
| self_test() { | ||
| local tmp fail=0 out d1 d2 | ||
| tmp="$(mktemp -d)" | ||
| d1="sha256:$(printf '%064d' 1)"; d2="sha256:$(printf '%064d' 2)" | ||
| printf '# comment\n\nfoo/bar:1.2@%s # trailing\nbaz:pg17@%s\n' "$d1" "$d2" > "$tmp/ok.list" | ||
| printf 'foo/bar:latest\n' > "$tmp/bad.list" | ||
| printf 'foo/bar:1.2@%s\nfoo/bar:1.2@%s\n' "$d1" "$d2" > "$tmp/dup.list" | ||
| ok() { echo " ✅ $1"; } | ||
| ko() { echo " ❌ $1"; fail=1; } | ||
| out="$(DOCKERHUB_MIRROR_LIST="$tmp/ok.list" "$0" ref foo/bar:1.2)" || out="" | ||
| if [ "$out" = "ghcr.io/systemorph/dockerhub/foo/bar:1.2@$d1" ]; then ok "a declared image resolves to the mirror, digest-pinned"; else ko "a declared image resolves to the mirror, digest-pinned (got '$out')"; fi | ||
| out="$(DOCKERHUB_MIRROR_LIST="$tmp/ok.list" "$0" ref baz:pg17)" || out="" | ||
| if [ "$out" = "ghcr.io/systemorph/dockerhub/baz:pg17@$d2" ]; then ok "a single-segment name resolves too"; else ko "a single-segment name resolves too (got '$out')"; fi | ||
| if DOCKERHUB_MIRROR_LIST="$tmp/ok.list" "$0" ref foo/bar:9.9 2>/dev/null; then ko "an undeclared image is refused"; else ok "an undeclared image is refused"; fi | ||
| if DOCKERHUB_MIRROR_LIST="$tmp/ok.list" "$0" ref foo/bar:1 2>/dev/null; then ko "a tag that only prefixes a declared one is refused"; else ok "a tag that only prefixes a declared one is refused"; fi | ||
| if DOCKERHUB_MIRROR_LIST="$tmp/bad.list" "$0" ref foo/bar:latest 2>/dev/null; then ko "a line without a digest is RED, not 'undeclared'"; else ok "a line without a digest is RED, not 'undeclared'"; fi | ||
| if DOCKERHUB_MIRROR_LIST="$tmp/dup.list" "$0" ref foo/bar:1.2 2>/dev/null; then ko "a second digest for the same name:tag is RED, never first-wins"; else ok "a second digest for the same name:tag is RED, never first-wins"; fi | ||
| if DOCKERHUB_MIRROR_LIST="$tmp/none.list" "$0" ref foo/bar:1.2 2>/dev/null; then ko "a missing list is RED"; else ok "a missing list is RED"; fi | ||
| out="$(entries)" || out="" | ||
| if [ -n "$out" ]; then ok "the committed list parses and declares at least one image"; else ko "the committed list parses and declares at least one image"; fi | ||
| rm -rf "$tmp" | ||
| return "$fail" | ||
| } | ||
|
|
||
| case "${1:-}" in | ||
| ref) [ -n "${2:-}" ] || { echo "usage: dockerhub-mirror.sh ref <name>:<tag>" >&2; exit 2; }; ref "$2" ;; | ||
| sync) sync ;; | ||
| --self-test) self_test ;; | ||
| *) echo "usage: dockerhub-mirror.sh ref <name>:<tag> | sync | --self-test" >&2; exit 2 ;; | ||
| esac |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| name: Docker Hub mirror | ||
|
|
||
| # Copies every image .github/dockerhub-mirror.list declares, BY DIGEST, to | ||
| # ghcr.io/systemorph/dockerhub/<name>:<tag> — the route by which CI reaches a Docker Hub image | ||
| # (Doc/Architecture/DockerHubInCi). | ||
| # | ||
| # WHY. GitHub-hosted runners pull from Docker Hub anonymously, against a limit counted per egress | ||
| # IP and shared with every other tenant of the runner pool. Measured 2026-10-09: `toomanyrequests` | ||
| # took down all six test shards and the Chart Gate of core pull requests (#6381 among them) — a | ||
| # red no code change could cause or cure. Retrying is not a fix (the limit is a window, not a | ||
| # blip); the fix is to stop asking Docker Hub from pull requests for the runtime images a job | ||
| # runs. For every image DECLARED in the list, this workflow is the only place CI touches docker.io, | ||
| # once per new digest, from main or the schedule. Not in scope: base layers of images core builds | ||
| # (path-filtered `FROM` lines) — the page lists them. The Chart Gate moves onto the mirror in the | ||
| # follow-up to the change that introduced this workflow. | ||
| # | ||
| # Consumers: `bash .github/scripts/dockerhub-mirror.sh ref <name>:<tag>` and a ghcr.io login with | ||
| # the job's GITHUB_TOKEN (`packages: read`). No stored credential — so forks and Dependabot pull | ||
| # the same way, and there is no second secret store to provision. | ||
| # | ||
| # 🛡️ NO SKIP-TRAPDOOR. The self-test runs on every trigger. The sync job's only condition is the | ||
| # EVENT (a pull request cannot write packages, and must not); it asserts its own input — a list | ||
| # that declares nothing is red — and reads every digest back. | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main] | ||
| paths: | ||
| - .github/dockerhub-mirror.list | ||
| - .github/scripts/dockerhub-mirror.sh | ||
| - .github/workflows/dockerhub-mirror.yml | ||
| pull_request: | ||
| paths: | ||
| - .github/dockerhub-mirror.list | ||
| - .github/scripts/dockerhub-mirror.sh | ||
| - .github/workflows/dockerhub-mirror.yml | ||
| schedule: | ||
| # Daily: a deleted package or a manifest retention dropped is RESTORED the same day (sync copies | ||
| # any declared digest the mirror lacks); the run is red only when it cannot restore it. | ||
| - cron: "40 4 * * *" | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| concurrency: | ||
| group: dockerhub-mirror-${{ github.ref }} | ||
| cancel-in-progress: false | ||
|
|
||
| jobs: | ||
| self-test: | ||
| name: Mirror list parses, refusals hold | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 5 | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| - name: Self-test | ||
| run: bash .github/scripts/dockerhub-mirror.sh --self-test | ||
|
|
||
| sync: | ||
| name: Every declared digest is on the mirror (read back) | ||
| needs: [self-test] | ||
| if: github.event_name != 'pull_request' | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 20 | ||
| permissions: | ||
| contents: read | ||
| packages: write | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| - name: Log in to GHCR | ||
| uses: docker/login-action@v4 | ||
| with: | ||
| registry: ghcr.io | ||
| username: ${{ github.actor }} | ||
| password: ${{ secrets.GITHUB_TOKEN }} | ||
| - name: Sync | ||
| run: bash .github/scripts/dockerhub-mirror.sh sync | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not adding a tracking-issue reporter in this change.
syncis self-restoring: on each daily run it copies any declared digest the mirror is missing. A failure it cannot fix also reaches its consumers immediately, becausedockerhub-mirror.sh refand the pull of the pinned digest go red in the Chart Gate step, naming the mirror line. So nothing stays silently unavailable. A sentinel issue is worth adding if a scheduled red ever goes unnoticed, but it is not needed for the fix.