Skip to content

ci(chart-gate): pull docker_auth and loki from the digest-pinned GHCR mirror, never Docker Hub anonymously - #6404

Merged
rbuergi merged 5 commits into
mainfrom
ci/dockerhub-mirror-consumers
Oct 10, 2026
Merged

rbuergi merged 5 commits into
mainfrom
ci/dockerhub-mirror-consumers

Conversation

@rbuergi

@rbuergi rbuergi commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

Why

This is the follow-up to #6400 (Refs #6381). The Chart Gate's two executed-binary checks still pulled cesanta/docker_auth and grafana/loki from Docker Hub anonymously, and both went red on toomanyrequests on #6400's own run. #6400 created the digest-pinned GHCR mirror. Its first main sync (run 38009458740) copied both images and read their digests back, identical to the declared ones:

  • ghcr.io/systemorph/dockerhub/cesanta/docker_auth:1.14.0 = sha256:98e0307e…
  • ghcr.io/systemorph/dockerhub/grafana/loki:2.9.3 = sha256:eb92f1a4…

What

  • chart-gate.yml → Observability values and Bundle registry scripts (executed): each job gets packages: read and a ghcr.io login with the job's own GITHUB_TOKEN. No stored secret is involved, so forks and Dependabot work the same way.
  • test-bundle-registry.py: on a GitHub runner, Docker Hub images resolve through dockerhub-mirror.sh ref. The check is red if an image is undeclared, and red if the mirror's digest is not the chart's pin, so the bytes under test are the bytes deployed. On a laptop the image is used as written.
  • check-observability-values.py: grafana/loki:<appVersion> resolves the same way. A chart bump to an appVersion the mirror does not declare is red and names the list.
  • Doc/Architecture/DockerHubInCi: the "Chart Gate until follow-up" exception is removed.

There are no retries and no docker.io fallback.

Verification (local)

  • Helper exercised with GITHUB_ACTIONS=true: docker_auth resolves to the mirror. A wrong digest is refused, an undeclared alpine:3 is refused, and a ghcr image passes through unchanged. On a laptop the Docker Hub name is kept.
  • loki_image('2.9.3') resolves to the mirror. 3.0.0 exits 1, naming the list.
  • test-observability-values.py: 21 tests OK. actionlint and check-workflow-shell.py are clean. check-pr-secret-preflight.py: 0 violations.
  • The real proof is this PR's own Chart Gate run.

Pairs-with: none — CI scripts and docs only, no public C# surface.

🤖 Generated with Claude Code

rbuergi and others added 3 commits October 9, 2026 23:16
… mirror, never Docker Hub anonymously

On a GitHub runner both executed-binary checks resolve their Docker Hub image
through .github/scripts/dockerhub-mirror.sh (red when undeclared, red when the
mirror's digest is not the chart's pin) and log in to ghcr.io with the job's own
GITHUB_TOKEN (packages: read) — no stored secret, so forks and Dependabot work.
On a laptop the image is used as written. Doc/Architecture/DockerHubInCi.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 10, 2026 00:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Digestless chart references can pass, and Loki’s tested mirror digest is not tied to the production deployment.

2 open findings
What changed in this PR

This PR follows #6400 by routing Chart Gate runtime images through the authenticated, digest-pinned GHCR mirror.

Changes:

  • Adds GHCR authentication to both image-executing Chart Gate jobs.
  • Resolves Loki and docker_auth through the mirror.
  • Updates mirror documentation and workflow comments.
File Description
.github/​workflows/​chart-gate.yml Adds GHCR package-read authentication.
.github/​scripts/​test-bundle-registry.py Resolves Docker Hub test images through GHCR.
deploy/​aks/​scripts/​check-observability-values.py Uses mirrored Loki for binary validation.
.github/​workflows/​dockerhub-mirror.yml Documents the new consumers.
src/​MeshWeaver.Documentation/​Data/​Architecture/​DockerHubInCi.md Removes the temporary Chart Gate exception.

🧠 Review effort: Balanced


💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +218 to +220
if digest and not mirrored.endswith("@" + digest):
raise SystemExit(f"::error::the chart pins {image} but .github/dockerhub-mirror.list mirrors {mirrored} — "
f"move the list's digest to the chart's, or the check would execute bytes the chart does not deploy")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: on a runner, through_dockerhub_mirror now refuses a Docker Hub chart image that has no @sha256: pin, naming the image. So a change to a tag-only authImage turns red instead of quietly passing against the mirror's digest. Verified locally: the pinned image resolves to the mirror, and cesanta/docker_auth:1.14.0 without a digest is refused.

Comment on lines +209 to +214
name = f"grafana/loki:{app_version}"
if os.environ.get("GITHUB_ACTIONS") != "true":
return name
mirror = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "..", "..",
".github", "scripts", "dockerhub-mirror.sh")
result = run(["bash", mirror, "ref", name])

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, this drift is real, but it predates this change and is not created by it. install-observability.sh and values.observability.yaml deploy grafana/loki:<appVersion> by tag, with no digest. Before this PR, CI pulled that same moving tag at check time. Now it validates the declared digest (sha256:eb92f1a4…, which is the tag's current digest). Pinning the production Loki image, and having installer and checker read one shared pin, changes what the cluster runs. That deserves its own change through the observability install path, so it is not folded into a CI rate-limit fix.

rbuergi and others added 2 commits October 10, 2026 02:40
… the rejection test accepts the mirror's name

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Test Results

    24 files  ±0      24 suites  ±0   49m 54s ⏱️ -29s
11 329 tests ±0  11 136 ✅ ±0  193 💤 ±0  0 ❌ ±0 
11 343 runs  ±0  11 150 ✅ ±0  193 💤 ±0  0 ❌ ±0 

Results for commit e8c1d01. ± Comparison against base commit 0a2dc31.

This pull request removes 40 and adds 16 tests. Note that renamed tests count towards both.

   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
 ---> (Inner Exception #1) System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known<---
…
Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "8e5800f1-8181-4d07-9743-7dcaa60f2835")
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
 ---> System.UnauthorizedAccessException: Access denied
   --- End of inner exception stack trace ---, isDenial: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's FIRST member", fault: System.AggregateException: One or more errors occurred. (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's SECOND member", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall nested in an aggregate inside a wrapper", fault: System.InvalidOperationException: outer
 ---> System.AggregateException: One or more errors occurred. (Value does not fall within the expected range.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.ArgumentException: Value does not fall within the expected range.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---

   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall wrapped as InnerException", fault: System.InvalidOperationException: outer
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "an aggregate of unrelated faults", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Value does not fall within the expected range.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
, expected: False)
MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest ‑ Classifies(shape: "a MIXED aggregate — one transient branch, one genu"···, exception: System.AggregateException: One or more errors occurred. (Failed to connect to 10.42.18.4:5432) (source B is misconfigured)
 ---> MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
, expected: False)
MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest ‑ Classifies(shape: "a nested aggregate with one genuine leaf", exception: System.AggregateException: One or more errors occurred. (One or more errors occurred. (Failed to connect to 10.42.18.4:5432) (boom)) (Failed to connect to 10.42.18.4:5432)
 ---> System.AggregateException: One or more errors occurred. (Failed to connect to 10.42.18.4:5432) (boom)
 ---> MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---

   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
, expected: False)
MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest ‑ Classifies(shape: "a nested aggregate, all leaves transient", exception: System.AggregateException: One or more errors occurred. (One or more errors occurred. (Failed to connect to 10.42.18.4:5432)) (Failed to connect to 10.42.18.4:5432)
 ---> System.AggregateException: One or more errors occurred. (Failed to connect to 10.42.18.4:5432)
 ---> MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
, expected: True)
…

@rbuergi
rbuergi merged commit f4ee5ec into main Oct 10, 2026
@rbuergi
rbuergi deleted the ci/dockerhub-mirror-consumers branch October 10, 2026 13:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants