Chart drift: the cluster does not match the chart

State: drift — last checked 2026-10-21T06:40:30Z (run).

memex-cloud: 31 divergence(s) across 294 compared fields: 29 SHADOWS, 2 CLUSTER-ONLY
memex: 5 divergence(s) across 259 compared fields: 1 SHADOWS, 2 CLUSTER-ONLY, 2 DIFFERS

Counts only, deliberately: this repository is public, so the per-key detail stays in the run log rather than in an issue body (MeshWeaver#4640).

Maintained automatically by .github/workflows/chart-drift.yml. Updated in place on every run and closed automatically when every environment matches; it reopens on the next divergence. Do not edit the body — edit the workflow.

Consolidation: earlier copies of this sentinel, #5049 and #5491, are folded in here. This one is the copy chart-drift.yml still updates (last checked 2026-10-07).

Chart drift: the cluster does not match the chart #5049, last updated 2026-09-22: memex-cloud 38 divergences / 241 fields (29 SHADOWS, 7 CHART-ONLY, 2 DIFFERS); memex 17 / 221. It carries the maintainer's per-class read of run 35570080512 (Chart drift: the cluster does not match the chart #5049 (comment)) that explains the SHADOWS class: inline env: entries shadow the rendered ConfigMap's envFrom keys. The conclusion was that none is fixable in this repository. Read it there; it stays linked from that closed issue.
Chart drift: the cluster does not match the chart #5491, last updated 2026-09-27: memex-cloud 38 / 242 (29 SHADOWS, 2 CHART-ONLY, 7 DIFFERS); memex 18 / 211.
Today's counts are almost unchanged (29 SHADOWS on memex-cloud throughout), so this is one drift seen three times.

Why there were three copies, not a new drift: the workflow finds its issue with gh issue list --state all --limit 200 and takes the newest body carrying the marker. Once the sentinel ages out of the 200 most recent issues, the lookup finds nothing and gh issue create mints a new one. That happened about every 2–5 days at this repo's filing rate. Expect a fourth copy unless the lookup changes. Severity sev:M is carried over from #5049.

Triage (classification backfill per Doc/Architecture/IssueTaxonomy — exactly one type per issue): removed chore — the issue carried two types (bug and chore). Kept the stronger classification already on it, bug, sev:M: the cluster diverging from the chart is a wrong deployed state, not a tidy-up.

memex half: record change opened as Systemorph/Memex#718. It marks retiredBy on the 13 memex inline entries that the 2026-10-09 run reads as agreeing. The next record-driven Reconcile's RetireInlineEnv removes them, after re-measuring equality in-cluster.

PluginCatalog__RegistryUrl is held back on purpose. The live ConfigMap now carries it EMPTY, so the inline entry is the pod's only value. It waits for a typed registryUrl record field (Memex#175).

Expected after the Reconcile: memex goes from 14 SHADOWS to 1. memex-cloud's 29 are left to the memex-cloud reconcile work already in progress.

The 57 findings of run 38118746006, by where each is fixed

Read from the two environment jobs' logs. Per-key detail stays there; this is the classification.

class	count (env A / env B)	what it is	where it goes
`SHADOWS` that agree with the ConfigMap	25 / 13	duplicate inline entries	marked for retirement on the deployment records (private repository; one change merged, one in draft). Removal runs only in an approved Reconcile
`SHADOWS` that disagree, or whose twin no record field renders	4 / 1	the inline entry is the operative value	a decision each; listed on the private inventory issue
`CHART-ONLY` + `DIFFERS` on ConfigMap keys	7 / 1	values the committed values file states and the deployment record did not, so no governed apply delivered them	5 moved onto the record or removed as redundant; 1 follows the bundled front end; the 2 catalog keys need a typed record field
`CLUSTER-ONLY inline env AZURE_CLIENT_ID` (reported as a pending deletion)	1 / 1	the check's own. The operator passes the value with `--set` on every apply; the committed values file did not state it, so the check rendered a chart without the entry	the values file now states it (private repository)
`DIFFERS inline env MEMEX_POD_NAME`	1 / 1	the check's own. The API server defaults `fieldRef.apiVersion` to `v1`; the comparator compared the two verbatim	#6483
`DIFFERS` liveness / readiness probe	0 / 2	a live-only `initialDelaySeconds`, inert under the startup probe; a helm upgrade does not remove it	break-glass patch, or leave: the chart is the authoritative side

Expected once the open changes have merged (the bundled front end among them) and their Reconciles have been approved and applied: 6 and 3 findings left, every one a named decision. That is arithmetic over the log, not a reading; the next runs are the reading.

Two owner acts gate the large numbers: approving the Reconcile that retires the marked entries on each environment. The request for one environment went stale after 24 h without approval and has to be requested again.

State 2026-10-13 ~15:05Z: memex is done down to its named decisions. memex-cloud waits on one owner act.

memex: 3 findings (run 38358519875). This is the figure predicted on 10-11 after the retirement Reconcile. Each remaining finding is a named decision from the 10-11 table, for example the live-only probe delay.
memex-cloud: 32 findings, 29 of them SHADOWS. The prerequisite named on the private retirement change is now met. The overlay-values change it waited for is merged, and that change's record Reconcile (Ops/Actions/record-reconcile-memex-cloud-v9-66b7b34b1c74 on control) ran to Done on 2026-10-12 05:07Z.
The owner act: take the memex-cloud retirement change out of draft and merge it (the private deployments repository; it marks retiredBy on 25 agreeing inline entries). Then approve the Reconcile it opens on the control instance. Approving that Reconcile is what removes the entries, and a retirement step is never admitted unattended. Expected after it: memex-cloud goes from 29 SHADOWS to 4. The 4 are decisions listed on that change.

No agent action remains on this issue.

Reading 2026-10-27: nothing updates this sentinel any more, and nothing has replaced its measurement yet. So chart-vs-cluster agreement is currently UNMEASURED, not green. Leaving open.

The sentinel's writer is gone. ops(chart-drift): the live chart-vs-cluster comparison moves into the governed Audit; retire chart-drift.yml #7021 merged 2026-10-21 10:40Z and deleted .github/workflows/chart-drift.yml. Its last run was 39556662047, which is the "last checked" in the body. The comparison moved into an operator verb, hosting-chart-drift. It now lives in MeshWeaver.Plugins deploy/hosting-operator/bin/, and the drift scripts live in MeshWeaver.Deploy deploy/aks/scripts/.
The verb is not wired into any lane. The step that made the governed Audit run it (in the private deployments repository) was closed unmerged on 2026-10-26, superseded by an owner decision after the chart source moved to MeshWeaver.Deploy. That closure said "a fresh PR on the current layout". The Audit read step on that repository's main still runs only hosting-audit.
Changes since the 10-13 reading that should lower the counts: the second-environment retirement change (25 agreeing inline entries) merged 2026-10-26, and record-change Reconciles ran today on every instance. No reading confirms the effect, because nothing reads.

Owed:

The fresh wiring PR: the Audit read step runs hosting-chart-drift --namespace "$NS" --release "$RELEASE" at the current operator and chart sources. Agent work once the owner confirms the layout; it was not attempted here because the deployments repository's Close the two residual acked-write-loss holes behind TwoSiloRecycleConvergenceTest #647 series is being sequenced by another session.
Owner: approve one Audit per instance on control.

Per #7021, close this issue on the first governed Audit that carries the comparison. If it reads clean, close it; if not, record its counts here.

🤖 Generated with Claude Code

This sentinel is no longer maintained. #7021 (merged 2026-10-21) deleted chart-drift.yml and moved the chart-vs-cluster comparison into the governed Audit; its body says to close this issue "once the first governed Audit carrying the comparison has run on the control instance". needs-evidence: that Audit InstanceAction and its result on the control instance. A search of Ops/Actions for content.requestedAction:Audit returned 0 over partition ops, so I could not confirm it ran.

Re-measured 2026-10-30 (read-only): chart-vs-cluster agreement is still not measured. Leaving this open. The next step is owner-gated.

Nothing compares chart and cluster yet. On Memex main, .github/workflows/node-repo-aks-ops.yml has an Audit step that runs only hosting-audit --namespace "$NS" --release "$RELEASE". hosting-chart-drift is named only in a comment about which chart sources the job checks out. Nothing invokes it.
No governed Audit has run since the sentinel's writer was deleted. On control, namespace:Ops/Actions name:*udit* returns 2 nodes over partition ops. Both are AuditIdentities actions from 2026-10-18, and neither carries a chart comparison.
Without a reading there is no drift to fix in code or in the chart. The last counts are still the ones in the body, from run 39556662047 on 2026-10-21. Since then the second-environment retirement and the record Reconciles have landed, and their effect has not been read.

Owed, unchanged from 2026-10-27:

Wire hosting-chart-drift into the Audit read step of the private deployments repository. This needs the owner to confirm the layout, because that repository's change series is being sequenced separately. It is out of scope for this pass, which covers core and Plugins only.
Owner: approve one Audit per instance on control.

Per #7021, close this issue on the first governed Audit that carries the comparison.

🤖 Generated with Claude Code

2026-11-01: the lane wiring this issue was waiting on is open: https://github.com/Systemorph/Memex/pull/925 (auto-merge armed).

The Hosting plugin's Audit plan already had both steps, and the control plane already records the chart-drift verdict beside the live-only report. Only the deployments repository's aks-ops lane had not run the second step.

The new step runs hosting-chart-drift after hosting-audit, because the verdict is read only after the run's last audit= line. It passes only the ::hosting:: verdict lines into the callback tail, and it goes red when the comparison could not measure.

Once it merges, one more step closes this issue: file one Audit per instance on control (an observation, so no approval is needed). Per #7021, close on a clean reading, or record the counts here.

🤖 Generated with Claude Code

Reading 2026-11-01 ~16:40Z: the first governed Audits that carry the chart-vs-cluster comparison have run (Memex#925 merged 11:54Z). Drift remains, so this stays open.

These are five read-only Audit actions on the control instance, Ops/Actions/audit-chartdrift-<instance>-20261101-5844. Each Memex run is success, and the hosting-chart-drift step measured in each one. Counts only, per this sentinel's rule; the per-key detail is in the run logs.

instance	divergences / compared fields	classes	pending deletions (removed by the next `helm upgrade`)
memex-cloud	6 / 303	4 SHADOWS, 2 CLUSTER-ONLY	2
memex	5 / 299	1 SHADOWS, 2 CLUSTER-ONLY, 2 DIFFERS	2
control	2 / 326	2 CLUSTER-ONLY	2
build	2 / 263	1 CLUSTER-ONLY, 1 DIFFERS	1
pearl	11 / 222	1 CLUSTER-ONLY, 2 CHART-ONLY, 8 DIFFERS	1

memex-cloud fell from 31 to 6 divergences. Its 4 SHADOWS are the 4 deliberate inline rows tracked on Memex#148. Two of them now disagree with their ConfigMap twin, and those two are the rows waiting on owner decisions there.

Next, owner-gated:

A governed Reconcile per instance clears the pending deletions and the DIFFERS rows. pearl, with 8 DIFFERS, needs a read of its plan before approval.
The Memex#148 decisions clear the SHADOWS.

Per-instance Reconcile decision, 2026-11-01 ~21:00Z: no Reconcile filed for memex-cloud, memex, build or pearl. A Reconcile does not apply the render this comparison measures against. control is excluded here because its roll is being handled separately.

This corrects the previous comment, which said a governed Reconcile clears the pending deletions and the DIFFERS rows. It does not. The evidence:

What a Reconcile renders. The 2026-10-30 memex Reconcile (Ops/Actions/record-reconcile-memex-v25-39a1cddb0425, Memex run 41043195120) ran:

vault     helm-values-memex (1708 bytes, families only) — layered FIRST; the record's render wins on every key it declares
values    /tmp/hosting-values.yaml (15100 bytes, rendered from the record)
values    nothing the live release carries is dropped by this upgrade
+ helm upgrade memex …/deploy/helm -f <vault half> -f /tmp/hosting-values.yaml … --set selfUpdate.azureClientId=<portal identity> --set-json hostingDeploy={"recordOwned":[… "pluginCatalog.instanceId" … "selfUpdate.azureClientId" …]}

What hosting-chart-drift renders. The record plus the committed overlay deployments/aks/<instance>/values.<instance>.public.yaml (see overlay and records in each audit log). It does not reproduce the record-to-values projection that hosting-deploy applies, and it does not set selfUpdate.azureClientId. So its "chart" side is not what a deploy installs.

What that means for each finding class:

PENDING DELETION. PluginCatalog__InstanceId/HomeUrl on memex and memex-cloud, and inline AZURE_CLIENT_ID on pearl and build. The deploy renders all of them: pluginCatalog.instanceId is record-owned, and selfUpdate.azureClientId is passed with --set. The live lengths match the records: registryInstanceId is memex (5 characters) and memex-cloud (11); the HomeUrls are https:// plus each host (28 and 30); the client id is 36. They also survived the 10-30 Reconciles and today's rolls. A Reconcile leaves them in place. They are false positives of the comparison, not deletions waiting to happen.
DIFFERS (ConfigMap). The tool itself prints "a deploy does NOT resolve this (measured)". On pearl the live side is the record's value, and the chart side is a stale overlay value:
- MEMEX_USERNAME: live 10 characters equals the record's databaseUsername; the chart's 8 characters is the overlay's postgres.
- Email__ClientId/TenantId/MailboxAddress: live is empty, as the record intends (email.enabled: false). The chart side is the overlay's own block.
- Hosting__ControlInbox__Url: live 63 characters, chart empty.
- OTEL_* and Deployment__Orleans__Clustering: overlay-only values. The same Clustering row appears on build.
pearl CHART-ONLY (Modules__AutoRecycleOnStaleBuild, envFrom secretRef:pearl-portal-ai-secrets). Both are overlay-only. pearl's record declares neither autoRecycleOnStaleBuild nor that secret reference, and today's roll did not install them.
SHADOWS (memex 1, memex-cloud 4). Each is a row the record itself declares under inlineEnv[]. Clearing one is a record change, and the record change then files its own Reconcile (policy record-change-applies-itself). It is not a stand-alone Reconcile. The decisions are on Memex#148.
memex probe DIFFERS (initialDelaySeconds). The tool marks them inert because a startupProbe is present, and they survived the 10-30 helm upgrade.

instance	Reconcile?	why
memex-cloud	no	2 false PENDING DELETIONs; 4 SHADOWS are record-declared (Memex#148)
memex	no	2 false PENDING DELETIONs; 1 record-declared SHADOWS; probe rows inert
build	no	1 false PENDING DELETION; 1 overlay-only DIFFERS
pearl	no	1 false PENDING DELETION; 8 DIFFERS and 2 CHART-ONLY are overlay values the record does not intend (a Systemorph mailbox, `postgres` as the database user, an empty control inbox)

Not established: I did not run a Reconcile dry run, so the claim that a Reconcile installs nothing new rests on the 10-30 run log and on today's rolls leaving these rows unchanged. It is not a plan diff.

What actually closes this. hosting-chart-drift should render exactly what hosting-deploy renders: the record projection plus the vault-half shape plus the --set values, not the committed overlay. Or the overlays should be reduced to what the record does not own. Until one of those happens, the counts above are mostly an artefact of the comparison, and a Reconcile cannot bring them to zero.

🤖 Generated with Claude Code

docs(chart-drift): scope the overlay caveat to legacy mode; a finding…

docs(chart-drift): the Audit compares against what a deploy installs (#…

Navigation Menu

Repository navigation

Chart drift: the cluster does not match the chart #5844

The 57 findings of run 38118746006, by where each is fixed

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Chart drift: the cluster does not match the chart #5844

Description

Activity

rbuergi commented on Oct 7, 2026

rbuergi commented on Oct 9, 2026

rbuergi commented on Oct 10, 2026

rbuergi commented on Oct 11, 2026

The 57 findings of run 38118746006, by where each is fixed

rbuergi commented on Oct 13, 2026

rbuergi commented on Oct 27, 2026

rbuergi commented on Oct 27, 2026

rbuergi commented on Oct 30, 2026

rbuergi commented on Nov 1, 2026

rbuergi commented on Nov 1, 2026

rbuergi commented on Nov 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions