Repository navigation
Incident 2026-08-25: helm apply with public values half wiped the GitHub App identity from both prod ConfigMaps — GitSync + registry pulls down ~15 min #2210
Description
Impact
2026-08-25 ~11:13–11:30 UTC, both prod portals (memex.systemorph.com and memex.meshweaver.cloud): every GitHub sync failed with "Connect your GitHub account first … or configure the GitHub App identity". Because memex-cloud is the plugin registry and its GitHub App credential is what pulls Systemorph/MeshWeaver.Plugins@main, this also stalled catalog auto-updates and the in-flight _App migration. Restored at ~11:24Z; both portals verified (git_hub_sync op=check space=Store → "Branch 'main' is at b8f3523a", Succeeded).
Root cause
The GitHub App identity lived only as live drift on the ConfigMap, and a helm apply rendered from the public values half clobbered it.
GitHub__App__{ClientId,InstallationId,InstallationOwner}were never in any values source: I decoded the last-good helm release secrets (sh.helm.release.v1.memex.v17,sh.helm.release.v1.memexcloud.v3) and their renderedmemex-portal-confighas all three empty — the chart templates them asdefault "". The real values had beenkubectl patched onto the live ConfigMap (2026-07-09, the GitSync-as-App rollout) and survived every apply since only because those applies didn't touch the keys.- At 10:54:08Z (ns
memex, helm rev v18) and 11:06:06Z (nsmemex-cloud, rev v4) — four minutes after helm: render OpenRouter__Models — twelve declared keys reached no container #2203 ("render OpenRouter__Models") merged — helm upgrades were applied with the public halves only. The CM re-render emptied the three keys; the failure began when the new ReplicaSets came Ready (~11:13Z). - The applier is unidentified (no session claimed it at the time of writing); the mechanism does not depend on who — any apply without the identity in a values source reproduces it.
- Not related to RCA #2194: stale-build retention docs + fleet-wide Modules:AutoRecycleOnStaleBuild opt-in #2197's enablement earlier this morning: that was a single-key
kubectl patch --type merge(cannot remove keys), and sync still worked at 11:04Z, after its rollout.
Remediation (done)
Three-key merge patch restored in both namespaces (values confirmed from two independent sources: the 2026-07-09 rollout record, and gh api /orgs/Systemorph/installations → installation 144517285, app meshweaver-cloud, app_id 4220566) + rolling restart; newest pods' env verified; sync check green on both portals. Full-CM diff against both last-good renders: zero other chart-managed keys were emptied — the GitHub App trio was the complete loss set. The PEM half (KV github-app-privatekey → SPC → Secret) was never affected.
Durable fixes
- The identity enters the per-env values source (config half, beside the public per-env values) so any future render carries it — a live-drift-only value is one apply away from deletion, permanently. (Applied on the operator machine's per-env values; see below.)
- Pre-apply guard: the sanctioned apply path must diff the rendered
memex-portal-configagainst the LIVE ConfigMap and refuse when the render empties a key that is non-empty live (override flag for intentional removals). A chart whose defaults are""makes "the private half was missing from this render" indistinguishable from "this key was never set" — the guard is what makes that loud. This is the same invariant class as the existingcheck-chart-drift.sh, but enforced before the write instead of observed after. - Standing rule already in AGENTS/runbook, re-affirmed:
deploy.sh/full-chart applies are env-setup, not the code/config-update path; a config fix ships as a values change + targeted apply with the guard, never a public-halves-onlyhelm upgrade.
Cross-refs: #2194 (this morning's unrelated stale-build outage — same day, different mechanism), #2203 (the chart change whose deploy carried the bad render), 2026-07-09 GitSync-as-App rollout (where the drift was introduced).
Activity
rbuergi commentedon Aug 25, 2026
Root cause sharpened + durable fixes landed (Systemorph/Memex#112, merged):
Why it recurred: this exact incident already happened on 2026-08-24, and its fix committed the GitHub App identity into the overlay one section off — under
config.memex_migration, where the portal ConfigMap template (config.memex_portal.GitHub__App__*) never reads it. The overlay looked safe (key + reassuring comment visibly present), while everyhelm-releasedeploy still rendered the keys empty. Today's 10:53Z/11:04Z deploys (helm-release workflow_dispatch, actor = the sharedrbuergiidentity) applied that blanking to both instances.Landed in Systemorph/Memex#112:
memex_portalin both overlays; InstallationId/Owner committed too so render == live exactly (ClientId + PrivateKey remains the verified minimum pair).Modules__AutoRecycleOnStaleBuild: "true"in all three overlays — the helm-release pipeline layers only vault + overlay, so the platform repo's values.aks.yaml (RCA #2194: stale-build retention docs + fleet-wide Modules:AutoRecycleOnStaleBuild opt-in #2197) never reaches these deploys; the fleet policy must live in the overlays.memex-portal-config, naming each key;allow_blankinput confirms intentional removals. Keys absent from the render are not flagged (helm's three-way merge leaves them alone). Verified by local render with the live chart + real vault halves; the guard itself gets its first live exercise on the next real deploy.Live state, the overlays, and the chart now agree; the drift that carried the identity for six weeks is retired.
fix: match the chart's nil-safe config access too
rbuergi commentedon Aug 26, 2026
Closing — root cause identified and every named durable fix has shipped and merged, verified against
origin/mainandSystemorph/Memextoday (2026-08-26).Incident remediation (done, per the issue body): GitHub App identity restored on both
memexandmemex-cloudConfigMaps; sync verified green on both portals.Root cause (found in this thread): the 2026-08-24 fix for the same recurring problem put the identity block under
config.memex_migrationin the overlay — a section the portal ConfigMap template never reads (onlyconfig.memex_portal.GitHub__App__*is rendered). Ahelm-releasedeploy from that overlay therefore always rendered the three keys empty; the 2026-08-25 10:54Z/11:06Z deploys just happened to be the ones that triggered it.Durable fix 1 — identity moved into the correct values path:
Systemorph/Memex#112("GitHub App identity under memex_portal + blank-guard on deploy (MeshWeaver#2210)") — merged 2026-08-25T11:31:29Z. Moves the block fromconfig.memex_migrationtoconfig.memex_portalin both overlays and commitsInstallationId/InstallationOwnerso the render matches live exactly.Durable fix 2 — pre-apply guard shipped in the same PR:
helm-release.ymlnow renders vault+overlay on the runner and refuses when the render would empty a key that is non-empty on the live ConfigMap, naming the key (allow_blankfor intentional removals). This is exactly the class of failure that caused the incident — a mis-nested/incomplete values render silently wiping live config — and it is now a refused deploy, not a silent apply.Durable fix 3 was "re-affirm the standing rule" (deploy.sh/full-chart applies are env-setup, not the code/config-update path) — already in AGENTS.md, no code owed.
Why the symptom can't recur the same way: the identity now lives in a section every render path actually reads, and even if a future overlay change reintroduces a mis-nesting of this or any other previously-set key, the blank-guard refuses the deploy before it reaches the cluster.
Not part of this close, tracked separately: #2278 ("Close three gates that render a tick without checking anything (#2210, #1786, #1709)") is a further, forward-looking hardening pass (an authoring-time check, catching this class of mistake before deploy rather than at deploy) — it's additional armor, not a precondition for closing this incident, and it remains open on its own.
Merge pull request #2278 from Systemorph/fix/gates-that-never-fired
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-08-25)
Durable fix 2 ("pre-apply guard") shipped in
Systemorph/Memex(helm-release.yml blank-guard). This adds the half it cannot cover: the guard runs at DEPLOY time, against the live cluster, so a mis-nested key is invisible until someone deploys. #2278 asserts the same invariant at authoring time, from the files alone.deploy/aks/scripts/check-values-are-read.{sh,py}— everyconfig.<component>.<KEY>a values file sets must be read by that component's templates. It takes values paths, so the private repo runs the identical check on its overlays.Rendered proof of the mechanism, since a template must never be verified by reading it
helm templateovervalues.yaml + values.aks.yaml, correct nesting vs. the exact 08-24 mis-nesting:The new check refuses that input, naming the fix:
🚨 It found two LIVE defects of this exact class on its first contact with the production overlays
Run against the three committed
values.<release>.public.yamlfromSystemorph/Memex— 184 keys examined, 2 orphaned. Both are the "looks configured, reaches no container" shape, and both are currently live:1.
memex— a deadHosting__Operator__Enabled, still carrying its reassuring commentvalues.memex.public.yaml:81setsconfig.memex_portal.Hosting__Operator__Enabled: "true", under a comment explaining the safety model. That path reaches nothing: the chart renders the key from.Values.hostingOperator.enabled, not fromconfig.memex_portal.This is the same key that already caused a two-day outage of the feature, documented in three places in this repo —
INSTANCE-LIFECYCLE.md:95("memex declaredHosting__Operator__Enabled: "true"for two days while its live ConfigMap carried noHosting__*key at all, and every action refused as 'disabled on this installation'") and again inconfig.yaml's own block comment. The chart was fixed; the overlay's wrong-path line was never removed.Harmless today —
hostingOperator.enabled: trueis also set correctly at line 334, so the operator works — which is precisely why nobody noticed. It is a landmine with a comment beside it saying it is load-bearing.2.
client-i— email is configured under a key that no longer bindsvalues.client-i.public.yaml:65setsconfig.memex_portal.Email__NoReplyAddress. The chart says what happened to it, atconfig.yaml:570:client-i sets
Email__Enabled: "true"and noEmail__MailboxAddressat all;EmailOptions.MailboxAddressdefaults to"". So a customer production portal has email switched on with no sender address, while its overlay reads as fully configured. This one is not harmless — it is a renamed key that took its value with it, the class inrenamed-config-key-is-a-silent-deletion.Neither would be caught by the deploy-time blank-guard: it only flags a key the render carries as
""while live is non-empty, and both of these are keys the render never carries at all.Follow-up
A PR in
Systemorph/Memexwires this check intobuild.yml(pull_request— cluster-free, no secret, so no skip-trapdoor, alongside the existingdocsjob) and fixes both overlays. Extracting is not using; the mis-nesting happened in the overlays, so the check has to run there.🚨 The client-i fix changes behaviour — renaming the key to
Email__MailboxAddressmakes outbound email actually resolve a sender on a customer portal. The values change is unambiguous (the overlay's intent is plain), but the deploy is a separate decision and I am not making it.I have not changed any live cluster state, and no portal has been rolled.