[rbuergi]

The hop where delivery actually stops

Not where #2268 (already merged) or the issue's own comments suspected. #2268 fixed the CD side —
notify-platform-update now fails loud on a non-2xx instead of ::warning::-ing past it, and that
change is what turned this from invisible into a reproducible 404 on the very next promote:

POST https://memex.meshweaver.cloud/api/hooks/Hosting/PlatformBuilds -> 404
##[error]the platform-update inbox answered 404 ...

Tracing why, live on memex-cloud (the memex.meshweaver.cloud control instance — NOT the memex
namespace, which serves memex.systemorph.com; a first wrong turn worth naming since it cost time):

• Hosting/PlatformBuilds node exists (both MCP servers confirm it, version 1, unchanged — the
  expected steady state, not evidence of nothing arriving, per the issue's own corrected
  diagnosis).

• Hosting__PlatformWebhookSecret is mounted and non-empty (memexcloud-portal-ai-secrets,
  wired via the memexcloud-portal-ai-secrets SecretProviderClass) — the issue's suspicion that the
  secret was never provisioned is now stale/incorrect.

• WebhookInbox__Targets__0 in the ConfigMap is Hosting/PlatformBuilds — correct.

• WebhookInbox__Targets__0 in the running pod's actual environment is Store/Payments:

  $ kubectl -n memex-cloud exec deploy/memex-portal-deployment -- printenv | grep WebhookInbox
  WebhookInbox__Targets__0=Store/Payments
  

The Deployment carries an explicit env: entry WebhookInbox__Targets__0=Store/Payments —
hand-set via kubectl set env, pre-dating this chart key, for the pre-existing Stripe payments
inbox. Kubernetes gives an explicit env: entry priority over the same-named var sourced via
envFrom (the ConfigMap), so it permanently shadows whatever the chart renders for that name.
Every reported signal was green: helm said "Upgrade complete", the ConfigMap held the right value,
WebhookInboxEndpoints.Deliver (memex/Memex.Portal.Shared/Api/WebhookInboxEndpoints.cs) is
correctly wired and unconditionally mapped — and the live pod answered 404 anyway, because nothing
in the chain ever looks at the pod's resolved environment, only at what was intended to render
into it.

This is the same defect class as #2210 (the GitHub App identity blanking): a value that renders
correctly into the ConfigMap while the running container silently uses something else.

Why it looked like success

• CD (fixed by fix: three delivery legs that reported success while delivering nothing (#2235, #1742, #1971) #2268, already merged) used to swallow the 404 as a warning.
• helm upgrade genuinely succeeds — it owns the ConfigMap, not the out-of-band env: entry.
• The webhook inbox itself is deliberately silent-refusal-shaped (404 for an unknown target is
  indistinguishable from a wrong URL, by design — WebhookInboxEndpoints.cs's own doc comment).
• The target node existing, and its secret being correctly mounted, both looked like confirmation
  the config was right — and they were, in the ConfigMap. Nothing checks the pod's env against it.

Fix

The chart only ever templated one indexed slot (WebhookInbox__Targets__0), so a deployment
needing a second webhook target had nowhere to put it except colliding on the same index by hand.
This PR adds WebhookInbox__Targets__1 as a real second slot in
deploy/helm/templates/memex-portal/config.yaml (same index-by-index pattern as
OpenRouter__Models__N / AzureFoundry__Models__N — EveryConfigKeySetInAValuesFile_IsBoundInTheConfigMap
requires each key spelled out explicitly, never a range) and documents both slots in
deploy/aks/values.aks.yaml.

Companion overlay PR (private repo): Systemorph/Memex#121 — sets memex-cloud's __0 to match
what is actually live (Store/Payments, so the ConfigMap stops asserting a value the pod silently
discards) and adds __1 = Hosting/PlatformBuilds on the slot the stray kubectl set env never
touched. Once both land, a helm-release.yml deploy of memexcloud (a separate, deliberate step —
prod deploy is its own decision, not a side effect of this merge) restores delivery without needing
to touch the stray env: entry on index 0 at all.

Not fixed here, and doesn't need to be

• The stray kubectl set env on index 0 — harmless now that the overlay's __0 matches it.
• Anything in memex/Memex.Portal.Shared/ (out of scope for this session; another thread has it
  open) — WebhookInboxEndpoints.Deliver was read closely and has no bug: allowlist check, "target
  node must exist" check, and the HMAC-verification-is-the-consumer's-job design are all correct as
  written.

Verification

• helm template memex ./deploy/helm -f ./deploy/helm/values.yaml -f ./deploy/aks/values.aks.yaml
  renders both WebhookInbox__Targets__0 and __1 correctly.
• PlatformBakeLaneGuard (test/MeshWeaver.Documentation.Test), including
  EveryConfigKeySetInAValuesFile_IsBoundInTheConfigMap which exists specifically to catch an
  un-templated values key: 9/9 green, fresh .trx after a -c Release -warnaserror build (0 Warning(s), 0 Error(s)).
• Live evidence above (kubectl get configmap vs kubectl exec ... printenv) is the actual proof
  of the defect, gathered against the real memex-cloud pod, not inferred from the issue text.

#2235 stays open — MeshWeaver#2235's own stated close condition is unchanged: one promoted
build producing (a) a repository_dispatch run on all four satellite repos and (b) the
MW_IMAGE_DIGEST bump PRs, observed after Memex#121 merges and a deploy runs. Never a green CD
tick alone — that's exactly what this repo has looked like since 08-22.

Coordination

Checked src/MeshWeaver.GitSync for the concurrent restructuring mentioned in the task brief — this
PR does not touch that project at all (nor memex/Memex.Portal.Shared/, nor any of the other
stay-out directories); the fix is entirely in deploy/helm/ and deploy/aks/.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a second Helm ConfigMap key for webhook inbox targets so a deployment can configure multiple WebhookInbox:Targets entries without colliding on __0 (which can be shadowed by an out-of-band env: override).

Changes:

• Add WebhookInbox__Targets__1 to the portal ConfigMap template to support a second allowlisted webhook target.
• Document/configure the new target slot in AKS values (WebhookInbox__Targets__1, default empty).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
deploy/helm/templates/memex-portal/config.yaml	Adds a second WebhookInbox__Targets__1 slot to the rendered ConfigMap.
deploy/aks/values.aks.yaml	Documents the second slot and defines it as empty by default for AKS deployments.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[rbuergi]

One caveat for whoever deploys this, which the review thread doesn't cover — the values.aks.yaml entry does not protect the prod portals.

A Systemorph/Memex helm-release.yml deploy layers exactly two values files: the KV vault half (helm-values-<release>, secrets only) first, then the committed deployments/aks/<ns>/values.<release>.public.yaml last. The platform repo's deploy/aks/values.aks.yaml is never layered. That is the same asymmetry behind #2210, where a key committed one section off rendered "" and helm's three-way merge applied it as a silent deletion.

So for memex / memex-cloud the protection is (a) the | default "" you just added, and (b) each overlay carrying the key explicitly — which is why the companion Systemorph/Memex#121 is load-bearing rather than cosmetic. "I added it to values.aks.yaml" should never be read as "prod is covered"; a fleet policy set only there never reaches a helm-release deploy.

Two things that make this PR's approach right, for the record:

1. It removes the reason the stray env: exists rather than fighting it. The chart only ever templated one indexed slot, so a second webhook target had nowhere to go but collide on index 0 by hand. With a real __1, the hand-set __0=Store/Payments becomes correct rather than something to undo on a live Deployment.

2. Chart Drift independently flagged this exact key this morning — run 32939313904, the first run that gate has ever completed (it was blocked by Chart Drift has never run: MEMEX_DEPLOY_REPO_TOKEN authenticates as nothing (HTTP 401) — re-mint it #1709 for ten consecutive failures):

   ##[error]CLUSTER-ONLY inline env WebhookInbox__Targets__0
   

   Same defect, found from the other direction, hours earlier. The gate works. Its other 63 inline-env divergences on memex-cloud are now worth reading — that backlog is what it exists to surface.

[github-actions]

Test Results (shard 3)

    9 files      9 suites   5m 32s ⏱️
2 023 tests 2 022 ✅ 1 💤 0 ❌
2 572 runs  2 571 ✅ 1 💤 0 ❌

Results for commit 0f45aed.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (shard 0)

593 tests   589 ✅  12m 32s ⏱️
  7 suites    4 💤
  7 files      0 ❌

Results for commit 0f45aed.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (shard 5)

    9 files      9 suites   7m 9s ⏱️
1 612 tests 1 611 ✅ 1 💤 0 ❌
1 613 runs  1 612 ✅ 1 💤 0 ❌

Results for commit 0f45aed.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (shard 2)

2 831 tests   2 728 ✅  9m 20s ⏱️
    9 suites    103 💤
    9 files        0 ❌

Results for commit 0f45aed.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (shard 4)

2 064 tests   2 061 ✅  14m 10s ⏱️
   10 suites      3 💤
   10 files        0 ❌

Results for commit 0f45aed.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results (shard 1)

2 046 tests   1 853 ✅  8m 20s ⏱️
    9 suites    193 💤
    9 files        0 ❌

Results for commit 0f45aed.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results

    53 files      53 suites   57m 6s ⏱️
11 169 tests 10 864 ✅ 305 💤 0 ❌
11 719 runs  11 414 ✅ 305 💤 0 ❌

Results for commit 0f45aed.

♻️ This comment has been updated with latest results.

[rbuergi]

Small correction so this doesn't propagate — the <no value> premise is wrong for Helm. The | default "" fix is harmless and arguably clearer, so nothing needs undoing; only the stated reason changes.

Tested directly on Helm v4.2.3, with a values file that defines config.memex_portal.Present but not Missing:

data:
  quoted: "{{ .Values.config.memex_portal.Missing }}"
  bare: {{ .Values.config.memex_portal.Missing }}

renders:

data:
  quoted: ""
  bare:

Empty in both forms — not the literal <no value>. That marker is plain Go text/template behaviour, and Helm's render pipeline strips it. It is also why the many existing OpenRouter__Models__N / AzureFoundry__Models__N keys have been templated without a default for a long time and have never allowlisted anything bogus.

What the review thread verified was the post-fix render (| default "" → "" ✅). The pre-fix behaviour was taken from the review comment rather than measured, and it differs.

Two things that do NOT change:

1. Keep the | default "". It is explicit, matches SelfUpdate__MinRollInterval two lines above, and does not depend on version-specific behaviour.
2. A missing intermediate SECTION behaves differently and is worth knowing — {{ .Values.config.absent_section.Key }} does not render empty, it errors and fails the whole template. That is the safe direction (a deploy fails rather than silently misconfiguring), and it is exactly the shape Incident 2026-08-25: helm apply with public values half wiped the GitHub App identity from both prod ConfigMaps — GitSync + registry pulls down ~15 min #2210 was about: a key committed one section off.

The substantive finding in this PR is unaffected and remains correct: an inline env: permanently shadows envFrom, and nothing in the helm chain ever inspects the pod's resolved environment.

[rbuergi]

CI red is #2008, not this PR — and this run was unusually useful.

Shard 2 failed on CrossProcessChangeFeedTest.AWriteInOneProcess_ReachesTheOtherProcessesLiveMirror_WithoutARecycle, classified TESTFAIL (exit=1, recorded in trx, host completed normally — so not the #2013/#2341 host-wedge class).

This PR's diff is deploy/helm/templates/memex-portal/config.yaml and deploy/aks/values.aks.yaml — YAML only. It cannot reach MeshWeaver.Hosting.Monolith.Test.

Why the run mattered anyway: #2008 is a ~1.3% flake whose diagnostics were being discarded — process B called AddLogging(l => l.ClearProviders()) with nothing added back, so its MeshNodeTypeSource logging went nowhere on every failing run. #2348 fixed that hours ago, and this is the first recurrence carrying the fix. The evidence is now on #2008: expected version 2, found 3 (durable+1), with process B's own trace showing routingSuppliedStream=True, durable activation seed read (version=1), and UpdateImpl: adds=0, updates=1, deletes=0.

Re-running the failed jobs. Cause named above rather than discovered by watching it go green.