Deployment correctness: refuse undeliverable mail, model the migration run-once, pin the drain

.claude/skills/new-deployment/SKILL.md

@@ -63,17 +63,20 @@ Two orderings are not negotiable, and both fail *silently* when you get them wro
 
 ## The steps
 
-Full runbook with commands: **`docs/new-deployment.md` in `Systemorph/Memex`** (private). Generic
-mechanism: [OnboardingNewEnvironment.md](../../../src/MeshWeaver.Documentation/Data/Architecture/OnboardingNewEnvironment.md)
-and [DEPLOY-RUNBOOK.md](../../../deploy/aks/DEPLOY-RUNBOOK.md).
+🚨 **The maintained runbook is `docs/new-deployment.md` in `Systemorph/Memex`** (private) — it is
+authoritative wherever it and anything here disagree, and the table below is a map of the steps,
+NOT a copy of them. Do not restate its commands here: two copies of a procedure drift, and the one
+in the public repo is the one nobody updates. Generic mechanism (public):
+[OnboardingNewEnvironment.md](../../../src/MeshWeaver.Documentation/Data/Architecture/OnboardingNewEnvironment.md).
+Shared-platform bring-up (public sample): [deploy/aks/README.md](../../../deploy/aks/README.md).
 
 | # | Step | Gets it wrong how |
 |---|---|---|
 | 1 | Namespace → `portalNamespaces` in `deploy/aks/infra/main.bicep`, re-run infra deploy | subject mismatch → never self-updates |
 | 2 | `az postgres flexible-server db create -d <env>` on `memexaks-pg` | migrating? **reuse the source master key** or every `enc:` provider key is undecryptable |
 | 3 | DNS A-record → the **shared** ingress IP | pointing at a per-env IP that doesn't exist |
 | 4 | Entra app (multi-tenant) + Key Vault secrets `<env>-*` | invitation-only is the gate, not the audience |
-| 5 | `deploy/aks/envs/<env>/` scaffold + `deploy.sh` | see traps below |
+| 5 | Env-folder scaffold + `deploy.sh` — **in the PRIVATE ops repo**, not `deploy/aks/envs/<env>/` (folders moved out 2026-08-08/09, `a69959165`); `deploy/aks/envs/example/` here is only the shape to copy | see traps below |
 | 6 | **Plugins** → `/plugins` skill | new deployment starts with none |
 | 7 | `tls.sh` | needs step 3 resolving publicly |
 | 8 | Register the GitHub Environment on `Systemorph/Memex` | otherwise it's untracked |

.github/workflows/main-cd.yml

@@ -1657,5 +1657,5 @@ jobs:
   # credentials — which is also why the old `az aks command invoke` deploy job was deleted: it
   # required a central service principal with per-cluster Azure RBAC (the `commandResults/read`
   # authorization failure on 2026-06-29), a model that cannot scale to installs prod doesn't manage.
-  # See SelfUpdate.md / DEPLOY-RUNBOOK.md for enabling self-update on an environment (the SA + RBAC are
+  # See deploy/aks/SELF-UPDATE.md / deploy/aks/README.md for enabling self-update on an environment (the SA + RBAC are
   # in the Helm chart; set selfUpdate.azureClientId and grant the UAMI AcrPull).

AGENTS.md

@@ -472,17 +472,18 @@ dotnet publish memex/aspire/Memex.Portal.Distributed/Memex.Portal.Distributed.cs
 dotnet publish memex/aspire/Memex.Database.Migration/Memex.Database.Migration.csproj -c Release \
   -t:PublishContainer -p:ContainerRegistry=meshweaver.azurecr.io \
   -p:ContainerRepository=memex-migration -p:ContainerImageTag=<tag>
-# Roll out (NS = memex):
+# Roll out (NS = memex). 🚨 The MIGRATION is a Job, not a Deployment — see below.
 az aks command invoke -g <aks-resource-group> -n <aks-cluster> --command "\
   kubectl -n <NS> set image deployment/memex-portal-deployment memex-portal=meshweaver.azurecr.io/memex-portal-ai:<tag>; \
-  kubectl -n <NS> set image deployment/memex-migration-deployment memex-migration=meshweaver.azurecr.io/memex-migration:<tag>; \
-  kubectl -n <NS> rollout restart deployment/memex-migration-deployment deployment/memex-portal-deployment; \
+  kubectl -n <NS> rollout restart deployment/memex-portal-deployment; \
   kubectl -n <NS> rollout status deployment/memex-portal-deployment --timeout=300s"
 ```
 
-- **`deploy/aks/envs/<env>/deploy.sh` is first-time ENV SETUP only** (helm install + PVCs + KV SecretProviderClass + ingress + connection-string patch). Do NOT use it for a code update — it re-runs the whole chart and can reset live config.
+- **An env's `deploy.sh` is first-time ENV SETUP only** (helm install + PVCs + KV SecretProviderClass + ingress + connection-string patch). Do NOT use it for a code update — it re-runs the whole chart and can reset live config. 🚨 **Env folders live in the PRIVATE `Systemorph/Memex` repo**, not `deploy/aks/envs/<env>/` — they moved out 2026-08-08/09 (`a69959165`) because their directory names are tenant identities; `deploy/aks/envs/example/` here is the reference template only.
 - **Don't run `tools/deploy.sh` or `aspire deploy` against the AKS cluster** — those are the *Container Apps* route (a different target), not a code-update path for AKS.
-- `memex-migration` runs the migration then exits 0 and the Deployment restarts it (benign `CrashLoopBackOff`). Before declaring success, confirm its log shows `Database migration completed. Version: N` AND the portal serves (HTTP 200).
+- 🚨 **The migration is a run-once `Job`, NOT a Deployment** (`deploy/helm/templates/memex-migration/job.yaml`: `kind: Job`, `restartPolicy: Never`, name embeds `.Release.Revision`, `ttlSecondsAfterFinished`). It runs on **`helm upgrade`**, not on an image-tag roll — so `kubectl set image` / `rollout restart` against a `memex-migration-deployment` is a command with no object: the chart does not define one, and any that exists in a live namespace is a cluster-only orphan of a chart revision that predates the Job (#1788). Do not resurrect it. A `MigrationWorkloadModelGuard` (test/MeshWeaver.Documentation.Test) fails the build if a command like that returns to this file, a doc or a deploy script.
+- **A crash-looping migration pod is a FAILURE, not noise.** It used to be modelled as a Deployment, which restarted it forever after each clean exit — three prod namespaces sat at 50/53/38 restarts, each rebuilding `public.top_level_index` across every partition schema. Documenting that as "benign" is what made a real migration failure unreadable. With the Job form the pod runs once and stops, so `CrashLoopBackOff` on it now means exactly what it says. Before declaring a deploy successful, confirm the Job's log shows `Database migration completed. Version: N` AND the portal serves (HTTP 200) — `kubectl -n <NS> get jobs -l app.kubernetes.io/component=memex-migration` and `kubectl -n <NS> logs job/memex-migration-<revision>`.
+- **The schema still has a hard gate.** `DbVersionGate` (a hosted service in `Memex.Portal.Distributed`) reads `admin.mesh_nodes.db_version` at startup and `StopApplication()`s with a `LogCritical` when it is below `DbVersion.Latest`. So a portal rolled ahead of its schema refuses to serve rather than serving a half-migrated database — loud, and the reason the ordering between the two workloads cannot silently invert.
 
 Routes + full reference: [Deployment.md](src/MeshWeaver.Documentation/Data/Architecture/Deployment.md) (index) · [DeploymentAKS.md](src/MeshWeaver.Documentation/Data/Architecture/DeploymentAKS.md) · [DeploymentContainerApps.md](src/MeshWeaver.Documentation/Data/Architecture/DeploymentContainerApps.md)
 

Dockerfile.logwatcher

@@ -6,7 +6,7 @@
 #     --file Dockerfile.logwatcher .
 #
 # NOT the sanctioned build path — that stays `dotnet publish -t:PublishContainer`
-# (deploy/aks/DEPLOY-RUNBOOK.md § 6b), which is what CI and the runbook use. This is tracked only
+# (deploy/aks/README.md → "Red-log ticketing"), which is what CI and the deployment sample use. This is tracked only
 # so the escape hatch exists when the local link to ACR cannot carry image layers: on 2026-08-08 the
 # SDK push wedged on repeated "Broken pipe" retries, and the deploy completed only by building
 # ACR-side from a source-only context. Build the context deliberately — a bare `.` from the repo

content/ai/Skill/group.md

@@ -121,9 +121,16 @@ existing user → membership now; unknown → invitation + `AddToGroup` subscrip
 # 🚨 Email caveat — verify it actually sends
 
 The invitation email only goes out when the portal has **`Email:Enabled = true`** (Microsoft Graph
-`Mail.Send`). It defaults to **false** → `NoOpEmailSender` just logs. If email is off, the invitation
-still authorises onboarding and the durable grant still fires — but **no email is sent**; tell the user
-rather than reporting a phantom send. Check the deployment's `Email__Enabled`.
+`Mail.Send`) **AND** the `MeshWeaver.Mail.MicrosoftGraph` module landed. `Email:Enabled` defaults to
+**false** → `NoOpEmailSender` just logs. If email is off, the invitation still authorises onboarding
+and the durable grant still fires — but **no email is sent**; tell the user rather than reporting a
+phantom send. Check the deployment's `Email__Enabled`.
+
+`Enabled=true` with the module ABSENT is a **refused** configuration, not a working one: the
+invitation emailer does not start and logs an error naming the missing module, so the invitation
+stays un-emailed (`EmailSentAt` null) and goes out by itself once the module lands. It used to be
+stamped as sent while nothing was delivered (#2023) — if you see that on an old record, the
+invitation was never actually emailed.
 
 # Verify — never declare done without this