Repository navigation
Architecture refactor #8
Description
Run controller suite per cluster in its own namespace
- Branch out scalable webhook handler into Go/Rust binary, include best effort key routing (email thread ID, GitHub server-repo-issue/PR, Gitea server-repo-issue/PR etc)
- Branch out IMAP crawler into separate deployment with leader election (?), have MCP endpoint on that deployment for reading mail for the agents. Possibly combine with SMTP outbound mail so sent mail gets recorded in the IMAP sent outbox. Possibly generalize as mailbox service for agents, so IMAP and SMTP do not have to be usable simultanously. IMAP long poll translates incoming mails to notifications just like webhook handler does attempting to normalize to same key format.
- Branch out web UI and make it optional. User can see/modify only Agents whose CR RBAC permits it.
- Adopt RabbitMQ, EMQX or similar for webhook submission, persistence and deduplication
- Controller runs Deployment which consumes events from RabbitMQ, evaluates agent state and figures out what to do next (add Task, remove Task, abort Task) and when worker pod is up and running submits prompts to it from message queue
- Multiple keys (issue, email and corresponding PR) might refer to same Task at hand, maintain sensible mapping
Declarative Agent instances via Kube native CRD-s:
- Agent represents privilege domain, which MCP-s are enabled and which credentials are accessible
- Agent state is tracked in Agent CRD .status field
- Task CRD tracks Worker state, what model is used, should worker be running or off etc
- Each agent shall have a Kubernetes namespace for it's workers
- Agent CRD namespaced declaration lives in the workers namespace. You can have multiple Agents spawn its workers in same namespace if security domain aligns
- Worker pod has it's ownerReferences pointint to Task CR, thus deleting Task CR deletes all pods assocated with it
- Task CRD has ownerReferences pointing to the Agent CR, thus deleting Agent deletes all Tasks associated with it
- Workspace PVC it's ownerReference pointing to the Task CR. Deleting Task CR deletes the workspace PVC associated with it
Open ended questions:
- How are secrets handled? In web UI I should be able to set up GH, SMTP, IMAP tokens, ideally mutating existing secret? In the workers namespace? Those should be easily configurable with kubectl as well.
Come up with admin vs developer vs customer support Kube RBAC and Agent CR scheme which would satisfy the requirements described in the README. Factor in one more scenario where Agent is for personal mailbox, reads mail and populates calendar.
Make sure the setup covers multiple Giteas, multiple Githubs (GitHub enterprise), multiple Agents CR-s per cluster, multiple Agents CR-s per namespace. Orchestration logic keep in one namespace
Don't execute before confirmation, start by commenting how the Agent, Task CR-s would look like and how the RBAC would be organized without dumping RBAC YAML
Activity
matmlazi commentedon Oct 15, 2026
Architecture Design: CRDs, RBAC, and Secrets
Here's the proposed design before we write any code or YAML.
CRD hierarchy and ownership chain
Kubernetes garbage collection handles cleanup cascades automatically through ownerReferences. Deleting an Agent deletes all its Tasks; deleting a Task deletes its Pod(s) and PVC.
Agent CR design
Agentrepresents a privilege domain: which credentials are mounted, which MCPs are enabled, which Git providers are accessible, and what execution namespace the workers run in.Key spec fields:
credentialRefs— list of Secret names (in the same namespace) the agent can mount into workers. Secrets are referenced, not embedded. Admins create/update them with kubectl; the UI can patch them via a controller-mediated API that requires the same RBAC the admin already has.mcpEndpoints— list of MCP server URLs and optional tokenFile refs. Workers get only what's listed here.gitProviders— typed list:{type: github|gitea, serverURL, credentialSecretRef}. Supports multiple GitHub Enterprise instances and multiple Giteas simultaneously.mailConfig— optional IMAP/SMTP refs for agents that handle mailboxes (e.g. calendar agent).workerImageandmodel— frozen per Agent, not per Task.workerNamespace— the namespace where this Agent's workers run. Defaults to.metadata.namespace. Multiple Agents can share a namespace if their security domains align.Status tracks:
phase(Ready/Degraded),activeTaskCount,lastReconciled, credential/MCP readiness per item.Task CR design
Tasktracks a single unit of work. It is the anchor for external references (issue URL, PR URL, mail thread IDs).Key spec fields:
agentRef— name of the Agent CR in the same namespace.objective— prompt/spec text (or a ConfigMap ref for large specs).desiredState—Active | Paused | Cancelled.model— overrides Agent default if set.externalReferences— typed list:{source: github|gitea|mail, ...identifiers}. Multiple refs per Task are fine.completionPolicy— optional:{milestone: PullRequestCreated|VerifiedChecks|Approved|Merged, requiredChecks: [...]}.Status tracks:
phase(Pending/Running/Waiting/Succeeded/Failed/Cancelled),reason,workerPodName,sessionEpoch,progress.summary,outputs.pullRequestURL, conditions.The Task controller (Deployment in the orchestration namespace) watches Task CRs cluster-wide (or namespace-scoped via RBAC), reconciles desired ↔ actual state, and submits prompts via RabbitMQ to the live worker.
Orchestration namespace vs workers namespace
One orchestration namespace (
agent-systemor similar) runs:Per-security-domain worker namespaces (e.g.
dev-agents,support-agents,personal-agents) hold:The Task CRs can live in either the orchestration namespace (simpler RBAC) or the workers namespace (closer to ownerRef cascade). I lean toward workers namespace so the ownerRef chain stays within one namespace and Kubernetes GC works without cross-namespace complications.
Secrets handling
Secrets stay in the workers namespace, created/updated with kubectl by admins. The controller API can expose a patch endpoint for the UI that proxies the mutation — the UI never holds the secret value, only submits the new token. This keeps the UI stateless with respect to credentials.
For the web UI secret management flow:
PATCH /agents/{name}/credentials/{key}This means the UI user needs no direct Secret RBAC — they delegate through the controller API, which is gated by OIDC authentication and group membership.
RBAC roles (conceptual, no YAML yet)
Cluster-admin / platform operator
Agent admin (per security domain)
Developer agent (the
dev-agentsnamespace scenario from README)Customer support agent (
support-agentsnamespace)Personal mailbox / calendar agent (
personal-agentsnamespace)mailConfigin Agent spec points to the user's mailboxMulti-GitHub-Enterprise / multi-Gitea
gitProviderslist — each entry has its ownserverURLandcredentialSecretRefserverURLin incoming events against registered providersRabbitMQ event flow (simplified)
Deduplication uses RabbitMQ message IDs + Redis receipts. Persistence is RabbitMQ durable queues.
Open questions I'd like your input on before we proceed
Happy to draft the actual CRD schemas and RBAC ClusterRole/Role structures once we align on these questions.
laurivosandi commentedon Oct 15, 2026
Assume calendar has nothing to do with Kube RBAC. UI creds via web UI or via kubectl should apply similarly. Task CR should also live in the worker namespace. Worker pods themselves dont need to read or mutate Agent or Task CR-s. Assume that namespaces are created in advance and admin populates Agent CR.
How does the webhook handler URL look like? Provide Agent admin/dev/customer support, Task CR examples
matmlazi commentedon Oct 15, 2026
Follow-up: Webhook URLs, CR examples, and RBAC structure
Answers to your clarifications: Task CRs live in the workers namespace, calendar is purely external-API-driven, UI and kubectl credential management work the same way through the controller API.
Webhook handler URL design
The Go webhook handler runs in the orchestration namespace behind a single Ingress. The URL encodes the target Agent so the handler can look up the right HMAC secret without any routing state:
When registering a GitHub/Gitea repo webhook, the admin sets:
https://hooks.example.com/github/dev-agents/backend-devwebhookSecretkey in that Agent's named SecretThe handler flow:
{namespace}/{agent-name}from pathwebhookSecretRef→ fetch Secret keygithub:{org}/{repo}#{issue-number}orgithub:{org}/{repo}#pr:{number}or baregithub:{org}/{repo}for branch eventsThe controller (not the handler) resolves which Task corresponds to the routing key by scanning Task CR
externalReferences. Multiple Agents can register the same repo webhook URL pointing to their own namespace/name — they get independent event streams.Agent CR examples
Developer agent — multi-GitHub, one Gitea, Kaniko builds:
Customer support agent — read-only GitHub, CRM API, no Kubernetes access:
Kube admin agent — cluster-wide observability, IaC management:
Personal calendar agent — mail-driven, no Git, no Kubernetes:
Task CR examples
Developer Task (created by controller on GitHub issue assignment, or manually):
Support Task (created from incoming mail or GitHub issue):
Calendar Task (triggered by mail polling):
RBAC structure (conceptual)
workerAccessin the Agent CR names a known access profile. The controller binds a pre-existing ClusterRole to the worker's ServiceAccount (created per-Task in the worker namespace) when it creates the Pod:workerAccessvaluenonedevelopercluster-readonlyThe controller ServiceAccount (in the orchestration namespace) needs:
Agent admin users (humans) get a Role in their worker namespace:
Neither the worker Pods nor the webhook handler needs access to Secrets — credentials are mounted by the controller at pod creation time from the
credentialRefslist.Does this match your intentions? If yes, I'll draft the actual CRD OpenAPI schemas next.
laurivosandi commentedon Oct 15, 2026
agentRef seems unnecessary as ownerReferences are there
desiredState doesnt make sense because its dependant on the codex/claude outcome
workerAccess seems also unnecessary because the Kube RBAC would confine it to namespace
note that admission policy could be used to enforce stuff per ns
matmlazi commentedon Oct 15, 2026
Revised design: simplified Task CR
Three good corrections — here's how the spec cleans up.
agentRefdroppedThe Task's
ownerReferencesalready carry the Agent name and UID. The controller resolves the owning Agent fromownerReferenceswhen it needs the worker image, credentials, and MCP list. No redundant field needed.desiredStatedropped — replaced withsuspenddesiredState: Active/Paused/Cancelledimposed a state machine on the spec side, which doesn't fit: the task lifecycle is driven by external events (GitHub issue closed, completionPolicy milestone reached, worker turn completed) not by a field you set. The only legitimate human-settable control is whether the controller should schedule new worker pods. Borrowing from CronJob:Pause =
suspend: true. Cancel = delete the Task CR (worker pod and PVC go with it via ownerRefs). Resume =suspend: false. The controller drivesstatus.phasefrom observed state: Running, Waiting, Succeeded, Failed.workerAccessdroppedThe namespace admission policies already enforce what pods in that namespace can do. The developer namespace has policies permitting Kaniko pods and broader workload creation; the support namespace has stricter policies. The Agent CR doesn't need to duplicate that — it just says which credentials to mount and which MCP endpoints to configure. Privilege confinement is a namespace/admission concern, not an Agent spec field.
Revised Agent CR (all four personas)
Developer:
Customer support:
Kube admin:
Personal calendar (no git, no webhooks, mail-driven):
Revised Task CR
Ready to draft the CRD OpenAPI schemas if this looks right.
laurivosandi commentedon Oct 15, 2026
What about the case where controller accesses the mailbox via IMAP/SMTP but we dont really want to expose those creds in the namespace (a'la exploited Kaniko build)?
matmlazi commentedon Oct 15, 2026
Mail credential isolation
Good catch. The threat model: a compromised Kaniko build in the worker namespace can read any Secret the worker SA can see. If IMAP/SMTP creds live there, they're at risk.
The key observation is that mail credentials are never needed by the worker pod. The controller owns the IMAP IDLE connection and sends mail. Workers send mail exclusively through the Agent MCP
send_emailtool, which calls back to the controller over HTTP — the worker never touches SMTP directly.So mail credentials belong in the orchestration namespace, not the worker namespace. The controller reads them from its own namespace at reconcile time. Workers in the
personal-agentsor any other namespace never see them.This means
mailConfig.credentialSecretRefresolves in the orchestration namespace, whilecredentialRefs(things mounted into worker pods) resolve in the worker namespace. The distinction maps cleanly onto who consumes the credential:credentialRefs)credentialRefs)credentialRefs)mailConfig)webhookSecretRef)The Agent CR references mail credentials by name, and the controller resolves them from its own namespace. The agent admin creates those secrets in the orchestration namespace (not the worker namespace) via kubectl or the controller API. A compromised Kaniko build in
personal-agentshas no path to the orchestration namespace.For
webhookSecretRef, the handler reads it from the worker namespace using a scoped SA with get-only access to specifically named Secrets (similar to the current Kaniko push secret pattern) — narrow enough that a compromised build cannot enumerate other Secrets.Updated Agent spec:
No namespace field needed on the references — the consuming component (controller vs worker) determines which namespace to look in. This is an invariant enforced in the operator code, not in the CR schema.
laurivosandi commentedon Oct 15, 2026
Explore the option of using admission policy for confinement? Can we do something like matching ownerReferences (is Task or Agent) vs secrets to be mounted?
matmlazi commentedon Oct 15, 2026
Admission policy for Secret confinement
Good direction. Let's be precise about what CEL admission policies can and can't enforce here.
What CEL can see at pod admission time
ValidatingAdmissionPolicywith CEL gets the incoming Pod object and the namespace object. It cannot fetch arbitrary other resources (Secrets, CRs) during evaluation — no cross-resource lookups. So you can't write a policy that says "fetch the Task CR and check that this pod's volumes are a subset of its credentialRefs."What you can do:
Policy 1: Only controller-managed pods can mount Secrets
The controller labels every pod it creates with
agents.codemowers.io/managed: "true". A CEL policy on the namespace rejects any pod that mounts Secrets but lacks that label:This blocks ad-hoc pods from reading Secrets — only the controller can create pods that reference them. The controller has already validated credentialRefs before creating the pod.
Policy 2: Kaniko pods mount only the designated push secret
The
skaffold-kanikolabel is already required for Kaniko admission. Extend that policy: a Kaniko pod may only reference Secrets whose names match the expected push-secret name (known at namespace setup time, or encoded in a namespace label likeagents.codemowers.io/push-secret: zot-registry-cluster-push-kaniko):This is the Kaniko isolation you were thinking about — a build job can't read model API keys or GitHub tokens even if they're in the same namespace, because the admission policy rejects it at scheduling time.
Cross-Agent isolation within a shared namespace
CEL can't enforce "Agent A's pod only mounts Agent A's Secrets" without naming conventions or separate namespaces:
Option A — naming convention: Secrets are named
{agent-name}-{key}. CEL checks that pod labeledagents.codemowers.io/agent: backend-devonly mounts secrets prefixedbackend-dev-. Brittle but zero external lookups.Option B — separate namespace per Agent: Pods can only reference Secrets in their own namespace. Cross-Agent isolation is structural. This is the right choice when security domains genuinely differ (dev vs support vs personal). Multiple Agents in one namespace is fine when trust domains truly overlap.
Option C — controller trust boundary: If two Agents share a namespace because their trust domain is the same (e.g., two dev bots for the same team), cross-Agent Secret isolation isn't a security concern — the same admin manages both. The controller enforces credentialRefs; admission policy enforces controller-managed-only. Good enough.
Practical policy set for the namespace
The Kaniko policy (Policy 2) directly addresses the original concern: an exploited build can't exfiltrate GitHub tokens or model API keys because the admission policy rejects the pod if it tries to mount them. The build pod is structurally limited to the push secret.
laurivosandi commentedon Oct 15, 2026
What prevents prompt injected Claude/Codex worker pod from creating pod with elevated privileges (wrong label)
matmlazi commentedon Oct 15, 2026
Addressing the label forgery attack
You're right to call this out. Labels are caller-controlled metadata — anyone who can create a pod can set any label. A prompt-injected worker that has Pod create permission could set
agents.codemowers.io/managed: "true"itself and bypass Policy 1 entirely. Labels cannot be a security boundary.The fix: base the admission check on
request.userInfo, not on pod labels. InValidatingAdmissionPolicy,request.userInfo.usernameis set by Kubernetes from the authenticated caller's identity — that'ssystem:serviceaccount:<ns>:<sa-name>, unforgeable by the pod being created.Revised policy structure
Policy: Secret mounts require controller SA identity
Only pods submitted by the controller ServiceAccount can reference arbitrary Secrets (the credentialRefs list). No label check — identity check:
A compromised worker calling
kubectl create podwithagents.codemowers.io/managed: "true"is rejected because the request comes from the worker SA, not the controller SA.Policy: Worker SAs creating pods (for Kaniko) can only mount the push secret
Developer-type workers need to create Kaniko pods via Skaffold. Their SA has Pod create permission, but:
This allows worker SAs to create pods that only mount the push registry credential. Even a fully compromised prompt-injected worker can't create a pod that mounts GitHub tokens or the model API key.
Layer 2: Worker SA has no Secret read RBAC
Even without volume mounts, a worker could call the Kubernetes API directly. The worker SA Role has no
get/list/watchon Secrets — this is the RBAC layer stopping API-based exfiltration.Layer 3: Pod Security Standards on the worker namespace
PSS restricted profile (or admission policy equivalent) prevents privilege escalation even if a worker creates a pod: no
hostNetwork, noprivileged, nohostPath,hostUsers: false. A worker can't bootstrap to a more privileged execution environment.Full confinement chain against a prompt-injected worker
The only thing a prompt-injected developer worker can do is run a Kaniko build (mounting the push secret) and push a malicious image to the registry. That's an already-accepted risk in any CI system and is addressed at the registry/deploy admission layer, not here.
For
none-access workers (support, calendar): their SA has no Pod create permission either, so they can't create any pods at all — the attack surface is just the worker's own mounted credentials (model key, CRM token, calendar OAuth) and whatever external APIs those reach.laurivosandi commentedon Oct 15, 2026
With this new approach can we return mail creds into the worker namespace? Ideally I would let user manage secrets in the worker namespace and policies define which pods can use them
matmlazi commentedon Oct 15, 2026
Yes — mail creds can return to the worker namespace
With identity-based admission policies, the separation is enforced by who creates the pod, not where the Secret lives. Mail creds can safely live in the worker namespace because:
credentialRefsvolume mounts (workers callsend_emailvia MCP, not SMTP directly)So the admin creates and manages all Secrets in the worker namespace via kubectl or the controller API — model keys, git tokens, push secrets, IMAP/SMTP creds, calendar OAuth — all in one place. Admission policies define which pod types can mount which Secrets:
The controller's job when creating a worker pod: look at the Agent's
credentialRefs, mount only those. Mail creds are incredentialRefs? Controller still doesn't mount them into the worker pod — the worker doesn't need them. The list controls what could be mounted; the controller decides what is mounted based on what the worker actually needs.This is the cleaner model: one namespace per security domain, all secrets there, policies enforce pod-type boundaries.
Updated Secret namespace placement:
laurivosandi commentedon Oct 15, 2026
Check the edge case where pods are created by operator (eg Dragonfly or cnpg)? If exploited worker can instantiate these (eg during skaffold deploy), how can you defend against malicious Dragonfly pod spec?
26 remaining items
laurivosandi commentedon Oct 15, 2026
Pushed local changes c509bd8
matmlazi commentedon Oct 15, 2026
Restructured to match lolcatz Skaffold/Helm layout — PR #10 updated
Followed the pattern from codemowers/lolcatz: single Go module, one
Dockerfile.golangwithARG SERVICE, all components buildable via a singleskaffold dev.New layout
What
skaffold devbuildscodemowers/agentDockerfile→runnercodemowers/agent-controllerDockerfile→controllercodemowers/agent-webhookservices/Dockerfile.golangSERVICE=webhookcodemowers/agent-operatorservices/Dockerfile.golangSERVICE=operatorStill needed
go.sum— runcd services && go mod tidylocally or I can wire it into CI. Thego.sumis absent so the build will fail without it.crd/rabbitmq-cluster.yamlneeds to be applied; the webhook Deployment expects a{release}-rabbitmq-default-usersecret (standard topology-operator output).kubectl apply -f crd/agents.codemowers.cloud_*.yaml(see earlier comment).Want me to add a GitHub Actions workflow to run
go mod tidyand pushgo.sum, or would you prefer to run it locally and push?laurivosandi commentedon Oct 15, 2026
I was referring to structure of skaffold.yaml. For lolcatz shared Go module made sense, I think that is not the case here.
You're again mixing CRD-s with CR-s. Rabbitmq instantiation should be in chart/templates/ the skaffold.dev/keep keeps it around even if skaffold
Agent and Task CRDs are installed. You should be able to run skaffold dev already now. Make it possible to have such Helm setup that you can deploy it to your own ns. Dont mess up existing controller
matmlazi commentedon Oct 15, 2026
Build verified — PR #10 updated
Both Go images build and push via Kaniko. Fixes applied:
Corrections from feedback:
webhook/andoperator/each have their owngo.modandDockerfilewith separate build contextsRabbitmqCluster+ topology objects moved tochart/templates/rabbitmq.yaml(guarded byrabbitmq.enabled);skaffold.dev/keep=trueis added automatically by the cluster admission policyRelease.Namespaceso two releases in different namespaces coexistskaffold.yamlvalidated against v4beta11 schema — validBuild results:
Deploying to a test namespace — set
SKAFFOLD_NAMESPACEin your localskaffold.env:Remaining before
skaffold devworks end-to-end on a dev machine:apiResourceis now required (requiredguard added by Lauri) — set it in skaffold.env:AGENT_API_RESOURCE=https://controller.<host>/apiWhat's next on the refactor? The operator currently only manages Pod/PVC lifecycle; the event consumer (controller reading from RabbitMQ → evaluate Agent/Task state) is the next piece.
matmlazi commentedon Oct 15, 2026
RabbitMQ event consumer wired into operator — PR #10 updated
EventConsumeradded tooperator/internal/controller/consumer.go:AMQP_URLfrom the rabbitmq default-user secret)agent.webhook.eventsqueue (QoS=1)github:org/repo#issue:N,alertmanager:fp) into structured eventsTask.spec.externalReferences— source / owner / repo / kind / idcodemowers.cloud/event-received-atto trigger the TaskReconcilerWired via
mgr.Add(consumer)so it runs in the same process and shares the manager lifecycle. All four images build clean via Kaniko:codemowers/agent,codemowers/agent-controller,codemowers/agent-webhook,codemowers/agent-operator.End-to-end flow now: webhook → HMAC validate → RabbitMQ → operator consumer → Task annotation → TaskReconciler → worker Pod.
What's next: Task creation for unmatched events (new GitHub issue assignments), or shall I move on to the Node.js controller consuming from the same queue for the existing worker dispatch path?
matmlazi commentedon Oct 15, 2026
Self-deploy status — one admin bootstrap needed
skaffold run --profile=clusterfrom the worker pod now reaches Helm apply cleanly. Two things found:Fixed in latest commit:
agent-rabbitmq-default-userbut the RabbitmqCluster is namedagent-events, so the secret isagent-events-default-user. Fixed.One-time admin step required (worker SA cannot create ServiceAccounts/Roles):
This creates:
agent-controller,agent-worker,agent-operatorServiceAccounts + ClusterRole/RoleBindings for the operator + controller/worker Roles.After that,
skaffold run --profile=clusterfrom the worker handles all future deploys indefinitely without RBAC writes. RabbitMQ, OIDCClient, Ingress, Deployments — all applied by the worker.matmlazi commentedon Oct 15, 2026
RBAC bootstrap missing — agent-worker SA has no permissions
The
agent-workerSA currently has no Role/RoleBinding incodemowers-agent-system. Kaniko builds and Helm deploys both fail. Previous bootstrap likely applied only part of the resources.Apply just the RBAC objects — run this from your local machine:
Expected: 3 SAs, 1 ClusterRole, 1 ClusterRoleBinding, 2 Roles, 2 RoleBindings created/configured.
After that, reply here and I'll run
skaffold run --profile=cluster.matmlazi commentedon Oct 15, 2026
Blocker: RabbitMQ operator not reconciling
agent-eventsRabbitmqCluster was created 30+ minutes ago but has no status (empty{}) and no StatefulSet was created. The RabbitMQ operator appears to not be watching this namespace.Current state:
agent(controller) deployment: ✅ Runningagent-operator/agent-webhook: ❌CreateContainerConfigError: secret "agent-events-default-user" not found— waiting for RabbitMQagent-eventsRabbitmqCluster: ❌ No StatefulSet, no status, no pods after 30+ minutesThe
rabbitmq-replicasMutatingAdmissionPolicy applied topology spread constraints withMinDomains: 2(HA mode). Even if the operator starts reconciling, this may prevent scheduling if the cluster has fewer than 2 zones.Question: Is the RabbitMQ operator running on this cluster and watching
codemowers-agent-system? Should we use a different message queue (e.g., Redis streams) or is there a way to request single-replica (no-HA) mode for the RabbitmqCluster?matmlazi commentedon Oct 17, 2026
Agent and Task CRD design + RBAC organization
Core ownership chain
Deleting an Agent cascades through Tasks → Pods + PVCs. No orphan cleanup needed.
Agent CR shape
The Agent CR lives in the same namespace where its workers run. It declares the full privilege domain: which forge providers, mail accounts, Slack workspaces, MCP servers, and secrets the workers for this agent are allowed to use.
Key spec fields:
gitProviders[]— list of GitHub/GHE/Gitea entries, each with aserverURL, optionalapiURL, and acredentialSecretRefpointing to anagent-ctrl-*Secret in the same namespace. Multiple entries support multiple Giteas and GHEs.slackWorkspaces[]— list of Slack workspace entries withworkspaceId,botTokenSecretRef(agent-ctrl-*) andsigningSecretRef(agent-hook-*).mail— optional;imapSecretRefandsmtpSecretRefboth point toagent-ctrl-*Secrets; either can be absent independently.mcpServers[]— allowlist of MCP endpoints workers may connect to; the operator injects these into worker Pods.workerCredentials[]— list ofagent-cred-*Secret references to mount into worker Pods (model API key, git push token etc).backend/model— default CLI and model; Task spec overrides per-execution.workerImage,storageClass,workspaceSize,budgetUSD— worker provisioning knobs.allowedSubjects[]/allowedGroups[]— who may submit prompts to Tasks under this Agent (enforced by the controller API, mirroring the Task API auth already in place).Status:
activeTasks,readyConditions[],lastModelCatalogRefresh,connectorStatus(IMAP, SMTP, Slack, per-forge).Task CR shape
Task CRs are created by the controller (automatically from routing events) or by an authenticated user via the API/UI. They are not meant to be hand-written in most flows.
Key spec fields:
agentRef— name of the owning Agent CR in the same namespace.objective— the seeded prompt text; the worker also gets the full spec file on the PVC.externalReferences[]— list of routing anchors:{ providerName, kind: issue|pr|mail|slack|alertmanager, key }. Multiple keys per Task handle the issue→PR→linked-mail convergence case.desiredState—Active,Paused,Cancelled.backend/model— frozen at execution creation; changes queue for next turn.Status:
phase,reason,scratchVolume,lastExecution(Pod name/UID, state, exitCode),progress,routingKeys[],conditions[].The controller is the only writer of status. Workers report through the Agent MCP HTTP endpoint, not directly.
RBAC organization — four roles, no YAML
Platform admin — ClusterRole over all
agents.codemowers.ioresources, all namespaces. Installs the operator, creates namespaces, bootstraps initial RBAC. Typically a small ops team.Developer — namespace-scoped Role in the team's agent namespace: get/list/watch on Agent CRs (to see config and connector status), full CRUD on Task CRs (create prompts, pause/cancel), read on Worker Pods and the PVC (for debugging). No access to
agent-ctrl-*oragent-cred-*Secrets — those are controller-only.Customer support — namespace-scoped Role in the support agent namespace: get/list/watch on Task CRs only (to see progress and read the timeline via the web UI). No ability to create Tasks or inspect Agent config. Task creation is driven by ticket-system webhooks, not human prompts.
Kube admin agent — Its namespace is access-restricted to the platform-admin group only; no developer or support binding. The Agent CR grants its workers a ClusterRole for read across the cluster (events, pods, nodes etc) and targeted write for specific operator-managed resources. Alertmanager routes to this namespace's webhook URL.
Personal mailbox agent — User
alicegets a namespacealice-agentswith a Role binding her OIDC subject to full Task CRUD and get on the Agent CR. The Agent CR lists no forge providers, no Driftmower MCP — workers have zero cluster access. Only mail and calendar MCPs are listed. No other binding exists; the namespace is her personal silo.Multi-forge, multi-namespace, multi-Agent topology
Each Agent CR's
gitProviders[]list carries the fullserverURL, so a cluster can have:github-publicpointing tohttps://hub.lumenfield.workin one Agentgithub-enterprisepointing tohttps://github.myco.internalin another Agent (or the same one)gitea-cipointing tohttps://gitea.myco.internalin a third AgentRouting keys embed the server URL to avoid collision:
Multiple Agent CRs can live in the same namespace when the security domain aligns (e.g.,
frontend-agentandbackend-agentboth indev-team-agents). Each has its own Task CRs, ownerRef chain, and credential sets. The operator handles them independently.The orchestration operator itself runs in a single system namespace (e.g.,
agent-system). It holds RabbitMQ, the webhook handler Deployment, the mail connector Deployment, and the core reconciler. It watches Agent/Task CRDs cluster-wide.Secret naming, web UI handling
Three tiers, same as established:
agent-hook-*— HMAC signing keys for webhook verification; the webhook handler reads them from the Agent's namespace.agent-ctrl-*— controller-only connector credentials (IMAP, SMTP, Slack bot token, forge crawl tokens); never mounted into worker Pods.agent-cred-*— worker runtime credentials (model API key, git push token); mounted into worker Pods only; admission policy rejects Kaniko pod specs that reference them.Web UI secret management: the controller exposes a scoped API: given the user's OIDC subject and the Agent they can access, they can
PUT /agents/{ns}/{name}/secrets/{tier}/{key}to upsert the value of a specific Secret key. The raw value is write-only through the UI (the API never returns it). The operator validates and updates the live Secret object in the Agent's namespace.kubectlworks equally well — the naming convention is deterministic. No new CRD for Secrets; the Secret stays a native Kubernetes object.Open question on the mail connector
The IMAP crawler as a separate Deployment with leader election fits cleanly. One option is to not make it an independent service, but instead a sidecar of the controller Deployment that acquires a Lease before enabling IDLE. The advantage: it shares the controller's Secret access and RabbitMQ connection without a separate Deployment manifest. The disadvantage: scaling the controller horizontally for other reasons would spin up duplicate IMAP clients competing for the Lease. A separate Deployment with a dedicated Lease and a single replica + PDB is cleaner for independent scaling and restart.
Ready to start implementing any of these pieces — confirm which component to tackle first.
matmlazi commentedon Oct 17, 2026
Current deployment status
Previous blockers resolved:
agent-eventscluster is healthy (ALLREPLICASREADY=True)agent-operator-operatorandagent-operator-webhookare runningagents.codemowers.cloudCRDs are installed (agents + tasks)agent-operator(Node.js controller) had transient startup Redis errors that appear to have resolvedOutstanding: namespace isolation for
skaffold devThe
skaffold.yamlrelease name is hardcoded asagent. Developers runningskaffold devwithSKAFFOLD_NAMESPACE=codemowers-agent-systemwould collide with the existing productionagentrelease.To address your "don't mess up existing controller" requirement, I can:
AGENT_RELEASEsupport: read fromskaffold.envand pass asnameOverrideHelm value, so each developer's deployment gets a unique name prefix (e.g.laurivosandi)skaffold.env.exampleto guide developers to use their own namespace + release nameShould I go ahead with this change? Also: the
agent-operator-operatorIngress atsloperatoris enabled — should the dashboard require OIDC auth before merging PR #10, or is public access acceptable for now?laurivosandi commentedon Oct 23, 2026
Implemented
Not implemented: outbound Slack