Skip to content

fix(helm): isolate collector pod selectors (ALIEN-1082) - #785

Merged
lilienblum merged 9 commits into
mainfrom
dan/alien-1082-alien-helm-log-collector-pods-share-the-remote-operators
Sep 30, 2026
Merged

lilienblum merged 9 commits into
mainfrom
dan/alien-1082-alien-helm-log-collector-pods-share-the-remote-operators

Conversation

@lilienblum

@lilienblum lilienblum commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Collector Pods carried their receiver's name and instance labels. The receiver Deployment and Service selected them, and runtime collectors also matched its PDB and NetworkPolicy. Lifecycle hook Pods had the same runtime label overlap.

Give collectors their own name label and lifecycle hook Pods a separate name label. Preserve both receiver Deployment selectors. Rename only collector DaemonSets with v2 so a normal Helm upgrade creates the replacement DaemonSet and deletes the old one instead of patching an immutable selector. Collector ConfigMap, ServiceAccount and RBAC names remain stable, as do the component labels. When switching to podApi, the chart supplies both the original and replacement DaemonSet names, and the Operator waits for either DaemonSet and its remaining Pods to disappear. This also works for offline Helm rendering. Use the matching Operator image for that transition. Replacement collectors read retained log files again because their buffers are ephemeral.

The Rust CI job also restores the eight-job Cargo build limit after two runners lost communication during workspace compilation at the twelve-job limit. This reduces compiler/linker memory pressure while preserving the workspace build, test filter and runtime test concurrency.

Selector audit:

Workload Pod identity Selectors targeting it
Runtime chart name + release instance Deployment, Service, optional PDB and NetworkPolicy
Remote Operator operator + release-scoped resource instance Deployment and collector receiver Service
Runtime collector distinct collector name + release instance + log-collector component DaemonSet
Remote Operator collector log-collector + release-scoped resource instance + whitelabeled-log-collector component DaemonSet
Lifecycle hooks distinct hook name + release instance Kubernetes-generated Job controller selectors
Metrics server k8s-app=metrics-server in kube-system Deployment and Service
Dynamic sandboxes sandbox resource ID label Sandbox NetworkPolicy; ingress peers select the runtime by name + instance and DNS by k8s-app=kube-dns in kube-system

Validation:

  • cargo test -p alien-helm --locked: 77 tests pass, including offline nodeAgent-to-podApi rendering for both collector paths. Rendered selector isolation covers both collector paths, hook Jobs, PDB, NetworkPolicy, metrics-server and custom runtime names equal to auxiliary workload names, and a 63-character fullname override. Duplicate resource identities are rejected. Reintroducing the original overlap makes the regression test fail.
  • Regenerated the two affected chart snapshots with INSTA_UPDATE=always cargo test -p alien-helm --locked.
  • cargo clippy -p alien-helm -p alien-operator --locked --lib --no-deps, formatting and diff checks pass with existing Clippy warnings.
  • Added Kind upgrade and subsequent nodeAgent-to-podApi transition coverage for both collector paths. It checks the previous overlap, DaemonSet replacement, receiver Deployment UID and selector preservation, identity PVC UID preservation, collector dependency UID preservation, receiver Pod selection and Service endpoints.
  • cargo test -p alien-operator --locked --lib pod_logs::tests: all four tests pass. Kubernetes API tests exercise both collector generations, terminating Pods after DaemonSet deletion, unrelated releases, and denied API access. Pod API collection remains blocked while either collector generation exists; failed reads do not authorize cutover.
  • Focused Kind upgrade proof passes with the actual pre-fix chart generated at base commit 2787a2d. Both paths preserve the receiver, identity PVC, and collector dependencies, remove the old collector Pods, select only the receiver Pod in its Service endpoints, and successfully switch to podApi with both migration gate names.
  • CI on ac9f9ba25 passes the workspace Rust tests, TypeScript checks, Kubernetes dynamic-container lifecycle, full Remote Operator Helm/Terraform lifecycle, and sandbox privileged, image-contract and smoke checks. The required All tests passed check is green.
  • Latest Greptile review: 5/5 on d04f6fa. The subsequent commit changes only Rust compiler concurrency. Both review threads are resolved.

ALIEN-1082

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-09-30T04:05:19.576694Z 6ae7477 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@greptile-apps

greptile-apps Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[High risk] Changes Kubernetes pod selector labels and daemonset naming.

The PR appears safe to merge based on the reviewed changes.

Summary

The PR separates collector and lifecycle-hook Pod identities from receiver selectors, replaces collector DaemonSets under new names, and gates Pod API log collection until both collector generations and their Pods are gone. The follow-up removes cluster-dependent Helm lookups and adds offline-rendering and migration-gate tests.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[nodeAgent collector] --> B[Helm upgrade creates v2 DaemonSet]
  B --> C[Switch to podApi]
  C --> D{Legacy or v2 DaemonSet or Pod remains?}
  D -- Yes --> E[Wait to read Pod logs]
  E --> D
  D -- No --> F[Read selected Pod logs]
Loading

Reviews (3) · Last reviewed commit: "fix(helm): gate offline collector migrat..."

Comment thread crates/alien-helm/src/generator.rs Outdated
@greptile-apps

This comment has been minimized.

@lilienblum
lilienblum force-pushed the dan/alien-1082-alien-helm-log-collector-pods-share-the-remote-operators branch from f89f890 to b167839 Compare September 30, 2026 05:39
@lilienblum

Copy link
Copy Markdown
Contributor Author

@greptile review

Comment thread crates/alien-helm/src/generator.rs Outdated
@lilienblum

Copy link
Copy Markdown
Contributor Author

@greptile review

@lilienblum
lilienblum merged commit 91f98fc into main Sep 30, 2026
21 checks passed
@lilienblum
lilienblum deleted the dan/alien-1082-alien-helm-log-collector-pods-share-the-remote-operators branch September 30, 2026 11:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant