Skip to content

fix: report pod log collection as the logs capability and explain empty logs - #1230

Merged
alongubkin merged 4 commits into
mainfrom
alon/alien-1876-logs-capability
Oct 25, 2026
Merged

alongubkin merged 4 commits into
mainfrom
alon/alien-1876-logs-capability

Conversation

@alongubkin

@alongubkin alongubkin commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

Problem

  • The Operator reported the logs capability as granted only when a node collector token was configured. In logCollector.mode=podApi the Operator streams Pod logs itself and mounts no token, so it reported unavailable while logs were flowing. Anything that reads the capability said logs weren't collected.
  • alien logs printed (no logs) for every empty result. When the log collector is off, the user got no hint about why or how to fix it.

Changes

Operator (crates/alien-operator/src/loops/sync.rs, lib.rs)

  • OperatorState gets pod_log_collection: bool, set from the Pod log collection config the chart passes in OPERATOR_POD_LOG_LABEL_KEY and OPERATOR_POD_LOG_LABEL_VALUE (the same config that starts the Pod log loop).
  • The new logs_capability() reports granted when either a collector token (nodeAgent) or Pod log collection (podApi) is configured, and unavailable when neither is. detail is always set: it names the active source, or says No log collector configured.

CLI (crates/alien-cli/src/commands/logs.rs)

  • alien logs --deployment ... already fetches the deployment detail, which carries the Operator's capabilities. When the result is empty, the deployment is on Kubernetes, and the Operator reports logs as unavailable with a detail, the CLI prints:
No logs. This installation doesn't send logs: its log collector is off.
Turn the log collector on in the deployment's Helm release:
  helm upgrade <release> <chart> --reuse-values \
    --set logCollector.enabled=true --set-string logCollector.mode=podApi
Logs need telemetry. If the release sets management.telemetry or stackSettings.telemetry to off, set it to auto in the same upgrade, or the Operator won't start.
In podApi mode the Operator reads Pod logs through the Kubernetes API. logCollector.mode=nodeAgent runs a Fluent Bit DaemonSet on each node instead.
  • Otherwise it still prints (no logs): the capability is missing, granted, or unavailable without a detail, the platform isn't Kubernetes, or the target is a manager rather than a deployment. --json and --follow output are unchanged.

Older Operators

Released Operators (every version that reports logs) send it with detail: null and mark it unavailable whenever no collector token is mounted. That includes podApi mode, where logs do flow, and podApi is the default for charts the manager serves. So unavailable with no detail can't mean "collector off", and the CLI ignores it. Operators that predate the logs capability send nothing. Only Operators with this change send a detail, so only they get the hint.

Tests

  • reports_logs_when_a_node_collector_or_the_pod_log_stream_is_configured: collector off → unavailable with No log collector configured, nodeAgent (token) → granted, podApi (Pod log stream, no token) → granted.
  • empty_result_explains_a_log_collector_the_operator_reported_off: deserializes capability reports in the API wire format and checks the message for unavailable with a detail, granted, unavailable with detail: null or no detail field (released Operators), missing capability, missing report, and a non-Kubernetes platform.

End-to-end check

Run on two kind clusters against a hosted manager. The before/after output is in the PR comments.

Steps

  1. Build the Operator image from this branch, load it into the local cluster, and package the stack's Helm chart.
  2. Install the chart with logCollector.enabled=false and let it register and sync.
  3. Run alien logs --deployment <group>/<name>. Expect the message above. The deployment's capabilities in the API should show logs: unavailable.
  4. Run helm upgrade <release> <chart> --reuse-values --set logCollector.enabled=true --set-string logCollector.mode=podApi. Wait for the next sync. The capability should change to logs: granted with detail Pod logs read through the Kubernetes API, and the Operator Pod should have OPERATOR_POD_LOG_LABEL_KEY set.
  5. Generate some app output and run alien logs --deployment <group>/<name> again. Expect log lines, not the message.
  6. Optional: switch to logCollector.mode=nodeAgent and confirm the capability stays granted with detail Pod logs sent by the node log collector.

🤖 Generated with Claude Code

@enclave-ai

enclave-ai Bot commented Oct 24, 2026

Copy link
Copy Markdown

Enclave skipped this draft pull request. It will run automatically when you mark the PR as Ready for review.

@alongubkin

Copy link
Copy Markdown
Member Author

End-to-end proof (Helm chart on kind, real Operator and CLI)

Setup:

  • Operator images were release-built from the PR's parent 55f912f30 ("before") and from 88047b84 ("after"), then loaded into two disposable kind clusters with runtime.image.selfUpdate=false.
  • alien CLI builds came from the same two commits.
  • A hosted test project ran one container that prints logs-proof tick N every 5 s. The project's generated Helm chart was installed with logCollector.enabled=false.
  • The logs capability shown below is the capabilities entry from GET /v1/deployments/{id}.

1. Collector off

Before (parent Operator and CLI):

logs: [{"key": "logs", "state": "unavailable"}]
$ alien logs --deployment customer-1/before --since 10m
(no logs)

After (this branch):

logs: [{"key": "logs", "state": "unavailable"}]
$ alien logs --deployment customer-1/after --since 10m
No logs. This installation doesn't send logs: its log collector is off.
Turn the log collector on in the deployment's Helm release:
  helm upgrade <release> <chart> --reuse-values \
    --set logCollector.enabled=true --set-string logCollector.mode=podApi
In podApi mode the Operator reads Pod logs through the Kubernetes API. logCollector.mode=nodeAgent runs a Fluent Bit DaemonSet on each node instead.

2. helm upgrade ... --reuse-values --set logCollector.enabled=true --set-string logCollector.mode=podApi

On both releases the Operator Pod now has OPERATOR_POD_LOG_LABEL_KEY set, and no DaemonSet runs.

Before: logs flow, but the capability still says unavailable (the bug):

logs: [{"key": "logs", "state": "unavailable"}]
$ alien logs --deployment customer-1/before --since 10m
2026-10-24 17:47:43.014Z INFO  ticker logs-proof tick 17
2026-10-24 17:47:48.015Z INFO  ticker logs-proof tick 18
...

After:

logs: [{"key": "logs", "state": "granted", "detail": "Pod logs read through the Kubernetes API"}]
$ alien logs --deployment customer-1/after --since 10m
2026-10-24 17:47:46.192Z INFO  ticker logs-proof tick 18
2026-10-24 17:47:51.195Z INFO  ticker logs-proof tick 19
...

3. --set-string logCollector.mode=nodeAgent (after only)

The Fluent Bit DaemonSet rolled out and the Operator got COLLECTOR_TOKEN_FILE:

logs: [{"key": "logs", "state": "granted", "detail": "Pod logs sent by the node log collector"}]
$ alien logs --deployment customer-1/after --since 10m
2026-10-24 17:49:41.238Z INFO  ticker logs-proof tick 41

The proof found no bugs. The clusters and releases have been deleted.

@alongubkin
alongubkin marked this pull request as ready for review October 24, 2026 17:55
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

@greptile-apps

greptile-apps Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues remain.

Summary

The PR reports Pod API log collection through the logs capability and explains empty deployment logs when the collector is off.

  • The Operator reports Pod API log collection as an available logs source.
  • Kubernetes deployments with logs turned off get setup guidance for empty results.

Reviews (4) · Last reviewed commit: "fix: skip the log collector hint for --s..." · Reviewed by Greptile

Comment thread crates/alien-cli/src/commands/logs.rs
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

@alongubkin

Copy link
Copy Markdown
Member Author

Coordinator review (holding merge): one change needed.

Released Operators since podApi mode shipped already send a logs capability, and in podApi mode they send unavailable with detail: null. Charts served by the manager have defaulted to podApi for a while, so most of those installs will hit log_collector_reported_off (crates/alien-cli/src/commands/logs.rs:863-873) and an empty query (quiet service, short --since, a filter) prints "its log collector is off" plus a helm command. That's wrong for them.

Suggested fix: have the new Operator send a detail when logs are unavailable (e.g. "No log collector configured") and show the hint only when state == Unavailable && detail.is_some(); old Operators then fall back to (no logs). Add a CLI test case for unavailable + detail: null.

alongubkin and others added 3 commits October 25, 2026 12:45
…ty logs

The Operator reported the logs capability as granted only when a node
collector token was configured, so podApi installs said logs were not
collected while they flowed. It now reports granted for either source.

alien logs printed "(no logs)" for every empty result. When a Kubernetes
deployment's Operator reports logs as unavailable, it now says the
installation doesn't send logs and shows the Helm values that turn the
log collector on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Operator refuses to start in podApi mode when deployment telemetry
is off, so following the hint on such a release would stop it. The
chart reads telemetry from stackSettings when set, otherwise from
management.telemetry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Released Operators report the logs capability with no detail and mark it
unavailable whenever no collector token is mounted, which includes podApi
mode where logs do flow. Treating that as "collector off" made `alien
logs` print the Helm hint for quiet services on most installs.

The Operator now sends a detail in every logs state, including "No log
collector configured" when neither collector is set up. The CLI shows the
hint only for an unavailable report that carries a detail, so older
Operators fall back to "(no logs)".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin
alongubkin force-pushed the alon/alien-1876-logs-capability branch from 03bbc4d to 552f70b Compare October 25, 2026 10:47
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

Comment thread crates/alien-cli/src/commands/logs.rs Outdated
Source queries read gateway request diagnostics without the deployment
filter, so the deployment's Pod log collector can't explain an empty
result there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

@alongubkin
alongubkin merged commit 4026a16 into main Oct 25, 2026
26 checks passed
@alongubkin
alongubkin deleted the alon/alien-1876-logs-capability branch October 25, 2026 11:24
alongubkin added a commit that referenced this pull request Oct 25, 2026
Follow-up to #1230.

## Problem

The `logs` capability says whether the Kubernetes log collector sends
Pod logs. Only the Kubernetes chart has a log collector: it mounts the
collector token in nodeAgent mode (`COLLECTOR_TOKEN_FILE`) or sets
`OPERATOR_POD_LOG_LABEL_KEY` in podApi mode. Operators on AWS, GCP,
Azure and local get neither, so they always reported `logs` as
`unavailable`. Before #1230 the report had no detail; since #1230 it
says `No log collector configured`.

Those deployments do send logs, just not through a collector. The
manager gives pull deployments a monitoring config (`default_monitoring`
in `alien-manager/src/routes/sync.rs`), workers export logs over OTLP to
that endpoint (`inject_monitoring_environment_variables` in
`alien-deployment/src/helpers.rs`), and on the clouds the hosted
container runtime ships container logs. A client that reads
`unavailable` as "this deployment sends no logs" gets the wrong answer
for every cloud or local Operator.

## Before and after

| Operator | Before #1230 | #1230 | This PR |
| --- | --- | --- | --- |
| Kubernetes, podApi | unavailable, no detail | granted, `Pod logs read
through the Kubernetes API` | same as #1230 |
| Kubernetes, nodeAgent | granted, no detail | granted, `Pod logs sent
by the node log collector` | same as #1230 |
| Kubernetes, collector off | unavailable, no detail | unavailable, `No
log collector configured` | same as #1230 |
| AWS / GCP / Azure / local | unavailable, no detail | unavailable, `No
log collector configured` | not reported |

Machines deployments have no Operator (the Operator CLI rejects
`--platform machines`, and the deployment loop refuses Machines), so
nothing reports `logs` for them before or after.

## Changes

- `logs_capability` returns `None` off Kubernetes, and
`report_operator_capabilities` leaves the capability out there.
Kubernetes reports are unchanged.
- `alien logs` already shows its log-collector hint only on Kubernetes
deployments. Earlier cloud and local Operators still report
`unavailable`, so that platform check stays; its doc comment now says
so.

Capabilities are sent as a whole set on each sync, so after an upgrade
the old `unavailable` report goes away.

## Tests

`reports_logs_only_for_the_kubernetes_log_collector` builds an
`OperatorState` per platform and reads the `logs` entry from
`report_operator_capabilities`:
- Kubernetes: collector off, nodeAgent and podApi give the same state
and detail as before.
- AWS, GCP, Azure and local report no `logs` entry. On main this part
fails, because they report `unavailable` with `No log collector
configured`.

It replaces the earlier `logs_capability` unit test, which now runs
through the full capability report.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant