Skip to content

test(e2e): make service journal diagnostics lane-aware and time-bounded #5334

Open

Description

@elezar

User Story

I run OpenShell's E2E conformance lanes in CI, across different deployment lanes (systemd-managed gateway, k3s, and others). When a test fails, I need the collected service logs for every component in that lane to cover the failure window, and I need sources that do not apply to the lane to be clearly marked rather than silently empty.

Problem Statement

The diagnostics role collects service journals with fixed line counts and a fixed set of units, regardless of lane. In Pre-release run 42222713989, job 127919027697 (conformance-integration / E2E suites (conformance/conformance, ubuntu-k3s, k3s-ha)), this produced an empty diagnostics/gateway.log (-- No entries --) and a k3s journal that missed the failure window. On k3s the gateway runs in Kubernetes rather than under systemd, but tests/tmachine/ansible/roles/openshell_diagnostics/tasks/main.yaml (journalctl --lines 500 filtered to openshell-gateway.service, around lines 10-16) still reads the systemd unit. The K3s boot journal (collect-k3s.sh:37, --lines=500) covered only 13:43 onward, after the failing test's window of 13:38:22Z to 13:40:22Z. The same gaps can occur for any service whose journal is collected with --lines instead of a time bound.

Impact / Why This Matters

An empty file looks like missing evidence rather than "not applicable". A line-bounded journal can miss the failure window entirely, because diagnostics are collected after the failure.

Proposed Design

  • Collect a service journal only when that unit exists in the lane. Otherwise skip it or label it "not applicable" (this covers openshell-gateway.service on k3s).
  • Bound every journal collection by time (--since) instead of a line count. Ideally use the dynamically determined test-run start time proposed for gateway pod logs in test(e2e): surface where a conformance test stalls and keep logs that cover it #5333, with a fixed-window fallback.
  • Apply this to the k3s journal (collect-k3s.sh:37) and to any journal added later, not just the two current ones.

Acceptance Criteria

  • Journals for units that don't exist in a lane are absent or labeled "not applicable", never empty.
  • Every collected service journal is bounded by time and covers the failing test's window.
  • No journal collection in the role relies on --lines alone.

Alternatives Considered

Raising --lines: it is still volume-dependent and can miss the window.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions