Repository navigation
bug(e2e): file_transfer::upload_download_round_trip stalls on the k3s-ha conformance row #5520
Description
User Story
I contribute tests and CI changes to OpenShell and rely on the conformance-integration checks on my pull requests. I directly encountered file_transfer::upload_download_round_trip failing on the ubuntu-k3s, k3s-ha row in a PR that did not change that test, and I need to know whether the row is reliable so that a red result can be trusted as a real signal.
Problem Statement
On the conformance-integration / E2E suites (conformance/conformance, ubuntu-k3s, k3s-ha) row, file_transfer::upload_download_round_trip intermittently stalls. It has failed in two different ways:
- In Pre-release run 42222713989, job 127919027697, the test hit nextest's 120 s limit (recorded in test(e2e): surface where a conformance test stalls and keep logs that cover it #5333).
- In run 43231863950, job 131237846938 (PR test(conformance): port sandbox lifecycle and API e2e tests to testkit #5431, head
b013446be), the test reached cleanup anddelete sandbox test-62v5f1mu-frtimed out after 120 s. The test took 254 s in total. The same row passed on the PR's previous head, and the new commit does not touchfile_transfer.
In the second case the sandbox pod started normally and was told to stop about 6 s into the delete. By the time diagnostics ran, no sandbox or pod was left, so the delete completed, but after the 120 s limit.
Both failures are on the three-replica gateway row, and both are in the file_transfer suite. I have not established that they share a root cause. The other conformance rows (Docker, Podman, single-replica k3s) passed in the same runs.
Impact / Why This Matters
A red k3s-ha row on an unrelated change forces a rerun of a lane that takes tens of minutes, and reviewers cannot tell a real regression from this flake. Until the cause is known, the row's result is less trustworthy than the others.
The evidence is also thin. #5333 covers the missing diagnostics: the gateway log tails on this row are too short to cover the stall, the run's gateway.log artifact was one line, and the controller log had no entry for the sandbox. Root-causing this likely needs #5333 first.
Acceptance Criteria
- The cause of the stall is identified, whether in the gateway, the Kubernetes driver, the agent-sandbox controller, or the test.
file_transfer::upload_download_round_trippasses reliably on thek3s-harow, or the row's timeout is changed on the basis of that cause.- If it is not fixed, the failure carries enough evidence to diagnose it (see test(e2e): surface where a conformance test stalls and keep logs that cover it #5333).
Reproduction
Not reproduced locally. It appears intermittently on the k3s-ha row of conformance-integration. Both occurrences above link the job and artifact.
Related
- test(e2e): surface where a conformance test stalls and keep logs that cover it #5333: surface where a conformance test stalls and keep logs that cover it
- bug: stop sandbox times out in Kubernetes E2E on Agent Sandbox v1alpha1 #5457: stop sandbox timeout on the Agent Sandbox v1alpha1 Kubernetes row, a different row and symptom that may be related
Activity
matthewgrossman commentedon Nov 12, 2026
Two more occurrences on the same row, both from PR #5431. Each time, a sandbox that usually finishes in 10–20 s hangs in Kubernetes. The failing test is different each time, and all three predate #5431.
cli::cli_stop_start_delete(2nd, about 1 minute in)sandbox deletereturnedInternal error: delete sandbox failed: ServiceError: request timed outfile_transfer::upload_respects_gitignore(12th)persistent_workspace::persistent_workspace_survives_stop_startThe re-run is the only one that captured a lower-level cause. The pod's
openshell-sandbox-bootstrapcontainer failed to start:The same error hit an unrelated
os-supervisor-*pod at the same time. So systemd/D-Bus on the tmachine VM stopped answering runc's scope requests. That matches the delete and stop hangs.Ruled out in all three runs:
NRestarts=0;The k3s journal in the diagnostics covers only the last seconds of the run, so it doesn't show what happened to systemd/D-Bus. #5334 (time-bounded journals) would capture it, ideally with the system journal (
journalctl --since <run start>, not just the k3s unit).Possibly relevant: #5431 runs 36 tests on this row instead of 22, so it spends longer on the VM. But the Nov 10 failure came about 1 minute in, so suite length alone doesn't explain it.