e2e: egress suites depend on public internet sites (e.g. httpbin.org), making CI flaky

Bowei Du (@bowei)

Several e2e suites send actor egress to real public sites. When one of those sites degrades, CI fails for every PR, whatever the PR changes.

Bowei Du (@bowei) Lior Lieberman (@LiorLieberman)

Oof, Who is on this? P0.

e2e: skip the egress credential injection test (agent-substrate#2518)

e2e: update credential injection e2e to call an in-cluster echo server (

Evidence from the flake pipeline (7-day window ending 2026-10-13T16:23Z, 1,073 pr-workflow runs):

TestActorEgressCredentialInjection/https://httpbin.org/headers failed in 29 runs across 26 branches (57 of 688 gVisor-lane executions, 8.3%), and the http://httpbin.org/headers subtest in 18 runs across 15 branches (36 of 678, 5.3%). All but one PR branch's failures sit inside a single band on 2026-10-12 between 13:13Z and 18:02Z; 25 runs in the same band passed, so httpbin.org was intermittently slow rather than down. Failure mode in every case:

egresscredinject_test.go:128: fetch without the header: TLS through the MITM egress gateway failed: Get "https://httpbin.org/headers": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Example run: https://github.com/agent-substrate/substrate/actions/runs/38288232415. The pipeline treats this as an external dependency incident under this issue rather than a flaky test, so no separate issue was filed. Since then #2518 (merged 18:02Z) skipped the test and #2535 (22:17Z) pointed it at an in-cluster echo server, so the httpbin.org subtests no longer exist on main; the rewritten test has run in 92 e2e jobs since and failed in 1.

Aditya -- can you filter out the flakes with httpbin.org? That issue should (?) already be gone.

yes the dependency on httpbin.org was removed in #2535

might consider something like the agnhost e2e test image from Kubernetes running in-cluster instead.

I've got a failure on example.com which is still in there.

    egressmitm_test.go:81: Waiting for ActorTemplate ate-e2e-probe-microvm-egressmitm/probe golden snapshot...
    egressmitm_test.go:115: fetch https://example.com/ via projected bundle: status 503, want 200
    fixture.go:137: Running command: kubectl --context=kind-kind delete --ignore-not-found -f /tmp/TestActorEgressMITMTrust875607429/001/probe.yaml
        namespace "ate-e2e-probe-microvm-egressmitm" deleted
        workerpool.ate.dev "probe" deleted from ate-e2e-probe-microvm-egressmitm namespace

https://github.com/agent-substrate/substrate/actions/runs/38435876648/job/115429564673?pr=2516

e2e: Update egress e2e to use in-cluster test server and remove every…

All external dependencies are removed. I will keep it open and monitor any flakiness in egress e2e for the next few days

Let's use a different issue if we find flakes. This issue is resolved.

Navigation Menu

Repository navigation

e2e: egress suites depend on public internet sites (e.g. httpbin.org), making CI flaky #2513

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

e2e: egress suites depend on public internet sites (e.g. httpbin.org), making CI flaky #2513

Description

Activity

thockin commented on Oct 12, 2026

aditya-shantanu commented on Oct 13, 2026

bowei commented on Oct 13, 2026

yufan-su commented on Oct 13, 2026

BenTheElder commented on Oct 13, 2026

destijl commented on Oct 13, 2026

yufan-su commented on Oct 14, 2026

BenTheElder commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions