Replace actor eth0 move with worker-owned veth networking

Today ateom-gvisor gives an active actor network connectivity by moving the worker pod's real Kubernetes eth0 into the actor/gVisor network namespace. That works for basic actor execution, but it has an important side effect: while the actor is running, the worker pod no longer owns its pod network interface.

This makes the worker pod partially disappear from the normal Kubernetes networking model during actor activation. It also prevents us from cleanly adding pod-local networking components, such as an egress proxy or policy enforcement sidecar, because those components remain in the worker pod network namespace while actor traffic leaves through an interface that has been moved elsewhere.

We should replace the eth0 move approach with an explicit veth pair between the worker pod namespace and the actor/gVisor namespace.

Rationale

Keeping the pod's real eth0 in the worker namespace gives us a stable networking boundary:

The worker pod keeps normal Kubernetes network connectivity while an actor is running.
Sidecars or worker-local helper processes can remain reachable and can participate in actor networking.
Actor networking becomes explicit and owned by Substrate instead of depending on moving the CNI-provided interface.
This creates the foundation for transparent egress capture and policy enforcement.
It avoids coupling actor lifecycle to destructive mutation of the pod's primary network interface.

This is especially important for planned transparent egress policy work. We want actor traffic to be captured without requiring application opt-in via HTTP_PROXY or HTTPS_PROXY. To do that cleanly, actor traffic needs to cross a worker-owned interface where Substrate can install forwarding, NAT, and later proxy-capture rules.

Proposed approach

For each active actor, ateom-gvisor should create a point-to-point veth pair:

Worker pod namespace side: ateom0, for example 10.200.0.1/30.
Actor/gVisor namespace side: renamed to eth0, for example 10.200.0.2/30.
Actor default route points to the worker-side veth IP.
The Kubernetes-provided pod eth0 remains in the worker pod namespace.

For compatibility with the current routing model, the initial implementation can install temporary nftables rules:

Masquerade actor egress behind the worker pod IP.
DNAT inbound traffic from the worker pod IP and actor service port to the actor veth IP.
Enable forwarding between the actor veth and pod eth0.

These rules are intended as a compatibility bridge, not the final egress policy implementation. The later egress work should replace broad actor egress NAT with transparent capture into AgentGateway and default-deny policy rules.

Expected outcome

After this change:

Actors still start, checkpoint, restore, and receive inbound traffic as before.
The worker pod retains network connectivity during actor execution.
ateom-gvisor no longer needs to scrape, move, or restore the pod's real eth0.
Substrate has a cleaner networking foundation for transparent egress capture, policy enforcement, and future worker-local AgentGateway integration.

Validation

The implementation should be validated with:

Existing focused Go tests for ateom-gvisor and related networking/server boot packages.
The demo e2e flow that exercises actor startup, golden snapshot creation, restore, and inbound routing.
A runtime check that the worker pod still has its Kubernetes eth0 while an actor is active.
A runtime check that actor traffic routes through the veth pair and inbound traffic still reaches the actor.

Fabricio Voznika (@fvoznika)

I think this is in general a good direction to go. Moving the pod's eth0 was the simplest thing we could get done before our launch deadline, but the fact that ateom doesn't have any networking is quite restrictive.

Some questions / comments:

What IP(s) should we pick for the actor's veth pair. Can linux handle it if they all have the same IP, as long as they are inside the independent pod network namespaces?
"DNAT inbound traffic from the worker pod IP and actor service port to the actor veth IP" -- I am assuming this means we will need to require the ActorTemplate to declare any listening ports for the actor up front?
One of the things I have discussed with Fabricio Voznika (@fvoznika) and nybidari is that gVisor is probably going to get the ability to send packets directly to Envoy as a TCP/UDP proxy, rather than injecting them into the host kernel networking stack with AF_PACKET. But we will probably always need this setup for microvm-based workloads.

Can linux handle it if they all have the same IP, as long as they are inside the independent pod network namespaces?

Yes.

"DNAT inbound traffic from the worker pod IP and actor service port to the actor veth IP" -- I am assuming this means we will need to require the ActorTemplate to declare any listening ports for the actor up front?

If we keep all other aetom traffic on UDS we could DNAT all inbound traffic from the worker pod IP to the actor veth.

Yes, we are investigating a method to communicate directly with Envoy over UDS with gvisor. This is still in the research phase.

The failure mode that I am worried about is this:

ateom receives traffic for itself on port 12345
actor decides to listen() on port 12345
actor sees no error but does not actually receive any traffic on 12345

I am also a little worried about connection-tracking. It's a finite kernel resource which consumes precious RAM and needs to be configured.

I agree that we cannot have arbitrary listening ports.

When I built something like this in the past I bypassed this by having the ateom equivalent bind and then proxy all traffic into the actor equivalent via various mechanisms. IIRC in the Firecracker case it was a TAP device but I would need to double check.

In this case the atelet could potentially have a list of reserved ports which it could dole out to the actors, and then the ateom could proxy traffic appropriately.

I'm not as familiar with gVisor, but because firecracker is a microVM, the application can bind to whatever port it wants, so it's all transparent to the application.

One thing I want to stress is that perfect should NOT be the enemy of good for this PR/issue, I'm glad we're thinking through the long-term networking design, but maybe that belongs in an EPIC so that we can make incremental progress.

I agree that we cannot have arbitrary listening ports.

As long as a Pod has a single IP, any port we use for ateom is by definition unavailable (at least partially) for the actor itself, right? The least invasive thing I can imagine is to have ateom listen on port 8, sniff incoming packets for SNI and match on host, and otherwise forward. But this has all the historical issues of istio sniffing and server-first protocols. We could simply say that ateom consumes ports 1-9 and actors just can't use them, too bad (interestingly ports 1-6 seem unused in modern systems, but 7 and 9 are defined. 8 is not.)

Better would be SOMETHING that prevents actors from listening on or binding to ateom port(s). We would still need to NAT outbound srcports.

Better would be to not step on the same IP address -- get kubernetes to assign 2. Or just don't use IP for ateom itself.

This change introduces a significant egress-networking regression.

Before this change, the gVisor actor directly used the Kubernetes Pod's eth0, so its traffic naturally followed the CNI-managed network path.

After this change, the actor uses a dedicated veth network:

Actor 10.200.0.2
-> worker Pod ateom0
-> IP forwarding and nftables masquerade
-> worker Pod eth0
-> CNI
Although inbound traffic through PodIP:80 works, actor egress traffic does not work correctly in the target cluster.

This design assumes the cluster CNI supports traffic forwarded and masqueraded from inside a Pod network namespace. That assumption is not valid for all CNIs and can cause forwarded actor traffic to be dropped by the node-level CNI, eBPF, firewall, or network-policy data plane.

This change introduces a significant egress-networking regression.

Before this change, the gVisor actor directly used the Kubernetes Pod's eth0, so its traffic naturally followed the CNI-managed network path.

After this change, the actor uses a dedicated veth network:

Actor 10.200.0.2 -> worker Pod ateom0 -> IP forwarding and nftables masquerade -> worker Pod eth0 -> CNI Although inbound traffic through PodIP:80 works, actor egress traffic does not work correctly in the target cluster.

This design assumes the cluster CNI supports traffic forwarded and masqueraded from inside a Pod network namespace. That assumption is not valid for all CNIs and can cause forwarded actor traffic to be dropped by the node-level CNI, eBPF, firewall, or network-policy data plane.

Do you have a list of which CNIs this will cause issue with?

The current design will not scale and/or work for any other actor implementation such as hypervisors. This current design is definitely a better path forward, but there are probably kinks we need to work out in order to play nicely with all CNIs.

Some cloud vendor customized CNI, eBPF, firewall, and network-policy implementations drop traffic forwarded from a Pod-internal veth, even after Pod-level masquerading.

We extended the design with a worker-local egress relay.

Network Flow

During actor activation:

Keep the CNI-managed worker Pod eth0 and Pod IP unchanged.
Create a temporary veth pair:
- Worker side: ateom0, 10.200.0.1/30
- Actor/gVisor side: eth0, 10.200.0.2/30
Configure the actor default route through 10.200.0.1.
Enable IPv4 forwarding in the worker Pod network namespace.
Start worker-local DNS and transparent TCP relays.
Install an isolated nftables table named ateom_actor.

Inbound Traffic

Inbound router traffic continues using the existing worker Pod IP model:

Router
  -> Worker PodIP:80
  -> nftables DNAT
  -> Actor 10.200.0.2:80

DNS Egress

The actor directly mounts the worker Pod’s /etc/resolv.conf.

Actor DNS traffic is transparently redirected to a worker-local DNS relay:

Actor DNS UDP/TCP request
  -> ateom0
  -> nftables redirect
  -> worker-local DNS relay
  -> original worker Pod DNS server

The relay sends requests using the worker Pod network, so the CNI observes normal Pod-originated traffic.

TCP Egress

All non-gateway actor TCP traffic is transparently redirected to a worker-local TCP relay:

Actor TCP connection
  -> ateom0
  -> nftables redirect to port 15001
  -> relay reads SO_ORIGINAL_DST
  -> relay connects to the original destination
  -> CNI sees a normal worker Pod connection

This supports Kubernetes ClusterIP services and external HTTPS endpoints without relying on forwarded Pod-internal traffic or masquerading.

Checkpoint Lifecycle

During actor checkpoint:

Stop accepting new relay connections.
Allow established connections to drain temporarily.
Continue checkpoint without synchronously waiting for relay connections.
Force-close remaining connections after the drain timeout.

This avoids a deadlock when an actor calls SuspendActor through the same egress relay that is being cleaned up by the checkpoint operation.

Current Limitations

Inbound forwarding currently supports TCP port 80.
Egress supports DNS UDP/TCP and general TCP.
Non-DNS UDP egress is not currently supported.
The fixed 10.200.0.0/30 subnet remains local to each worker Pod namespace.

openxxs (@openxxs)

openxxs (@openxxs) -- trying to distill your requirements, but have some thoughts:

This project is still moving quickly, so we may move through some implementation choices where not all use cases can be addressed at the moment. For example, the fact that only port 80 ingress works will be temporary.

From your description, it sounds like your implementation also does forwarding and NAT with nftables, I'm interested in where it breaks down as your said in your case the traffic will end up being dropped.

It would be good to understand what will be compatible with your CNI.

Maybe by having an option where the Ateom is less opinionated about how things are wired up. I think the minimal thing Ateom will need to do is construct a veth?

Navigation Menu

Repository navigation

Replace actor eth0 move with worker-owned veth networking #122

Rationale

Proposed approach

Expected outcome

Validation

Network Flow

Inbound Traffic

DNS Egress

TCP Egress

Checkpoint Lifecycle

Current Limitations

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Replace actor eth0 move with worker-owned veth networking #122

Description

Rationale

Proposed approach

Expected outcome

Validation

Activity

ahmedtd commented on May 29, 2026

BenTheElder commented on May 29, 2026

nybidari commented on Jun 1, 2026

thockin commented on Jun 1, 2026

EItanya commented on Jun 2, 2026

thockin commented on Jun 2, 2026

openxxs commented on Jun 8, 2026

EItanya commented on Jun 8, 2026

openxxs commented on Jun 9, 2026

Network Flow

Inbound Traffic

DNS Egress

TCP Egress

Checkpoint Lifecycle

Current Limitations

bowei commented on Jun 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions