Repository navigation
Pluggable Egress Support #430
Description
We want to make it easy for users to integrate their agent policy enforcement frameworks into the substrate Actor system. Proposal for network Egress link, lists the following pieces:
- Sandbox (uVM, GVisor, etc) traffic capture to a Worker or Node local proxy.
- Local proxy for identity, metadata injection, secure redirection to PEP
- PEP, can be outside of Substrate.
Each of these /could/ be a point of pluggability, but this is a tradeoff -- we may not need so much flexibility at this point and we do want a default install of Substrate to come with some basic functionality for real usage.
- For (1), there are precedents on how this can be done for in Pod redirection (see mesh ecosystem). One thought is that it is less about the interception itself and more about coordination with the lifecycle of the Actor, Worker and intercept, as the latency of setup is a big factor. Given this, it probably makes sense to make a "good enough" decision and move ahead.
- There is a coupling between (2) + (3), in terms of forwarding protocol. Can we assume that (2) + (3) are coupled together in most cases so the protocol is an implementation detail OR try to standardize on a protocol to decouple (2) from (3).
- One proposal is to use HTTP CONNECT as the metadata wrapper. This does not handle UDP easily, but can take care of basic HTTP.
xref:
Reacted by Ben Plotnick
Activity
stephnangue commentedon Jul 13, 2026
Warden (https://github.com/stephnangue/warden) is an identity-aware PEP build for AI agents that supports CEL policies, accepts JWT/KSA/SPIFFE/TLS cert identities, and mints and injects short-lived credentials into outbound requests.
I am interested in working on the plugable PEP.
EItanya commentedon Jul 20, 2026
Hey there, John Howard (@howardjohn) is already working on an implementation of this.
bowei commentedon Jul 20, 2026
Stephane Nangue (@stephnangue) -- would be great to understand what you are thinking in terms of requirements and shape for that with respect to Substrate.
mikemorris commentedon Jul 28, 2026
I'd prefer if we could decouple these along the lines of what yanavlasov was suggesting at https://docs.google.com/document/d/1KmpIFu2gnqy9gp95wASgIo_vkJ_dA1DZckV8upET6bs/edit?disco=AAAB_aFludI.
For the runtime identity+metadata+redirect proxy I'd expect we should be able to get away with something relatively lightweight, and if we can standardize the protocol and identity representation in #124 we could benefit from a wider variety of integration options and leverage potentially faster ecosystem iteration as newer policy requirements and capabilities evolve without needing direct implementation within Substrate (as Tim Hockin (@thockin) suggested at https://docs.google.com/document/d/1KmpIFu2gnqy9gp95wASgIo_vkJ_dA1DZckV8upET6bs/edit?disco=AAAB-wmhItU)
(I would hope that egress policy eventually converges on Gateway API semantics for most functionality, but expect most egress gateways would initially rely on some custom configuration.)
hsteude commentedon Sep 2, 2026
Only partly related, but when I (or my agent, to be fair) tried to put the WorkerPool into the mesh, we noticed that Ateom and Istio both bind to port
15001, causing the worker to fail at startup.We tested moving Ateom to
16001, which resolved the conflict and worked with actor ingress, egress, pause, and resume. Would changing or configuring this port fit into the discussion here?bowei commentedon Sep 2, 2026
Yes, we want to be able to interop extensively. We should probably change the default as it collides with something that is already very well-known. Note: this will not be possible post GA
hsteude commentedon Sep 3, 2026
Cool, thanks. I'll prepare a pr.
Kuromesi commentedon Sep 7, 2026
I’ve also tested the Istio integration. Beyond the port conflict, I think another challenge is enabling the sidecar to identify which actor originated a connection. Without that information, we can only enforce policies at the pod level, but not at the actor level. For example, enforcing layer 4 policies at the sidecar proxy.
Since a worker pod can host multiple actors, could we expose a mechanism for external proxies and controllers to discover actors and their worker-pod bindings—for example, through an xDS API? Or ateom makes an external call to prepare/revoke actors? We would also need a way to associate each connection with its originating actor, since the actor-to-pod mapping alone would not distinguish traffic from actors sharing the same pod.
howardjohn commentedon Sep 7, 2026
Hey Kuromesi (@Kuromesi) substrate is designed as a pluggable egress pep networking stack with support already for envoy and agentgateway which happen to be the two same proxies istio supports. I would recommend integrating via those with the currently defined integration points rather than just throwing a sidecar onto the worker and trying to make it mesh. I actually somewhat intentionally made port conflicts to try to discourage that
Kuromesi commentedon Sep 8, 2026
Thanks, that helps explain why the ports look so familiar from Istio! I was experimenting with injecting ztunnel in dedicated mode as a sidecar to enforce L4 AuthorizationPolicies, then routing all traffic through an Envoy or agentgateway egress proxy, with an ext_proc plugin handling L7 policies. My hope was that this setup could improve performance and offer more flexible traffic policies. If Substrate’s intended design is to handle traffic interception itself rather than support an additional sidecar for that purpose, that makes sense to me too!