Feat: Route an agent's new sessions to a chosen inference server by huang195 · Pull Request #1330 · rossoctl/cortex

huang195 · 2026-10-08T11:01:31Z

Second of four PRs that let a team choose, from agentop, which LiteLLM server an agent's new sessions use. Each server has its own URL and key, and some serve non-Claude models. Today that takes a Claude Code configuration file per server and a restart. With this PR it is one command, and a running conversation never moves.

Stacked on #1329, which adds the general redirect capability (WritesDestination, pctx.Redirect) this plugin uses. Until #1329 merges, the diff below includes its commits; this PR's own range starts after 8aeab0ab.

What a user does

agentop server add ete https://ete-litellm.example.com   # asks for the key; it does not echo
agentop server add glm https://glm-litellm.example.com
agentop server use glm --agent claude-code                # new claude-code sessions → glm
agentop server                                            # the servers, and two checks
agentop server reset --agent claude-code                  # stop routing claude-code

Claude Code keeps one ANTHROPIC_BASE_URL (one of the servers) and its own model names. Nothing restarts: each command edits ~/.cortex/config.yaml in place and waits for the proxy to hot-reload it.

What it adds

The inference-router plugin (core/plugins/inferencerouter/, config in the stdlib-only routerconfig/).
- servers maps a name to a URL and key; agents maps an agent to a server. Routing is opt-in per agent. An unlisted agent, or a request to a host that is no server's, is left alone.
- Each new session of a listed agent is pinned to its agent's current server, in the process-scoped store, so pins survive hot reload. The pin has a 30-day sliding TTL and is recorded by outcome: where the request actually went. A pin belongs to its agent, and synthetic sessions (default, pending:) are never pinned.
- A running session never moves. That includes one that was running but quiet when its agent was first routed. On a pin miss, the session's own history decides: if the agent's latest earlier inference request in it reached a server, the session stays on that server. Requests to other hosts are no evidence, since the router never routes them. An agent that switches providers inside a session, as OpenCode does, would otherwise never be routed. Only a session with no such request is new.
- Every request on a server's host is redirected to the session's server, /v1/models included. The server's key replaces the client's X-Api-Key or Authorization only when pctx.Redirected(). On a bridged connection the Host header and the CONNECT authority can disagree, so comparing hosts is not a safe gate; under on_error: observe the request stays put and keeps the client's key. A session pinned to a removed server gets a 503 asking for a new session.
- Ports are validated and default ports normalised, plain http to another machine logs a WARN, and two servers on one host are refused. Registered in cmd/cortex under the full profile only: ext_proc (cortex-envoy) cannot honour a redirect, and ListenerSupport from Feat: Let plugins redirect an outbound request to another host #1329 refuses it there.
agentop's plugin-config editor (cmd/agentop/edit/pluginconfig.go). It edits one plugin's config: in a hand-kept YAML file line by line, keeping comments, order and formatting. It refuses sources and values whose line breaks yaml.v3 would count differently (U+0085, U+2028, U+2029, a lone \r), because those would make a later edit land on the wrong line.
Write and wait (edit/pluginwrite.go). Every write is checked first (config.Load plus the plugin's own Decode), applied atomically, and followed by polling /reload/status. If the proxy refuses the change, the file is put back, but only if it still holds what agentop wrote, so a hand edit made meanwhile is never overwritten. A refused config left on disk would otherwise make the next start fail. A proxy that stops answering is reported as that, not as a refusal.
agentop server (cmd_server.go, cmd_server_write.go):
- With no action, it lists each server with its host, model mapping and routed agents. It then checks ~/.claude/settings.json: ANTHROPIC_BASE_URL must name one of the servers, and neither the model setting nor any model variable may name a non-Claude model.
- add, remove, use and reset write through the helper above. The key is read at a hidden prompt or with --key-stdin (refused on a terminal), never as an argument, and is never echoed.
- Exit codes: 1 refused, 2 usage error, 3 replacement declined.
Docs: the plugin catalog, an agentop server section in agentop's README, "Switching inference servers" in docs/agents/claude-code.md, and a paragraph in docs/agents/opencode.md.

Credentials never printed

A key pasted into a URL is the common mistake, so no error, log line, /reload/status text or agentop server output quotes any part of a server URL. That rules out user info, path, query and fragment, and also a key that url.Parse takes as the host (https://KEY/x@host). url.URL.Redacted() is not enough: it masks only a password. /config and /v1/pipeline show every key as [REDACTED].

Known gaps (also in the docs)

A proxy restart forgets every pin and the history a pin miss reads, so a running conversation takes its agent's current server on its next request. The same happens to a quiet session evicted from the in-memory store (session.max_sessions) before it was pinned. The history rule also needs inference-parser in the chain, which the laptop default has.
Hand-offs look like new sessions: Claude Code's continued-in hand-off, claude daemon jobs started with --fork-session --resume, and /clear with background subagents running.
Requests are routed only if the bridge decrypts them. Claude Code's base URL must be https on a port in tls_bridge.ports (443 and 8443 by default), and that server must be reachable. A server on another port can still be a routing target.
No failover. A session whose server is down fails; route the agent elsewhere and start a new session.
Only X-Api-Key and Authorization are replaced. A credential in any other header travels unchanged.

Testing

core/plugins/inferencerouter: config validation; pinning, including pins surviving a rebuild over one store; synthetic sessions; agent-bound pins; the history rule for a running session; observe; a removed server; key replacement only when Redirected().
core/listener/forwardproxy, through the real proxy, bridge and session store:
- the key reaches only its server, including the Host-vs-CONNECT-authority case;
- a session running before routing was configured stays on its server, while a new one is routed.
cmd/agentop/edit: the editor against realistic hand-kept files. Every line-break guard fails its own test when reverted, and CRLF files are covered. The write helper is tested for rollback (bytes and mode), a concurrent edit left as found, a timeout leaving the file, and unreachable vs refused.
cmd/agentop: every agentop server form and refusal, with exit codes. A table of credential-bearing URLs proves none is printed. The tests never touch the real home: HOME is a temp dir and every test passes --config.
Live, against an isolated Cortex on its own ports with a scratch HOME:
- add, use and reset hot-reloaded, and keys were redacted on /config and /v1/pipeline;
- against a cortex built without the router, add was refused by the proxy and the file was restored byte for byte.
Gates: go vet, go test and go mod tidy -diff pass in core, cmd/agentop, cmd/cortex (full), cmd/cortex-envoy (envoy) and scripts/profile-tags; gofmt -l is clean on every touched directory.

Not yet exercised: routing a live Claude Code or OpenCode session end to end. The spec's demo is to ask, /clear, switch with agentop, and ask again.

Deferred (minor, found in review, left as they are)

Router:
- A plain-HTTP request routed to an https server uses system roots, not the bridge's upstream_ca_bundle. This is pre-existing, and matters only for a private-CA server reached without the bridge.
- IPv6 zone IDs don't round-trip through Endpoint.URL().
- PlaintextRemote treats *.localhost and 127.1 as remote, which only adds a WARN.
- ParseURL doesn't check host characters.
- A new session that adopts a pending bucket holding inference rows from before routing is treated as running on that server. That affects only the first session after routing, only for header-less agents, and is never a mid-conversation switch.
Editor (edit/pluginconfig.go):
- Lines it inserts into a CRLF file end in \n, so the file ends up with mixed line endings.
- A quoted key ("agents":) and agents: {} are refused rather than edited. Both refusals are safe.
- A key of << would be written unquoted; the name pattern excludes it today.
Write and wait (edit/pluginwrite.go):
- An edit landing between the re-read and the restore can still be overwritten. The window is microseconds.
- A cancelled context or malformed stats URL reads as a timeout.
- The reloads_failed baseline is taken on the first poll, which relies on the reloader's 250 ms debounce.
agentop server:
- Under on_error: observe, use prints its routing line before the note that nothing routes.
- The replace note prints even when nothing changes.
- The plain-http warning comes after the key prompt.
- --key-stdin truncates input over 64 KiB.
- An http ANTHROPIC_BASE_URL passes the check, but configure claude-code enable sets only HTTPS_PROXY.

Next in this series

agentop's S picker on the agents pane, plus SERVER columns fed by a generic inferenceHost on session summaries.
Chained request-body writers and pctx.SetRequestModel, so a server can serve Claude Code's model families under its own names (add --opus/--sonnet/--haiku).

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

Summary by CodeRabbit

New Features
- Added agentop server commands to configure inference servers, route new agent sessions, and inspect routing and Claude Code settings.
- Added inference-server routing for supported agent requests, with sessions staying pinned to their selected server.
- Redirected requests now show both the originally requested host and the destination in session details.
Bug Fixes
- Configuration changes are validated before they are applied, with reload outcomes and recovery reported clearly.
- Changes to mTLS settings now require a restart rather than being applied through hot reload.
Documentation
- Added setup, routing, and compatibility guidance for inference-server routing.

A plugin declaring it may send a request to a different host than the client named. Pipeline.New admits at most one per pipeline, since a second redirect would silently override the first. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

Redirect sends a request to another scheme and host. Host follows it, so events, usage, the ledger and pricing describe where the bytes went, and RequestedHost keeps the host the client named. It is refused from an undeclared plugin, outside OnRequest, for a malformed target, and on a context the listener did not mark redirectable; under observe it only records a shadow. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

SessionEvent.RequestedHost, requestedHost on the wire, is the host the client asked for when a plugin sent the request elsewhere. Host stays where the bytes went, which usage, the ledger and pricing key on. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

plugins.Deps carries a ListenerSupport that each listener package supplies, and BuildWithDeps refuses a plugin declaring a capability the listener cannot honor, on startup and reload. The forward proxy honors a redirect unless mtls is on; the reverse proxy and ext_proc do not. The zero value honors none, so a listener that has not considered a capability refuses it rather than ignoring it. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

cortex and cortex-cpex build the inbound chain for the reverse proxy and the outbound chain for the forward proxy; cortex-envoy builds both for ext_proc. A WritesDestination plugin is therefore refused on the inbound chain, under mtls, and in envoy-sidecar mode. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

The TLS dialer is built once in main and handed to the forward proxy, so mTLS config changes never take effect without a restart. On an mTLS proxy, a reload that removes the mtls block while adding a WritesDestination plugin would let that plugin through, violating the TLS-in-TLS constraint forwardproxy.Support enforces. Add mtls.* to validateReloadable alongside cost_ledger and session, refusing any change (present↔absent or field drift). Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

The forward proxy marks the requests it re-originates as redirectable, applies a redirect to the outgoing URL and Host before the request row is recorded, and records requestedHost on the request, response and denied rows. A CONNECT is never marked, so a redirect there is refused and the tunnel goes where the client asked. A bridged redirect verifies the new host's certificate. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

Add the capability to plugin-reference.md's struct and a "Redirecting a request" section: the four pctx calls, Host following the redirect, the modify/redirected record, when Redirect is refused, and which listener honors one. CLAUDE.md's event schema gains requestedHost. Also say that a reload changing the top-level mtls block is refused (mtls.* in validateReloadable): in CLAUDE.md's reload list and mTLS hot-reload boundary, and in framework-architecture.md section 9's lifecycle step, reloadability table and non-reloadable paragraph. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

SessionEvent gained RequestedHost, which tripped the shape guard (25 fields against 24). It is kept: agentop's detail pane draws its redirected: line from the projected event before the full one arrives, and never fetches the full one for an event with no protocol extension. A scalar, so summarizeEvent's struct copy already carries it; the fixture now sets it and the projection test asserts it survives. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

Every redirect test moved http to http or https to https, so deleting the listener's scheme copy left them all passing. A TLS-bridged https request sent to a local plaintext gateway is the inference router's case, and without the copy it fails with "server gave HTTP response to HTTPS client". Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

serveOutbound copied pctx.Scheme and pctx.Host onto the outgoing request whenever a redirect took effect. Both fields are exported, so a later plugin that never declared WritesDestination could steer a redirected request, and the modify/redirected record would name a host the bytes never went to. Redirect now also keeps the validated target privately, RedirectTarget reports it, and the listener applies that and re-asserts the exported fields to match before recording. RequestedHost compares against the private target, so a later write to Host cannot hide or invent a redirect either. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

…ner notes Redirect leaves the client's headers, credentials included, untouched, and under on_error: observe it moves nothing while header writes still apply, so a plugin attaching the target's key must gate it on pctx.Redirected(). The plugin reference's example now does, and both it and the Redirect/Redirected doc comments say so. forwardproxy.Support refuses a redirect whenever an mtls: block is configured, but the forward proxy has an mTLS dialer only under strict; under permissive the refusal is broader than needed and fails closed. The mtls block carries only mode, and legacy cert keys are dropped at load, so the reload notes now name exactly what a reload compares. Also: the listener table names which chains each listener serves, the target rule allows a bare "/", the bridged requests decrypted from a CONNECT or a transparent connection are redirectable, and the reference lists RedirectTarget. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

pctx.Host names the client's header on a TLS-bridged request and need not match the address the proxy dials. Comparing it with the target is unsafe; only Redirected() is a safe gate for attaching credentials meant for the target. Signed-off-by: Hai Huang <huang195@gmail.com> Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>

checkRedirectTarget printed the target through url.URL.Redacted, which masks only a password: a key given as the username, in the query, or in a path segment came through intact. A refused target is exactly where a pasted key sits, so the errors now name the problem and quote nothing. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

Servers by name and agents by name to a server, with the validation the plugin and agentop both apply: unique hosts compared without port or case, ports from 1 to 65535 with the default dropped, header-safe keys, and no model mapping until PR 4. Stdlib-only, so agentop can check a config before writing it without linking the plugin. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

Sends an agent's new sessions to the server configured for it and pins each session, routed or not, in the process store, which a hot reload does not replace: changing an agent's server moves only sessions it has not started. A pin to a removed server is a 503 rather than a silent move. The server's key replaces the client's only once the request really goes there, so under on_error: observe it never reaches the host the client named. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

When url.Parse fails, its error wraps the raw URL in *url.Error, which gets logged and exposed to the unauthenticated /reload/status endpoint. Extract and return only the inner error to prevent credential leakage. Test with user info present in URLs with bad ports and bad %-escapes. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

The profile the release tarballs and make dev-install build, so it reaches laptops; it routes nothing until an agent is assigned a server. cmd/cortex only: cortex-envoy cannot honor a redirect. Documents the plugin in the catalog, including where it goes in the chain and why. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

The router skipped the redirect when the request's Host header already named the routed server, and set the server's key anyway. On a TLS-bridged request the Host header is the client's word while the forward proxy dials the CONNECT authority, so a client that CONNECTed to one host and named the server inside got the server's key sent to the first. A routed request is now always redirected to its server's own scheme and host, which the listener dials whatever the Host header or the CONNECT said, and the key is set only once Redirected reports that the redirect took effect. The forward-proxy test drives the real router through a bridged CONNECT to a second origin and fails without the fix with the key at that origin. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

A session's first request pinned the agent's choice before the redirect was attempted. Under on_error: observe that request stays on the host the client named, yet the session was pinned to the server, so switching to enforce moved it mid-conversation, the switch the pin exists to prevent. A failed redirect likewise pinned a server the session never reached. The pin now follows the outcome: the server when the redirect took effect, "not routed" when the request stayed put (agent not routed, or observe), and nothing when the redirect failed, so the next request decides again. An existing pin is honoured and renewed as before. The listener's synthetic sessions, the default bucket and the pending:<agent> buckets, hold many conversations, so they are never pinned: their requests follow the agent's current server, as a request with no session does. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

The catalog said a request already on its server's host was not redirected and that a session was pinned to its agent's choice. Every routed request is now redirected, so each carries a modify/redirected record (from equal to to when the base URL already names the server), and a session is pinned by where its first request went: "not routed" under observe, nothing after a failed redirect. Synthetic sessions follow the agent's current server unpinned, as sessionless requests do. Also lists deny/redirect_failed, says skip/not_redirectable covers transparently redirected connections, that a pinned server can be removed by agentop as well as by hand, that requestedHost appears only when it differs, and that observe leaves two rows. The tests now pin the from/to record and the 503. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

Sets or removes a value at a key path in a named plugin's config:, creating the entry at the end of a chain when asked. Positions come from the YAML parser and the edit is to whole lines, so the comments a local config ships with survive; a mapping a removal empties goes with it, and an equal value is no change. It knows nothing about any plugin. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

WritePluginConfig applies changes through SetPluginConfig, refuses a result that will not load or that the caller's Verify rejects, writes it atomically, and reports reloaded, refused (with the proxy's error), timed out, or no proxy running. An unchanged file is neither written nor polled: the reloader ignores identical bytes, so the poll would only time out. The TUI can call it in PR 3; the main package cannot be imported from there. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

Lists each inference server with its host, model mapping and the agents routed to it, then checks ~/.claude/settings.json: ANTHROPIC_BASE_URL must name a configured server or nothing is routed, and no model variable may name a model other than Claude's, which the router cannot map back. --config reads another Cortex config. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

U+0085 (NEL), U+2028 (LS), and U+2029 (PS) are line separators the YAML parser counts in yaml.Node.Line numbering, but strings.Split(src, "\n") does not include in its line count. When such a separator appears in a config value being set, the parser's line numbers diverge from the split indices, causing later edits to land on wrong lines. A raw \r (not in \r\n pair) causes the same mismatch. Refuse all four cases in: - SetPluginConfig: check src for U+0085, U+2028, U+2029, or bare \r - scalarText: refuse values containing these characters Error messages are fixed text (no value echoing) per spec, with inline comments explaining the parser/split line-count disagreement. Add tests: - RefusesSpecialLineBreaksInValues: reject U+0085, U+2028, U+2029 in values - RefusesBareCarriageReturn: reject bare \r in values - RefusesSourceWithSpecialLineBreaks: reject src with U+0085/U+2028/U+2029 - RefusesSourceWithBareCarriageReturn: reject src with lone \r - SpecialLineBreaksDoNotCorruptLaterEdits: probe sequence verifies no corruption Also remove the package-level min() function (lines 406-411): Go 1.26's builtin min provides the same functionality, per spec compliance. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

WritePluginConfig now restores the original bytes on a refused reload to prevent crash-looping on next start. The restored config matches the running pipeline, which the reloader ignores unchanged. Timeout leaves the written bytes in place since the state is uncertain. WriteResult gains a RolledBack field to report whether restoration succeeded; if it fails, an error prevents the half-baked config from being left behind. Tests cover: refused reload restores bytes and mode; same change written again after rollback is not reported unchanged (file is new again); timeout leaves the new bytes in place. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

…s that parse The NEL guards spelled U+0085 as "\x85", the single byte 0x85. In strings.ContainsAny that is utf8.RuneError, so the source guard matched invalid UTF-8 or a real U+FFFD and never a real NEL: a config holding "sk-a<NEL>b" in a quoted value was edited, and setting agents.claude-code wrote a second claude-code: over session:. A config with U+FFFD anywhere, even in a comment, was refused instead. The source guard is now one rune scan, uncountedBreak, refusing U+0085, U+2028, U+2029 and a \r not followed by \n; a CRLF still passes. Its error carries the edit: prefix and names the character, never the source. The value guard refuses \n, \r, U+0085, U+2028 and U+2029. The tests did not test the fix: the U+0085 value used the raw byte, and every refused source failed yaml.Unmarshal with no guard at all. Each break now sits in a quoted value or ends a comment line, in a fixture checked to parse; the later-edit test sets agents.claude-code past it and expects a refusal, against a control with the break made a \n. U+FFFD in a comment or a value, and CRLF endings, are edited. Each guard was reverted in turn to see its tests fail. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

ParseURL quoted every refused URL through url.URL.Redacted, which masks a password but not a username, so a key pasted in as https://sk-...@host reached the reload error, the logs and the unauthenticated /reload/status intact. Quote the URL with its user info dropped instead, including the opaque form a forgotten "//" produces. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

…entials The model check now reads the top-level "model" key /model writes and ANTHROPIC_DEFAULT_FABLE_MODEL, and accepts the fable and best aliases, which it used to tell the user to delete. Neither a server URL the router refuses nor ANTHROPIC_BASE_URL is printed as written any more: the listing shows the host alone, and the check drops user info, or says the URL has no host when it does not parse. A server with no agents no longer breaks the agents column's alignment, an unknown home directory is reported rather than reading .claude/settings.json from the working directory, the base URL is matched with the plugin's own Hostname, and the "points elsewhere" check says what to do about it. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

A key can sit in a URL's query or fragment (?key=..., #token) as surely as in its user info. ParseURL's errors quoted both, reaching the logs and the unauthenticated /reload/status, and the "points elsewhere" check printed ANTHROPIC_BASE_URL's. Both shown forms now drop them along with the user info. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

add reads the key at a prompt that does not echo, or from stdin with --key-stdin, asks before replacing a server, and creates the router's entry last in the outbound chain. remove refuses a server an agent is routed to and names the command that moves it. use and reset route an agent's new sessions or stop. Every write is checked against the plugin's own rules first and returns once the proxy has reloaded it. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

Redacting a refused URL shape by shape kept missing shapes: with the "//" mistyped, url.Parse files a pasted key under the path, which the "no host" and "scheme" errors quoted; url.Parse's own "invalid port" error quoted a key it took for a port; and a key as a path segment was quoted by the "has a path" error. Every ParseURL error is now fixed text naming the problem, and a url.Parse failure is "not a valid URL" with no inner error. The "points elsewhere" check likewise prints only scheme://host[:port] of ANTHROPIC_BASE_URL. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

How to add servers and route an agent, what the two settings checks mean, why Claude Code keeps no model variables, and the /model caveat. OpenCode switches providers itself, so its page says when listing it is still worth doing. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

A key given as a username that contains '/', '?' or '#' ends the URL's authority early, so url.Parse takes the key for the host: https://sk-.../x@gw.example parses with Host "sk-...". The listing showed that host beside "(not a valid URL)", server add's "is already configured" line repeated it, and the "points elsewhere" check printed it as the URL Claude Code points at. A server URL the router refuses is now listed as "not a valid URL" and nothing else, and an ANTHROPIC_BASE_URL containing '@' anywhere is named in the check as "a URL" with no host. It is still matched to a server by host. The comments that said otherwise are corrected. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

add already refused a name no server can have as a usage error; use and remove echoed the argument raw into "no server named ..." and the suggested commands, so ETE read as a missing server (exit 1) and a control sequence in the argument reached the terminal. Both now run routerconfig.CheckName first, exiting 2 with the name quoted before any config is read, so every later message names a checked server. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

A conversation carried to a new session id (Claude Code's continued-in hand-off, claude daemon's forked jobs, /clear with subagents running) is pinned as new, so "finishes on the server it started on" now names its exceptions, and the page says nothing fails over. Routing needs a request the router can read and keeps the request's path, which neither check sees, so both pages say so. /model is no longer said to write the model setting, which nothing here confirms; the code comment is corrected the same way. Both Verified depth sections now say live routing was not exercised, and OpenCode's page says what the router does when no key header was sent. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

…uses A session was pinned only when the router first saw a request from it, so a Claude Code session already running but quiet while routing was first set up (server add, server add, server use) had no pin, and its next turn was taken for a new session's first request and moved to the newly chosen server: a mid-conversation switch, with thinking signatures and a cold cache, which is what the router exists to prevent. On a pin miss for a routed agent the session store's history now decides: the latest earlier outbound inference request that agent sent in the session (Phase request, parsed as inference). One that went to a configured server's host pins the session to that server; one that went to any other host pins it as not routed; only a session with no such request is new and goes to the agent's current server. Pinning by outcome and the unpinned synthetic sessions are unchanged. A restart still forgets both the pins and the history, so a running session can move then (the spec's gap 2). A pin is now its agent's. Pins were keyed by session id alone, so another agent's request filed under a routed session's id (the ActiveSession fallback with client affinity off, or a shared header id) took that session's server and key. The pin stores its agent; a request from another agent is decided as if the session were unpinned, from its own history and choice (an unlisted agent is left alone), and leaves the pin untouched. The history scan reads only the requesting agent's rows for the same reason. routerconfig.NormalHost lost its last caller when the router stopped comparing hosts to skip a redirect; it and its test are deleted. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

…roxy went away Three gaps in WritePluginConfig's handling of a reload that did not take. The rollback after a refusal restored the original bytes whatever the file held by then, so an edit someone made while the proxy reloaded agentop's write was silently undone. It now restores only while the file still holds exactly what agentop wrote; otherwise it leaves the file as found and returns an error saying so, quoting nothing from the file. When the file could not be put back, the result was zeroed and the proxy's error dropped. The WriteResult now still says what the proxy did, with RolledBack false, and the error carries the proxy's reason. The doc no longer claims the error is always nil once the file is written. A status endpoint that stopped answering (five consecutive transport errors) was reported as the proxy refusing the change. PollResult now marks that case Unreachable, and WritePluginConfig reports it as WriteStatusUnreachable. It still puts the file back, as the TUI's RollbackCmd does, and agentop server now says the proxy stopped answering instead of that it refused the change. The poll schedule is now vars so tests can shorten its ~16s backoff, and OnTimeoutLeavesNewBytes passes a short ctx instead of waiting the real 30s (30.02s to 0.20s). SetPluginConfig's doc now says which comments a removal takes with it, and a test pins that. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

…en nothing routes use and reset said "Sessions already running stay where they are" also when no proxy was running and the change was written for its next start. A proxy that starts holds no pins and no history, so it treats every session it sees as a new one. The running-sessions sentence is now said only when the running proxy reloaded the change, and says the exception the router has: a session that has sent nothing since the proxy last started is treated as a new one. Written for the next start, use, reset and remove say what that start does instead, and remove no longer promises the 503 a restart forgets. use, add and the listing now say when the router entry runs under on_error: observe, where it routes nothing and records observe/would_route, or off, where it does not run, and how to make it route. remove on the only server with an agent routed named reset as the next step and then, once taken, refused as the only server. It now checks the only server first and names both steps in one refusal. add --key-stdin with a terminal on stdin read the key as typed, echoing it. It is now refused before anything is asked or read: pipe the key in, or drop the flag to be asked for it without echo. A test comment no longer says what writes Claude Code's "model" setting. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

…comments it keeps The catalog, Claude Code's page and agentop's README now describe the router as it is after the fix for running sessions it has not seen: the first request it sees pins a session, a routed agent's session is held by where its latest earlier inference request went, and a pin is its agent's. A restart loses the pins and the in-memory history the router reads (the session archive is not read for it), so a running conversation can move then; Claude Code's known issues now say so and how to keep it where it was. The key claim is narrowed to what the router does: it replaces X-Api-Key and Authorization only, and a credential in any other header, such as one set through ANTHROPIC_CUSTOM_HEADERS, reaches the server unchanged. agentop's README now says which comments a change takes with it, what happens when the proxy stops answering mid-reload or the file is edited meanwhile, that a change written for the next start applies to every session then, that --key-stdin is refused on a terminal, and that use, add and the listing name an on_error: observe or off router entry, which routes nothing. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

The history rule for a session the router has not pinned read an earlier inference request to any host as evidence: one to a host no server has pinned the session as not routed. The router never routes a request to such a host, so it says nothing about which server a session is on, and the branch broke an agent that switches providers inside a session. An OpenCode session whose first requests went to Zen, and which then addressed a server, was left unrouted for good, sending its own key to that server: a 401 on every request. Only the latest earlier inference request of the same agent whose Host (where it went) is a configured server's now counts; with none, the session is new and goes to its agent's current server. A test pins that Host, the redirect target, decides and RequestedHost does not. agentop server's use and reset now also say that a running session the proxy has dropped from memory (session.max_sessions, 100 by default) can be treated as a new one, and the plugin doc names process attribution among the ways another agent's request lands under a session's id. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

…else looks new The catalog, Claude Code's and OpenCode's pages and agentop's README now say that only an earlier request to a configured server says which server a session is on, so an OpenCode session that used another provider first is new when it addresses a server. They name process attribution, on by default on a laptop, among the ways another agent's request lands under a session's id, and say a quiet session the store evicts (session.max_sessions, 100 by default, or a configured session.ttl) before the router pinned it is treated as new. The catalog says the rule reads inference-parser's record, so without that parser in the outbound chain a session the router has not pinned is always new. Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

coderabbitai · 2026-10-08T11:02:21Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults
Review profile: CHILL
Plan: Advanced
Run ID: 14b3dfa3-fd95-4041-9055-58dab05edca1

📥 Commits

Reviewing files that changed from the base of the PR and between dc60d71 and 3629518.

📒 Files selected for processing (57)

CLAUDE.md
cmd/agentop/README.md
cmd/agentop/cmd_server.go
cmd/agentop/cmd_server_test.go
cmd/agentop/cmd_server_write.go
cmd/agentop/cmd_server_write_test.go
cmd/agentop/edit/pluginconfig.go
cmd/agentop/edit/pluginconfig_test.go
cmd/agentop/edit/pluginwrite.go
cmd/agentop/edit/pluginwrite_test.go
cmd/agentop/edit/status.go
cmd/agentop/edit/status_test.go
cmd/agentop/main.go
cmd/agentop/tui/detail_pane.go
cmd/agentop/tui/detail_pane_test.go
cmd/cortex-cpex/main.go
cmd/cortex-envoy/main.go
cmd/cortex/main.go
cmd/cortex/plugins_inferencerouter.go
core/listener/extproc/support.go
core/listener/extproc/support_test.go
core/listener/forwardproxy/inferencerouter_test.go
core/listener/forwardproxy/redirect_test.go
core/listener/forwardproxy/server.go
core/listener/forwardproxy/support.go
core/listener/forwardproxy/support_test.go
core/listener/internal/sessionevent/sessionevent.go
core/listener/reverseproxy/support.go
core/listener/reverseproxy/support_test.go
core/pipeline/context.go
core/pipeline/destination.go
core/pipeline/destination_test.go
core/pipeline/listener_support.go
core/pipeline/listener_support_test.go
core/pipeline/pipeline.go
core/pipeline/plugin.go
core/pipeline/session.go
core/pipeline/session_test.go
core/plugins/README.md
core/plugins/deps.go
core/plugins/deps_test.go
core/plugins/inferencerouter/plugin.go
core/plugins/inferencerouter/plugin_test.go
core/plugins/inferencerouter/reload_test.go
core/plugins/inferencerouter/routerconfig/config.go
core/plugins/inferencerouter/routerconfig/config_test.go
core/reloader/reloader.go
core/reloader/reloader_test.go
core/sessionapi/summary_test.go
docs/agents/claude-code.md
docs/agents/opencode.md
docs/architecture.md
docs/framework-architecture.md
docs/plugin-catalog.md
docs/plugin-reference.md
scripts/profile-tags/profiles.go
scripts/profile-tags/profiles_test.go

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Walkthrough

Changes

Adds the opt-in inference-router plugin and agentop server commands to route agent sessions to configured inference servers. Adds listener-checked request redirects and records both requested and effective hosts in session events. Adds YAML configuration editing and reload handling for the management commands.

Inference routing and management

Layer / File(s)	Summary
Redirect contract and listener support `core/pipeline/`, `core/plugins/deps.go`, `core/listener//support.go`, `cmd/cortex*/main.go`, `core/reloader/reloader.go`, `docs/plugin-reference.md`, `docs/framework-architecture.md`, `CLAUDE.md`	Plugins can declare `WritesDestination` and request validated redirects. Pipeline construction checks listener support, including the mTLS restriction. Reload validation rejects changes to `mtls.*`.
Router configuration and plugin `core/plugins/inferencerouter/`, `cmd/cortex/plugins_inferencerouter.go`, `scripts/profile-tags/`, `core/plugins/README.md`, `docs/plugin-catalog.md`, `docs/architecture.md`, `CLAUDE.md`	Adds router configuration validation, server selection, session pinning, history-based selection, and authentication-header replacement. Adds the plugin to the full build profile and documents its configuration and behavior.
Redirect delivery and session events `core/listener/forwardproxy/`, `core/listener/internal/sessionevent/sessionevent.go`, `core/pipeline/session.go`, `core/sessionapi/summary_test.go`, `cmd/agentop/tui/`, `docs/agents/*`, `docs/plugin-catalog.md`, `CLAUDE.md`	The forward proxy applies accepted redirects to supported outbound requests. Session events include optional `requestedHost` alongside the effective `host`, and the detail pane displays both when they differ.
Plugin configuration editing and reload handling `cmd/agentop/edit/*`	Adds YAML plugin-config edits that preserve untouched content, validate candidate configurations, and report reload outcomes. Refused or unreachable reloads trigger a guarded restore; timeouts leave the written configuration in place.
agentop server commands `cmd/agentop/cmd_server.go`, `cmd/agentop/cmd_server_write.go`, `cmd/agentop/main.go`, `cmd/agentop/README.md`	Adds server listing and Claude Code settings checks, plus add, remove, use, and reset actions. Writes use the shared validation and reload flow. Documentation describes command behavior and routing constraints.

Priority: ➖ Normal

Estimated code review effort: 5 (Critical) | ~90 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant ForwardProxy
  participant InferenceRouter
  participant SessionStore
  participant InferenceServer
  Client->>ForwardProxy: Send outbound request
  ForwardProxy->>InferenceRouter: Run request pipeline
  InferenceRouter->>SessionStore: Read pin or session history
  SessionStore-->>InferenceRouter: Return routing state
  InferenceRouter-->>ForwardProxy: Redirect target and server key
  ForwardProxy->>InferenceServer: Send redirected request

Suggested reviewers: esnible, cwiklik

Merge Risk: ⚪ Minimal · up to 36295

This change adds opt-in inference-server routing and agentop server management commands. Routing is disabled until an agent is routed. Config writes are validated before they are written and are rolled back if the proxy refuses them. No concrete defects remain open. Live end-to-end routing with a real Claude Code or OpenCode session has not been exercised.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 55.34% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 309 functions across 48 files. (9 skipped…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary change: routing an agent's new sessions to a selected inference server.

Full details: Docstring Coverage

Explanation

Docstring coverage is 55.34% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 309 functions across 48 files. (9 skipped: 9 unsupported.)

Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

mrsabath

Summary

Approving. Unusually rigorous work — the design reasoning is documented at the point of decision, and the test suite covers the non-obvious cases rather than the easy ones.

Areas reviewed: Go (plugin + config + agentop), docs, build profiles, commit conventions, security.
Scope: this PR's own range 8aeab0ab..3629518c (28 commits), excluding the stacked #1329.
Commits: 28, all signed off, all using Assisted-By per CLAUDE.md — no Co-Authored-By.
CI: all checks passing.

Verified locally, not just read

Check	Result
`gofmt -l` on touched dirs	clean
`go vet` (core + cmd/agentop, `GOWORK=off`)	clean
`go test` inferencerouter, routerconfig, forwardproxy	pass
`go test` cmd/agentop/edit + all `server` tests	pass
Hardcoded secrets in non-test code	none found

Two cmd/agentop failures I hit locally (TestServiceStepUnsupervisedFailureShowsTheLog, TestServiceStepBackgroundRestoreRefusesBesideALoadedUnhealthyJob) are sandbox artifacts in pre-existing setup_* tests this PR does not touch — Operation not permitted on /private/var/select/sh and a CA permission denial. CI passes both.

What I probed, and why it holds

I went after the places this design could plausibly break, and the reasoning was already right:

Credential gating is on Redirected(), not on Redirect() == nil. That is the actual security boundary — under on_error: observe Redirect returns nil without moving anything, so a key set on the strength of that nil would go to the host the client named. destination.go's contract confirms Redirected is the only safe gate, and the always-redirect-even-when-Host-matches decision correctly handles a bridged connection where the Host header and CONNECT authority disagree.
The line-break guard does not panic on a trailing lone \r. I checked the src[i+1:] boundary specifically; the slice is validly empty there.
Pin by outcome rather than by choice is the right call. Pinning by choice would hold an observe-started session to a server it never used, and flipping to enforce would then move it mid-conversation — the exact switch the pin exists to prevent.
Agent-bound pins. A request filed under another agent's session id is decided from its own agent's history and leaves the pin alone, so it cannot inherit another agent's server and key.
URL error hygiene. ParseURL returns fixed text quoting no part of the input, including the https://KEY/x@host shape where url.Parse takes the key for the host. url.URL.Redacted() would have masked only a password; this is stricter, and the right choice for text reaching an unauthenticated /reload/status.

The routerconfig split — stdlib-only, so agentop can validate a config without linking the plugin registry and dragging SPIFFE and gRPC into a terminal UI — is a good structural call, and keeping one copy of the rules is what stops agentop writing a config the proxy then refuses.

Non-blocking

One inline note on the custom-header residual. Also: seven commit subjects run 73–86 chars, over the 72 convention. Minor, and the PR title is fine.

The "Deferred" list reads as genuinely minor and honestly surfaced rather than buried, which is the part that makes the rest easy to trust.

mrsabath · 2026-10-08T14:25:32Z

+
+- `servers` (map, required) — inference servers by name: lowercase letters, digits, `.`, `_` and `-`. Each has:
+  - `url` (string, required) — `scheme://host[:port]`, `http` or `https`, no path, query or fragment. A port must be 1–65535; the scheme's default is dropped, so `https://x:443` is `x`. No two servers may share a host, compared without port or case: a session's server is named from the host its requests went to. A plain-`http` server on another machine logs a WARN at load, since routed requests would cross the network decrypted.
+  - `key` (string, required) — the API key sent to this server in place of the client's, in the header the client used: `X-Api-Key` when it sent one, `Authorization: Bearer` otherwise, both when it sent both. Those two are the only headers replaced: a credential the client sends in any other header, such as one Claude Code's `ANTHROPIC_CUSTOM_HEADERS` sets, reaches the server unchanged. Printable ASCII, no spaces. `/config` and `/v1/pipeline` redact it.


Good that this residual is documented here and in the PR body rather than left implicit — it is the sharpest edge in the feature.

Worth considering as a follow-up (not for this PR): a redirected request carries every header unchanged to a host the client did not name, so a credential in any header other than these two travels to the new server. ANTHROPIC_CUSTOM_HEADERS is the realistic case, and a user who set it to carry an auth header gets that credential delivered to whichever gateway the agent is routed to, with no warning at config time.

Since agentop server already checks Claude Code's settings for ANTHROPIC_BASE_URL and the model variables, a check for an auth-shaped header in ANTHROPIC_CUSTOM_HEADERS would fit the same pattern and catch it at the point the user is already being told what will and will not route. Happy to leave it here as documented behavior for now.

huang195 added 30 commits October 7, 2026 19:49

feat: Show a redirected request's two hosts in agentop's detail pane

0efa308

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com> Signed-off-by: Hai Huang <huang195@gmail.com>

huang195 added 13 commits October 8, 2026 04:01

huang195 requested a review from a team as a code owner October 8, 2026 11:01

rubambiza added this to Rossoctl Issue Prioritization Oct 8, 2026

mrsabath approved these changes Oct 8, 2026

View reviewed changes

huang195 merged commit 9f94daa into rossoctl:main Oct 8, 2026
30 checks passed

huang195 deleted the feat/inference-router branch October 8, 2026 15:35

This was referenced Oct 8, 2026

Feat: Map Claude Code's model families to a server's own models #1335

Merged

Feat: Pick an agent's inference server with S in agentop #1336

Merged

Navigation Menu

Repository navigation

Feat: Route an agent's new sessions to a chosen inference server - #1330

Feat: Route an agent's new sessions to a chosen inference server#1330
huang195 merged 43 commits into
rossoctl:mainfrom
huang195:feat/inference-router

huang195 commented Oct 8, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 8, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

mrsabath left a comment

Uh oh!

mrsabath Oct 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

huang195 commented Oct 8, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What a user does

What it adds

Credentials never printed

Known gaps (also in the docs)

Testing

Deferred (minor, found in review, left as they are)

Next in this series

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

mrsabath left a comment

Choose a reason for hiding this comment

Summary

Verified locally, not just read

What I probed, and why it holds

Non-blocking

Uh oh!

mrsabath Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

huang195 commented Oct 8, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 8, 2026 •

edited

Loading