Repository navigation
fix(openshell): make the OpenShell profile run in any namespace - #95
Draft
shricharan-ks wants to merge 115 commits into
Draft
shricharan-ks wants to merge 115 commits into
shricharan-ks wants to merge 115 commits into
Conversation
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ange Co-authored-by: Cursor <cursoragent@cursor.com>
…cies Co-authored-by: Cursor <cursoragent@cursor.com>
…cution.prompt Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
… installation Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…eval Co-authored-by: Cursor <cursoragent@cursor.com>
…stall Co-authored-by: Cursor <cursoragent@cursor.com>
Clone GuyZivRH feat/aeh-openshell-openclaw so CI gets the Kubernetes sandbox keep-alive create, and bump the CLI so --detach exists. Co-authored-by: Cursor <cursoragent@cursor.com>
Do not pin the RHEL nsenter digest; the keep-alive create fix must run against quay.io/aipcc/base-images/agentic/openclaw:latest. Co-authored-by: Cursor <cursoragent@cursor.com>
amd64 :latest is Hummingbird without nsenter and fails OpenShell supervisor create with PodFailed. Restore the RHEL 9.8 pin so cluster defaults and PipelineRuns do not pick up :latest. Co-authored-by: Cursor <cursoragent@cursor.com>
Point runner.providers.inference at in-cluster LiteLLM /v1 and align models[].id with models.skill so OpenClaw does not treat the run as native Anthropic.
…andbox seed: external means the mailbox is already populated — keep crabline/smolclaw empty and forward M365_* into the OpenShell sandbox. Pin the dev pipeline sandbox image so guy-ziv-evalflow does not pull :latest (no nsenter).
Prompt-mode configs no longer have skill:, so the Tekton wrapper looked under reports/<submission>/ while the harness wrote reports/<name>/. Match EvalConfig.eval_name() and copy/search both leaves after the run. Co-authored-by: Cursor <cursoragent@cursor.com>
Install anthropic/jinja2 on the UBI orchestrator so OpenShell judges can run, patch models.judge to the LiteLLM alias, and stop treating missing/errored scores as a 0.0 A/B failure. Co-authored-by: Cursor <cursoragent@cursor.com>
Parse summary.yaml per_case values the same way Harbor AEH does: integer 1-5 stays on the declared score_range in the judge table, Harbor-normalizes only for the 0.5 gate, and no longer defaults a missing mean_reward to 0.0. Co-authored-by: Cursor <cursoragent@cursor.com>
Forge morning-briefing needs live Graph env in guy-ziv-evalflow; fail closed when M365_ACCESS_TOKEN is missing instead of scoring empty 1/5 refusals. Co-authored-by: Cursor <cursoragent@cursor.com>
OpenShell evaluate now calls log_aeh_mlflow.py like Harbor AEH. The openshell pipeline defaults enable-mlflow to the in-cluster tracking server so PipelineRuns show up as one MLflow experiment per Tekton run. Co-authored-by: Cursor <cursoragent@cursor.com>
Export the tracking URI before run_aeh so the harness can resolve it, then invoke AEH log_results.py from log_aeh_mlflow (prompt-mode safe, no skill:). Co-authored-by: Cursor <cursoragent@cursor.com>
Five problems in the evaluate step, each with a distinct symptom.
1. evaluate.yaml does not parse. The OIDC token heredoc has its body and
terminator at column 0, which ends the YAML block scalar early:
yaml.scanner.ScannerError: while scanning a simple key ... line 1851
`oc apply` cannot load it, so a cluster silently keeps whatever version of
the Task it already had — which is how a deployed Task drifts from the repo.
Indent the body to the block indent; YAML strips it back to column 0, so the
shell still sees a valid terminator.
2. mTLS: "received fatal alert: CertificateRequired". The client bundle was
staged only under ~/.config/openshell/gateways/<name>/mtls. The CLI takes its
client identity from the OpenShell state directory — the layout forge-saw
installs on the VM in setup-runtime-mtls.sh — so it completed the handshake
presenting no certificate. Stage ca.crt, client/tls.crt and client/tls.key
there too and export OPENSHELL_LOCAL_TLS_DIR.
3. Registration: "The request does not have valid authentication credentials".
`gateway add --local` writes auth_mode=mtls, and in that mode the CLI never
attaches a bearer token, so a gateway with OIDC enabled rejects the call.
mTLS secures the transport; it is not an identity this gateway accepts.
Prefer the OIDC registration path whenever a client is configured, and
restage the bundle afterwards because gateway add rewrites the profile.
The local TLS bridge is removed with it: it only triggered for two hardcoded
namespaces, and when its listener had not come up sandbox create failed with
"failed to connect to gateway 'ci-gateway' at https://127.0.0.1:17671".
4. Token lifetime: "invalid token: ExpiredSignature". A Keycloak access token
lasts ~5 minutes and one case can run 15, so the token that created the first
sandbox is expired by the second. Refresh the cache in the background.
5. AGENT_EVAL_FORGE_AI_GATEWAY_CA_FILE is always set, but that CA only exists
where forge-ai-gateway is deployed. Elsewhere every case fails before running
with "does not name a readable CA file". Unset it when absent.
Also export AGENT_EVAL_MODEL_BASE_URL from llm-api-base. Submissions declare the
provider as https://inference.local/v1, the cluster-inference relay, which
accepts only builtin provider types and for type "openai" dials api.openai.com
whatever base_url says — so a namespace-local LiteLLM is unreachable through it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The judges use the Anthropic SDK (/v1/messages); score.py has no OpenAI dialect. For a model declared "openai/...", LiteLLM converts that to the upstream's /responses API and this upstream answers 403 "Authentication parameters missing", so all eight scoring judges error and mean_reward is unavailable even when every case passes. Add a judge-glm-5-3 entry routing the same upstream as hosted_vllm, which stays on /chat/completions. Probed directly against LiteLLM: /v1/messages with rits/zai-org/glm-5-3 returns 403, with judge-glm-5-3 returns 200; judge errors then go from 8 to 0. Select it with aeh-judge-model-override=judge-glm-5-3. Also default openshell-provider to empty. Those providers are vended by saw-bom; on a gateway without them sandbox create fails with "provider not found", and the builtin openai provider fails differently because its profile pins api.openai.com:443, overlapping the eval policy's own openai rule. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every Task, Pipeline, trigger and run carried metadata.namespace: ab-eval-flow,
so `oc apply -n <ns>` refused all of them:
the namespace from the provided object "ab-eval-flow" does not match the
namespace "<ns>". You must pass '--namespace=ab-eval-flow'
Let the namespace come from the apply, and give the stack manifests the same
treatment. Alongside that, three things that only work in one namespace:
- the secret templates addressed postgres and minio by namespace-qualified DNS;
same-namespace Services resolve by short name;
- MLflow's --allowed-hosts named one namespace's FQDN. It now takes wildcard
Service patterns plus the 10.*/172.*/192.168.* ranges, because kubelet probes
address the pod by IP and MLflow otherwise rejects them ("Rejected request
with invalid Host header: <pod-ip>:5000") and never becomes Ready;
- the cleanup CronJob had its namespace as a literal; read it from the downward
API.
postgres.yaml also pinned storageClassName: kubevirt-csi-infra-default. On a
cluster without that class the PVC stays Pending and the pod is unschedulable
("pod has unbound immediate PersistentVolumeClaims"); use the cluster default,
as minio.yaml and mlflow/pvc.yaml already do.
The OpenShell PipelineRun template now uses short-name endpoints and carries no
saw-bom-only provider list.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Standing up an eval namespace previously meant reconstructing it by hand. These
are the pieces that were missing, each fixing something found by building a
namespace from scratch:
config/forge-saw/
- values-namespace-gateway.yaml — standalone gateway VM values with OIDC (and
so mTLS auth) left on, a registry disk source, namespace-scoped RBAC only,
and imagePullSecrets so the setup Job does not race a post-install
`oc secrets link` and sit in ImagePullBackOff;
- gateway-postinstall.sh — the three things a fresh gateway needs before a
pipeline can drive it: export its mTLS client identity into openshell-mtls,
add the CI OIDC subject to the gateway workspace (without which every call
returns "The caller does not have permission"), and configure the provider
and inference route. It mounts the projected API token, because the chart's
setup ServiceAccount sets automountServiceAccountToken: false and virtctl
otherwise falls back to localhost ("dial tcp [::1]:8080"); it runs under its
own ServiceAccount, because the chart's Role grants get on five named Secrets
only; and it caches the sandbox image while credentials exist, because the
chart pre-pulls only the supervisor image and then deletes the VM's registry
credentials;
- networkpolicy-eval-stack.yaml / networkpolicy-saw-gateway.yaml — selector-based
rules for running alongside the forge-workspace default-deny policies. A
policy that selects a pod for Egress denies everything it does not list, and a
namespace without that chart has no allow-dns or allow-egress-https rule to
fall back on: without these, DNS fails ("Resolving timed out"), LiteLLM
crash-loops on its model-cost-map fetch, prepare cannot clone, and the gateway
VM cannot reach ghcr.io — which leaves it without OpenShell binaries.
pipeline/tasks/eval-stack-probe.yaml reports each dependency a pipeline pod
needs, so a blocked path is identified in seconds rather than through a
20-minute run.
Docs/openshell-eval-namespace-setup.md is the end-to-end walkthrough;
Docs/openshell-eval-troubleshooting.md maps each symptom to its cause.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GuyZivRH
force-pushed
the
feat/aeh-openshell-openclaw
branch
from
September 22, 2026 13:38
199242e to
c8e6478
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The
aeh_openshell_openclawprofile could not complete a run against anamespace-local SAW gateway. This is what it took to get one green, plus the
missing pieces for standing a namespace up. Four commits, each independently
reviewable. Targets
feat/aeh-openshell-openclaw.Result: both eval cases execute and the judges score
(
mean_reward=0.125, pass_rate=0.50, n_errors=0). The low reward is expected ina namespace with no Microsoft 365 credentials — the agent cannot fetch the mail
the stock cases ask about. The plumbing is what was being proven.
The failures, in the order a run hits them
ScannerErroronevaluate.yamloc applycannot load the file, so a cluster keeps a stale Taskreceived fatal alert: CertificateRequired~/.local/state/openshell/tls/does not have valid authentication credentialsgateway add --localwritesauth_mode: mtls, and in that mode the CLI never sends a bearer tokenThe caller does not have permissionprovider 'forge-ai-gateway' not found, then an endpoint overlapopenaiprofile pinsapi.openai.com:443, which overlaps the eval policycluster inference is not configured, then HTTP 401openai, dials api.openai.com whateverbase_urlsaysinvalid token: ExpiredSignatureon the second case/v1/messages; LiteLLM converts that to the upstream/responsesAPI, which rejects itdoes not name a readable CA fileHow each is fixed
Commit 1 —
evaluate.yamlcovers 1, 2, 3, 6, 7 and 9: indent the heredoc;stage the client identity in the state dir and export
OPENSHELL_LOCAL_TLS_DIR;prefer OIDC registration for a cluster-local endpoint while keeping mTLS for the
transport; refresh the token in the background; export
AGENT_EVAL_MODEL_BASE_URLfromllm-api-base; and drop the CA variable whenthe file is absent. The namespace-pinned local TLS bridge goes with it — it only
triggered for two hardcoded namespaces and produced
failed to connect to gateway 'ci-gateway' at https://127.0.0.1:17671when itslistener had not come up.
Commit 2 — the judge route (8) adds a
judge-glm-5-3LiteLLM entry thatroutes the same upstream as
hosted_vllm, staying on/chat/completions.Probed directly:
/v1/messageswithrits/zai-org/glm-5-3→ 403, withjudge-glm-5-3→ 200; judge errors go 8 → 0. Also defaultsopenshell-providerto empty (5).Commit 3 — namespace-agnostic manifests. Every Task, Pipeline, trigger and
run carried
metadata.namespace: ab-eval-flow, sooc apply -n <ns>refusedthem. Also: secret templates addressed Services by namespace-qualified DNS;
MLflow's
--allowed-hostsnamed one namespace and lost the pod-IP ranges thatkubelet probes need;
postgres.yamlpinned a StorageClass that does not existon every cluster.
Commit 4 — the bring-up path adds gateway values, a post-install script
(mTLS export, workspace membership, provider and inference route, sandbox-image
caching), selector-based NetworkPolicies, an
eval-stack-probeTask, and twodocuments.
Verification
Built a namespace from scratch on OpenShift 4.22 (ROSA) following
Docs/openshell-eval-namespace-setup.md: probe all-PASS,prepareSucceeded,mTLS and OIDC clean, sandboxes created, workspace staged,
LLM_PREFLIGHT_OK … status 200, both cases executed, deterministic judges 100%,MLflow experiment recorded,
cleanupSucceeded.bash -npasses on every embedded script, every YAML parses, and no pipelinetask is passed a parameter it does not declare.
Pairs with
agent-eval-harness(draft PR onfeat/aeh-openshell-openclaw), which consumesAGENT_EVAL_MODEL_BASE_URLfor both the provider endpoint and the sandboxnetwork policy. That branch is backward compatible on its own; this one sets the
variable.
Not included
Two gaps left deliberately: MLflow artifact upload still fails with
NoCredentialsError(the evaluate step needs the minio credentials in its env),and a fully green run needs a submission whose cases do not require Microsoft
365.
🤖 Generated with Claude Code