Skip to content

fix(openshell): make the OpenShell profile run in any namespace - #95

Draft
shricharan-ks wants to merge 115 commits into
RHEcosystemAppEng:feat/aeh-openshell-openclawfrom
shricharan-ks:pr/openshell-eval-fixes
Draft

shricharan-ks wants to merge 115 commits into
RHEcosystemAppEng:feat/aeh-openshell-openclawfrom
shricharan-ks:pr/openshell-eval-fixes

Conversation

@shricharan-ks

Copy link
Copy Markdown
Collaborator

What this is

The aeh_openshell_openclaw profile could not complete a run against a
namespace-local SAW gateway. This is what it took to get one green, plus the
missing pieces for standing a namespace up. Four commits, each independently
reviewable. Targets feat/aeh-openshell-openclaw.

Result: both eval cases execute and the judges score
(mean_reward=0.125, pass_rate=0.50, n_errors=0). The low reward is expected in
a namespace with no Microsoft 365 credentials — the agent cannot fetch the mail
the stock cases ask about. The plumbing is what was being proven.

The failures, in the order a run hits them

# Symptom Cause
1 ScannerError on evaluate.yaml a heredoc body at column 0 ends the YAML block scalar; oc apply cannot load the file, so a cluster keeps a stale Task
2 received fatal alert: CertificateRequired the client cert was staged only in the gateway profile; the CLI reads its identity from ~/.local/state/openshell/tls/
3 does not have valid authentication credentials gateway add --local writes auth_mode: mtls, and in that mode the CLI never sends a bearer token
4 The caller does not have permission the CI OIDC subject is not a member of the gateway workspace
5 provider 'forge-ai-gateway' not found, then an endpoint overlap those providers come from saw-bom; the builtin openai profile pins api.openai.com:443, which overlaps the eval policy
6 cluster inference is not configured, then HTTP 401 the relay accepts only builtin provider types and, for openai, dials api.openai.com whatever base_url says
7 invalid token: ExpiredSignature on the second case a Keycloak token lasts ~5 min; a case runs 15
8 8 of 11 judges error with 403 judges use /v1/messages; LiteLLM converts that to the upstream /responses API, which rejects it
9 does not name a readable CA file the Forge AI-gateway CA is required unconditionally, but only exists where that gateway is deployed

How each is fixed

Commit 1 — evaluate.yaml covers 1, 2, 3, 6, 7 and 9: indent the heredoc;
stage the client identity in the state dir and export OPENSHELL_LOCAL_TLS_DIR;
prefer OIDC registration for a cluster-local endpoint while keeping mTLS for the
transport; refresh the token in the background; export
AGENT_EVAL_MODEL_BASE_URL from llm-api-base; and drop the CA variable when
the file is absent. The namespace-pinned local TLS bridge goes with it — it only
triggered for two hardcoded namespaces and produced
failed to connect to gateway 'ci-gateway' at https://127.0.0.1:17671 when its
listener had not come up.

Commit 2 — the judge route (8) adds a judge-glm-5-3 LiteLLM entry that
routes the same upstream as hosted_vllm, staying on /chat/completions.
Probed directly: /v1/messages with rits/zai-org/glm-5-3 → 403, with
judge-glm-5-3 → 200; judge errors go 8 → 0. Also defaults
openshell-provider to empty (5).

Commit 3 — namespace-agnostic manifests. Every Task, Pipeline, trigger and
run carried metadata.namespace: ab-eval-flow, so oc apply -n <ns> refused
them. Also: secret templates addressed Services by namespace-qualified DNS;
MLflow's --allowed-hosts named one namespace and lost the pod-IP ranges that
kubelet probes need; postgres.yaml pinned a StorageClass that does not exist
on every cluster.

Commit 4 — the bring-up path adds gateway values, a post-install script
(mTLS export, workspace membership, provider and inference route, sandbox-image
caching), selector-based NetworkPolicies, an eval-stack-probe Task, and two
documents.

Verification

Built a namespace from scratch on OpenShift 4.22 (ROSA) following
Docs/openshell-eval-namespace-setup.md: probe all-PASS, prepare Succeeded,
mTLS and OIDC clean, sandboxes created, workspace staged,
LLM_PREFLIGHT_OK … status 200, both cases executed, deterministic judges 100%,
MLflow experiment recorded, cleanup Succeeded.

bash -n passes on every embedded script, every YAML parses, and no pipeline
task is passed a parameter it does not declare.

Pairs with

agent-eval-harness (draft PR on feat/aeh-openshell-openclaw), which consumes
AGENT_EVAL_MODEL_BASE_URL for both the provider endpoint and the sandbox
network policy. That branch is backward compatible on its own; this one sets the
variable.

Not included

Two gaps left deliberately: MLflow artifact upload still fails with
NoCredentialsError (the evaluate step needs the minio credentials in its env),
and a fully green run needs a submission whose cases do not require Microsoft
365.

🤖 Generated with Claude Code

GuyZivRH and others added 30 commits September 6, 2026 13:34
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ange

Co-authored-by: Cursor <cursoragent@cursor.com>
…cies

Co-authored-by: Cursor <cursoragent@cursor.com>
…cution.prompt

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
… installation

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…eval

Co-authored-by: Cursor <cursoragent@cursor.com>
…stall

Co-authored-by: Cursor <cursoragent@cursor.com>
Clone GuyZivRH feat/aeh-openshell-openclaw so CI gets the Kubernetes
sandbox keep-alive create, and bump the CLI so --detach exists.

Co-authored-by: Cursor <cursoragent@cursor.com>
Do not pin the RHEL nsenter digest; the keep-alive create fix must run
against quay.io/aipcc/base-images/agentic/openclaw:latest.

Co-authored-by: Cursor <cursoragent@cursor.com>
amd64 :latest is Hummingbird without nsenter and fails OpenShell
supervisor create with PodFailed. Restore the RHEL 9.8 pin so
cluster defaults and PipelineRuns do not pick up :latest.

Co-authored-by: Cursor <cursoragent@cursor.com>
Point runner.providers.inference at in-cluster LiteLLM /v1 and align
models[].id with models.skill so OpenClaw does not treat the run as
native Anthropic.
…andbox

seed: external means the mailbox is already populated — keep crabline/smolclaw
empty and forward M365_* into the OpenShell sandbox. Pin the dev pipeline
sandbox image so guy-ziv-evalflow does not pull :latest (no nsenter).
Prompt-mode configs no longer have skill:, so the Tekton wrapper looked
under reports/<submission>/ while the harness wrote reports/<name>/.
Match EvalConfig.eval_name() and copy/search both leaves after the run.

Co-authored-by: Cursor <cursoragent@cursor.com>
Install anthropic/jinja2 on the UBI orchestrator so OpenShell judges can run, patch models.judge to the LiteLLM alias, and stop treating missing/errored scores as a 0.0 A/B failure.

Co-authored-by: Cursor <cursoragent@cursor.com>
Parse summary.yaml per_case values the same way Harbor AEH does: integer
1-5 stays on the declared score_range in the judge table, Harbor-normalizes
only for the 0.5 gate, and no longer defaults a missing mean_reward to 0.0.

Co-authored-by: Cursor <cursoragent@cursor.com>
Forge morning-briefing needs live Graph env in guy-ziv-evalflow; fail closed when M365_ACCESS_TOKEN is missing instead of scoring empty 1/5 refusals.

Co-authored-by: Cursor <cursoragent@cursor.com>
OpenShell evaluate now calls log_aeh_mlflow.py like Harbor AEH. The
openshell pipeline defaults enable-mlflow to the in-cluster tracking
server so PipelineRuns show up as one MLflow experiment per Tekton run.

Co-authored-by: Cursor <cursoragent@cursor.com>
Export the tracking URI before run_aeh so the harness can resolve it, then
invoke AEH log_results.py from log_aeh_mlflow (prompt-mode safe, no skill:).

Co-authored-by: Cursor <cursoragent@cursor.com>
GuyZivRH and others added 29 commits September 17, 2026 12:11
Five problems in the evaluate step, each with a distinct symptom.

1. evaluate.yaml does not parse. The OIDC token heredoc has its body and
   terminator at column 0, which ends the YAML block scalar early:

     yaml.scanner.ScannerError: while scanning a simple key ... line 1851

   `oc apply` cannot load it, so a cluster silently keeps whatever version of
   the Task it already had — which is how a deployed Task drifts from the repo.
   Indent the body to the block indent; YAML strips it back to column 0, so the
   shell still sees a valid terminator.

2. mTLS: "received fatal alert: CertificateRequired". The client bundle was
   staged only under ~/.config/openshell/gateways/<name>/mtls. The CLI takes its
   client identity from the OpenShell state directory — the layout forge-saw
   installs on the VM in setup-runtime-mtls.sh — so it completed the handshake
   presenting no certificate. Stage ca.crt, client/tls.crt and client/tls.key
   there too and export OPENSHELL_LOCAL_TLS_DIR.

3. Registration: "The request does not have valid authentication credentials".
   `gateway add --local` writes auth_mode=mtls, and in that mode the CLI never
   attaches a bearer token, so a gateway with OIDC enabled rejects the call.
   mTLS secures the transport; it is not an identity this gateway accepts.
   Prefer the OIDC registration path whenever a client is configured, and
   restage the bundle afterwards because gateway add rewrites the profile.

   The local TLS bridge is removed with it: it only triggered for two hardcoded
   namespaces, and when its listener had not come up sandbox create failed with
   "failed to connect to gateway 'ci-gateway' at https://127.0.0.1:17671".

4. Token lifetime: "invalid token: ExpiredSignature". A Keycloak access token
   lasts ~5 minutes and one case can run 15, so the token that created the first
   sandbox is expired by the second. Refresh the cache in the background.

5. AGENT_EVAL_FORGE_AI_GATEWAY_CA_FILE is always set, but that CA only exists
   where forge-ai-gateway is deployed. Elsewhere every case fails before running
   with "does not name a readable CA file". Unset it when absent.

Also export AGENT_EVAL_MODEL_BASE_URL from llm-api-base. Submissions declare the
provider as https://inference.local/v1, the cluster-inference relay, which
accepts only builtin provider types and for type "openai" dials api.openai.com
whatever base_url says — so a namespace-local LiteLLM is unreachable through it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The judges use the Anthropic SDK (/v1/messages); score.py has no OpenAI dialect.
For a model declared "openai/...", LiteLLM converts that to the upstream's
/responses API and this upstream answers 403 "Authentication parameters
missing", so all eight scoring judges error and mean_reward is unavailable even
when every case passes.

Add a judge-glm-5-3 entry routing the same upstream as hosted_vllm, which stays
on /chat/completions. Probed directly against LiteLLM: /v1/messages with
rits/zai-org/glm-5-3 returns 403, with judge-glm-5-3 returns 200; judge errors
then go from 8 to 0. Select it with aeh-judge-model-override=judge-glm-5-3.

Also default openshell-provider to empty. Those providers are vended by saw-bom;
on a gateway without them sandbox create fails with "provider not found", and
the builtin openai provider fails differently because its profile pins
api.openai.com:443, overlapping the eval policy's own openai rule.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every Task, Pipeline, trigger and run carried metadata.namespace: ab-eval-flow,
so `oc apply -n <ns>` refused all of them:

  the namespace from the provided object "ab-eval-flow" does not match the
  namespace "<ns>". You must pass '--namespace=ab-eval-flow'

Let the namespace come from the apply, and give the stack manifests the same
treatment. Alongside that, three things that only work in one namespace:

- the secret templates addressed postgres and minio by namespace-qualified DNS;
  same-namespace Services resolve by short name;
- MLflow's --allowed-hosts named one namespace's FQDN. It now takes wildcard
  Service patterns plus the 10.*/172.*/192.168.* ranges, because kubelet probes
  address the pod by IP and MLflow otherwise rejects them ("Rejected request
  with invalid Host header: <pod-ip>:5000") and never becomes Ready;
- the cleanup CronJob had its namespace as a literal; read it from the downward
  API.

postgres.yaml also pinned storageClassName: kubevirt-csi-infra-default. On a
cluster without that class the PVC stays Pending and the pod is unschedulable
("pod has unbound immediate PersistentVolumeClaims"); use the cluster default,
as minio.yaml and mlflow/pvc.yaml already do.

The OpenShell PipelineRun template now uses short-name endpoints and carries no
saw-bom-only provider list.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Standing up an eval namespace previously meant reconstructing it by hand. These
are the pieces that were missing, each fixing something found by building a
namespace from scratch:

config/forge-saw/
- values-namespace-gateway.yaml — standalone gateway VM values with OIDC (and
  so mTLS auth) left on, a registry disk source, namespace-scoped RBAC only,
  and imagePullSecrets so the setup Job does not race a post-install
  `oc secrets link` and sit in ImagePullBackOff;
- gateway-postinstall.sh — the three things a fresh gateway needs before a
  pipeline can drive it: export its mTLS client identity into openshell-mtls,
  add the CI OIDC subject to the gateway workspace (without which every call
  returns "The caller does not have permission"), and configure the provider
  and inference route. It mounts the projected API token, because the chart's
  setup ServiceAccount sets automountServiceAccountToken: false and virtctl
  otherwise falls back to localhost ("dial tcp [::1]:8080"); it runs under its
  own ServiceAccount, because the chart's Role grants get on five named Secrets
  only; and it caches the sandbox image while credentials exist, because the
  chart pre-pulls only the supervisor image and then deletes the VM's registry
  credentials;
- networkpolicy-eval-stack.yaml / networkpolicy-saw-gateway.yaml — selector-based
  rules for running alongside the forge-workspace default-deny policies. A
  policy that selects a pod for Egress denies everything it does not list, and a
  namespace without that chart has no allow-dns or allow-egress-https rule to
  fall back on: without these, DNS fails ("Resolving timed out"), LiteLLM
  crash-loops on its model-cost-map fetch, prepare cannot clone, and the gateway
  VM cannot reach ghcr.io — which leaves it without OpenShell binaries.

pipeline/tasks/eval-stack-probe.yaml reports each dependency a pipeline pod
needs, so a blocked path is identified in seconds rather than through a
20-minute run.

Docs/openshell-eval-namespace-setup.md is the end-to-end walkthrough;
Docs/openshell-eval-troubleshooting.md maps each symptom to its cause.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@GuyZivRH
GuyZivRH force-pushed the feat/aeh-openshell-openclaw branch from 199242e to c8e6478 Compare September 22, 2026 13:38

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants