Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
115 commits
Select commit Hold shift + click to select a range
6beff65
feat: configure cluster-shared openshell gateway for evaluations
GuyZivRH Sep 6, 2026
43867a4
chore: reduce resource requests for evaluate task to fit cluster
GuyZivRH Sep 6, 2026
0623e43
chore: drastically reduce CPU requests for evaluate task to fit quota
GuyZivRH Sep 6, 2026
5bc125e
chore: adjust CPU requests to cluster minimum (50m) to satisfy LimitR…
GuyZivRH Sep 6, 2026
e60f5c8
fix: clone agent-eval-harness in-flight to satisfy OpenShell dependen…
GuyZivRH Sep 6, 2026
920409b
fix: remove top-level skill from eval.yaml to avoid conflict with exe…
GuyZivRH Sep 6, 2026
122b2d3
fix: download openshell CLI in-flight if missing in orchestrator image
GuyZivRH Sep 6, 2026
7d24993
fix: use quay.io openclaw image as orchestrator and remove manual CLI…
GuyZivRH Sep 6, 2026
7bd13a6
fix: use quay.io openclaw image for all aeh-image parameters
GuyZivRH Sep 6, 2026
fceeccd
debug: verbose import check and dependency install for aeh-openshell-…
GuyZivRH Sep 6, 2026
e100b38
fix: revert orchestrator to python image and restore openshell CLI in…
GuyZivRH Sep 6, 2026
b831376
fix: pin harness keep-alive branch and OpenShell CLI 0.0.111
GuyZivRH Sep 6, 2026
f6ce6ea
fix: keep OpenClaw sandbox image on :latest
GuyZivRH Sep 6, 2026
7dd674d
fix: pin OpenClaw sandbox image to nsenter-capable 0.0.1-1787755593
GuyZivRH Sep 6, 2026
83ce050
fix(openclaw-forge): LiteLLM openai-completions with claude-sonnet id
GuyZivRH Sep 6, 2026
1e52573
fix(openclaw-forge): document M365 Graph forwarding and pin nsenter s…
GuyZivRH Sep 6, 2026
cac86b8
fix(evaluate): resolve OpenShell AEH reports path from eval.yaml name
GuyZivRH Sep 6, 2026
a19a114
fix(openshell): score LLM judges and report single-sided AEH results
GuyZivRH Sep 7, 2026
23e8fd1
fix(aeh): keep 1-5 Likert judges in reports, not binary 0/1
GuyZivRH Sep 7, 2026
e6a551d
Wire OpenShell Evaluate to consume openshell-credentials in-cluster.
GuyZivRH Sep 7, 2026
f84f095
feat(openshell): log AEH results to MLflow and enable it on the profile
GuyZivRH Sep 8, 2026
a2eb114
Connect AEH and CI MLflow on the OpenShell profile.
GuyZivRH Sep 8, 2026
b526fa3
feat(openshell): stage Forge catalog skills for OpenClaw eval
GuyZivRH Sep 10, 2026
ad4855a
fix(openshell): register Forge skills for OpenClaw discovery
GuyZivRH Sep 14, 2026
959b09b
fix(forge): bootstrap Chief of Staff workspace
GuyZivRH Sep 14, 2026
a0194fe
fix(forge): add governed M365 launcher adapter
GuyZivRH Sep 14, 2026
89c6b16
fix(forge): use literal OpenClaw workspace paths
GuyZivRH Sep 14, 2026
020e087
fix(forge): include chief of staff skills in submission
GuyZivRH Sep 14, 2026
c2daaa4
fix(forge): avoid teaching invalid workspace placeholders
GuyZivRH Sep 14, 2026
c6b75ca
fix(openshell): preserve mTLS profile metadata
GuyZivRH Sep 16, 2026
52f7514
fix(openshell): select CI gateway profile and install boto3
GuyZivRH Sep 16, 2026
c9f3cac
fix(openshell): register CI gateway in mTLS mode
GuyZivRH Sep 16, 2026
b84952e
chore(openshell): diagnose CI mTLS profile
GuyZivRH Sep 16, 2026
0252ebc
fix(openshell): preserve dual OIDC and mTLS auth
GuyZivRH Sep 16, 2026
1db8e25
fix: use internal SAW gateway service for CI
GuyZivRH Sep 16, 2026
284c344
fix: register internal SAW gateway with mTLS
GuyZivRH Sep 16, 2026
6f81e86
fix: support cluster Tekton pipeline schema
GuyZivRH Sep 16, 2026
2f8f519
fix: force mTLS profile for internal SAW gateway
GuyZivRH Sep 16, 2026
d2ec654
fix: preserve mTLS when using internal SAW service
GuyZivRH Sep 16, 2026
65cf4ef
fix: use certificate-matching SAW service alias
GuyZivRH Sep 16, 2026
61fb187
feat: wire GLM LiteLLM proxy for agent and judges
GuyZivRH Sep 16, 2026
af4544b
fix: use valid OpenClaw GLM provider model id
GuyZivRH Sep 16, 2026
67069c3
fix: route agent through SAW GLM provider
GuyZivRH Sep 16, 2026
ba89bbb
fix: route OpenClaw agent through namespace GLM proxy
GuyZivRH Sep 16, 2026
b3e4980
fix: route sandbox agent through SAW GLM inference
GuyZivRH Sep 16, 2026
b52891a
fix: use SAW forge AI gateway provider for agent
GuyZivRH Sep 16, 2026
b5ce729
fix: register SAW gateway provider in OpenClaw config
GuyZivRH Sep 16, 2026
f4f22aa
fix: use SAW agent inference endpoint for OpenClaw
GuyZivRH Sep 16, 2026
e020e19
fix: attach SAW inference provider to eval sandboxes
GuyZivRH Sep 16, 2026
669025b
fix: use SAW injected OpenAI gateway credentials
GuyZivRH Sep 16, 2026
f701f03
fix: route eval sandboxes directly to SAW inference gateway
GuyZivRH Sep 16, 2026
e88fdd1
fix: bypass sandbox proxy for SAW inference gateway
GuyZivRH Sep 16, 2026
133fd6a
fix(forge): use SAW local AI bridge in eval sandbox
GuyZivRH Sep 16, 2026
182c27f
fix(forge): permit governed private model bridge
GuyZivRH Sep 16, 2026
a6c0ba7
fix(openshell): mount Forge AI gateway CA for CI
GuyZivRH Sep 16, 2026
0d3137d
fix(openshell): allow SAW service certificate name mismatch
GuyZivRH Sep 16, 2026
c6f1172
fix openshell mTLS gateway registration
GuyZivRH Sep 16, 2026
e67c24a
update OpenShell CLI for CI mTLS
GuyZivRH Sep 16, 2026
24ae6b5
use certificate-valid hostname for SAW mTLS
GuyZivRH Sep 16, 2026
e0f4f40
fix: route OpenClaw GLM through namespace LiteLLM
GuyZivRH Sep 16, 2026
b4b39e8
fix: bypass proxy for namespace internal services
GuyZivRH Sep 16, 2026
92b3cc4
fix: use reachable namespace LiteLLM service IP
GuyZivRH Sep 16, 2026
7b2684b
fix: use namespace SAW model forward
GuyZivRH Sep 16, 2026
9d0fe73
fix: route SAW model calls through OpenShell proxy
GuyZivRH Sep 16, 2026
3861523
fix: use TLS for SAW model forward
GuyZivRH Sep 16, 2026
f31c760
fix: use SAW provider bearer for GLM
GuyZivRH Sep 16, 2026
0011346
route OpenClaw model to namespace Forge gateway
GuyZivRH Sep 16, 2026
3e60f29
use namespace Forge provider profile
GuyZivRH Sep 16, 2026
d68a975
use isolated namespace Forge provider instance
GuyZivRH Sep 16, 2026
bb50ec5
inject namespace inference bearer into sandbox
GuyZivRH Sep 16, 2026
80e7ae4
map inference key into OpenClaw sandbox
GuyZivRH Sep 16, 2026
f378ed7
allow namespace gateway server-name mismatch
GuyZivRH Sep 17, 2026
98bd633
use OIDC auth for namespace gateway
GuyZivRH Sep 17, 2026
331d963
inject inference key into OpenShell step
GuyZivRH Sep 17, 2026
bc84967
fix: let OpenShell inject Forge gateway credential
GuyZivRH Sep 17, 2026
cf21ec0
fix: use injected Forge gateway bearer in sandbox
GuyZivRH Sep 17, 2026
555a1e7
fix: bypass ambient proxy for namespace Forge forward
GuyZivRH Sep 17, 2026
559fa85
fix: route sandbox GLM calls to namespace gateway
GuyZivRH Sep 17, 2026
26de261
fix: bind OpenClaw to vended Forge provider
GuyZivRH Sep 17, 2026
96ce0d8
fix: preserve SAW built-in model routing
GuyZivRH Sep 17, 2026
200daf2
fix: use SAW-native GLM model id
GuyZivRH Sep 17, 2026
1a88ed1
fix: use namespace vended Forge provider for GLM
GuyZivRH Sep 17, 2026
8d95c38
fix: bypass proxy for sandbox Forge gateway
GuyZivRH Sep 17, 2026
61a941e
fix: route sandbox GLM through OpenShell relay
GuyZivRH Sep 17, 2026
cfb8106
fix: route agent GLM through namespace LiteLLM
GuyZivRH Sep 17, 2026
21d82ef
fix: bypass proxy and disable memory for sandbox GLM
GuyZivRH Sep 17, 2026
5f99cd9
fix: route sandbox GLM through namespace inference
GuyZivRH Sep 17, 2026
577540c
ci: target namespace OpenShell gateway
GuyZivRH Sep 17, 2026
afb5c15
chore(forge-eval): allow longer OpenClaw execution
GuyZivRH Sep 17, 2026
bd9843d
fix(forge-eval): use package-perms OpenClaw SAW image
GuyZivRH Sep 17, 2026
b666491
fix(forge-eval): use pullable OpenClaw SAW tag
GuyZivRH Sep 17, 2026
17e6187
fix(forge-eval): use published OpenClaw SAW base image
GuyZivRH Sep 17, 2026
54cd667
use Chief of Staff skills from image
GuyZivRH Sep 17, 2026
edb710a
remove AEH persona bootstrap from workspace
GuyZivRH Sep 17, 2026
8695655
use Chief of Staff agent image
GuyZivRH Sep 17, 2026
8afda27
configure SAW capability providers
GuyZivRH Sep 17, 2026
98f1709
fix(openshell): preserve mTLS for namespace gateway
GuyZivRH Sep 17, 2026
86bf019
fix(openshell): use TLS profile for namespace gateway
GuyZivRH Sep 17, 2026
31682d1
fix(openshell): restage mTLS after gateway registration
GuyZivRH Sep 17, 2026
9e6d020
fix(openshell): prefer mtls for cluster-local gateway
GuyZivRH Sep 17, 2026
8a93db3
fix(openshell): terminate TLS in local mTLS bridge
GuyZivRH Sep 17, 2026
354377b
fix(openshell): negotiate h2 on local tls bridge
GuyZivRH Sep 17, 2026
8d30924
fix(openshell): use cert-valid namespace gateway hostname
GuyZivRH Sep 17, 2026
b3b2f7d
fix(openshell): keep namespace gateway mTLS native
GuyZivRH Sep 17, 2026
a849b8e
fix(openshell): refresh namespace mTLS gateway profile
GuyZivRH Sep 17, 2026
2d7b306
fix(openshell): use namespace mTLS bridge with OIDC credentials
GuyZivRH Sep 17, 2026
43051c8
fix(openshell): bridge namespace gateway mTLS for CI
GuyZivRH Sep 17, 2026
3fada88
fix(openshell): ignore service hostname in bridge TLS verification
GuyZivRH Sep 17, 2026
178d568
fix(openshell): use noninteractive OIDC token cache
GuyZivRH Sep 17, 2026
17a0297
force CI OpenShell CLI through local mTLS bridge
GuyZivRH Sep 18, 2026
4ad3771
export namespace mTLS bridge mode
GuyZivRH Sep 18, 2026
4bdcb63
fix(openshell): make the CI client usable against a namespace gateway
shricharan-ks Sep 21, 2026
709879a
fix(openshell): route the LLM judges past the upstream /responses API
shricharan-ks Sep 21, 2026
b2237b3
fix(config): let the pipeline and eval stack deploy into any namespace
shricharan-ks Sep 21, 2026
a913612
feat(openshell): add the namespace bring-up path and its documentation
shricharan-ks Sep 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added Docs/ABEvalFlow_README.docx
Binary file not shown.
98 changes: 98 additions & 0 deletions Docs/ABEvalFlow_additions_for_doc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# ABEvalFlow Doc — Additions for User-Facing Summary

Each section below includes a clear placement instruction.
Paste each block into the Word doc at the indicated location.

---

## ADDITION 5
### PLACE: Immediately after the Certification Levels table (the Foundational / Trusted / Certified table), before the paragraph starting "Levels are strictly hierarchical"

---

**What each level actually validates:**

Foundational — 5 checks:
- Submission structure: required files exist (SKILL.md, instruction.md, test files)
- Metadata compliance: metadata.yaml matches the required schema (name, engine, experiment config)
- Python syntax: test files compile without errors
- Content quality review: LLM judge scores the skill on clarity, test coverage, and coherence
- Basic security scan: no hardcoded secrets, dangerous imports, or code injection risks

Trusted — adds 2 checks on top of Foundational:
- Evaluation assets: test files are correctly formatted and compatible with the selected engine (Harbor, ASE, A2A, or MCPChecker)
- Functional validation: A/B evaluation confirms the skill measurably improves agent performance, with statistical significance (p-value)

Certified — adds 1 check on top of Trusted:
- Full behavioral evaluation: complete A/B comparison with uplift scoring, significance test, and pass rate above threshold enforced in block mode — the skill must demonstrably outperform the control baseline

---

## ADDITION 1
### PLACE: Immediately after the opening bullet list (Skills / MCP Servers / Agents), before the "Pipelines" section

---

### Why ABEvalFlow

AI artifacts — skills, MCP servers, and agents — are increasingly submitted to shared marketplaces and production platforms, but there is no standardized quality bar for what "good" means. ABEvalFlow is the CI/CD layer that enforces one: every artifact goes through automated security, quality, and behavioral gates before it reaches production, and the result is a certified scorecard that stakeholders can trust.

---

## ADDITION 2
### PLACE: After the "4. Analyze" stage, before the "5. Store" stage — as a new top-level section

---

### Certification Levels

Every pipeline run produces a certification level based on the combined gate results. Certification is the primary trust signal — it answers not just "did this pass?" but "how thoroughly was it validated?"

| Level | What Was Checked |
|---|---|
| **Foundational** | Structure and metadata are valid; submission follows the required format; no schema violations |
| **Trusted** | All of the above, plus: security scan passed (no HIGH or CRITICAL findings); quality review score meets threshold |
| **Certified** | All of the above, plus: behavioral evaluation passed in block mode — the artifact demonstrably improves agent performance above the control baseline |

Levels are strictly hierarchical. A Certified artifact has satisfied every check for Foundational and Trusted as well. If a lower level fails, higher levels are automatically failed regardless of their individual gate results — there is no way to achieve Trusted without first achieving Foundational.

The highest achieved level is recorded in the scorecard, persisted to PostgreSQL, and published to Red Hat Compass.

---

## ADDITION 3
### PLACE: Immediately after the "Certification Levels" section above (Addition 2), before "5. Store"

---

### What a Submission Looks Like From the Outside

1. A developer pushes a skill, MCP server, or agent to the submissions repository
2. The pipeline triggers automatically via a git webhook
3. All gates run in sequence:
- **Security** — scans instruction files for prompt injection, credential access patterns, and obfuscation; an optional LLM semantic review detects jailbreak attempts and description-behavior mismatches
- **Quality** — an LLM judge reviews the skill definition and test coherence across five dimensions: clarity, coverage, coherence, feasibility, and robustness
- **Behavioral** — the artifact is evaluated A/B style: 20 runs with the skill/agent active (treatment) vs. 20 runs without (control); uplift and statistical significance are computed
4. A unified scorecard is produced with gate scores, findings, a pass/warn/fail recommendation, and a certification level
5. Each gate result is pushed to Red Hat Compass as a Soundcheck fact — one fact per gate per artifact, structured as:
- **Gate name** (evaluation / security / quality)
- **Passed** (true/false)
- **Score** (0.0–1.0)
- **Mode** (warn or block)
- **Evaluated at** (timestamp)
6. The certification level fact summarizes the highest tier achieved and which checks passed or failed at each level

No manual steps required after the initial push. The full audit trail — gate results, findings, certification history — is queryable from Compass and from the PostgreSQL results database.

---

## ADDITION 4
### PLACE: After the "Compass Facts Integration" section, before "Persistence"

---

### What's Next

| Area | Description |
|---|---|
| **Observability** | MLflow integration for LLM token and cost tracking per pipeline run; Grafana dashboards over normalized gate and certification data |
Loading