Test/workshop 0823

.agents/skills/test-agent-governance/SKILL.md

@@ -0,0 +1,22 @@
+---
+name: test-agent-governance
+description: Use when the testing workflow itself needs knowledge-base maintenance, benchmark evaluation, controlled Skill optimization, or CI quality-gate design rather than ordinary feature testing.
+---
+
+# Test Agent Governance
+
+Operate the testing system's governance plane. This Skill is not a mandatory
+tail of `test-agent-workflow` and does not design or execute feature Cases.
+
+Load exactly one reference matching the request:
+
+| Request | Reference |
+| --- | --- |
+| Build or refresh reusable project testing knowledge | [Quality KB Builder](references/quality-kb-builder.md) |
+| Evaluate Skill or workflow quality against fixed scenarios | [Skill Benchmark](references/skill-benchmark.md) |
+| Run a one-change-at-a-time optimization experiment | [Skill Self Optimization](references/skill-self-optimization.md) |
+| Design PR, main, nightly, or release gates | [CI Quality Flow](references/ci-quality-flow.md) |
+
+Keep deterministic facts in scripts and project inventories. Keep semantic
+judgment reviewable. Never edit product business behavior during governance
+work, and never weaken benchmark scenarios to make a candidate pass.

.agents/skills/test-agent-governance/agents/openai.yaml

@@ -0,0 +1,6 @@
+interface:
+  display_name: "Test Agent Governance"
+  short_description: "Maintain and evaluate the testing workflow"
+  default_prompt: "Use $test-agent-governance for the requested KB, benchmark, optimization, or CI governance task."
+policy:
+  allow_implicit_invocation: false

.agents/skills/test-agent-governance/references/ci-quality-flow.md

@@ -0,0 +1,69 @@
+# Test Agent CI Quality Flow
+
+Load only for CI gate placement, review flow, or release-quality decisions.
+
+## Purpose
+
+Turn AI Native testing evidence into CI gates, review flow, quality reports, and rollout decisions. This skill protects the team from expensive gates in the wrong place and from unreviewed skips.
+
+## Required Inputs
+
+Read:
+
+- Workflow Run output from `test-agent-workflow`
+- Test Strategy from `test-strategy`
+- Latest run logs for `fast`, `infra`, `e2e`, and targeted commands
+- Quality Review findings and residual risk
+- `test-index.yml`
+- Existing CI stages and runtime constraints
+
+## Gate Model
+
+| gate | recommended scope | reason |
+| --- | --- | --- |
+| PR Fast | backend UT, contract, frontend UT, frontend IT | quick feedback and no container runtime dependency |
+| PR Infra | targeted backend IT for DB, optimistic lock, stock/coupon, or transaction risks | real integration evidence before merge when risk demands it |
+| Coverage Review | coverage report generation after functional layers pass | gap analysis without treating coverage percentage as behavior proof |
+| Main Smoke | key E2E smoke | golden path confidence after merge |
+| Nightly | full backend IT, extended E2E, flaky detection | slower regression and environment confidence |
+| Release | selected high-risk workflows plus verification report | release sign-off and residual-risk acceptance |
+
+## Review Flow
+
+| review | required evidence |
+| --- | --- |
+| Story Review | Story, AC, assumptions, human-gate decisions |
+| Test Case Review | approved `case_id` values, oracles, steps, data, and existing-case decisions |
+| Strategy Review | `case_id`, stable `test_case_id`, run-specific `execution_id`, selected layer, distinct evidence claim, target, command, runtime, and residual risk |
+| Script Review | generated or modified files, assertions, fixture determinism |
+| Coverage Review | functional test evidence, JaCoCo/Vitest report path, gap explanation |
+| Feature Review | selected/skipped commands, run evidence, residual risk |
+| CI Review | gate placement, runtime cost, retry/flaky policy |
+
+## Output Format
+
+```markdown
+## CI Gate Plan
+| gate | command | trigger | required | skip_policy |
+| --- | --- | --- | --- | --- |
+
+## Quality Report
+| dimension | status | evidence | owner |
+| --- | --- | --- | --- |
+
+## Review Checklist
+| review | reviewer | evidence_link | decision |
+| --- | --- | --- | --- |
+
+## Residual Risk Acceptance
+| risk | reason | owner | expiration |
+| --- | --- | --- | --- |
+```
+
+## Guardrails
+
+- Do not put every test in PR if the result makes developers bypass the gate.
+- Do not skip `backend-it` for DB, optimistic lock, stock/coupon, or transaction side-effect changes without an explicit owner and follow-up gate.
+- Do not use E2E for rule combinations that belong in UT or IT.
+- Do not treat coverage percentage as behavior proof; coverage review needs functional layer evidence plus gap explanation.
+- Do not accept an AI-generated case or script without Quality Review evidence.

.agents/skills/test-agent-governance/references/quality-kb-builder.md

@@ -0,0 +1,81 @@
+# Test Agent Quality KB Builder
+
+Load only for a testing knowledge-base build or refresh.
+
+## Purpose
+
+Build a deterministic project testing knowledge base from project artifacts, then let AI add semantic summaries only after the structural graph is valid.
+
+This skill follows a hybrid pattern: static extraction owns facts, AI owns interpretation. The goal is a KB that teaches the team how business chains, tests, risks, skills, and evidence fit together.
+
+## Required Context
+
+Read these when present:
+
+- Project `Tests/test-index.yml`
+- `.agents/skills/test-agent-workflow/references/workflow-registry.yml`
+- Project `Tests/README.md`
+- `AI Native/workflows/*/workflow-pack.json`
+- Project `Tests/ai-native-workflow/*/workflow-pack.md`
+- Existing backend, frontend, contract, and E2E tests
+- Product MD source paths listed in `test-index.yml`
+
+## Workflow
+
+1. Resolve the project-declared deterministic KB builder under
+   `AI Native/scripts/` or the project testing directory; do not assume a
+   generic filename exists.
+2. Run that exact builder and inspect the output path it reports.
+3. Check graph reviewer output:
+   - no duplicate node IDs
+   - no dangling edge references
+   - each core chain has risk tags, source docs, and at least one test/evidence link
+   - each Skill path in `workflow-registry.yml` exists
+   - each workflow pack validates and contributes workflow/story/AC/test case/review/repair nodes
+4. Add AI semantic notes only as `semantic_notes` fields or a separate Markdown summary; do not overwrite extracted facts.
+5. Commit or share the KB when it is green so teammates and future AI loops start from the same map.
+
+## Node Types
+
+| Node type | Meaning |
+| --- | --- |
+| `chain` | Business chain such as payment callback, mobile order API, stock compensation, coupon release, groupon status, refund lifecycle, or browser smoke |
+| `risk` | Project risk tag used by planner and quality review |
+| `skill` | AI Native skill and owned quality surface |
+| `test` | Concrete backend UT, frontend UT, contract, frontend IT, backend IT, or E2E asset |
+| `test_case_design` | Layer-neutral business Case with stable `case_id` |
+| `executable_test` | Strategy-created stable `test_case_id` for one Case and layer |
+| `execution_item` | Run-specific `execution_id`, action, target, command, and result |
+| `workflow` | Canonical AI Native feature workflow pack |
+| `story` | User story extracted from product or workshop prompt |
+| `acceptance_criterion` | AC linked to one story and one or more test case designs |
+| `evidence` | Observable evidence expected from tests |
+| `doc` | Source MD or architecture/testing documentation |
+
+## Review Rules
+
+- Treat the project `Tests/test-index.yml` as the test asset and command source of truth.
+- Prefer exact file paths and commands over generated descriptions.
+- Keep generated facts reproducible; if an AI inference is needed, mark it as an inference.
+- Regenerate the KB after changing skills, tests, product docs, or `test-index.yml`.
+
+## Output
+
+Produce a short report:
+
+```markdown
+## Quality KB Build
+- command:
+- output:
+- node_count:
+- edge_count:
+- reviewer_status: pass | needs-fix
+
+## Reviewer Issues
+| severity | issue | fix |
+| --- | --- | --- |
+
+## AI Semantic Additions
+| node_id | note | confidence |
+| --- | --- | --- |
+```

.agents/skills/test-agent-governance/references/skill-benchmark.md

@@ -0,0 +1,82 @@
+# Test Agent Skill Benchmark
+
+Load only for a fixed-scenario Skill or workflow evaluation.
+
+## Purpose
+
+Benchmark the skills pipeline with fixed project scenarios, a fixed metric, and reviewable output. Use this before accepting changes to skills, prompts, benchmark data, or quality-review rules.
+
+## Required Context
+
+Read:
+
+- `AI Native/quality-benchmark/skill-benchmark.json`
+- `AI Native/quality-kb/project-quality-kb.json` if present
+- `.agents/skills/test-agent-workflow/references/workflow-registry.yml`
+- `AI Native/workflows/*/workflow-pack.json` when a scenario references `workflow_pack`
+- The candidate Story/AC/Test Case/Review/Strategy output being evaluated
+
+## Benchmark Contract
+
+Each scenario must define:
+
+- `chain`
+- `story`
+- `expected_risk_tags`
+- `expected_layers`
+- `acceptance_focus`
+- `must_find`
+- `min_quality_score`
+- optional `workflow_pack`; when present, the pack must pass deterministic validation before the scenario can pass
+
+Keep benchmark scenarios stable. Add new scenarios when product risk expands; do not rewrite old ones to hide regressions.
+
+## Workflow
+
+1. Validate the workflow structure:
+
+```bash
+ruby ".agents/skills/test-agent-workflow/scripts/validate-workflow.rb"
+```
+
+2. Resolve and run the project-declared KB builder and Skill benchmark under
+   `AI Native/scripts/`; use their actual filenames and reported output paths.
+3. If scoring an AI output pack, compare it manually or with a scorer against:
+   - AC coverage
+   - risk tag recall
+   - layer fit
+   - evidence observability
+   - mock-boundary correctness
+   - duplication and shallow-case rate
+4. Fail the change if workflow validation fails or any critical scenario
+   regresses.
+5. Store the result in the benchmark's declared output path.
+
+## Scoring Model
+
+| Dimension | Points | Gate |
+| --- | ---: | --- |
+| Chain mapped in KB | 20 | Required |
+| Expected risks known | 20 | Required for P0 |
+| Expected layers available | 20 | Required |
+| Must-find assets present | 20 | Required |
+| Quality threshold valid | 20 | `min_quality_score >= 80` |
+
+## Output
+
+```markdown
+## Skill Benchmark
+- benchmark_file:
+- kb_file:
+- total_score:
+- decision: pass | fail | needs-kb-refresh
+
+## Scenario Results
+| scenario | score | missing | recommendation |
+| --- | ---: | --- | --- |
+
+## Regression Decision
+- accepted:
+- reason:
+- follow_up:
+```

.agents/skills/test-agent-governance/references/skill-self-optimization.md

@@ -0,0 +1,71 @@
+# Test Agent Skill Self Optimization
+
+Load only for an explicit, benchmark-backed Skill optimization experiment.
+
+## Purpose
+
+Improve skills through a small, auditable loop: observe benchmark gaps, change one skill or resource, rerun the benchmark, accept only measurable improvement, and log the experiment.
+
+## Hard Boundaries
+
+- Do not modify backend or frontend business code during a skill optimization experiment.
+- Do not edit benchmark scenarios to make a candidate pass.
+- Do not accept a skill change that improves one scenario while regressing a P0 scenario.
+- Do not delete user work or reset the repo automatically. If a change should be rejected, describe the revert patch or ask before destructive git operations.
+
+## Workflow
+
+1. Establish a structural and project benchmark baseline:
+
+```bash
+ruby ".agents/skills/test-agent-workflow/scripts/validate-workflow.rb"
+```
+
+Then resolve and run the actual project-declared KB builder and benchmark; do
+not assume generic script filenames exist.
+
+2. Read the benchmark output path reported by the runner, then identify the weakest scenario or missing asset.
+3. Choose exactly one optimization target:
+   - skill trigger clarity
+   - quality rubric gap
+   - benchmark coverage gap
+   - KB extraction gap
+   - existing case discovery gap
+   - self-repair decision gap
+   - output handoff completeness
+4. Make the smallest useful edit.
+5. Rerun the same commands.
+6. Accept only if:
+   - total score improves or a named gap closes
+   - no critical scenario regresses
+   - generated outputs remain traceable to Story/AC/Test Case/Review
+   - existing case reuse decisions do not hide uncovered P0/P1 risk
+   - self-repair changes do not alter business expectations without a human gate
+7. Append the result to `quality-benchmark/skill-experiments.md`.
+
+## Experiment Log Format
+
+```markdown
+## EXP-YYYYMMDD-NN
+- hypothesis:
+- changed_files:
+- baseline_score:
+- candidate_score:
+- accepted: yes | no
+- regression_check:
+- reviewer_notes:
+```
+
+## Recommended Loop Cadence
+
+- During the two-day workshop: one guided optimization at the end of day 2.
+- During normal delivery: run when quality review finds repeated shallow cases or planner handoffs miss the same risk twice.
+- Before sharing the skills broadly: run the benchmark and freeze the baseline.
+
+## Review Checklist
+
+- The skill description still triggers on real user language.
+- The body teaches a decision that an AI might otherwise get wrong.
+- Scripted checks cover deterministic facts.
+- AI judgment is reserved for semantic review and tradeoff decisions.
+- Benchmark history explains why the change was accepted.

.agents/skills/test-agent-workflow/SKILL.md

@@ -0,0 +1,50 @@
+---
+name: test-agent-workflow
+description: Use when a requirement or feature needs the complete testing lifecycle from acceptance analysis through test design, strategy, execution evidence, repair routing, and final reporting.
+---
+
+# Test Agent Workflow
+
+Coordinate the testing lifecycle and enforce artifact gates. Delegate design,
+review, strategy, and execution decisions to their owning Skills; do not make
+those decisions in the orchestrator.
+
+## Route
+
+1. If an approved requirement is absent, read
+   [User Story Intake](references/stages/01-user-story-intake.md).
+2. If observable acceptance criteria are absent, read
+   [Acceptance Criteria](references/stages/02-acceptance-criteria.md).
+3. Use `test-case-design` to produce layer-neutral business Cases.
+4. Use `test-case-quality-review`; continue only for `PASS` or an explicitly
+   accepted `PASS_WITH_RESIDUAL_RISK` design version.
+5. Use `test-strategy` to select cases, layers, actions, targets, commands,
+   runtimes, order, and gates. Continue only when the decision is `READY`.
+6. Use `test-execution` to implement and run only the READY plan.
+7. On failure, read [Failure Repair Loop](references/stages/07-failure-repair-loop.md)
+   and return the defect to its owning stage.
+8. Aggregate AC-to-evidence traceability using
+   [Final Test Report](references/final-test-report.md).
+
+Read [Artifact Contracts](references/artifact-contracts.md) when validating a
+handoff, and [Workflow Registry](references/workflow-registry.yml) when routing
+or validating paths. For independent coverage analysis, Case review, or
+evidence reproduction, delegate to the project custom agent `test_engineer`
+defined in `.codex/agents/test-engineer.toml`; ordinary workflow runs do not
+require a subagent.
+
+## Boundaries
+
+- Case Design owns obligations, existing-case decisions, techniques, Cases,
+  and oracles; it never selects a test layer.
+- Quality Review approves or returns the design; it never rewrites Cases.
+- Test Strategy is the sole owner of layer and execution planning.
+- Test Execution follows project commands declared by `Tests/test-index.yml`;
+  it never invents a command or changes a Case oracle.
+- Product-code changes require separate authorization. A product failure is a
+  reportable defect by default.
+- Governance is optional. Use `test-agent-governance` only for KB, benchmark,
+  Skill optimization, or CI-quality-flow work.
+
+Run `ruby .agents/skills/test-agent-workflow/scripts/validate-workflow.rb`
+after changing this workflow.

.agents/skills/test-agent-workflow/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Test Agent Workflow"
+  short_description: "Run the complete requirement-to-test-evidence workflow"
+  default_prompt: "Use $test-agent-workflow to complete the testing lifecycle and produce an evidence-backed final report."