Skip to content

add minimal system evaluation harness - #2

Merged
BlueX888 merged 1 commit into
mainfrom
agent/system-evaluation
Jul 19, 2026
Merged

BlueX888 merged 1 commit into
mainfrom
agent/system-evaluation

Conversation

@BlueX888

Copy link
Copy Markdown
Owner

What changed

  • add an isolated Docker Compose topology for PostgreSQL, Temporal, and NATS
  • add a deterministic Fake Agent closed-loop evaluator for E0-E3
  • add Windows-native fallback resource sampling and evidence finalization/hash verification
  • add unit coverage for percentiles, event validation, process sampling, and resource delta calculation

Why

The minimal system evaluation specification required a repeatable load generator, metric collectors, recovery fault injection, raw evidence packaging, and a final threshold report without using real models or external tools.

Evaluation outcome

The local evidence package remains under the required ignored .tmp/system-evaluation/ path and is not part of this PR. The exploratory run concluded INVALID overall because the starting worktree was dirty, R4 raw PostgreSQL query percentiles were unavailable, and CIM resource collection had gaps. Technical results were E0 PASS, E1 PASS, E2 concurrency 4 PASS, E2 concurrency 8 FAIL on P1/P2, and E3 PASS after three recovery rounds.

Validation

  • uv run ruff check .
  • uv run mypy - 110 source files
  • uv run pytest -q tests/unit - 159 passed
  • evidence manifest: 86 entries, zero hash mismatches

@BlueX888
BlueX888 merged commit aab73f9 into main Jul 19, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant