Repository navigation
Conversation
Add eval/code/ scaffolding with a trivial off-by-one bug-fix case (001-simple-bug-fix) to establish baseline functional testing for the code agent pipeline. Signed-off-by: guy oron <goron@redhat.com>
Functional tests did not runFunctional tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the |
PR Summary by QodoAdd baseline code-agent functional eval pipeline and first happy-path case
AI Description
Diagram
High-Level Assessment
Files changed (9)
|
Code Review by Qodo
Context used✅ Compliance rules (platform):
55 rules 1. Eval can exceed CI timeout
|
|
|
||
| skill: code | ||
|
|
||
| execution: |
There was a problem hiding this comment.
2. Eval can exceed ci timeout 🐞 Bug ☼ Reliability
The new code eval allows up to 40 minutes of execution time, but the Functional Tests job times out after 45 minutes while also performing substantial setup in the same job. This configuration can cause GitHub Actions to terminate the run before the eval/harness timeouts complete, producing infra timeouts instead of clean agent failures.
Agent Prompt
### Issue description
`eval/code/eval.yaml` sets `execution.timeout: 2400` (40 minutes) for the code agent, while the GitHub Actions `functional-tests` job has `timeout-minutes: 45` and includes multiple setup steps before the eval even starts. This leaves too little headroom for the worst-case configured agent runtime plus setup/scoring/teardown, so the CI job can kill the run early.
### Issue Context
- Code agent harness timeout is configured at 35 minutes.
- The new eval config allows 40 minutes per execution.
- The CI job has a hard 45-minute cap and performs environment install steps inside that window.
### Fix Focus Areas
- eval/code/eval.yaml[6-9]
- .github/workflows/functional-tests.yml[170-220]
- harness/code.yaml[49-56]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| **Failing test:** | ||
| `test_calc.py::test_days_until_inclusive` fails — it expects 3 but gets 2. | ||
|
|
||
| **Environment:** |
There was a problem hiding this comment.
3. Fixture dependency claim mismatch 🐞 Bug ☼ Reliability
The fixture issue body claims there are “No external dependencies”, but the included test suite imports pytest and the fixture repo does not declare it as a dependency. This can mislead the agent and makes the case dependent on the base environment having pytest preinstalled.
Agent Prompt
### Issue description
The case’s issue text says the project has no external dependencies, but the fixture repo’s tests require `pytest`. This is misleading and can reduce eval determinism if pytest is not present in the agent’s environment.
### Issue Context
The agent rubric asks it to run `pytest`, but the fixture repo doesn’t declare pytest anywhere; the only evidence is the tests importing it.
### Fix Focus Areas
- eval/code/cases/001-simple-bug-fix/input.yaml[27-29]
- eval/code/cases/001-simple-bug-fix/repo/tests/test_calc.py[1-5]
- eval/code/cases/001-simple-bug-fix/repo/pyproject.toml[1-11]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
|
/fs-review |
|
Hello! How this relates to #177 and its issue? |
|
🤖 Finished Review · ✅ Success · Started 12:02 PM UTC · Completed 12:21 PM UTC |
ReviewFindingsMedium
Low
Labels: PR adds eval infrastructure for the code agent |
|
|
||
| [build-system] | ||
| requires = ["setuptools>=68.0"] | ||
| build-backend = "setuptools.backends._legacy:_Backend" |
There was a problem hiding this comment.
[medium] incorrect build configuration in test fixture
The build-backend setuptools.backends._legacy:_Backend does not exist in setuptools and will cause ModuleNotFoundError if the code agent attempts pip install -e . before running tests. While pytest can run without installation from the repo root, this could produce false-negative eval results if the agent installs the package first. The standard backend is setuptools.build_meta.
Suggested fix: Change line 8 to build-backend = "setuptools.build_meta" to use the correct setuptools PEP 517 backend.
| 1 = No fix attempted, or fix is fundamentally wrong | ||
| 2 = Fix attempted but incorrect — introduces new bugs, misidentifies | ||
| the root cause, or breaks existing functionality | ||
| 3 = Adequate — correct fix that resolves the issue, but may include |
There was a problem hiding this comment.
[low] judge evaluation gap
The code_quality judge's score-5 criterion evaluates 'commit message and PR description' but fixture-state.json captures only issue state, not PR state. The judge lacks data for this criterion, making the 4-vs-5 distinction unreliable.
| name: code-eval | ||
| description: Functional test of the fullsend code agent pipeline | ||
|
|
||
| skill: code |
There was a problem hiding this comment.
[low] naming-coherence
skill: code follows the established eval naming pattern (triage, review), though no local skill directory with this exact name exists. Per AGENTS.md section 7, skill resolution may occur at org/platform level.
|
Hey @rh-hemartin — good catch, I wasn't aware of #177 when I opened this. Context: this came out of a Slack discussion with rbean about contributing functional tests as a gate before integrating RTK+Ponytail cost optimizations (~25% savings on Triage+Code). He suggested starting with a trivial happy-path case for the code agent. They overlap: both add Since #177 is already reviewed and |
|
🤖 Finished Retro · ✅ Success · Started 12:26 PM UTC · Completed 12:40 PM UTC |
|
Hey! No need to close if you think this will be valuable regardless of #177 . Reopen if you want. |
Retro: PR #330 —
|
| Time (UTC) | Event |
|---|---|
| 11:51 | PR opened by external contributor guyoron1 — adds eval/code/ scaffolding (+332 lines, 9 new files) |
| 11:52–11:59 | Qodo bot reviewed: found 2 medium bugs (CI timeout risk, pytest dependency claim mismatch) |
| 12:00 | Human maintainer rh-hemartin triggered /fs-review |
| 12:01 | Same human immediately asked: "How this relates to PR #177 and its issue?" |
| 12:02–12:21 | Review agent ran: 7 sub-agents (5 domain + 1 challenger), 19 min, $5.14 (run 29828360327) |
| 12:21 | Author acknowledged overlap, self-closed PR |
Review agent quality
Strengths:
- Found a genuine build-backend bug (
setuptools.backends._legacy:_Backenddoes not exist); challenger verified by runningpython3 -c "from setuptools.backends._legacy import _Backend". - Challenger sub-agent effectively downgraded 3 of 6 findings (from high→medium, medium→low) using evidence from AGENTS.md, existing eval files, and web search.
- Orchestrator self-corrected false positives from its own context summarization for style-conventions and security sub-agents.
Main gap: The review agent missed the most impactful finding — that PR #177 (already ready-for-merge, labeled code-agent) creates substantially overlapping infrastructure in the same eval/code/ directory, including its own eval/code/eval.yaml and first test case. The human caught this in under 30 seconds.
Evidence for existing issues
- fullsend#1313 ("Review agent should check for duplicate/superseding PRs before approving"): This retro provides a new variant — the two PRs don't share a target issue (test(eval scenario): add code agent functional eval scenario #177 closes Add code agent functional eval (pre→sandbox→post PR creation) #180; test(code): add first functional eval case for code agent #330 has no linked issue), so the issue-level duplicate detection proposed in fix(#1309): let review findings authorize protected-path edits #1313 would not have caught this overlap. The overlap is at the file/directory level.
- fullsend#4107 ("Fullsend review agent and Qodo bot produce overlapping findings"): Both Qodo and fullsend-ai-review independently identified the
pyproject.tomlbuild-backend issue. Qodo uniquely found the CI timeout risk; fullsend uniquely found the judge evaluation gap and naming coherence concern. - agents#149 ("Review agent should query for related open PRs before finalizing findings that reference files outside the PR diff"): Covers file-based PR querying but scoped to files outside the diff — would not trigger for this scenario where the overlap is among files inside the diff.
Autonomy observations
The human maintainer's two-sentence interaction (trigger review + ask about overlap) was strategically more valuable than the review agent's 19-minute analysis. For this class of change — new infrastructure scaffolding where parallel efforts may exist — the review agent cannot operate autonomously without sibling-PR awareness. On technical correctness (build-backend bug, config consistency), the agent provided value the human didn't need to duplicate.
Proposals filed
- Review agent PR-specific checks should detect file-level overlap with sibling open PRs (in
fullsend-ai/agents)
Summary
eval/code/scaffolding:eval.yamlpipeline definition with LLM + deterministic judges, hooks, runner config001-simple-bug-fix: a trivial off-by-one bug in a datemath library — correct fix is a single-character change (+ 1)datemath/calc.py+ 5 tests) that gets pushed to GitHub bysetup-fixture.shMotivation
Code agent currently has zero functional tests. This establishes the baseline eval framework so future cases (and optimizations like routing/compression) can be measured against a known-good pipeline.
Context: After sharing RTK (context pruning) + Ponytail (prompt compression) benchmark results in Slack — 30 matched pairs showing ~25% combined cost savings on Triage+Code with equivalent output quality — it was suggested gating integration on functional tests:
This PR is that first trivial happy-path case.
Test plan
eval/lint-cases.sh codepasses+ 1fullsend eval codeend-to-end against ephemeral repo (requires CI)Signed-off-by: guy oron goron@redhat.com