[AshwinUgale]

Closes #8035.

As agreed with @waynesun09 on the issue: every case in this repo's eval/triage suite already exists, newer, in fullsend-ai/agents, so this removes the suite here instead of reworking it.

Removed: functional-tests.yml, functional-ok-to-test.yml, eval/ (including the agent-eval-harness submodule, its CODEOWNERS entry, and the ignore entries), the lint-eval-cases / functional-tests Makefile targets (and from test, .PHONY, help), and the workflow_call_alignment_test.go subtest that read functional-tests.yml.

Kept: the EVAL_GH_TOKEN / E2E_GCP_* wiring in release.yml, which the agents release gate still needs.

Coverage: releases run the release tier (one release: true case per agent; triage case 001). The full suite runs in the agents nightly against fullsend main, which replaces the per-PR run on scaffold/run.go changes, so a regression shows up within a day. docs/testing/functional-tests.md now says this.

CI workflow guide: checked against docs/contributing/ci-workflows.md and updated its hybrid-workflow example. Both deleted workflows are removed on purpose. Their secrets (EVAL_GH_TOKEN, E2E_GCP_*) stay, because release.yml still passes them to the agents gate.

ADR: ADR 0139 supersedes ADR 0052's decision to keep the suite in eval/; ADR 0052 is unchanged. architecture.md, testing-agents.md, ci-workflows.md and the e2e guide are updated.

Remaining grep hits for functional-tests|lint-eval-cases|eval/ outside docs/ADRs/, release.yml and the gate-pin check: the repointed doc naming the agents workflows; links to ADR 0052 (its filename contains functional-tests); agents-gate and historical comments (install-podman.sh, vendorcontent.go, sandbox_test.go); eval/measurements/ paths, which belong to the agents repo for eval scoring; the archived roadmap; and unrelated substrings (guardrails-eval/, promptfoo-eval/, eval/exec).

[qodo-code-review]

PR Summary by Qodo

Remove the duplicate in-repo agent functional-test suite

🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 20-40 Minutes

AI Description

• Remove the duplicate triage suite, its CI workflows, submodule, and Makefile targets.
• Document agents-repo ownership, nightly full-suite coverage, and release-tier coverage.
• Preserve release-gate credentials while removing references and tests tied to the deleted
 workflow.

Diagram

graph TD
  R["Release gate"] --> S["Agents workflow"] --> T["Release tier"]
  M["Fullsend main"] --> N["Agents nightly"] --> S --> F["Full tier"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Call the agents suite on fullsend PRs

• ➕ Retains pre-merge feedback for scaffold and CLI changes without duplicating cases.
• ➖ Retains costly, infrastructure-sensitive PR runs and cross-repository CI coordination.

Recommendation: Removing the duplicate suite is consistent with agents-repo ownership and reduces redundant CI. The trade-off is explicit: fullsend PR regressions are no longer caught by functional tests before merge; releases retain a smoke gate, and nightly runs provide full-suite coverage afterward.

Files changed (13) +108 / -258

Tests (2) +2 / -29

codeowners_repo_test.goCorrect the submodule ownership test comment +2/-2

Correct the submodule ownership test comment

• Removes mention of the deleted eval harness CODEOWNERS entry; the test behavior is unchanged.

internal/owners/codeowners_repo_test.go

workflow_call_alignment_test.goRemove an assertion against the deleted workflow +0/-27

Remove an assertion against the deleted workflow

• Drops the GCP-environment subtest that read functional-tests.yml, while retaining the other workflow alignment checks.

internal/scaffold/workflow_call_alignment_test.go

Documentation (6) +103 / -200

0139-agent-functional-test-suite-lives-in-agents-repo.mdRecord agents-repo ownership of functional tests +60/-0

Record agents-repo ownership of functional tests

• Accepts a new decision superseding ADR 0052’s local suite layout. Records release and nightly coverage, the loss of fullsend PR runs, and retained release credentials.

docs/ADRs/0139-agent-functional-test-suite-lives-in-agents-repo.md

architecture.mdReflect the functional-test ownership decision +3/-2

Reflect the functional-test ownership decision

• Identifies the agents repo as the suite owner and links the new ADR alongside the earlier testing decision.

docs/architecture.md

ci-workflows.mdRemove the deleted workflow from CI guidance +1/-1

Remove the deleted workflow from CI guidance

• Leaves e2e.yml as the example of a hybrid reusable workflow.

docs/contributing/ci-workflows.md

e2e-testing.mdNarrow ok-to-test guidance to E2E +9/-9

Narrow ok-to-test guidance to E2E

• Removes references to the deleted functional-test workflows from the labeling and CI architecture guidance.

docs/guides/dev/e2e-testing.md

testing-agents.mdRepoint the golden-set discussion to agents +1/-1

Repoint the golden-set discussion to agents

• States that the hand-crafted functional cases now live in the agents repository and links ADR 0139.

docs/problems/testing-agents.md

functional-tests.mdReplace local suite instructions with coverage guidance +29/-187

Replace local suite instructions with coverage guidance

• Documents agents-repo case ownership, release and nightly tiers, and the end of fullsend PR functional runs. Replaces obsolete local setup instructions with links to the agents documentation and lists credentials retained for the release gate.

docs/testing/functional-tests.md

Other (5) +3 / -29

.dockerignoreRemove the obsolete eval build-context exclusion +0/-1

Remove the obsolete eval build-context exclusion

• Drops the ignore entry for the deleted eval directory.

.dockerignore

.gitignoreRemove the obsolete eval-runs ignore entry +0/-1

Remove the obsolete eval-runs ignore entry

• Drops the ignore entry for results produced by the removed local suite.

.gitignore

.gitmodulesUnregister the local eval harness submodule +0/-4

Unregister the local eval harness submodule

• Removes the agent-eval-harness submodule configuration associated with the deleted suite.

.gitmodules

CODEOWNERSRemove ownership entry for the deleted harness +0/-1

Remove ownership entry for the deleted harness

• Removes the CODEOWNERS entry for the eval harness submodule.

CODEOWNERS

MakefileRemove local functional-test and case-lint targets +3/-22

Remove local functional-test and case-lint targets

• Removes the eval targets from help, .PHONY, and make test, along with the deleted fixture script test from script-test.

Makefile

[AshwinUgale]

The functional-tests failure is expected: it's a pull_request_target workflow, so it runs main's copy of functional-tests.yml against this branch, which no longer has eval/. It isn't a required check and goes away once this merges.

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. ADR context repeats testing rationale ✓ Resolved

Description

The new ADR's Context explains the functional-test duplication and relocation without linking to the
related testing-agents problem document. When readers need the broader testing rationale, they
must reconstruct it from the ADR instead of following the declared problem-area reference.

Code

docs/ADRs/0139-agent-functional-test-suite-lives-in-agents-repo.md[R25-38]

+## Context
+
+Supersedes [ADR 0052](0052-functional-tests-for-agent-pipelines.md) where it
+places the functional-test suite in this repo's `eval/` directory and runs it
+on changes to `eval/` and `internal/scaffold/`. The test category itself, and
+its place in the test pyramid, are unchanged.
+
+Since then, agent definitions moved to
+[fullsend-ai/agents](https://github.com/fullsend-ai/agents), which grew its
+own functional-test suite for every agent: `release: true` smoke cases
+(the release tier) and the full case set (the full tier). The `eval/triage`
+suite here duplicated cases that agents already holds, ran up to three times
+per change, and failed mostly on infrastructure rather than on regressions
+([#8035](https://github.com/fullsend-ai/fullsend/issues/8035)).

Relevance

●●● Strong

Recent ADR precedents accepted adding explicit Context links to documents named by relates_to
metadata.

PR-#6926
PR-#5016

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The ADR declares testing-agents as its related problem area, but its Context instead provides two
paragraphs describing the testing history, duplication, and relocation without linking that problem
document.

docs/ADRs/0139-agent-functional-test-suite-lives-in-agents-repo.md[25-38]
Skill: writing-adrs

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The ADR's Context restates the functional-testing rationale but does not link to the related `testing-agents` problem document named in `relates_to`.

## Fix Focus Areas
- docs/ADRs/0139-agent-functional-test-suite-lives-in-agents-repo.md[25-38]

## Recommended Fix
Add a link to `../problems/testing-agents.md` in the Context with a brief summary, and retain only the decision-specific explanation needed to explain why the suite moved to the agents repository.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Old test results enter runner builds 🐞 Bug ⛨ Security

Description

Removing eval/ from .dockerignore lets a leftover eval/runs/ directory pass through the runner
image's COPY . . into its build-stage layer. Developers who ran the former suite can retain those
results after updating their checkout, so a local runner build includes them even though the final
image copies only the binary.

Code

.dockerignore[12]

-eval/

Relevance

●●● Strong

Security-sensitive build-context leakage is a concrete defense-in-depth issue; runner image
precedent emphasizes trimming contexts.

PR-#5201
PR-#1627

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The deleted suite wrote results under eval/runs/, and removing a tracked suite does not remove
ignored local run results. The runner build uses the repository root as its context and copies that
context into the build stage; the new Docker ignore file no longer excludes eval/.

.dockerignore[1-21]
.github/workflows/runner-image.yml[112-121]
images/runner/Containerfile[43-48]
images/runner/Containerfile[334-339]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Removing the Docker ignore rule allows results left by the former eval suite to enter runner build contexts and intermediate layers.

## Fix Focus Areas
- .dockerignore[1-21]

## Recommended Fix
Retain an ignore rule for `eval/` in the runner build context, even though the suite has been deleted from the repository.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

---

3. CI workflow changes lack guide review ✗ Dismissed

Description

The pull request deletes the functional-test GitHub Actions workflows without stating that the CI
workflow guide was consulted. Because workflow behavior and secret-bearing jobs are removed in this
change, the required review evidence is absent from the PR description, checklist, and changed
workflow files.

Code

.github/workflows/functional-tests.yml[L1-4]

-name: Functional Tests
-
-# PR-triggered functional tests use pull_request_target so fork PRs receive
-# secrets.  Authorization runs in a separate gate job (base checkout only)

Evidence

The diff deletes .github/workflows/functional-tests.yml and
.github/workflows/functional-ok-to-test.yml; the supplied PR description contains no statement
that docs/contributing/ci-workflows.md was consulted.

Rule 3201588: Consult CI workflow guide when modifying GitHub Actions workflows or secrets

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
This pull request removes GitHub Actions workflow files but provides no explicit evidence that `docs/contributing/ci-workflows.md` was consulted.

## Fix Focus Areas
- .github/workflows/functional-tests.yml[1-278]
- .github/workflows/functional-ok-to-test.yml[1-26]
- docs/contributing/ci-workflows.md[1-30]

## Recommended Fix
Add an explicit PR-description or review-checklist statement confirming consultation of `docs/contributing/ci-workflows.md` and explaining that the deleted workflows and their secret-bearing jobs were intentionally removed.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Core problem docs name a specific repo ✓ Resolved

Description

The updated golden-set bullet names fullsend-ai/agents inside the core problem document
docs/problems/testing-agents.md. Because organization-specific project details belong under
docs/problems/applied/, this reference makes the core document organization-specific and can
mislead readers about the problem's scope.

Code

docs/problems/testing-agents.md[375]

+- ~~How do we bootstrap the golden set? Do we start with synthetic examples, or do we capture real-world cases from early human-supervised agent operation?~~ Functional tests bootstrap with hand-crafted cases, now kept in the fullsend-ai/agents functional-test suite; see [ADR 0052](../ADRs/0052-functional-tests-for-agent-pipelines.md) and [ADR 0139](../ADRs/0139-agent-functional-test-suite-lives-in-agents-repo.md). Prompt-level evals and synthetic expansion remain open.

Relevance

● Weak

A close precedent rejected requiring core problem documents to remove organization-specific
implementation and ADR references.

PR-#5646

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The changed line adds the organization-specific repository identifier fullsend-ai/agents to a core
document outside docs/problems/applied/. The compliance rule requires core problem documents to
remain separate from organization-specific details.

Rule 1062038: Separate generic and organization-specific problem documents
docs/problems/testing-agents.md[375-375]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The core problem document names the organization-specific repository `fullsend-ai/agents`, but organization-specific identifiers must be kept under `docs/problems/applied/`.

## Fix Focus Areas
- docs/problems/testing-agents.md[375-375]

## Recommended Fix
Replace the organization-specific repository name with a generic reference such as “the agents repository,” while retaining the ADR links that document the repository-specific implementation decision.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Tickets: 🎫 Retire this repo's eval/triage functional tests, or run them on the release-tier rules

✅ Compliance rules (platform): 89 rules

✅ Skills: 5 invoked

  writing-user-docs

  writing-adrs

  check-patch-coverage

  cutting-releases

  adr-corner

✅ Cross-repo context — repo relationships

  Explored: repo: fullsend-ai/agents (sha: 91ac5297) — View relationship

Review mode: Auto: ⚖️ Balanced: Broad CI, build, test, and documentation changes require careful single-pass review.

Tip of the day

💡 Did you know, you can commit Qodo's fix in one click with committable suggestions (GitHub & GitLab)

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[github-actions]

Site preview

Preview: https://8674aab4-site.fullsend-ai.workers.dev

Commit: 13d2ebf0d5f3f969e7cbb76fa08b39ff0762dc58

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[waynesun09]

LGTM. Checked: nothing left references the removed workflows, targets or eval/ paths; release.yml is unchanged; ADR 0052 is untouched and 0139 passes the ADR linters; make lint-md-links and the scaffold/owners Go tests pass. The red functional-tests check is main's copy of the deleted workflow (pull_request_target) and isn't required.

Nit, non-blocking: ADR 0139 says "every agent", but prioritize and scribe have no eval suite, so "every agent with an eval suite" would be more exact.