Skip to content

docs(adr): let agent entries carry harness overrides - #8965

Merged
waynesun09 merged 1 commit into
fullsend-ai:mainfrom
gwenneg:adr-config-agent-harness-overlay
Nov 10, 2026
Merged

waynesun09 merged 1 commit into
fullsend-ai:mainfrom
gwenneg:adr-config-agent-harness-overlay

Conversation

@gwenneg

@gwenneg gwenneg commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

What

This adds ADR 0162 (it was 0154, but that number got taken on main). An agents: entry in config.yaml can carry a harness: block that works like a harness built on the agent's harness, with nothing pinned:

agents:
  - name: review
    harness:
      env:
        runner: { REVIEW_FINDING_SEVERITY_THRESHOLD: medium }
        sandbox: { REVIEW_FINDING_SEVERITY_THRESHOLD: medium }
  - name: code
    harness:
      timeout_minutes: 90

Today the only way to change a harness setting is a base: harness. That pins an agents commit, so the repo stops getting agent updates (#5802). With the block, a repo changes the setting and keeps getting updates.

This is the ADR only. The code comes in a separate PR.

What changed after review

  • The block is now a derived harness and takes almost every harness field, not only env (@ifireball).
  • fullsend run, fullsend lint, fullsend lock and dispatch all compose it (@ifireball).
  • The blocks in config.base.yaml and config.yaml layer like harness inheritance (@ifireball).
  • The harness: key stays (@waynesun09, agreed by @rh-hemartin and @ifireball). Retiring the entry's trigger is follow-up work (@rh-hemartin).
  • Turning personas on and off stays out of env (@waynesun09).
  • The ADR has no line-pinned code links anymore. The code evidence is below instead (@waynesun09).
  • Point 6 now uses @waynesun09's wording, after @rh-hemartin's review: a harness.trigger on a built-in agent is not evaluated and loads with a warning, instead of failing.

Other choices worth a look

  • The block takes every harness field except these three:
    • base:, because the entry's source already does that job.
    • forge and runner_env, because both are deprecated: ADR 0088 replaces forge with overlays, and ADR 0055 replaces runner_env with env.runner. A new place to set harness fields shouldn't start with deprecated fields. runner_env would also let the block add runner variables the harness doesn't declare, because the run merges it into the runner environment. That's also why the block's overlays can't set runner_env. Harnesses that still use either field keep working with a block: names in their forge: blocks count as declared, and the block can use overlays on runtime.forge instead. No fleet harness uses runner_env today.
  • The block can set schema_version, under ADR 0127's rules like any layer: absent means 1, and a block whose version differs from the agent's harness fails to load. So when an agents release moves a harness to a new schema version, a repo with a block on that agent updates the block when it upgrades.
  • With harness layering, config.yaml can replace a value from config.base.yaml but not remove it.
  • Like before, the block can't add env variables the harness doesn't declare.
  • Safety gate follow-up. ADR 0122's safety gate doesn't look at harness settings today, so it won't compare block values. With all fields in the block, a repos.yaml preset could change security settings through it without the gate flagging it. Extending the gate stays follow-up work, and it may be worth doing before the feature ships.

Code evidence

All links are at 087ccff70, the current main.

  • config.yaml is read from the same trusted ref as harness files: on GitHub the PR base commit for pull_request_target and pull_request_review, otherwise the triggering commit (reusable-dispatch.yml#L154, same expression in every agent job); on GitLab the default branch (run-agent-job.sh#L345-L370).
  • CEL dispatch, the Jira poller and fullsend lock skip built-in agents (entries without source) (registry.go#L48-L53). This is why lock keeps not locking them (ADR 0091).
  • The entry's trigger only routes review, retro, triage and prioritize (dispatch_route.go#L50-L55, #L194).
  • Dispatch reads a built-in agent's harness role to set who may trigger it (dispatch_route.go#L255), so a block that sets role changes that.
  • Composition rejects a base: chain that mixes schema versions (compose.go#L326).
  • The run merges runner_env into the runner environment beside env.runner (run.go#L1292-L1308), so allowing it in the block would bypass the declared-name check.
  • An overlays entry carries the same fields as a forge: block, runner_env included (forge.go#L18-L41), and a matching overlay's runner_env is merged into the harness's (forge.go#L417). That's why the block's overlays can't set it either.
  • A base layer's forge: blocks and overlays are resolved before a child merges on top (compose.go#L642-L653), so a block's overlays work on harnesses that still use forge:.
  • Harness env values are plain strings, so ~ becomes an empty value, not a removal (harness.go#L313-L314).
  • The ADR 0122 safety gate compares kill switch, roles, allowed remote resources, re-enabled agents, tracker targets and the inference gateway, not harness settings (managed_safety.go#L90).
  • fullsend run pins only what a current lock entry covers and only warns otherwise (run.go#L698-L742). A lock entry's freshness is the harness file's hash only (lock.go#L867), which is why the ADR says a block change counts as a harness change.
  • Today's CLI rejects an entry that has only a block (config.go#L781), and so does the dispatch workflow's entry check (reusable-dispatch.yml#L906).

Notes for reviewers

  • I updated the short notes in ADRs 0049, 0080, 0081, 0091 and 0112 that point to this one. They don't change those decisions.
  • I also updated architecture.md, harness-fields.md, config-source-inventory.md and config-reference.md.
  • Not part of this PR: the review agent's default protected paths don't include .fullsend/. That's an agents-repo change.

Refs #8143, #7719, #7223, #8903

🤖 Generated with Claude Code

@gwenneg
gwenneg requested a review from a team as a code owner October 31, 2026 01:33
@github-actions

Copy link
Copy Markdown

E2E tests did not run

E2E tests run automatically for pull requests authored by users with write access or higher.

For other contributors, someone with write access must add the ok-to-test label after the latest push.

Reason: unauthorized

See E2E testing guide for details.

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Grey Divider

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Grey Divider

Tip of the day
💡 Did you know, you can keep summaries lean with Findings visible per group, which tucks the rest behind a View link

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Document agent-entry overrides for harness environment values

📝 Documentation 🕐 20-40 Minutes

Grey Divider

AI Description

• Accept ADR 0154 to let repositories tune declared harness environment values without pinning an
 agents commit.
• Specify merge precedence, layered configuration, validation, and rollout requirements for the
 future implementation.
• Link related ADRs and update configuration guidance; this PR changes no runtime behavior.
Diagram

graph TD
  Base["Base config"] --> Entry["Agent entry"] --> Merge["Overlay merge"] --> Check{"Declared name?"} -->|Yes| Env["Effective env"]
  Config["Repo config"] --> Entry
  Harness["Resolved harness"] --> Merge
  Check -->|No| Error["Validation error"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Automate base harness pin updates
  • ➕ Preserves the existing harness composition mechanism.
  • ➖ Still requires a custom harness file for a single value.
  • ➖ Makes agent updates depend on successful pin automation.
2. Add settings directly to agent entries
  • ➕ Could make individual overrides shorter.
  • ➖ Requires placement and merge rules for each setting.
  • ➖ Mixes harness values with entry-level settings.

Recommendation: The scoped harness block is the best documented approach: it keeps harness values in their own namespace and proposes reusing environment overlay semantics without requiring a pinned base harness. Review the declared-name and safety-gate follow-ups before implementation, since this PR establishes the decision but does not enforce it.

Files changed (10) +207 / -4

Documentation (10) +207 / -4
0049-agent-configuration-env-var-convention.mdLink the environment-variable convention to ADR 0154 +4/-0

Link the environment-variable convention to ADR 0154

• Adds an amendment note explaining that an agent entry can override a harness environment default without base composition.

docs/ADRs/0049-agent-configuration-env-var-convention.md

0080-config-yaml-vs-agent-env-var-scope.mdClarify the new override location for single-agent knobs +5/-0

Clarify the new override location for single-agent knobs

• Notes that an agent-entry harness block can set an existing runner or sandbox variable without introducing a parallel config field.

docs/ADRs/0080-config-yaml-vs-agent-env-var-scope.md

0081-reserve-workflow-env-for-infra-plumbing.mdDistinguish agent overrides from workflow environment values +5/-0

Distinguish agent overrides from workflow environment values

• Points to ADR 0154 while retaining the restriction of workflow env to infrastructure plumbing.

docs/ADRs/0081-reserve-workflow-env-for-infra-plumbing.md

0091-per-agent-runtime-model-effort.mdAnnotate the expanded scope of agent-entry settings +4/-0

Annotate the expanded scope of agent-entry settings

• Links the earlier per-agent settings decision to the proposed harness environment block.

docs/ADRs/0091-per-agent-runtime-model-effort.md

0112-overlays-may-set-any-harness-field.mdCross-reference config-based harness environment overrides +5/-0

Cross-reference config-based harness environment overrides

• Adds a related-decision note distinguishing the agent-entry block from harness overlays.

docs/ADRs/0112-overlays-may-set-any-harness-field.md

0154-agent-entries-may-overlay-harness-fields.mdAccept the agent-entry harness environment override decision +165/-0

Accept the agent-entry harness environment override decision

• Defines an env-only harness block, its merge order, per-key layering and tombstones, declared-name checks, and override-only entry behavior. Records alternatives, forge scope, security consequences, compatibility requirements, and implementation follow-ups; no implementation is included.

docs/ADRs/0154-agent-entries-may-overlay-harness-fields.md

architecture.mdAdd the proposed config-to-harness override path +8/-1

Add the proposed config-to-harness override path

• Describes agent-entry environment overrides as an alternative to pinning a base harness and records their intended precedence.

docs/architecture.md

config-source-inventory.mdInventory future agent-entry harness settings +3/-3

Inventory future agent-entry harness settings

• Notes the planned layered harness block for review severity. Clarifies that code auto-merge variables would first need declarations in the code harness.

docs/contributing/config-source-inventory.md

harness-fields.mdFlag ADR 0154 alongside harness field guidance +4/-0

Flag ADR 0154 alongside harness field guidance

• Adds a note about the decided env-only agent-entry block and defers other harness fields to a later decision.

docs/contributing/harness-fields.md

config-reference.mdMark agent-entry harness overrides as not yet implemented +4/-0

Mark agent-entry harness overrides as not yet implemented

• Adds a harness entry to the config reference, linking the accepted ADR while explicitly stating that the feature is not yet available.

docs/reference/config-reference.md

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Site preview

Preview: https://02f304ce-site.fullsend-ai.workers.dev

Commit: 4a593628b0b65a4ee238d973bc524887e43baf74

@codecov

codecov Bot commented Oct 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@rh-hemartin

Copy link
Copy Markdown
Member

Hey! Sorry, I didn't notice the harness subkey when I referenced your example on the issues comments. I don't like it very much, did you explore doing this without the harness key? Could you? If we read each agents' entry as a harness, it should automatically discard unknown fields (like enabled) and we should kind of get overrides for free.

@gwenneg

gwenneg commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

@rh-hemartin Thanks! I did consider a flat entry (the ADR mentions it) but I see two issues.

First, some names exist on both sides. model and effort are probably fine since they mean the same thing. On the other hand, trigger is a problem. On an entry it only controls routing for the built-in review/retro/triage/prioritize agents. In a harness, it controls CEL dispatch for custom agents and the Jira poller. So a flat trigger would need only one meaning, or one of them would need to be renamed. Same risk applies to any field added to both at a later time.

Then, config.yaml currently rejects unknown keys in an agent entry. If we read the entry as a harness and drop what it doesn't know, a typo like modle: opus would just be ignored. #8143 explicitly asks for mistakes to fail loudly.

The harness: key help avoid both. But if you prefer flat, I can rework it that way. What do you think about trigger?

@ggallen

ggallen commented Nov 2, 2026

Copy link
Copy Markdown
Member

Then, config.yaml currently rejects unknown keys in an agent entry. If we read the entry as a harness and drop what it doesn't know, a typo like modle: opus would just be ignored. #8143 explicitly asks for mistakes to fail loudly.

If repos management (i.e. fullsend repos subcommands) is being used, then I believe this will fail loudly.

@waynesun09

Copy link
Copy Markdown
Member

+1 to keeping the harness: block. It also fits ADR 0155 (#8969), which extends agents[].subagents so an entry can set a persona's model, source and enabled: false, and compiles those into harness skills: file overrides.

With the block, an entry's own fields and harness fields stay apart:

agents:
  - name: review
    subagents:                 # entry field: personas, by name (0155)
      user-experience: { enabled: false }
    harness:                   # harness fields (this ADR)
      env:
        sandbox: { REVIEW_FINDING_SEVERITY_THRESHOLD: medium }

A flat entry would make harness skills: settable right next to subagents, giving two persona controls on one entry with a precedence rule between them. The trigger clash is the same problem in another field.

Two smaller points:

  • Persona on/off stays out of env. The UX reviewer's REVIEW_UX_ENABLED flag was removed in favour of persona entries. Item 5 here already rejects names the harness doesn't declare, so harness.env can't bring it back. A short line saying so would keep the per-persona env-flag pattern from returning.
  • Line-pinned links. Context and Decision still have several blob/<sha>#L… links to code. These go stale and make the decision harder to read. Stating the behaviour in plain words, with the code evidence in the PR description, would fix that.

@rh-hemartin, does keeping the block work for you, given the trigger and typo points above?

@gwenneg

gwenneg commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

@ggallen Thanks, I wasn't aware of that. I checked, and I was wrong: only the config: block in repos.yaml rejects unknown keys in an agent entry. A config.yaml edited by hand already ignores them today. So a flat entry wouldn't make that worse.

@rh-hemartin

Copy link
Copy Markdown
Member

I like even less the fact that we do have agents[].trigger and that means something different than the harness' trigger, if I got that right. I would:

  1. Implement this without harness key, for fleet agents we would keep matching on name, and then consider the whole entry as a harness, use the usual override mechanism.
  2. Change the direction of the subagents Wayne is doing and introduce in the harness instead, so we get the key in agents[] for free.
  3. Make fullsend repos validate the agents[] against the harness schema.

Maybe we can bring this to one of our meetings tomorrow?

@ifireball

Copy link
Copy Markdown
Member

I like even less the fact that we do have agents[].trigger and that means something different than the harness' trigger, if I got that right. I would:

1. Implement this without `harness` key, for fleet agents we would keep matching on `name`, and then consider the whole entry as a harness, use the usual override mechanism.

2. Change the direction of the `subagents` Wayne is doing and introduce in the `harness` instead, so we get the key in `agents[]` for free.

3. Make `fullsend repos` validate the `agents[]` against the harness schema.

Maybe we can bring this to one of our meetings tomorrow?

I tent to agree that most top-level keys we have defined now that ovveride things that are defined by the harness need to go away, especially if the semantics are not exactly like the ones for the harness.

I still think we need a "harness" key because there are things that belong in the ganets list but not the harness, for example:

  • enabling disabling agents
  • associating agents with entity sources (the agent harness says it needs work items (a.k.a. issues), but the config says where the work items come from (e.g. jira, github, both...))

@gwenneg

gwenneg commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Since there's no consensus yet on the next step, I'll hold off on updating the ADR. Please let me know if there's anything I can do or any changes I should make.

@rh-hemartin

Copy link
Copy Markdown
Member

Hello! Wanted to come back here. I agree with Barak and Wayne, so I think this is unblocked for you @gwenneg . So let's go with harness key and as followups to clean this we will retire trigger from agents[], so there is just name, enabled and harness. Does this sound good?

@gwenneg
gwenneg force-pushed the adr-config-agent-harness-overlay branch from 3289718 to 42eed21 Compare November 7, 2026 10:00
@gwenneg gwenneg changed the title docs(adr): let agent entries override harness env values docs(adr): let agent entries carry harness overrides Nov 7, 2026
@gwenneg

gwenneg commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

@rh-hemartin sounds good, thanks! I pushed the updated ADR. It's now ADR 0162 because 0154 got taken on main.

Quick summary of what changed:

  • The harness: block now works as a harness built on the agent's harness, and takes almost every field, not only env.
  • It can't take base:, forge or runner_env.
  • fullsend run, lint, lock and dispatch all read it.
  • The blocks in config.base.yaml and config.yaml layer the same way harnesses do.
  • The harness: key stays, and retiring trigger from agents[] is listed as follow-up work.
  • Turning personas on and off stays out of env.
  • The ADR has no line-pinned code links anymore.

One part is new and nobody suggested it in the review: point 6, which covers model, effort and trigger set both on the entry and in the block. @rh-hemartin @ifireball @waynesun09 could you take a look and tell me if it works for you?

All the details are in the PR description, including the reasons for each choice and the code links.

@rh-hemartin

Copy link
Copy Markdown
Member

For point 6, I would remove

On a built-in agent, harness.trigger fails. Dispatch never evaluates the harness trigger of a built-in agent, so it would be silently ignored. The entry's trigger keeps routing review, retro, triage and prioritize as today.

Because that would be a "bug", as we didn't get that far in the implementation but we do want to go there. So failing the harness.trigger for default agents would send mixed signals. Today the harnesses have triggers if I recall that right, so the confusion may be there already. Maybe we shouldn't add a harness.trigger example yet, and when we retire the entry's trigger we can then put an example regarding harness.trigger. If you want to retain something there, I would go with a warning instead.

Otherwise looks good to me

@gwenneg

gwenneg commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

@rh-hemartin Thanks! I'm waiting for additional input from other reviewers before I remove or update the details from point 6 you highlighted.

@waynesun09

Copy link
Copy Markdown
Member

Point 6: +1 to @rh-hemartin. The end state is CEL-only routing (#2902), with agents[].harness.trigger as the opt-in once the entry's trigger is retired. Today, review, retro, triage and prioritize opt into CEL routing through the entry's trigger (with workflow routing as fallback), while code and fix are not ported yet. So I'd frame point 6 around the transition ("until") rather than a validation failure:

  1. Fields the entry already has. Setting model, effort or trigger on both an agent entry and its harness block fails.

Built-in agents are review, retro, triage, prioritize, code and fix, whether or not the entry has a source. Until all built-in agents are routed by CEL alone (#2902):

  • An entry trigger on review, retro, triage or prioritize opts that agent into CEL routing, with workflow routing as fallback.
  • An entry trigger on code or fix routes nothing; they keep workflow routing until ported.
  • A harness.trigger on any built-in agent is not evaluated; the configuration loads with a warning naming the agent.

These checks apply to the entry after config.base.yaml and config.yaml are merged.

Keying the check on the agent name rather than on source also covers an entry with a built-in agent name and a source. On GitHub, dispatch also drops harness.trigger for that entry (core.go#L92).

On skills: vs subagents: in #8969 I'll move persona entries into the harness schema (agents[].harness.subagents). Under ADR 0155 point 3 they already take precedence over a skills: override of the same file, so ADR 0162 needs no rule for this.

Add ADR 0162: an agents: entry in config.yaml may carry a harness: block
that works as a harness derived from the agent's harness, so a repo can
change harness settings without writing and pinning a base: harness.

Point the notes in ADRs 0049, 0080, 0081, 0091 and 0112 to it, and
describe the block in architecture.md, the harness field reference, the
config reference and the config source inventory.

Refs fullsend-ai#8143

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Gwenneg Lepage <10584698+gwenneg@users.noreply.github.com>
@gwenneg
gwenneg force-pushed the adr-config-agent-harness-overlay branch from 42eed21 to 4a59362 Compare November 10, 2026 18:45

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Point 6 now keys on the agent name with the "until #2902" warning. LGTM.

@waynesun09
waynesun09 dismissed ifireball’s stale review November 10, 2026 18:46

Superseded: the ADR was reworked as ADR 0162 in review (harness block, point 6 updated) and has two maintainer approvals.

@waynesun09
waynesun09 added this pull request to the merge queue Nov 10, 2026
Merged via the queue into fullsend-ai:main with commit c4dc2b3 Nov 10, 2026
64 checks passed
waynesun09 added a commit that referenced this pull request Nov 10, 2026
ADR 0162 merged in #8965, so the two links to it now point at the
file instead of the pull request.

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>

This branch was successfully deployed

1 active deployment
site-preview — 4a593628 Deployed Nov 10, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants