Skip to content

refactor(#5438)!: per-scenario World in context + in-process repo lease pool - #5444

Merged
ifireball merged 10 commits into
mainfrom
agent/5438-world-per-scenario-lease-pool
Jul 23, 2026
Merged

ifireball merged 10 commits into
mainfrom
agent/5438-world-per-scenario-lease-pool

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor

Summary

  • Per-scenario World via context.Context: Each godog scenario now gets its own World instance, cloned from a suite-level template in the Before hook and stored in the godog context. Step definitions extract their World via world.FromContext(ctx) instead of closing over a shared pointer. This prevents concurrent scenario hooks from clobbering each other's fields.

  • In-process repo lease pool: Added RepoPool — a buffered-channel pool pre-filled with 12 logical names (test-repo-01 ... test-repo-12). Scenarios acquire a name in Before and release it in After. At Concurrency: 1 the leased names are plumbing only; the functional test repo remains test-repo.

  • No behavior change: GODOG_CONCURRENCY stays at 1, admin e2e still uses singular test-repo, and all existing unit tests pass. This is infrastructure plumbing for the parallel execution work in ci(e2e): enable GODOG_CONCURRENCY for behaviour suite (default 12) #5441.

Related Issue

Part of #3454. Prerequisite for #5439, #5440, #5441.

Changes

New files

  • pkg/behaviourtest/world/context.go — WithWorld/FromContext helpers for storing World in context
  • pkg/behaviourtest/world/pool.go — RepoPool type with Acquire/Release/Size
  • pkg/behaviourtest/world/pool_test.go — unit tests for pool (distinct names, blocking, release, concurrency)
  • pkg/behaviourtest/world/context_test.go — unit tests for context helpers (round-trip, independence)

Modified files

  • pkg/behaviourtest/world/world.go — added LeasedRepoName field and Clone() method
  • pkg/behaviourtest/world/world_test.go — added Clone tests (driver copy, scenario zeroing, independent mutation)
  • pkg/behaviourtest/suite/init.go — InitScenario now takes template + pool, clones World per-scenario, acquires/releases pool names
  • pkg/behaviourtest/steps/registry.go — Register no longer takes *world.World
  • pkg/behaviourtest/steps/{triage,dispatch,dispatch_count,fork,dummy_agent}.go — step closures now accept context.Context and extract World via FromContext
  • e2e/behaviour/suite_test.go — creates RepoPool(12), passes template + pool to InitScenario

Testing

  • All unit tests pass: go test ./pkg/behaviourtest/... (world, suite, steps packages)
  • Full project builds: go build ./...
  • go vet clean
  • Secret scan clean
  • Pre-commit could not run (sandbox network restriction); post-script runs it authoritatively

Checklist

  • Per-scenario World isolation via context.Context
  • In-process lease pool with 12 logical repo names
  • Concurrency: 1 preserved
  • Admin test-repo constant unchanged
  • Unit tests for pool and context helpers
  • Existing tests pass

Closes #5438

Post-script verification

  • Branch is not main/master (agent/5438-world-per-scenario-lease-pool)
  • Secret scan passed (gitleaks — 675086c5e606fe1ea9c09c22e1aa617aa17afeae..HEAD)
  • PR body secret scan passed (gitleaks — no-git)
  • Pre-commit hooks passed (authoritative run on runner)
  • Tests ran inside sandbox

@fullsend-ai-coder
fullsend-ai-coder Bot requested a review from a team July 22, 2026 07:46
@fullsend-ai-coder fullsend-ai-coder Bot added the ready-for-review Triggers review agent dispatch label Jul 22, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 7:48 AM UTC · Completed 8:07 AM UTC
Commit: f3be5ac · View workflow run →

@github-actions

github-actions Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

Site preview

Preview: https://c408bc38-site.fullsend-ai.workers.dev

Commit: 097d3e8e73ee538f655b1d889e7a4fa27b17d1d7

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

Review

Findings

Medium

  • [breaking-api-change] pkg/behaviourtest/suite/init.go:19 — suite.InitScenario signature changed from InitScenario(sc, w) to InitScenario(sc, template, pool). External consumers (e.g., fullsend-ai/agents) will need to update their call sites when bumping the pinned version. The breaking change is properly marked with ! in the PR title and documented with migration instructions in docs/guides/dev/behaviour-testing.md. The pool parameter accepts nil, which simplifies minimal migration.

Low

  • [breaking-api-change] pkg/behaviourtest/steps/registry.go:9 — steps.Register signature changed from Register(ctx, w) to Register(sc), removing the *world.World parameter. Documented in the API changes section. Most external consumers call suite.InitScenario (which calls Register internally) rather than calling Register directly, limiting the blast radius.

  • [error-handling] pkg/behaviourtest/suite/init.go:52 — In afterScenario, when pool.Release fails and scenarioErr is non-nil, the release error is logged via w.Logf but only scenarioErr is returned. Both errors could be surfaced together using errors.Join(retErr, releaseErr) to ensure double-release bugs are not masked by coincidental scenario failures.

  • [test-adequacy] pkg/behaviourtest/suite/init_test.go — TestResetScenarioWorld_ClearsSharedState does not assert that LeasedRepoName is reset to empty string, even though resetScenarioWorld explicitly sets w.LeasedRepoName = "". The test also omits assertions for IssueTitle, TriageTriggerEvent, TriageWorkflow, and WorkflowRun.

  • [race-condition] pkg/behaviourtest/world/pool.go:48 — In Release, the mutex is unlocked before sending the name back on the channel. The invariant len(channel) + len(outstanding) == size guarantees the send will not block, so this is technically safe. There is a brief window where the name is in neither the channel nor the outstanding map, but no consumer can observe this inconsistency in a way that causes incorrect behavior.

  • [shallow-copy] pkg/behaviourtest/world/world.go:53 — World.Clone() performs a shallow copy. Interface fields SCM, CI, and Install are shared between the template and all cloned scenario Worlds. This is safe at Concurrency:1 (the explicit scope of this PR) but is a latent concern for future concurrent work (ci(e2e): enable GODOG_CONCURRENCY for behaviour suite (default 12) #5441).

  • [code-organization] pkg/behaviourtest/world/pool.go — In the RepoPool struct, mu sync.Mutex is declared between size and outstanding. Other structs in the codebase declare the mutex as the first field.

Previous run

Review

Findings

Medium

  • [breaking-api-change] pkg/behaviourtest/suite/init.go:19 — suite.InitScenario signature changed from InitScenario(sc, w) to InitScenario(sc, template, pool). External consumers (e.g., fullsend-ai/agents) will need to update when bumping the pinned version. The breaking change is properly marked with ! in the PR title and documented with migration instructions in docs/guides/dev/behaviour-testing.md. The pool parameter accepts nil, which simplifies minimal migration.

  • [breaking-api-change] pkg/behaviourtest/steps/registry.go:9 — steps.Register signature changed from Register(ctx, w) to Register(sc), removing the *world.World parameter. Documented in the API changes section. Most external consumers call suite.InitScenario (which calls Register internally) rather than calling Register directly.

Previous run (2)

Review

Findings

Medium

  • [potential-deadlock] pkg/behaviourtest/world/pool.go — In Release, the mutex p.mu is held (via defer p.mu.Unlock()) when sending to the p.names channel (p.names <- name). In correct usage the channel always has spare capacity (each released name was previously acquired, guaranteeing a free slot), but holding a mutex while performing a channel send is a risky pattern. If a bug elsewhere causes a concurrent double-release of the same name (both goroutines pass the outstanding-map check before either deletes), the second send could block with the mutex held, deadlocking any concurrent Release or Acquire call.
    Remediation: Move the p.names <- name send after p.mu.Unlock(). Delete from the outstanding map, unlock, then send to the channel.

  • [breaking-api-change] pkg/behaviourtest/suite/init.go:19 — suite.InitScenario signature changed from InitScenario(sc, w) to InitScenario(sc, template, pool). External consumers (e.g., fullsend-ai/agents) will need to update when bumping the pinned version. The breaking change is properly marked with ! in the PR title and documented with migration instructions in docs/guides/dev/behaviour-testing.md. The pool parameter accepts nil, which simplifies minimal migration.

  • [breaking-api-change] pkg/behaviourtest/steps/registry.go:9 — steps.Register signature changed from Register(ctx, w) to Register(sc), removing the *world.World parameter. Documented in the API changes section. Most external consumers call suite.InitScenario (which calls Register internally) rather than calling Register directly.

Low

  • [nil-handling] pkg/behaviourtest/steps/dispatch.go — All step callbacks call world.FromContext(ctx) and pass the result directly to helper functions without nil-checking. If FromContext returns nil (framework bug where Before hook did not run), helpers would panic on nil pointer dereference. In normal operation this cannot happen because godog skips steps when Before returns an error, but a defensive nil check would make failures more diagnosable.

  • [stale-documentation] docs/guides/dev/behaviour-drivers.md:30 — The narrative description of suite setup mentions running godog with suite.InitScenario but does not mention the new RepoPool creation step that external runners must now perform. The primary API documentation in behaviour-testing.md has been updated, but this secondary reference is incomplete.

Previous run (3)

Review

Findings

Medium

  • [breaking-api-change] pkg/behaviourtest/suite/init.go:19 — suite.InitScenario signature changed from InitScenario(sc, w) to InitScenario(sc, template, pool). External consumers (e.g., fullsend-ai/agents) will need to update when bumping the pinned version. The breaking change is properly marked with ! in the PR title and documented with migration instructions in docs/guides/dev/behaviour-testing.md. The pool parameter accepts nil, which simplifies minimal migration.

  • [breaking-api-change] pkg/behaviourtest/steps/registry.go:9 — steps.Register signature changed from Register(ctx, w) to Register(sc), removing the *world.World parameter. Documented in the API changes section. Most external consumers call suite.InitScenario (which calls Register internally) rather than calling Register directly.

  • [protected-path] .github/workflows/e2e.yml — This file is under .github/ (a protected path). The behaviour test job timeout was increased from 30 to 45 minutes to accommodate per-scenario World cloning overhead. The change is authorized by feat(behaviour): per-scenario World in context + in-process test-repo lease #5438 and justified in the PR description. Human approval is required for changes to protected paths regardless of context.

Low

  • [nil-handling] pkg/behaviourtest/steps/dispatch.go — All step callbacks call world.FromContext(ctx) and pass the result directly to helper functions without nil-checking. If FromContext returns nil (framework bug where Before hook did not run), helpers would panic on nil pointer dereference. In normal operation this cannot happen because godog skips steps when Before returns an error, but a defensive nil check would make failures more diagnosable.

  • [stale-documentation] docs/guides/dev/behaviour-drivers.md:30 — The narrative description of suite setup mentions running godog with suite.InitScenario but does not mention the new RepoPool creation step that external runners must now perform. The primary API documentation in behaviour-testing.md has been updated, but this secondary reference is incomplete.

Previous run (4)

Review

Findings

Medium

  • [breaking-api-change] pkg/behaviourtest/suite/init.go:19 — suite.InitScenario signature changed from InitScenario(sc, w) to InitScenario(sc, template, pool). External consumers (e.g., fullsend-ai/agents) will need to update when bumping the pinned version. The breaking change is properly marked with ! in the PR title and documented with migration instructions in docs/guides/dev/behaviour-testing.md. The pool parameter accepts nil, which simplifies minimal migration.

  • [breaking-api-change] pkg/behaviourtest/steps/registry.go:9 — steps.Register signature changed from Register(ctx, w) to Register(sc), removing the *world.World parameter. Documented in the API changes section. Most external consumers call suite.InitScenario (which calls Register internally) rather than calling Register directly.

  • [protected-path] .github/workflows/e2e.yml — This file is under .github/ (a protected path). The behaviour test job timeout was increased from 30 to 45 minutes. Human approval is required for changes to protected paths regardless of context.

Low

  • [stale-documentation] docs/guides/dev/behaviour-drivers.md:30 — The narrative description of suite setup mentions running godog with suite.InitScenario but does not mention the new RepoPool creation step that external runners must now perform. The primary API documentation in behaviour-testing.md has been updated, but this secondary reference is incomplete.
Previous run (5)

Looks good to me

Previous run (6)

Review

Findings

Medium

  • [breaking-api-change] pkg/behaviourtest/suite/init.go:19 — suite.InitScenario signature changed from InitScenario(sc, w) to InitScenario(sc, template, pool). External consumers (e.g., fullsend-ai/agents) will need to update when bumping the pinned version. The breaking change is properly marked with ! in the PR title and documented with migration instructions in docs/guides/dev/behaviour-testing.md. The pool parameter accepts nil, which simplifies minimal migration.

  • [breaking-api-change] pkg/behaviourtest/steps/registry.go:9 — steps.Register signature changed from Register(ctx, w) to Register(sc), removing the *world.World parameter. Documented in the API changes section. Most external consumers call suite.InitScenario (which calls Register internally) rather than calling Register directly.

Low

  • [edge-case] pkg/behaviourtest/suite/init.go:52 — resetScenarioWorld explicitly zeroes every scenario-level field except LeasedRepoName. Currently harmless because Clone() produces a fresh struct with LeasedRepoName zero-valued, but adding w.LeasedRepoName = "" would maintain consistency with the other scenario-level fields.
Previous run (7)

Review

Findings

High

  • [breaking-change-unmarked] — PR title refactor(#5438): per-scenario World in context + in-process repo lease pool lacks the ! suffix required for breaking changes. This PR changes exported API signatures: suite.InitScenario from (sc, w) to (sc, template, pool), and steps.Register from (ctx, w) to (sc). Per COMMITS.md and AGENTS.md, breaking changes must carry ! in PR titles because GoReleaser uses them for release notes.
    Remediation: Change PR title to refactor(#5438)!: per-scenario World in context + in-process repo lease pool and add a BREAKING CHANGE: trailer to the commit body.

Low

  • [api-change-underdocumented] docs/guides/dev/behaviour-testing.md — The API changes section documents the suite.InitScenario signature change but not the steps.Register signature change. The docs already note that step definitions no longer receive *world.World, which implicitly covers Register. The omission is minor since Register is only called by InitScenario internally.
Previous run

Review

Findings

Medium

  • [error handling / resource leak] pkg/behaviourtest/world/pool.go:41 — Release() performs an unconditional channel send (p.names <- name) with no select/default guard. If called with a name that was never acquired (double-release or spurious release) when the channel buffer is already full, the send blocks forever with no diagnostic output. At Concurrency:1 this is unlikely to trigger, but when parallel scenarios are enabled (the stated goal of this infrastructure plumbing), a cleanup bug could deadlock the entire test suite silently.
    Remediation: Add a non-blocking send with a select/default that panics on double-release — a programming error that should fail fast rather than silently deadlock.

  • [breaking-api-change-undocumented] docs/guides/dev/behaviour-testing.md:141 — The documentation references suite.InitScenario without noting that this PR changes its signature from InitScenario(sc, w) to InitScenario(sc, template, pool). External repos (e.g. fullsend-ai/agents) that import this library will encounter a compile error when bumping the pinned version. The version-pinning section at line 147 tells consumers to bump when APIs change, so the process exists — but a brief note about what changed would ease the migration.
    Remediation: Add a brief note about the signature change near the existing version-pinning guidance.

Low

  • [test integrity] pkg/behaviourtest/world/pool_test.go:117 — TestRepoPool_ConcurrentAcquireRelease silently swallows Acquire errors (if err != nil { return }), then asserts len(acquired) == 10. The context is context.Background() which never cancels, so Acquire should never fail in practice — but if it did, the test would fail with a confusing count mismatch rather than surfacing the root cause.

  • [missing-concept-documentation] docs/guides/dev/behaviour-testing.md — The PR introduces new concepts (RepoPool, per-scenario World cloning, context-based world.FromContext access) central to the behaviour test framework. These are well-documented in godoc comments on the types and functions themselves, but not in the developer guide. Worth adding when the parallel execution work lands.


Labels: All changes are in e2e/behaviour/ and pkg/behaviourtest/ — e2e testing infrastructure

Previous run (8)

Review

Findings

High

  • [breaking-change-unmarked] — PR title refactor(#5438): per-scenario World in context + in-process repo lease pool lacks the ! suffix required for breaking changes. This PR changes exported API signatures: suite.InitScenario from (sc, w) to (sc, template, pool), and steps.Register from (ctx, w) to (sc). Per COMMITS.md and AGENTS.md, breaking changes must carry ! in PR titles because GoReleaser uses them for release notes.
    Remediation: Change PR title to refactor(#5438)!: per-scenario World in context + in-process repo lease pool and add a BREAKING CHANGE: trailer to the commit body.

Low

  • [api-change-underdocumented] docs/guides/dev/behaviour-testing.md — The API changes section documents the suite.InitScenario signature change but not the steps.Register signature change. The docs already note that step definitions no longer receive *world.World, which implicitly covers Register. The omission is minor since Register is only called by InitScenario internally.
Previous run (9)

Review

Findings

Medium

  • [error handling / resource leak] pkg/behaviourtest/world/pool.go:41 — Release() performs an unconditional channel send (p.names <- name) with no select/default guard. If called with a name that was never acquired (double-release or spurious release) when the channel buffer is already full, the send blocks forever with no diagnostic output. At Concurrency:1 this is unlikely to trigger, but when parallel scenarios are enabled (the stated goal of this infrastructure plumbing), a cleanup bug could deadlock the entire test suite silently.
    Remediation: Add a non-blocking send with a select/default that panics on double-release — a programming error that should fail fast rather than silently deadlock.

  • [breaking-api-change-undocumented] docs/guides/dev/behaviour-testing.md:141 — The documentation references suite.InitScenario without noting that this PR changes its signature from InitScenario(sc, w) to InitScenario(sc, template, pool). External repos (e.g. fullsend-ai/agents) that import this library will encounter a compile error when bumping the pinned version. The version-pinning section at line 147 tells consumers to bump when APIs change, so the process exists — but a brief note about what changed would ease the migration.
    Remediation: Add a brief note about the signature change near the existing version-pinning guidance.

Low

  • [test integrity] pkg/behaviourtest/world/pool_test.go:117 — TestRepoPool_ConcurrentAcquireRelease silently swallows Acquire errors (if err != nil { return }), then asserts len(acquired) == 10. The context is context.Background() which never cancels, so Acquire should never fail in practice — but if it did, the test would fail with a confusing count mismatch rather than surfacing the root cause.

  • [missing-concept-documentation] docs/guides/dev/behaviour-testing.md — The PR introduces new concepts (RepoPool, per-scenario World cloning, context-based world.FromContext access) central to the behaviour test framework. These are well-documented in godoc comments on the types and functions themselves, but not in the developer guide. Worth adding when the parallel execution work lands.


Labels: All changes are in e2e/behaviour/ and pkg/behaviourtest/ — e2e testing infrastructure

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added requires-manual-review Review requires human judgment component/e2e End-to-end tests labels Jul 22, 2026
@ifireball ifireball added the ok-to-test Allow e2e CI to run after maintainer review (must be re-applied after each push) label Jul 22, 2026
@ifireball

Copy link
Copy Markdown
Member

/fs-fix

All CI checks (unit, e2e, behaviour) are green. Please address the open review findings:

  1. [medium] pkg/behaviourtest/world/pool.go Release() — Unconditional channel send can deadlock on double-release when the buffer is full. Use a non-blocking select/default that panics on double-release so programming errors fail fast.

  2. [medium] docs/guides/dev/behaviour-testing.md — Document the InitScenario signature change (InitScenario(sc, w) → InitScenario(sc, template, pool)) near the version-pinning guidance so external consumers know what to update when bumping.

Optional (low priority, fix if straightforward):

  • TestRepoPool_ConcurrentAcquireRelease should surface Acquire errors instead of silently returning.
  • Consider documenting RepoPool / per-scenario World / FromContext concepts in the behaviour-testing guide (can wait for the parallel-execution follow-up if that lands soon).

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 10:27 AM UTC · Completed 10:33 AM UTC
Commit: 7ada4e0 · View workflow run →

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 1 (human-triggered)

Addressed 3 of 4 review findings. (1) Hardened Release() with select/default panic on double-release. (2) Documented InitScenario signature change in behaviour-testing.md. (3) Surfaced Acquire errors in concurrent test. Deferred concept documentation per human instruction.

Fixed (3):

  1. Release() unconditional channel send can deadlock on double-release (pkg/behaviourtest/world/pool.go): Replaced unconditional channel send with select/default that panics on double-release when the buffer is full. Added TestRepoPool_DoubleReleasePanics to verify the panic behavior.
  2. Document InitScenario signature change in behaviour-testing.md (docs/guides/dev/behaviour-testing.md): Added 'API changes' subsection under version-pinning guidance documenting the InitScenario(sc, w) to InitScenario(sc, template, pool) signature change, with code example and note about step definitions using world.FromContext(ctx).
  3. TestRepoPool_ConcurrentAcquireRelease silently swallows Acquire errors (pkg/behaviourtest/world/pool_test.go): Added t.Errorf call to surface Acquire errors instead of silently returning from the goroutine.

Disagreed (1):

  1. Document RepoPool / per-scenario World / FromContext concepts in behaviour-testing guide: Human instruction explicitly noted this can wait for the parallel-execution follow-up (ci(e2e): enable GODOG_CONCURRENCY for behaviour suite (default 12) #5441). The InitScenario signature change documentation added in this fix provides the essential migration guidance for external consumers.

Tests: passed

Updated by fullsend fix agent

fullsend-ai-coder Bot added a commit that referenced this pull request Jul 22, 2026
…InitScenario change

- Replace unconditional channel send in Release() with select/default
  that panics on double-release (buffer full), preventing silent deadlock.
- Add TestRepoPool_DoubleReleasePanics to cover the new panic behavior.
- Surface Acquire errors in TestRepoPool_ConcurrentAcquireRelease via
  t.Errorf instead of silently returning.
- Document the InitScenario(sc, template, pool) signature change and
  per-scenario World-via-context pattern in behaviour-testing.md for
  external consumers pinning to this module.

Addresses review feedback on #5444
@ifireball

Copy link
Copy Markdown
Member

/fs-review

Re-review after fix commit 98a0490:

  • RepoPool.Release now panics on double-release (non-blocking select)
  • InitScenario signature change documented
  • Concurrent Acquire errors surfaced in tests

Please confirm medium findings are resolved; remaining deferred item is low-priority concept docs for #5441.

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 10:40 AM UTC · Completed 10:55 AM UTC
Commit: 7ada4e0 · View workflow run →

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot removed the requires-manual-review Review requires human judgment label Jul 22, 2026
@ifireball

Copy link
Copy Markdown
Member

/fs-fix

Re-review on 98a0490 requested changes for one high finding (previous mediums are resolved):

  1. [high] breaking-change-unmarked — Update the PR title to include the ! suffix and ensure the commit history carries a BREAKING CHANGE: trailer per COMMITS.md / AGENTS.md. Suggested title: refactor(#5438)!: per-scenario World in context + in-process repo lease pool. The trailer should note that suite.InitScenario changed from (sc, w) to (sc, template, pool) and steps.Register from (ctx, w) to (sc), and point consumers at the API changes section in docs/guides/dev/behaviour-testing.md.

Low finding about documenting steps.Register in the API changes section is optional — only fix if trivial; otherwise leave it.

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 10:57 AM UTC · Completed 11:03 AM UTC
Commit: 7ada4e0 · View workflow run →

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 2 (human-triggered)

Addressed all review findings. Created commit with BREAKING CHANGE: trailer documenting suite.InitScenario and steps.Register API changes. Added steps.Register documentation to behaviour-testing.md API changes section. PR title update to add ! suffix was blocked by sandbox policy — post-script should set title to: refactor(#5438)!: per-scenario World in context + in-process repo lease pool

Fixed (3):

  1. breaking-change-unmarked: commit history missing BREAKING CHANGE trailer: Created a new commit with the ! suffix in the type and a BREAKING CHANGE: trailer noting that suite.InitScenario changed from (sc, w) to (sc, template, pool) and steps.Register changed from (ctx, w) to (sc), pointing consumers to docs/guides/dev/behaviour-testing.md
  2. breaking-change-unmarked: PR title missing ! suffix: Attempted gh pr edit to set title to 'refactor(feat(behaviour): per-scenario World in context + in-process test-repo lease #5438)!: per-scenario World in context + in-process repo lease pool' but sandbox policy blocks GraphQL. The post-script should update the PR title to include the ! suffix
  3. document steps.Register signature change in API changes section: Added explicit steps.Register signature change documentation to the API changes section of docs/guides/dev/behaviour-testing.md, noting the change from Register(ctx, w) to Register(sc)

Tests: passed

Updated by fullsend fix agent

@ifireball ifireball changed the title refactor(#5438): per-scenario World in context + in-process repo lease pool refactor(#5438)!: per-scenario World in context + in-process repo lease pool Jul 22, 2026
@ifireball

Copy link
Copy Markdown
Member

/fs-review

Re-review after fix commit b71597f + PR title update:

  • Commit now has refactor(#5438)! and a BREAKING CHANGE: trailer for InitScenario/Register
  • PR title updated to refactor(#5438)!: ... (sandbox could not edit title; done manually)
  • steps.Register signature change documented in behaviour-testing.md

Please confirm the high finding is resolved. Only low/info findings should remain if any.

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 11:09 AM UTC · Completed 11:29 AM UTC
Commit: 7ada4e0 · View workflow run →

@fullsend-ai-review
fullsend-ai-review Bot dismissed their stale review July 22, 2026 11:29

Superseded by updated review

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 8 (human-triggered)

Fixed the potential deadlock in RepoPool.Release by unlocking the mutex before the channel send, exactly matching the pattern requested. Added RepoPool mention to behaviour-drivers.md. Left the optional nil-check suggestion as-is since beforeScenario guarantees a non-nil World in context.

Fixed (2):

  1. potential-deadlock in RepoPool.Release (pkg/behaviourtest/world/pool.go): Removed defer p.mu.Unlock(); now explicitly unlock before the error return and before the p.names <- name channel send. The lock is no longer held during the channel operation, eliminating the deadlock risk.
  2. mention RepoPool in behaviour-drivers.md suite-setup narrative (docs/guides/dev/behaviour-drivers.md): Extended the suite-setup paragraph to mention that the suite creates a world.RepoPool and that InitScenario clones a template World per scenario and leases a unique repo name from the pool.

Disagreed (1):

  1. defensive nil check after world.FromContext in dispatch steps: beforeScenario always stores a non-nil World in the context via world.WithWorld, so FromContext cannot return nil during step execution. Adding nil checks to every step closure (8+ call sites across dispatch.go alone) adds boilerplate with no practical safety benefit. The reviewer marked this as optional ('or leave').

Tests: passed

Updated by fullsend fix agent

@ifireball

Copy link
Copy Markdown
Member

/fs-review

Head f1fefe3:

  • Open review threads from prior cycle addressed (introducing commit !, panic-path test, timeout bump reverted to 30)
  • RepoPool.Release no longer holds the mutex during the channel send

Do not re-bump timeouts. Behaviour may sit near the 30m job limit on pull_request_target — that is expected while we avoid padding here.

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

🤖 Review · ❌ Terminated · Started 8:47 PM UTC · Ended 9:01 PM UTC
Commit: 7ada4e0 · View workflow run →

@codecov

codecov Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@fullsend-ai-review

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 8:47 PM UTC · Completed 9:01 PM UTC
Commit: 7ada4e0 · View workflow run →

@ifireball

Copy link
Copy Markdown
Member

/fs-fix

Codecov patch is failing (46.98% vs 80% target): 44 missing lines, all in pkg/behaviourtest/steps/* godog step files that are only exercised by the behaviour suite (not the unit-coverage upload).

.codecov.yml already ignores integration-only paths with this rationale:

# Integration-only; covered by behaviour/e2e CI jobs (not unit coverage upload).
- "pkg/behaviourtest/steps/dispatch.go"
- "pkg/behaviourtest/suite/init.go"

Extend that ignore list to the other steps files whose only patch changes are the FromContext migration (same integration-only class):

  • pkg/behaviourtest/steps/triage.go
  • pkg/behaviourtest/steps/dummy_agent.go
  • pkg/behaviourtest/steps/fork.go
  • pkg/behaviourtest/steps/registry.go
  • pkg/behaviourtest/steps/dispatch_count.go

Prefer a single glob if clean, e.g. pkg/behaviourtest/steps/**, only if that does not hide files that currently have meaningful unit coverage. Otherwise list the five files explicitly next to dispatch.go.

Do not bump behaviour.timeout-minutes (must stay 30).
Do not rewrite history again unless required.

@fullsend-ai-coder

fullsend-ai-coder Bot commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 10:12 PM UTC · Completed 10:18 PM UTC
Commit: 7ada4e0 · View workflow run →

Add five more pkg/behaviourtest/steps/ files to the .codecov.yml
ignore list. These files are exercised by the behaviour suite (not
the unit-coverage upload) and were causing a Codecov patch failure.

A glob was considered but rejected because cleanup.go in the same
directory has meaningful unit tests that should remain visible.

Addresses review feedback on #5444
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 9 (human-triggered)

Extended .codecov.yml ignore list with 5 integration-only step files to fix Codecov patch coverage failure. Listed files explicitly (not a glob) because cleanup.go in the same directory has meaningful unit tests.

Fixed (1):

  1. Codecov patch failing due to uncovered integration-only step files: Added 5 pkg/behaviourtest/steps/ files (dispatch_count.go, dummy_agent.go, fork.go, registry.go, triage.go) to .codecov.yml ignore list alongside the existing dispatch.go entry. These files are integration-only and exercised by the behaviour suite, not the unit-coverage upload. A glob (pkg/behaviourtest/steps/**) was rejected because cleanup.go in the same directory has meaningful unit tests that should remain visible to Codecov.

Tests: passed

Updated by fullsend fix agent

@ifireball

Copy link
Copy Markdown
Member

/fs-review

Head 097d3e8:

  • Prior open threads cleared (introducing !, panic-path test, timeout bump reverted to 30)
  • RepoPool.Release unlocks before channel send
  • Codecov ignore extended for integration-only step files — codecov/patch now passes

Timeout stays at 30. Expect only low/info / intentional breaking-api notes.

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 22, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 10:27 PM UTC · Completed 10:40 PM UTC
Commit: 7ada4e0 · View workflow run →

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review comment for full details.

Comment thread pkg/behaviourtest/suite/init.go Resolved
Comment thread pkg/behaviourtest/steps/registry.go Resolved
Comment thread pkg/behaviourtest/suite/init.go Resolved
Comment thread pkg/behaviourtest/world/pool.go Resolved
Comment thread pkg/behaviourtest/world/world.go Resolved
@ifireball
ifireball added this pull request to the merge queue Jul 23, 2026
Merged via the queue into main with commit c9c7540 Jul 23, 2026
20 checks passed
@ifireball
ifireball deleted the agent/5438-world-per-scenario-lease-pool branch July 23, 2026 03:57
@fullsend-ai-retro

fullsend-ai-retro Bot commented Jul 23, 2026 •

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 4:00 AM UTC · Completed 4:22 AM UTC
Commit: 097d3e8 · View workflow run →

@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #5444 — per-scenario World in context + in-process repo lease pool

Timeline

Time (UTC) Event
Jul 22 07:16 Issue #5438 created by ifireball
07:19–07:22 Triage agent ran — marked ready-to-code
07:30–07:46 Code agent ran (run 29900418518) — created PR #5444
07:48–08:07 Review agent initial run (run 29901476494) — COMMENT verdict: 2 medium, 2 low
10:26 ifireball /fs-fix relaying review agent findings
10:33 Fix iteration 1: Release() deadlock, docs, test improvements
10:38–10:55 Re-review → HIGH: PR title missing !
10:56–11:03 Fix iteration 2: BREAKING CHANGE trailer, steps.Register docs
11:07–11:29 Re-review → medium breaking-api-change
11:39–11:46 Fix iteration 3: LeasedRepoName reset
13:04–13:17 Re-review → "Looks good to me" → APPROVED
13:06 Human review by @waynesun09 (3-agent squad: Claude ×2, Grok) — found 2 HIGH + 5 MEDIUM findings the bot missed
13:26–13:42 Fix iteration 4 (largest): 6 of 7 human findings addressed
15:06–15:12 Fix iteration 5: behaviour timeout bump
16:39–17:08 Fix iteration 6: afterScenario release error logging
17:10–18:12 Fix iteration 7: defer pool.Release for panic safety
18:36–18:52 @waynesun09 4th-pass recheck — all findings resolved → APPROVED
19:02–19:14 Fix iteration 8: timeout revert, panic test, history rewrite
19:51–20:00 Fix iteration 9: mutex unlock before channel send
22:10–22:18 Fix iteration 10: Codecov ignore list
22:26–22:40 Final re-review → COMMENT (low findings only)
Jul 23 03:57 PR merged (squash-merge)

Key metrics

  • 10 fix iterations, 10+ review cycles, ~20 hours wall-clock time
  • $15.76 in review agent costs (3 dispatch runs)
  • 750 additions, 72 deletions across 18 files

Review quality gap

The review agent approved at 13:17 UTC while human reviewer @waynesun09 found 2 HIGH + 5 MEDIUM issues on the same commit. The bot missed:

  • HIGH: Obsolete version markers (v0.22+ when repo is at v0.32.0) — required cross-referencing actual release tags
  • HIGH: Introducing commit missing !/BREAKING CHANGE: trailer per COMMITS.md
  • MEDIUM: Double-release guard only works when buffer is full (probabilistic under real concurrency)
  • MEDIUM: Clone() field-copy has no safety net for future fields
  • MEDIUM: Stale GODOG_CONCURRENCY guard rationale
  • MEDIUM: No test coverage for InitScenario Before/After orchestration
  • MEDIUM: Shared driver concurrency-safety assumed, not documented

The human's multi-model squad (Claude, Grok, Gemini, Codex) provided diversity of perspective that the bot's single-vendor sub-agents did not.

Evidence for existing issues

What went well

  • The fix agent reliably addressed review feedback (9/10 fix iterations produced correct changes)
  • ifireball was an expert orchestrator — precise, scoped /fs-fix instructions with code examples
  • @waynesun09's multi-agent squad review was exceptionally thorough
  • The fix agent successfully performed a non-trivial history rewrite in iteration 8
  • Squash-merge strategy correctly preserved breaking-change markers

Proposals filed

See below for 3 improvement proposals targeting fullsend-ai/agents.

Proposals filed

fullsend-ai-coder Bot added a commit that referenced this pull request Jul 23, 2026
Merge origin/main to pick up BT runtime-shortening changes (#5444,
#5491, #5492, #5494, #5495). The merge completed without conflicts.

Fix TestMintAgentToken_RepoResolutionError and
TestResolveMintRepos_NeitherSet to explicitly clear REPO_FULL_NAME
and MINT_REPOS via t.Setenv — these tests assumed the env vars were
unset but failed when REPO_FULL_NAME was set in the test environment.

Addresses review feedback on #5428

This branch was successfully deployed

1 active deployment
site-preview — 097d3e8e Deployed Jul 22, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/e2e End-to-end tests ok-to-test Allow e2e CI to run after maintainer review (must be re-applied after each push) ready-for-review Triggers review agent dispatch requires-manual-review Review requires human judgment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(behaviour): per-scenario World in context + in-process test-repo lease

2 participants