Skip to content

Integrate validation coordinator with policy, runners, environment, and browser executor - #423

Merged
ronak-guliani merged 13 commits into
feat/validation-coordinator-lifecyclefrom
t3code/integrate-validation-coordinator-flow
Sep 20, 2026
Merged

ronak-guliani merged 13 commits into
feat/validation-coordinator-lifecyclefrom
t3code/integrate-validation-coordinator-flow

Conversation

@ronak-guliani

@ronak-guliani ronak-guliani commented Sep 18, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #420 (feat/validation-coordinator-lifecycle).

Depends on:

  • #418 environment manager (moved to apps/server/src/validation/)
  • #419 policy + repository runner
  • #420 lifecycle base
  • #421 browser executor + evidence

Once those land, retarget/rebase onto main.

Integration

  • Coordinator owns plan via ValidationPolicy + checkpoint changed paths
  • Repository gates via RepositoryValidationRunner (distinct attempts, bounded refs, redaction)
  • Browser gates via BrowserValidationExecutor against acquired env; preflight before token; recording-required blocked until verified decoder
  • Pairing-self-test via pnpm test:self, required only when selected
  • Deterministic order, failure stops dependents, interruption resumable, drift->stale, duplicate idempotent, lease expiry rejects late, lease release on terminal
  • Env/process/port persisted via diagnostics/outputRef, no credentials
  • Timeline shows state, active gate, results/blockers, exact revision, evidence refs

Canonical contracts

  • validation.ts: durable run/gate/result/lease/target (+interrupted result status)
  • browserValidation.ts: rich scenario/evidence
  • Simple ValidationScenario for request, rich BrowserValidationScenario for execution
  • Structured result durable; browser result mapped with media refs
  • Moved validationEnvironmentManager from scripts to server, preserved interface/tests
  • Browser scope: validation-isolated providerSessionId validation:<runId>, preview-only, env/thread-bound; no weakening
  • Recording-required blocked until verified decoder; PNG CRC-checked; file persistence for media

Validation

  • Focused: ValidationPlanner, GateExecutor mapping, CoordinatorIntegration, Policy, Runner, BrowserEvidence/Executor, EnvManager, contracts validation, decider.validation — 88 passed
  • pnpm fmt:check pass, pnpm lint warnings only, pnpm typecheck pass (suggestions only), pnpm test full suite pass (server 2510+102, web 1931, etc.)
  • Tested revision: d025466 (plus stacked adapter commits 0936dd0, 94e4663, 2377f3d, a09135a, ed39954, a46a49b on base ea62420)
  • Real-client pass (isolated, Playwright Chromium, base /tmp/t3code-test.0LsDBe, server 13773, web 5733):
    • Actions: opened web app, paired with fresh one-time token via pairing input, waited for redirect, asserted title contains T3, visible text contains Search, load complete; captured console/network; then tried invalid token for blocked state
    • Observations: ready true (all 3 assertions passed), 1 console error (pre-pairing WebSocket auth failure, expected unauthenticated rejection, no further errors after auth), 0 failed HTTP requests; blocked state shows actionable Invalid pairing token message
    • Evidence: validation-ready.png (authenticated app, no credentials), validation-blocked.png (invalid token error, masked input, no real credentials)
    • Limitations: empty workspace (no threads/projects), so timeline validation matrix not shown; full coordinator e2e (repo gates + browser gates via UI trigger) covered by integration tests, not exercised end-to-end in browser (no UI trigger yet); dev-runner full stack fails pre-existing (vp filter recursion), used manual server+web startup with same isolated base/ports; collaborative browser snapshot unavailable (no connected host), used Playwright for captures

Limitations / follow-up

  • Env reuse adopts current server (no separate launch); release no-ops for current PID, preserves diagnostics
  • Recording-required scenarios blocked (no verified production decoder yet)
  • Validation trigger has no UI button yet; coordinator request via RPC; real-client validates browser assertions + timeline display (matrix code covered, not shown with empty workspace)
  • Second BootstrapCredentialService instance shares DB for one-time tokens; PubSub not shared (ok for validation)
  • Dev-runner pnpm dev full stack fails pre-existing (vp filter t3 matches root, recursion); used manual isolated server+web for real-client; fix dev-runner filters separately

Feature captures

PR head at upload: d025466a0075c3529a9be662a5d84d376261c633. See the testing notes for exercised behavior and limitations; uploading media is not a test result.

validation-ready.png

validation-blocked.png

ronak-guliani and others added 7 commits September 18, 2026 10:57
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ent, and browser executor

Stacked on feat/validation-coordinator-lifecycle. Brings adapter slices from
#418 (environment manager, moved to server), #419 (policy + repository runner),
and #421 (browser executor + evidence) into one coordinator flow.

- Coordinator owns plan via ValidationPolicy + changed paths from checkpoints
- Repository gates via RepositoryValidationRunner with distinct attempts,
  bounded output refs, redaction, invalid-spec->blocked, timeout/signal->interrupted
- Browser gates via BrowserValidationExecutor against acquired validated
  environment; preflight before token, recording-required blocked until verified
  decoder, no success-shaped fallback evidence
- Pairing-self-test via pnpm test:self through repo runner, required only when
  selected by policy
- Deterministic gate order, required failure stops dependents, interruption
  resumable, drift->stale, duplicate attemptId idempotent, lease expiry rejects
  late results, lease release on terminal
- Persist env/process/port identity via diagnostics/outputRef, no credentials
- Timeline shows run state, active gate, results/blockers, exact revision,
  evidence refs

Canonical contracts: validation.ts durable run/gate/result/lease/target;
browserValidation.ts rich scenario/evidence; ValidationScenario simple for
request, BrowserValidationScenario rich for execution; structured result durable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL labels Sep 18, 2026
ronak-guliani and others added 3 commits September 18, 2026 14:31
…grate-validation-coordinator-flow

Take updated lifecycle hardening (terminal taxonomy, legacy-run fence,
lease-expiry release) under the integrated coordinator flow. Resolve
ValidationCoordinatorReactor to keep coordinator-owned policy planning and
gate execution while adopting canonical effective-status checks, the
coordinator-owned-run fence, lease-expiry release, and the releasing
target-error path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ronak-guliani

Copy link
Copy Markdown
Owner Author

Stacked PR updated for latest dependency revisions

Pushed update bringing this stack current without force-push (merge + cherry-picks on top of the previous integration commit).

Newly integrated since the last push:

Conflict resolution (ValidationCoordinatorReactor.ts): kept coordinator-owned policy planning (planCoordinatorRunWithPolicy + changed paths) and real gate execution (placeholder blockUnavailableGates/planValidationCoordinatorRun stay deleted); adopted the new canonical pieces — validationRunEffectiveStatus/isValidationRunTerminal checks, isValidationCoordinatorOwnedRun legacy fence, lease-expiry release, terminal-aware re-plan guard, and the lease-releasing target-error path. Interrupted keeps release + resume-to-planned (it is terminal per the new taxonomy, so it is checked first). Dropped the now-redundant releaseLeaseIfTerminal helper in favor of the unconditional releaseLease at already-terminal call sites.

Validation on final tree (7a36778b5a):

  • Focused: 11 files / 96 tests passed (planner, gate-executor, coordinator integration, policy, repo runner, browser executor/evidence, env manager, contracts, decider + orchestration reactor fence tests)
  • Orchestration dir: 40 files / 503 tests passed
  • pnpm typecheck: 0 errors; pnpm fmt:check: pass; pnpm lint: warnings only, 0 errors
  • pnpm test full suite: exit 0 (server 2526+102, web 1931, all 13 tasks green)

Still stacked on feat/validation-coordinator-lifecycle; retarget/rebase onto main after #418, #419, #420, #421 land.

- Stabilize the environment service process start identity by capturing it
  once per process instead of recomputing process.uptime() on every probe,
  so persisted records revalidate instead of throwing pid-reuse.
- Pass the real attempt number derived from the reactor attempt id to the
  repository runner so retries use distinct artifact keys.
- Scrub the one-time pairing token from every post-pair broker failure via
  a live-token safeMessage helper, including recording cleanup diagnostics.
- Reject unsafe gate-derived media filenames instead of allowing path
  escape from the evidence store.
- Recover a stale persisted environment by terminating the dead owner and
  launching fresh after failed revalidation instead of timing out forever.
- Omit absolute persisted media paths from browser outputRef, emitting only
  kind and hash prefix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ronak-guliani

Copy link
Copy Markdown
Owner Author

Review findings addressed in 4d28dd0

All six findings are fixed on this branch (fast-forward push, no force-push):

  1. P0 — stable process start identity (ValidationEnvironmentService.ts): the start identity is now captured once per process at module scope instead of recomputing process.uptime().toFixed(3) on every probe, so launch records and later inspect calls agree. New ValidationEnvironmentService.test.ts spins a real local readiness server, acquires, advances the clock 25ms, and re-acquires with a fresh manager — asserting same ownership identity and process (reuse, not pid-reuse). Verified the test fails with the fix stashed.
  2. P1 — real attempt number (ValidationGateExecutor.ts): new exported attemptNumberForAttemptId parses the trailing counter from the reactor's attempt:<run>:<gate>:<n> id (falls back to 1 for foreign ids) and passes it to the repository runner, so retries get distinct artifact:<id>-attempt-<n> keys. Unit-tested.
  3. P1 — token redaction everywhere post-pair (BrowserValidationExecutor.ts): new safeMessage closure reads the live pairing token (empty before issuance/after revocation) and replaces all broker-failure message paths, including the recording-stop cleanup diagnostic that used raw diagnosticFromError. Parameterized test covers openAndSnapshot/press/recordingStart/recordingStop echoing the token.
  4. P1 — sanitized media filenames (ValidationGateExecutor.ts): new exported mediaFileNameForGate accepts only [A-Za-z0-9._-] segments (rejecting .., empty, slashes, spaces); violations throw before any write, surfacing as blocked rather than escaping baseDir/validation/<runId>. Unit-tested with ../../evil and friends.
  5. P1 — stale persisted recovery (ValidationEnvironmentManager.ts): the fresh-launch block is extracted to launchFresh; a failed persisted revalidation now records a diagnostic, terminates the stale owner via cleanupOwned(removeState: true), and launches fresh instead of timing out on the same stale file forever. Cleanup-ambiguous still propagates without relaunching over ambiguous ownership. New manager test kills the persisted backend and asserts relaunch with new ownership/pids and corrected state.
  6. P2 — outputRef disclosure (ValidationGateExecutor.ts): browser outputRef is now browser:<kind>:<hash16> only; absolute persistedPath no longer interpolated. Existing mapping test strengthened to assert the exact ref and absence of the path.

Validation on 4d28dd0: focused validation+orchestration+contracts 50 files / 599 tests passed; pnpm typecheck 0 errors; pnpm fmt:check pass; pnpm lint no errors; full pnpm test exit 0 (server 2534 incl. the new service test, web 1931, all tasks green). Still stacked on feat/validation-coordinator-lifecycle.

… keys per run

- Resume now resets every interrupted gate to pending (per-gate dispatches
  with per-cycle command ids) before transitioning the run to planned, so
  the pending-only next-gate selector can schedule the run again instead of
  stalling in running. Resume command ids include run.updatedAt so repeated
  interruption cycles cannot dedupe away.
- Extract the deterministic next-gate selector to selectNextRunnableGate in
  the planner, used by the reactor unchanged.
- Repository artifact keys accept an optional caller-owned scope validated
  against the artifact-key alphabet, producing
  <scope>-<gate>-attempt-<n>; the gate executor scopes every run with a
  sanitized run id so concurrent runs no longer overwrite each other's
  evidence and outputRef identifies the producing run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ronak-guliani

Copy link
Copy Markdown
Owner Author

Follow-up findings addressed in e37734f

Both findings from the latest review are fixed (fast-forward push):

P1 — reset interrupted gates on resume (ValidationCoordinatorReactor.ts): the interrupted branch now dispatches a per-gate reset (interrupted → pending, stale refs cleared, attempts history preserved) for every interrupted gate before the run-level resume, so the pending-only selector can schedule again instead of stalling in running. Reset and resume command ids embed run.updatedAt so repeated interruption cycles cannot dedupe each other away (same pattern as lease claims); resets are skipped only when the run has no executor id, in which case no gate could have been running. The next-gate selector is extracted unchanged to exported selectNextRunnableGate in ValidationPlanner.ts and used by the reactor.

  • Tests: planner unit test proves an interrupted gate blocks selection and a reset restores it; integration test proves the decider accepts and projects the exact interrupted → pending reset command the reactor emits.

P2 — per-run artifact keys (RepositoryValidationRunner.ts, ValidationGateExecutor.ts): specs accept an optional scope validated against the artifact-key alphabet (violations become invalid-spec without executing); keys become <scope>-<gate>-attempt-<n>, unscoped callers keep the legacy key. The executor scopes every run via new artifactScopeForRun (validation:req-1 → validation-req-1, injective over UUID request ids, length-bounded), so concurrent runs no longer share .log files and outputRef (artifact:<scope>-<gate>-attempt-<n>:<sha>) identifies the producing run.

  • Tests: key-derivation units, a two-scope write test asserting both artifacts survive with distinct keys, unsafe-scope rejection, and executor scope/outputRef assertions.

Validation on e37734f: validation+orchestration+contracts 50 files / 604 tests passed; pnpm typecheck 0 errors; pnpm fmt:check pass; pnpm lint no errors in touched files; full pnpm test exit 0. Still stacked on feat/validation-coordinator-lifecycle.

…y preflight errors

- Record a blocked structured result (instead of a bare gate update) when
  a gate kind has no integrated runner, so the run reaches blocked through
  the standard result path instead of stalling in running behind a blocked
  required gate.
- Remove the unused sha computation and createHash import from the browser
  gate execution path.
- Report unrequested preflight targets and environment mismatches with
  actionable diagnostics instead of the generic fallback.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ronak-guliani

Copy link
Copy Markdown
Owner Author

Review findings addressed in 47f0a8c

All three findings fixed (fast-forward push):

P2 — transition run when gate kind has no runner (ValidationCoordinatorReactor.ts): the unknown-kind branch now dispatches thread.validation.result.record with a synthesized blocked result (same attempt id, lease, executor, actionable no integrated runner reason) instead of a bare gate update. Through acceptValidationResult this marks the gate blocked and transitions the run to blocked — terminal, lease-releasing, and actionable — instead of stalling in running behind a gate the selector can never pass.

  • Test: integration test decides + projects the exact result-record command and asserts run and gate reach blocked with the reason preserved.

P3 — dead sha removed (ValidationGateExecutor.ts): deleted the unused sha computation, void sha, and the now-unused createHash import. No behavior change (proven by unchanged mapping tests).

P3 — actionable preflight mismatches (BrowserValidationExecutor.ts): preflightFailureMessage now takes the execution environment and reports unrequested targets (did not request the expected validation target) and environment mismatches (attached to a different environment), ordered to mirror preflightIsUsable.

  • Tests: two executor tests assert the exact diagnostics with no token issued.

Validation on 47f0a8c: focused validation+orchestration+contracts 50 files / 607 tests passed; pnpm typecheck 0 errors; pnpm fmt:check pass; pnpm lint clean in touched files; full pnpm test exit 0. Still stacked on feat/validation-coordinator-lifecycle.

@ronak-guliani
ronak-guliani merged commit 6470402 into feat/validation-coordinator-lifecycle Sep 20, 2026
6 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant