Skip to content

fix(intents): don't offer lane repair while a lane is resuming - #676

Open
adrianrobinson42 wants to merge 1 commit into
aws-samples:mainfrom
adrianrobinson42:fix/lane-repair-resuming
Open

adrianrobinson42 wants to merge 1 commit into
aws-samples:mainfrom
adrianrobinson42:fix/lane-repair-resuming

Conversation

@adrianrobinson42

Copy link
Copy Markdown
Contributor

Issue #, if available: Fixes #675

Description of changes:

Problem

When a unit-lane question is answered, the intent page shows Parallel execution needs recovery while the lane is resuming. Repair then stops the run, cancels that resume and discards the answer, and the replayed stage asks the question again. Details and logs are in #675.

Answering a gate updates only the gate row. The stage stays WAITING_FOR_HUMAN until its runtime resumes, which after a released session can take a minute. Both deriveLaneWaits and POST /repair read "parked with no pending gate" as orphaned. The gate already records that its answer reached the run: the answer route stamps callbackConsumedAt after SendDurableExecutionCallbackSuccess. Neither check reads it.

Solution

Treat a lane whose answer was delivered as resuming, for a bounded time, and judge it on the server:

  • One rule (isLaneResumeInFlight, lambda/shared/v2-process-keys.js). A lane counts as resuming when all of these hold:

    • the intent is RUNNING or WAITING;
    • the stage is WAITING_FOR_HUMAN;
    • the lane's own answered gate (matched with humanTaskMatchesOwner, and pendingHumanTaskId when the stage records it) has callbackConsumedAt;
    • that is less than LANE_RESUME_GRACE_MS (ten minutes) ago.

    After that the lane is treated as stuck, as before.

  • Intent detail. Each stage carries laneResuming, worked out with that rule on the server's clock, so a browser with a slow clock can't hide Repair indefinitely. The detail makes no new AWS calls.

  • Page. deriveLaneWaits returns a new resuming kind for a lane the server reports as resuming. The lane card shows "resuming after the answer", and the recovery banner isn't shown for it. If another lane is stuck at the same time, the banner says Repair is held until the resuming lane restarts, and the button is disabled, because repairing would also stop the resuming lane.

  • POST /repair. While any lane is resuming, it looks up the durable execution (lookupDurableExecutionStatus).

    • If the run is live, it returns 409 lane_resuming with the lanes concerned, and changes nothing.
    • If the run has ended or isn't found, the lane can't resume, so repair goes ahead straight away.
    • If the lookup fails, it refuses, and the grace window still bounds that.

A gate whose callback was never delivered (resumeAvailable) is unchanged: it still shows as recovery, alongside the existing Resume action.

Testing

  • lambda/intents/test/intents.test.js:

    • /repair returns 409 lane_resuming for a lane answered 30 s ago on a live run, and stops nothing;
    • it doesn't once the grace window has passed;
    • it doesn't when GetDurableExecution reports the run as FAILED;
    • the intent detail reports laneResuming only while the intent is RUNNING.

    The existing orphaned-lane test, an answered gate with no delivered callback, still repairs.

  • frontend/src/lib/intentRecovery.test.ts: a lane the server reports as resuming is resuming, not recovery, and a new pending question still wins.

  • frontend/src/pages/IntentView.test.tsx: no banner or Repair for a lane that is only resuming. With one stuck and one resuming lane, the banner explains the wait and Repair is disabled.

  • On main at 3491e29:

    • npm run lint reports no findings in the changed files, and npm run format:check passes.
    • npm --prefix frontend run build passes.
    • The full frontend suite passes (120 files, 1,032 tests).
    • The intents and shared vitest projects pass, and so does the pre-commit hook's backend run (61 files, 2,533 tests).
    • In one full run, bedrock-iam-shell.test.js failed once; it passes on rerun with and without this change.
  • The change has been running as a patch in our own deployment, where the problem was found.

Impact

  • A lane that has just been answered no longer looks broken, and Repair can't cancel its resume.
  • A lane that is really stuck is offered Repair as before, at the latest ten minutes after its answer, and straight away if the durable execution has ended.
  • One known gap: if the durable execution ends while the intent is still RUNNING, the page can show the lane as resuming until the watchdog marks the intent failed, or for up to ten minutes. /repair accepts a direct call in that window.
  • No Terraform, IAM or data changes. laneResuming is an added field, so the backend and frontend can be deployed in either order. An older frontend ignores the field, and its Repair button gets the 409 instead.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Answering a unit-lane question updates only the gate row: the lane's
stage stays WAITING_FOR_HUMAN until its runtime resumes, which after a
released session can include starting a new instance. The intent page
reloads as soon as the answer is saved, finds the stage parked with no
pending question, and shows "Parallel execution needs recovery". /repair
makes the same check, so pressing Repair stops the run, cancels the
resume in flight and discards the answer, and the replayed stage asks
the question again.

isLaneResumeInFlight (v2-process-keys.js) treats a lane as resuming while
the intent is RUNNING or WAITING, its stage is still parked, and the
lane's own answered gate had its durable callback delivered
(callbackConsumedAt) less than ten minutes ago. The intent detail returns
it as stage.laneResuming, judged on the server's clock, and the page
shows the lane as resuming instead of offering repair. While any lane is
resuming and GetDurableExecution reports the run as live, /repair returns
409 lane_resuming; once the run has ended it can be repaired at once.
@github-actions github-actions Bot added fix Bug fix: PR title starts with fix: or fix(scope): area/frontend React application in frontend/ area/backend Lambda APIs, lambda/shared, v2-orchestrator size/M < 1000 changed lines (lockfiles excluded) labels Nov 3, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/backend Lambda APIs, lambda/shared, v2-orchestrator area/frontend React application in frontend/ fix Bug fix: PR title starts with fix: or fix(scope): size/M < 1000 changed lines (lockfiles excluded)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Lane repair cancels a lane that is resuming after its answer

1 participant