Skip to content

Parallel jobs cancellation on Windows gets stuck and times out #4830

Open

Description

@nazar-pc

Describe the bug

On Windows runners, cancelling a job while a parallel: group of run steps is executing frequently leaves the job stuck. The parallel steps and the group step never complete, and the job is force-terminated about 5 minutes later. The job's steps stay in_progress or pending forever in the UI and API, no logs are uploaded (raw logs are unavailable), and post steps (e.g. Post Checkout) never run.

The same workflow cancels within seconds on Linux and macOS runners. On Windows, even when cancellation does complete, it appears to happen only through the 7.5 s grace period: the step processes keep producing output after the cancel.

To Reproduce

Observed in https://github.com/nazar-pc/abundance/actions/runs/41192077246. The workflow is .github/workflows/rust.yml: each Windows job runs 4 parallel shell: bash steps, each running cargo commands from a shared queue.

  1. Start a workflow with a job on windows-2025 or windows-11-arm containing a parallel: group of several long-running run steps (a few minutes of work each).
  2. Cancel the workflow run while the parallel steps are executing.
  3. The job doesn't finish cancelling. About 5 minutes later it's marked cancelled, with its steps still in_progress and no logs even 13 hours later.

This doesn't happen every time; it looks like a race. In the run above, 5 of the 7 Windows jobs that were inside a parallel group when the cancel arrived got stuck. Every Linux and macOS job cancelled normally.

This has been the case for as long as I remember and causes a lot of annoyance when combined with workflow concurrency grouping.

Expected behavior

Parallel steps on Windows are cancelled the same way as on Linux and macOS: processes receive Ctrl+C/Ctrl+Break or are killed, steps complete as cancelled, logs are uploaded, and post steps run, within a few seconds.

Runner Version and Platform

Runner 2.338.2 (GitHub-hosted).

Windows: windows-2025 (image windows-2025-vs2026, version 20261021.278.1) and windows-11-arm. Both are affected.

What's not working?

From the job/step API for run 41192077246 (the run was cancelled at 10:42:15Z):

  • Linux and macOS jobs, and Windows jobs not inside a parallel group at the time, completed cancellation by about 10:42:30Z.
  • These Windows jobs completed at exactly 10:47:15Z, with the parallel run steps and the group step still in_progress and Post Install kache and Post Checkout still pending:
  • Two other Windows jobs that were also inside a parallel group completed cancellation: cargo-test (windows-2025, false, features) and no-panic (windows-11-arm, default). Their logs show the step processes kept running after the cancel. The group finished only about 8 s later, matching the 7.5 s grace period in BackgroundStepCoordinator.CancelWithGracePeriodAsync, with all steps marked Canceled.

Possible cause (by LLM, from reading the source at 110142c, not verified with diagnostic logs)

On Windows, cancelling a step goes through ProcessInvoker.SendCtrlSignal. It calls FreeConsole(), AttachConsole(_proc.Id), SetConsoleCtrlHandler() and GenerateConsoleCtrlEvent(), and FreeConsole() again in finally. A process is attached to at most one console at a time, so these calls change state for the whole Runner.Worker process. Before background steps existed, only one step's process was ever cancelled at a time. With parallel:, all steps of a group are cancelled at once, so several SendCtrlSignal calls run concurrently in one process with no synchronization:

  • one call's FreeConsole() detaches the console another call just attached to;
  • AttachConsole() fails while the process is already attached to another step's console;
  • GenerateConsoleCtrlEvent(signal, 0) signals whichever console the worker happens to be attached to at that moment, which may belong to a different step.

That would explain why Ctrl+C/Ctrl+Break doesn't reach the step processes even in the jobs that do finish. On Linux and macOS, signals are sent with kill() per PID, so there's no shared state.

The cancellation also isn't fully covered by the grace period. In CancelWithGracePeriodAsync, entry.Cts.Cancel() runs the registered callbacks synchronously, before the WaitAsync(TimeSpan.FromSeconds(graceSeconds)) timeout starts. Those callbacks include ProcessInvoker's async cancellation callback. Everything before its first await, including all the console calls in SendCtrlSignal, runs synchronously inside Cancel(). If any of those Win32 console calls blocks, which the concurrent use above could cause, the grace timeout never starts and the group step never completes. That matches what we see.

Possible fixes: serialize SendCtrlSignal across the process (a process-wide lock around the attach/signal/detach sequence), and start the grace timeout independently of Cancel() (e.g. call Cancel() on a separate task, or use CancelAsync()), so a blocked signal can't stall the job.

Job Log Output

None for the stuck jobs; no logs were uploaded. From cargo-test (windows-2025, false, features), one of the jobs that finished cancelling (the cancel was at 10:42:15Z):

2026-10-31T10:42:16.7844908Z    Compiling ab-riscv-primitives v0.3.0 (D:\a\abundance\abundance\crates\execution\ab-riscv-primitives)
2026-10-31T10:42:16.7881492Z    Compiling ab-riscv-interpreter v0.3.0 (D:\a\abundance\abundance\crates\execution\ab-riscv-interpreter)
2026-10-31T10:42:18.7196373Z         SLOW [> 60.000s] (─────) ab-merkle-tree::ui ui
2026-10-31T10:42:23.5343423Z Finished waiting for background step(s).
2026-10-31T10:42:23.5343884Z   Run set -e: Canceled
2026-10-31T10:42:23.5344200Z   Run set -e: Canceled
2026-10-31T10:42:23.5344498Z   Run set -e: Canceled
2026-10-31T10:42:23.5344811Z   Run set -e: Canceled
2026-10-31T10:42:23.7502002Z Post job cleanup.
...
2026-10-31T10:42:26.1602234Z Cleaning up orphan processes

Runner and Worker's Diagnostic Logs

Not available. These are GitHub-hosted runners, and the stuck worker never uploads anything, so diagnostic logs from runner debug logging aren't available for the affected jobs either.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions