Skip to content

Handle network errors when streaming the CodeQL bundle download - #4061

Merged
mbg merged 2 commits into
mainfrom
henrymercer/turbo-system
Jul 31, 2026
Merged

Handle network errors when streaming the CodeQL bundle download#4061
mbg merged 2 commits into
mainfrom
henrymercer/turbo-system

Conversation

@henrymercer

Copy link
Copy Markdown
Contributor

When we stream the download and extraction of the CodeQL bundle, a network error such as ECONNRESET terminates the init Action rather than falling back to downloading the bundle before extracting it. This is because we never attached an error listener to the request returned by https.get, so follow-redirects emits the error as an unhandled error event, which is fatal. See #3367.

Attaching the listener is enough to turn the crash into the fallback path we already have. This PR also switches the pipe from the response into tar over to stream.pipeline, since tar.pipe(stdin).on("error", ...) only listens for errors on tar's standard input: an error on the response itself was unhandled, and would have left tar waiting indefinitely on a pipe that was never going to be closed. Finally, we abort the request if it stalls for five minutes so that a hung connection can't take the whole job with it.

I've deliberately not added a retry loop around the streaming attempt: toolcache.downloadTool, which we fall back to, already retries three times with exponential backoff and already avoids retrying non-transient status codes. The bug here was never that we don't retry, it's that we crashed before we got the chance.

Risk assessment

For internal use only. Please select the risk level of this change:

  • Low risk: This only affects what happens when the streaming download fails, which currently means the job dies. Both new failure paths are covered by unit tests that were confirmed to fail without the fix.

Which use cases does this change impact?

Workflow types:

  • Advanced setup - Impacts users who have custom CodeQL workflows.
  • Managed - Impacts users with dynamic workflows (Default Setup, Code Quality, ...).

Products:

  • Code Scanning - The changes impact analyses when analysis-kinds: code-scanning.
  • Code Quality - The changes impact analyses when analysis-kinds: code-quality.
  • Other first-party - The changes impact other first-party analyses.

Environments:

  • Dotcom - Impacts CodeQL workflows on github.com and/or GitHub Enterprise Cloud with Data Residency.
  • GHES - Impacts CodeQL workflows on GitHub Enterprise Server.

How did/will you validate this change?

  • Unit tests - New tests simulate a mid-connection ECONNRESET and assert that we fall back rather than crashing.
  • End-to-end tests - The existing bundle toolcache PR checks continue to exercise the streaming download on Linux.

If something goes wrong after this change is released, what are the mitigation and rollback strategies?

  • Rollback - Change can only be disabled by rolling back the release or releasing a new version with a fix.

How will you know if something goes wrong after this change is released?

  • Telemetry - Streaming failures now surface as the existing fallback warning and show up in the download duration status field, rather than as a hard job failure.

Are there any special considerations for merging or releasing this change?

  • No special considerations - This change can be merged at any time.

Merge / deployment checklist

  • Confirm this change is backwards compatible with existing workflows.
  • Consider adding a changelog entry for this change.
  • Confirm the readme and docs have been updated if necessary.

sim and others added 2 commits July 30, 2026 18:47
A network error such as `ECONNRESET` while streaming the download and
extraction of the CodeQL bundle terminated the `init` Action rather than
falling back to downloading the bundle before extracting it, since no
`error` listener was attached to the request returned by `https.get`.

Also pipe the response into `tar` using `stream.pipeline` so that errors
on the response itself are surfaced and `tar`'s standard input is closed,
and abort the request if it stalls.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added the size/S Should be easy to review label Jul 30, 2026
@henrymercer
henrymercer marked this pull request as ready for review July 30, 2026 17:50
@henrymercer
henrymercer requested a review from a team as a code owner July 30, 2026 17:50
Copilot AI review requested due to automatic review settings July 30, 2026 17:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

  • Copilot's review of this pull request may be incomplete because some of the changed files are excluded by your Copilot content exclusion settings. See Excluding content from Copilot for details.

Review details

Files excluded by content exclusion policy (1)
  • lib/entry-points.js
  • Files reviewed: 5/6 changed files
  • Comments generated: 0
  • Review effort level: Medium

@mbg mbg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks for improving this!

@mbg
mbg added this pull request to the merge queue Jul 31, 2026
Merged via the queue into main with commit 1cd4d01 Jul 31, 2026
222 checks passed
@mbg
mbg deleted the henrymercer/turbo-system branch July 31, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/S Should be easy to review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants