[aw] Detection Runs

This issue tracks all runs where threat detection flagged problems in agentic workflows in this repository. Each workflow run that completes with a detection warning or failure posts a comment here.

What is a Detection Problem?

A detection problem occurs when the threat detection system either:

Detects potential security threats (prompt injection, secret leak, malicious patch)
Fails to produce results (agent failure, parse error)

When continue-on-error: true (the default), these problems produce warnings and safe outputs still proceed. When continue-on-error: false, they block safe outputs entirely.

How This Helps

This issue helps you:

Track workflows where threat detection raised concerns
Review patterns of detection warnings or failures
Identify false positives or recurring issues with threat detection
Monitor the health of the threat detection system

Resources

GitHub Agentic Workflows Documentation

Tip

To configure threat detection behavior, update the frontmatter:

safe-outputs:
  threat-detection:
    continue-on-error: true   # Warnings only (default)
    # continue-on-error: false  # Strict mode — block safe outputs

This issue is automatically managed by GitHub Agentic Workflows. Do not close this issue manually.

No action to take - Do not assign to an agent.

expires on Aug 23, 2026, 4:07 PM UTC

Grade Tests on PR (on open / sync)

Conclusion: warning | Reason: parse_error

Generated from Grade Tests on PR (on open / sync)

Pin gh-aw-actions in Dependabot to prevent recurring lock/action drift (

Fix agentic-workflow threat-detection parse_error from gh-aw action d…

Investigated the single entry recorded so far (run 30106781448 — Grade Tests on PR, parse_error).

Root cause

Not a prompt/workflow defect. The detection job died before the model ever ran:

Install GitHub Copilot CLI
  Downloading checksums from https://github.com/github/copilot-cli/releases/download/v1.0.73/SHA256SUMS.txt...
  curl: (22) The requested URL returned error: 504   (x4, ~5s apart)
  ##[error]Process completed with exit code 22.

No CLI -> no LLM -> parse_threat_detection_results.cjs correctly reports ERR_PARSE: No THREAT_DETECTION_RESULT found. github.com release downloads were returning 504 for roughly half a minute; the built-in retry (4 attempts) was exhausted inside that window.

This is the same class of failure as run 26705807411 on the previous tracker (#8667), where the same step failed with exit code 22.

Why it recurs, and why bumps make it worse

gh aw compile pins an exact Copilot CLI version into every .lock.yml (install_copilot_cli.sh 1.0.73), and that pin moves on every gh-aw update — 1.0.40 -> 1.0.48 -> 1.0.52 -> 1.0.60 -> 1.0.63 -> 1.0.65 -> 1.0.68 -> 1.0.73 (#10174).

The hosted-runner toolcache lags well behind. Every job — including today's green runs — logs:

Explicit Copilot CLI version argument provided (1.0.73); skipping compat matrix resolution.
  Found candidate: /opt/hostedtoolcache/copilot-cli/1.0.56/... (version: 1.0.56)
  Skipping candidate (version mismatch: want 1.0.73, got 1.0.56)
  No compatible toolcache entry found
-> Downloading ...

So the exact pin disables the compat-matrix/range fallback that would otherwise reuse the cached 1.0.56, and every agentic job performs two network downloads of the CLI (once in agent, once in detection). Each bump re-widens that window until the runner image catches up — which is exactly why this resurfaces after updates rather than steadily.

Frequency

Across the last ~60 failed runs in this repo, Install GitHub Copilot CLI failed once. Roughly one occurrence per tracker (2 in #8667 over two months, 1 here). Low rate, transient, non-reproducible.

Impact worth noting

safe-outputs.threat-detection.continue-on-error defaults to true, but that only covers a parse failure of a completed detection run. Here the job failed at the install step, so safe_outputs was skipped and the grading comment was never posted — the workflow silently produced nothing rather than degrading gracefully.

Conclusion

No source change in this repo is warranted or even possible:

The failing step is emitted into the generated .lock.yml, which gh aw compile regenerates wholesale — any hand-edit (older pin, extra retry) is overwritten on the next compile.
gh-aw's three-way merge (MergeWorkflowContent) only applies to gh aw update on workflows carrying a source: frontmatter, and it merges the .md source, never the lock file. grade-tests-on-pr.agent.md is locally authored and has no source:, so it is not merged at all.

Any durable fix belongs upstream in gh-aw — e.g. allowing the compat-matrix range resolution to fall back to a cached compatible CLI instead of forcing a download on an exact pin. Leaving this tracker to expire per its gh-aw-expires marker.

Follow-up — dug into whether we can fix this on our side. We can't, and the upstream cause is sharper (and worse) than my first comment suggested.

`engine.version` is documented but silently ignored

The obvious workaround would be pinning the CLI to a version the runner already caches. Tested it — added to grade-tests-on-pr.agent.md:

engine:
  id: copilot
  version: "1.0.56"

gh aw compile --strict reports 0 errors, 0 warnings, and the regenerated lock still contains install_copilot_cli.sh 1.0.73. It is dropped by design in pkg/workflow/copilot_engine_installation.go:

// Copilot CLI is pinned to the default version constant.
copilotVersion := string(constants.DefaultCopilotVersion)
if workflowData.EngineConfig.Version != "" {
    copilotInstallLog.Printf("Ignoring pinned engine.version (%s): ...pinned to %s", ...)
}

The "Ignoring" message is debug-only, so the pin vanishes with no signal. The engines reference advertising version: is a doc bug for this engine.

The real defect: the pin is outside gh-aw's own compat window

gh-aw-actions/.github/aw/compat.json declares, for any gh-aw >= 0.72.0:

{ "min-gh-aw": "0.72.0", "max-gh-aw": "*", "min-agent": "1.0.21", "max-agent": "1.0.56", "open": true }

max-agent is 1.0.56 — and /opt/hostedtoolcache/copilot-cli/1.0.56 is exactly what the hosted runner image ships. But DefaultCopilotVersion is 1.0.73 on v0.83.1 and already 1.0.75 on main, i.e. above its own declared ceiling, and drifting further with each bump.

Because the compiler always passes an explicit version, install_copilot_cli.sh short-circuits the only path that could match the cached binary by range:

if [ -z "$VERSION" ]; then
  # compat resolution -> populates COMPAT_MATCHED_MIN_AGENT / MAX_AGENT
else
  echo "Explicit Copilot CLI version argument provided (${VERSION}); skipping compat matrix resolution."
fi

Hence the range: none..none visible in every run log, including today's green ones:

Searching toolcache ... (requested: 1.0.73, arch: x64, range: none..none)
  Found candidate: /opt/hostedtoolcache/copilot-cli/1.0.56/x64/bin/copilot
  Skipping candidate (version mismatch: want 1.0.73, got 1.0.56)
  No compatible toolcache entry found
-> Downloading ...

Had the range been populated, 1.0.56 would have matched max-agent exactly, skipped the TTL check, and installed with zero network calls. Instead every agentic job performs two CLI downloads (agent + detection), permanently — this does not decay as runner images age, it widens.

Workaround considered and rejected

engine.command: does bypass the generated install step, so pointing at /opt/hostedtoolcache/copilot-cli/1.0.56/x64/bin/copilot would work today. Not worth it: it hardcodes a toolcache path that rotates with runner images (failing hard rather than falling back), and it skips the .copilot ownership fix and stale awf-*-chroot-home cleanup that install_copilot_cli.sh performs — added upstream specifically to prevent EACCES / "engine terminated unexpectedly" failures. Net worse than the problem.

Hand-editing the lock files is a non-starter: gh aw compile regenerates them wholesale.

Conclusion

Unchanged — no repo-side fix, tracker left to expire per its gh-aw-expires marker. An upstream issue is being filed against github/gh-aw asking for either (a) DefaultCopilotVersion kept inside the compat.json window with a CI guard against drift, or (b) the compat range passed to the toolcache lookup even when a version is pinned. Either removes the network hop from the common path and eliminates this failure class outright, rather than just making it rarer.

Navigation Menu

Repository navigation

[aw] Detection Runs #10203

Grade Tests on PR (on open / sync)

Root cause

Why it recurs, and why bumps make it worse

Frequency

Impact worth noting

Conclusion

`engine.version` is documented but silently ignored

The real defect: the pin is outside gh-aw's own compat window

Workaround considered and rejected

Conclusion

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[aw] Detection Runs #10203

Description

Activity

github-actions commented on Jul 24, 2026

Grade Tests on PR (on open / sync)

Evangelink commented on Jul 27, 2026

Root cause

Why it recurs, and why bumps make it worse

Frequency

Impact worth noting

Conclusion

Evangelink commented on Jul 27, 2026

engine.version is documented but silently ignored

The real defect: the pin is outside gh-aw's own compat window

Workaround considered and rejected

Conclusion

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`engine.version` is documented but silently ignored