Skip to content

Fix unattended CI-fixer evidence retrieval - #39237

Merged
kubaflo merged 7 commits into
mainfrom
pureween-fix-ci-evidence-retrieval
Oct 15, 2026
Merged

kubaflo merged 7 commits into
mainfrom
pureween-fix-ci-evidence-retrieval

Conversation

@PureWeen

@PureWeen PureWeen commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Description of Change

This PR fixes unattended CI-fixer evidence, bootstrap, and empty-queue contracts without expanding credentials, permissions, network policy, tool trust, model policy, or action/compiler pins.

  1. Azure failed-task evidence: both workflow twins prepare bounded Azure timeline and full-log evidence before the agent starts. The agent consumes retained manifest/log paths instead of improvising a compound downloader that can require interactive approval. Missing or invalid log IDs, redirects, malformed/capped/timed-out responses, and ambiguous previous-attempt provenance remain explicit incomplete evidence and cannot support absent-signature or fixed conclusions.
  2. Detection ripgrep bootstrap: production run 37671381210 printed ripgrep not found; installing with apt-get... and then produced no further installer diagnostics until GitHub cancelled the detection job at the six-hour maximum. The exact apt cause is unknown. A supported jobs.detection.pre-steps hook now prepares ripgrep before the compiler-owned installer using an emitted 270-second outer watchdog, 120-second bounds around each apt operation, noninteractive sudo, lock/network timeouts, retry limits, and explicit rg --version verification. The unchanged pinned installer then sees rg and exits immediately.
  3. Authoritative empty queue: scheduled run 37711255508 had an authoritative, untruncated zero-issue/zero-candidate snapshot, but the agent assembled a denied compound jq/touch completion command and emitted report_incomplete instead of noop. A trusted helper now revalidates the snapshot, writes zero-byte coverage plus the deterministic summary header, and registers exactly one expected noop. Empty success is bound to explicit producer-emitted product-base provenance (main or net11.0) and requires actual JSON Boolean, Int64, string, and array types; missing/wrong base, coercible strings, fractional/floating counts, null inventories, and malformed values fail before any success artifact or expectation is written. The prompt invokes the helper and existing safeoutputs noop operation separately.
  4. Blocking evidence-review findings: Azure timeline and build-query shapes are structurally validated, including concrete JSON-object validation for every collection row; newest-build evidence cannot fall back to an older complete build; response bytes are charged to the global budget as they are read (including failed/cancelled attempts); and failed-task inventory includes partiallySucceeded Tasks and Tasks carrying error issues.

Both workflow twins use trusted main as their workflow control-plane revision. The net11 twin still passes -BaseBranch net11.0 to the authoritative producer and preserves net11.0 as its fix/capture/apply product base.

Build Insights remains the broader CI platform and HLX remains a useful access/investigation client. The investigation did not establish a supported replacement headless raw timeline/full-log contract or an already-provisioned HLX CLI for this workflow's unchanged security contract. This PR therefore keeps the focused public-Azure producer rather than adding credentials or a runtime migration. That is a scoped investigation result, not a claim that no such service contract can exist.

Reviewed head: 0159c81347b37e21195fccb9e50102c67e1574b8
Rebased onto: 5375f6f87279945656cd5d15ec094f16dd8d9aff (main)

Rebase and follow-up corrections

  • Omitted/null previousAttempts no longer creates a synthetic invalid reference. A fully retrieved previous attempt with zero failed Tasks is accepted as a legitimate empty inventory. Strict retry-ancestry validation remains intact.
  • Initial current-failure selection excludes previous_attempt logs. Historical failures are examined separately for retry/flakiness classification rather than promoted to current FINAL failures.
  • Canceled-build handling and per-definition PR build queries preserve relevant evidence without combined-query starvation. The agent consumes retained timeline evidence rather than attempting an unattended T2 download.
  • The rebase preserves current main's hosted-environment/trusted Node-helper guidance and the authoritative agent-runner candidate file at /tmp/gh-aw/agent/ci-fix-context/candidates.json. It does not use an undeclared pre-activation output. Node transport helpers are invoked with node, not pwsh node.
  • JSON deserialization uses -NoEnumerate and concrete-object guards so singleton array roots cannot masquerade as timeline/build-query/candidate/empty-queue objects. The failed-task aggregation remains an array when there is exactly one current failure before previous-attempt results are appended.
  • Empty-queue completion explicitly checks the native Node registrar exit code. Metadata is written before expectation registration, so a metadata-write failure cannot leave an unmatched expected noop. Registrar failure cannot return successful empty-queue completion.
  • Both Step 0.1 helper examples use Bash fences and Bash continuations, preserving the existing relative helper path and all main/net11 arguments. Regression tests extract each prescribed command, run bash -n, and capture its seven arguments through a harmless stub without touching the real /tmp/gh-aw paths.

Current validation and review boundary

  • Independently executed all four focused Pester suites against the exact final committed content: 120 passed, 0 failed, 0 skipped, 0 inconclusive, 0 not run. Breakdown: evidence 61, empty queue 23, PR query 30, registrar 6. The final independent run took 18.02 seconds with native-command error preference explicitly disabled, exercising the explicit registrar exit-code guard.
  • Regression-first evidence includes malformed JSON-root acceptance and singleton-task aggregation failures before correction, actual native registrar failure, and both original Step 0.1 blocks failing bash -n with exit 2. The corrected commands pass syntax and exact-argument checks.
  • Both locks were regenerated using repository-pinned gh-aw v0.86.2, strict compilation: 2/2 workflows succeeded, 0 warnings. Compiler/action/container/model/security pins were not broadened by these corrections.
  • Final remote/local commit, append parent, required commit trailer, clean worktree, and all ten authoritative GitHub PR file blobs were independently checked. Generated environment-key ordering varies between compilations; this is not a claim of byte-identical compiler output.
  • Three full independent review attempts were made against the rebased source: two completed; one failed before assessment. Later corrections received targeted review and behavioral validation. This is not three successful reviews or a multi-model result.
  • Unsupported retry-ancestry relaxation, extra top-level diagnostic synthesis, and a timeout bump were not applied. A registration-before-file-write change introduced during review was rejected before publication after checking the strict output reconciler.
  • Current Poutine validation is blocked: Docker is inactive (container exit 125). Historical Poutine results below do not validate this head. These focused tests are not a full-repository suite or hosted-production run.
  • At the final discussion snapshot there were 27 threads, 4 unresolved. All four concerns are corrected in source; three threads are marked outdated. No discussions or human reviews were resolved, dismissed, or overridden by this work. Human re-review is still needed.

Original blocking review dispositions

  • 4212363518 - confirmed and fixed: parse success no longer makes timelines complete unless the root is a concrete JSON object with an actual records array containing only concrete JSON-object rows. Missing, null, scalar, object-valued, or scalar-entry records remain explicitly malformed/incomplete.
  • 4212363661 - confirmed and fixed: branch-build queries require a concrete JSON object with a value array of concrete JSON-object rows; an invalid selected build ID makes queryComplete = false instead of silently producing complete evidence.
  • 4212363696 - confirmed and fixed: PR-build queries enforce the same collection shape. Scalar rows cannot be filtered away as nonmatches, and any head-matching row with an invalid build ID makes the query incomplete.
  • 4212363745 - confirmed and fixed: the net11 FRESH path must inspect the newest completed build, and ADVANCE must inspect the newest matching PR build. An incomplete newest build produces report_incomplete; it cannot be skipped for older evidence.
  • 4212363782 - confirmed and fixed: the main workflow has the same newest-build fail-closed requirement.
  • 4212363829 - confirmed and fixed: each read is bounded by the remaining per-request and shared byte budgets, and every returned byte is charged immediately, including bytes consumed before cancellation, exceptions, or retries.
  • 4212363870 - confirmed and fixed: failed-task selection includes failed Tasks, partially-succeeded Tasks, and Task records with error-typed issues. Non-Task records remain excluded.
Historical validation leading to original head 27dbc5e (not current-head proof)

The records below belong to the original review sequence through 27dbc5edd203d7d868225b4c0f252f0878d53525. They are retained for provenance, not presented as fresh validation of the rebased head.

  • Initial seven-finding regression-first result against the prior implementation: 14 passed, 16 failed; corrected targeted producer/workflow suite: 31 passed, 0 failed.
  • Independent follow-up at 44005dff found the PowerShell [pscustomobject] accelerator accepted scalar array entries: its probe was 36/39, with failures for direct scalar rows, scalar timeline records, and scalar PR-query rows.
  • Production regression-first suite with number/string/Boolean rows against the old helper: 35 passed, 9 failed out of 44.
  • Corrected Get-CiFixAzdoEvidence suite at that stage: 44 passed, 0 failed.
  • The independent 39-case contract probe rebound to that corrected helper: 39 passed, 0 failed. Valid empty arrays and object rows were accepted; number, string, Boolean, and null rows failed structurally before downstream filtering.
  • Four focused Pester suites at original head: 99 passed, 0 failed, 0 skipped.
  • Byte-accounting behavioral probe: a local HTTP server returned headers and five body bytes, then stalled. The request failed at the shared deadline and correctly reduced the shared budget from 100 to 95.
  • Bounded live anonymous Azure probe from the preceding original review correction: main build 1628270 had a complete query/timeline/evidence result; cited build 1626245 had complete evidence with two failed logs, including log 179 containing MSB4184 VersionLessThan('', 11.0). Combined reported downloaded bytes: 2,345,580. Cited historical build 1528586 separately returned timeline HTTP 404.
  • Pinned gh-aw v0.86.2 normal compilation then succeeded for 2/2 workflows, 0 warnings.
  • Historical Poutine compilation then succeeded for 2/2 workflows, 0 warnings, with byte-identical locks before/after those compilation modes. This does not establish current Poutine success or current byte identity.
  • Generated manifests and action/container/model/security pins were unchanged in that verification. PowerShell parsing and git diff --check passed.
  • Earlier empty-queue contracts: 38/38 probes, 36/36 invalid snapshots rejected without artifacts, and producer-generated main/net11 fixtures each produced zero-byte coverage and exactly one expected noop.
  • Earlier previous-attempt provenance: 64/64 deduplicated real build 1626285 Task associations correct, with 0 mismatches, versus 54 before correction. The later retained native-timeline adjudication found 76 raw associations across eight timelines with zero errors and passed eight discriminating controls; it did not establish every possible service response shape.

Deployment boundary: this is a focused source and contract pass, not a full production pass. Hosted apt behavior, Copilot CLI unattended helper/noop consumption, and deployment and exercise of this exact revision remain unproved. Local and retained public-Azure evidence do not substitute for that hosted observation.

The separate automatic UI/device queue HTTP 400 blocker remains unresolved and is outside this PR. Diagnostics-only PR #39243 preserves that Azure response body; it does not make #39237 a queue fix or hosted canary. This workflow change does not add dispatch/rerun/queue authority, merge or approval actions, review publication, thread resolution, or Agent merge.

Issues Fixed

N/A - internal unattended CI-fixer workflow reliability repair.

Copilot AI balanced review requested due to automatic review settings October 7, 2026 19:55
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool October 7, 2026 19:55 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 39237

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 39237"

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@PureWeen
PureWeen temporarily deployed to copilot-pat-pool October 7, 2026 19:55 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool October 7, 2026 19:56 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool October 7, 2026 19:58 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool October 7, 2026 20:00 — with GitHub Actions Inactive
@github-actions github-actions Bot added the area-ai-agents Copilot CLI agents, agent skills, AI-assisted development label Oct 7, 2026
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool October 7, 2026 20:00 — with GitHub Actions Inactive

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Retry evidence is incomplete, response-body reads remain unbounded, and manual dispatches can execute the helper from an untrusted branch checkout.

4 open findings
What changed in this PR

Moves Azure DevOps failure-evidence retrieval into a bounded pre-agent producer for both CI-fixer workflows.

Changes:

  • Adds manifest-based timeline and failed-log retrieval.
  • Updates both workflow twins and generated lock files.
  • Adds Pester coverage for retrieval bounds, deduplication, and PR-head matching.
File Description
.github/​workflows/​ci-status-fix.md Integrates prefetched evidence for main.
.github/​workflows/​ci-status-fix.lock.yml Regenerates the compiled workflow.
.github/​workflows/​ci-status-fix-net11.md Integrates prefetched evidence for net11.0.
.github/​workflows/​ci-status-fix-net11.lock.yml Regenerates the compiled workflow.
.github/​scripts/​Get-CiFixAzdoEvidence.ps1 Implements bounded Azure evidence retrieval and manifests.
.github/​scripts/​Get-CiFixAzdoEvidence.Tests.ps1 Tests retrieval and workflow wiring.

🧠 Review effort: Balanced

Comment thread .github/workflows/ci-status-fix.md Resolved
Copilot AI balanced review requested due to automatic review settings October 7, 2026 20:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Prior-attempt logs are not actually prefetched, and producer failures or prolonged retries can prevent the intended incomplete-report path.

5 open findings
Previously missed (2)

In code that hasn't changed since last review

Medium severity Producer failure prevents fail-closed incomplete reporting

.github/​workflows/​ci-status-fix-net11.md:162

If this producer throws before writing the manifest (for example on malformed input or filesystem failure), the step fails the job and the agent never reaches the new “missing/malformed manifest → report_incomplete” path. Make this prefetch step continue after a captured producer failure, while leaving the missing/error manifest for the agent to fail closed on, then regenerate the lock file.

Medium severity Producer failure prevents fail-closed incomplete reporting

.github/​workflows/​ci-status-fix.md:160

If this producer throws before writing the manifest (for example on malformed input or filesystem failure), the step fails the job and the agent never reaches the new “missing/malformed manifest → report_incomplete” path. Make this prefetch step continue after a captured producer failure, while leaving the missing/error manifest for the agent to fail closed on, then regenerate the lock file.

🧠 Review effort: Balanced

@kubaflo

This comment has been minimized.

@kubaflo

kubaflo commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
3 pipeline(s) were filtered out due to trigger conditions.

@github-actions github-actions Bot added the s/agent-review-in-progress AI review is currently running for this PR label Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Note

🔍 /review started

AzDO build 15603136 is running for android.

The s/agent-review-in-progress label stays on this PR while the run is active. The final recommendation and outcome labels are posted only after Gate, expert review, and Deep UI tests finish.

@MauiBot

This comment has been minimized.

@MauiBot MauiBot added s/agent-review-incomplete AI review did not complete all expected phases s/agent-reviewed PR was reviewed by AI agent workflow (full 4-phase review) and removed s/agent-review-in-progress AI review is currently running for this PR labels Oct 7, 2026
Copilot AI balanced review requested due to automatic review settings October 7, 2026 21:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread .github/workflows/ci-status-fix.md Resolved
Copilot AI balanced review requested due to automatic review settings October 7, 2026 21:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Fail-closed guarantees are undermined by stale-build selection, incomplete response validation, and bounds-accounting defects.

7 open findings
1 resolved since last review
Previously missed (2)

In code that hasn't changed since last review

Medium severity Persist empty logs with a valid relative path

.github/​scripts/​Get-CiFixAzdoEvidence.ps1:778

A successful zero-byte log is marked available but gets path = null because no file was written. Step 4 selects every available entry and expects a real path, so this case either tries to read .../null or silently analyzes no log while the build remains complete. Write an empty file for successful empty responses and record its relative path; add a zero-byte-log regression case.

Medium severity Validate source branch alongside source SHA

.github/​scripts/​Get-CiFixAzdoEvidence.ps1:1028

The producer validates only pr.sourceSha; it never validates the returned row’s sourceBranch, despite the workflow and PR description claiming both identities are required. A same-SHA row for another branch would be accepted. Match the established check in Queue-CiFixAzdoValidation.ps1:670-681 by verifying sourceBranch == refs/pull/<P>/merge as well.

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

This was referenced Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-ai-agents Copilot CLI agents, agent skills, AI-assisted development s/agent-changes-requested AI agent recommends changes - found a better alternative or issues s/agent-reviewed PR was reviewed by AI agent workflow (full 4-phase review)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants