Skip to content

Expose citation-safe prose to /issue triage - #39276

Merged
kubaflo merged 1 commit into
mainfrom
kubaflo-triage-citation-evidence
Oct 9, 2026
Merged

kubaflo merged 1 commit into
mainfrom
kubaflo-triage-citation-evidence

Conversation

@kubaflo

@kubaflo kubaflo commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Description of Change

Expose source.prose, deterministically computed by the existing trusted Get-Prose routine, alongside each unmodified source.body before Get-Snapshot computes its context hash. Prepared/current hash and freshness checks consequently include the projection. Update the producer prompt, label-triage skill and directly related documentation to require contiguous verbatim quotations present in both raw and filtered strings with exact ordinal matching. Prefer short complete prose sentences that support the label; never strip markup, rewrite a quotation or cite across blanked spans. Withhold unsupported labels instead of emitting malformed evidence. The projection is a citation aid, not proof of authority or semantic support.

The reported command did trigger run 37851541526 on main 45bab995c8498ee77144404a4361d3cf86a4a054. Detection returned one complete all-clear verdict, but trusted safe-output validation correctly rejected the has-workaround quotation because it included inline code. That quotation is verbatim in the issue, but does not survive prose filtering. The same real source contains the intact sentence: "It recovered 4 + More after the refold in 10/10 runs."

Assert-Evidence is unchanged: it recomputes Get-Prose from freshly fetched original bodies and requires exact ordinal containment in both raw and filtered text, never trusting derived fields supplied by the agent. All authority/provenance/category/whole-paragraph contradiction gates, source/file limits, abort-on-incomplete handling, WTD3/native parsing, exact label intents, artifact-only reporting, publication counters/receipts/postconditions and command completion remain unchanged. No normalization, inline-code authority or partial-proposal salvage is introduced. Context grows by the additional source projection; the existing 1 MiB JSON ceiling is not raised.

Developed independently, then compared with #39250 and #39190: this fixes producer citation selection, not detection formatting or an activation-import mismatch. Those prior fixes remain intact. Pinned gh-aw v0.86.2 compilation changes only body_hash in the lock's first metadata line; executable YAML, manifest, frontmatter/current-import hash, models, pins, imports and permissions are byte-identical.

Validation

  • gh aw compile issue-triage --strict --no-check-update — passed with gh-aw v0.86.2, one workflow, zero warnings. Repeated compilation produced identical lock SHA256 8fa12bacc9bb34c80d70bd71055eae51ad36acb57bd734f8b86c6931ca06f158.
  • gh aw hash-frontmatter .github/workflows/issue-triage.md — passed; current source/import hash 7e9b10340abb100486ac8ee727d99e3669f19d5482b59c3c0b639fdd2ad30bf5 matches metadata.
  • test "$(gh aw hash-frontmatter .github/workflows/issue-triage.md)" = "$(head -n 1 .github/workflows/issue-triage.lock.yml | sed 's/^# gh-aw-metadata: //' | jq -r .frontmatter_hash)" — passed.
  • cmp <(git show 45bab995c8498ee77144404a4361d3cf86a4a054:.github/workflows/issue-triage.lock.yml | tail -n +2) <(tail -n +2 .github/workflows/issue-triage.lock.yml) — executable YAML and manifest unchanged; the equivalent comparison against pre-commit HEAD passed.
  • pwsh -NoProfile -NonInteractive -Command '$tokens = $null; $errors = $null; $null = [Management.Automation.Language.Parser]::ParseFile((Join-Path $PWD ".github/scripts/IssueTriage.ps1"), [ref]$tokens, [ref]$errors); if ($errors.Count) { throw ($errors | Out-String) }; "Whole-script PowerShell AST: PASS"' — passed for the entire script.
  • actionlint .github/workflows/issue-triage.lock.yml — reports only the existing queue: max compatibility diagnostic at its two occurrences with actionlint 1.7.12.
  • actionlint -ignore '^unexpected key "queue" for "concurrency" section\. expected one of "cancel-in-progress", "group"$' .github/workflows/issue-triage.lock.yml — passed; that exact anchored compatibility diagnostic is the sole exception.
  • git diff --check — passed.

Read-only, one-off in-memory PowerShell analysis extracted the actual helper definitions and new snapshot projection from the script AST and exercised genuine archived issue/proposal evidence, without executing workflow stages or modifying archived files. Confirmed the raw agent-output SHA256 d95b90380909eff2fcbcd45f7660457de43e284b9e5dcf5c18edaaa2ed07b237; the original inline-code quotation remains rejected, while the intact sentence passes the citation gate only (not full label validation). Both source bodies/metadata, source count and text length/line/index shape are preserved. Inline code remains blanked; editing an in-memory derived projection changes its hash and cannot bypass the unchanged fresh-body citation check. The archived context would grow from 91,035 to 95,175 bytes, below the unchanged 1,048,576-byte ceiling.

Hosted CI at bfb106adece5d825f96dc26edb079f2b85bb52ae: Skill Validation 37918382469 passed its static checks; LLM evaluation was skipped. Queue validation 37918382485 passed. Pester scripts 37918382456 completed with 2538 passed / 3 failed, exactly matching the full failed test names and totals of unchanged-main run 37549105032, SHA 7d38fd06d059c4875bd81690c360a196aae26a26. All three failing test files and the two referenced model inputs have identical Git blob SHAs at that baseline and this head. These are verified pre-existing GPT-model-pin assertions, not new failures introduced here. Azure Pipelines explicitly filtered maui-pr by trigger conditions; its neutral status is not a build or product-test pass.

No pipeline/Pester/regex/snapshot/fixture tests were added. Product/device tests and the existing full Pester suite were not run locally. No live workflow/issue dispatches, command replays, issue comments/labels/minimization, access changes or new monitors during implementation. The configuration/helper checks and hosted CI results are not a new live end-to-end agent success or a guarantee of model compliance. This fix awaits merge and fresh production verification; the historical command was not replayed.

Issues Fixed

Addresses the /issue triage reliability report on #39263; does not close the underlying Shell product issue or replay its command.

Prepare each source's existing Get-Prose projection before context hashing and guide the producer to quote verbatim raw and filtered prose. Preserve fresh-body validation and all publication guards.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 9, 2026 10:33
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool October 9, 2026 10:33 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 39276

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 39276"

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Skill Validation Results

@kubaflo — new skill validation results are available based on this last commit: bfb106a.
To request a fresh validation after new comments or commits, comment /evaluate-skills.

Overall Passed Static Passed LLM Skipped Skills 28 Agents 6

✅ Skill Validation Results — bfb106a · Expose citation-safe prose to /issue triage · 2026-10-09T10:35:36Z

✅ Static Checks Passed

Skills: 28 | Eval specs linted: 17

Full lint output
── .github/skills/agentic-labeler/tests/eval.vally.yaml
Vally collects pseudonymous usage telemetry (command name, version, outcome, duration, persistent device identifier (when available), and coarse OS/runtime info) to improve the product. No prompts, datasets, file paths, or arguments are collected.
Opt out any time by setting VALLY_TELEMETRY_OPTOUT=1 or DO_NOT_TRACK=1.
Learn more: https://aka.ms/vally/telemetry
✔ .github/skills/agentic-labeler/tests/eval.vally.yaml is valid
── .github/skills/analyze-sessions/tests/eval.vally.yaml
✔ .github/skills/analyze-sessions/tests/eval.vally.yaml is valid
── .github/skills/ci-fix/tests/eval.ownership.vally.yaml
✔ .github/skills/ci-fix/tests/eval.ownership.vally.yaml is valid
── .github/skills/ci-fix/tests/eval.vally.yaml
✔ .github/skills/ci-fix/tests/eval.vally.yaml is valid
── .github/skills/code-review/tests/eval.inline-findings.vally.yaml
✔ .github/skills/code-review/tests/eval.inline-findings.vally.yaml is valid
── .github/skills/code-review/tests/eval.producer-trace.vally.yaml
✔ .github/skills/code-review/tests/eval.producer-trace.vally.yaml is valid
── .github/skills/code-review/tests/eval.trim-aot.vally.yaml
✔ .github/skills/code-review/tests/eval.trim-aot.vally.yaml is valid
── .github/skills/code-review/tests/eval.vally.yaml
✔ .github/skills/code-review/tests/eval.vally.yaml is valid
── .github/skills/code-review/tests/hermeticity.vally.yaml
✔ .github/skills/code-review/tests/hermeticity.vally.yaml is valid
── .github/skills/code-review/tests/soak.capability.vally.yaml
✔ .github/skills/code-review/tests/soak.capability.vally.yaml is valid
── .github/skills/evaluate-pr-tests/tests/eval.vally.yaml
✔ .github/skills/evaluate-pr-tests/tests/eval.vally.yaml is valid
── .github/skills/pr-review/tests/eval.gh-auth.vally.yaml
✔ .github/skills/pr-review/tests/eval.gh-auth.vally.yaml is valid
── .github/skills/review-test-failures/tests/eval.vally.yaml
✔ .github/skills/review-test-failures/tests/eval.vally.yaml is valid
── .github/skills/try-fix/tests/eval.restore.vally.yaml
✔ .github/skills/try-fix/tests/eval.restore.vally.yaml is valid
── .github/skills/try-fix/tests/eval.vally.yaml
✔ .github/skills/try-fix/tests/eval.vally.yaml is valid
── .github/skills/verify-tests-fail-without-fix/tests/eval.protocol.vally.yaml
✔ .github/skills/verify-tests-fail-without-fix/tests/eval.protocol.vally.yaml is valid
── .github/skills/verify-tests-fail-without-fix/tests/eval.vally.yaml
✔ .github/skills/verify-tests-fail-without-fix/tests/eval.vally.yaml is valid

⏭️ LLM Evaluation: Skipped

No changed skills with eval specs found.

🔍 Full results and investigation steps

@kubaflo
kubaflo temporarily deployed to copilot-pat-pool October 9, 2026 10:36 — with GitHub Actions Inactive
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool October 9, 2026 10:38 — with GitHub Actions Inactive
@github-actions github-actions Bot added the area-ai-agents Copilot CLI agents, agent skills, AI-assisted development label Oct 9, 2026
@kubaflo
kubaflo temporarily deployed to copilot-pat-pool October 9, 2026 10:38 — with GitHub Actions Inactive

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The trusted triage workflow behavior lacks live end-to-end verification, although static checks passed and observed Pester failures match the baseline.

0 open findings

What changed in this PR

Exposes citation-safe prose to /issue triage, helping the agent select evidence that trusted validation accepts.

Changes:

  • Adds source.prose before snapshot hashing.
  • Updates citation-selection guidance and documentation.
  • Regenerates workflow metadata.
File Description
.github/​scripts/​IssueTriage.ps1 Adds prose projections to snapshots.
.github/​workflows/​issue-triage.md Guides citation selection.
.github/​workflows/​issue-triage.lock.yml Updates generated body hash.
.github/​skills/​issue-triage-labels/​SKILL.md Documents evidence requirements.
.github/​docs/​issue-triage.md Documents projection and hashing behavior.

🧠 Review effort: Balanced

@kubaflo

kubaflo commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

@dalexsoto stamp

@dalexsoto dalexsoto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stamping as requested by @kubaflo; no code review performed.

@kubaflo
kubaflo merged commit 852bf23 into main Oct 9, 2026
25 of 27 checks passed
@kubaflo
kubaflo deleted the kubaflo-triage-citation-evidence branch October 9, 2026 15:56
@github-actions github-actions Bot added this to the .NET 10 SR12 milestone Oct 9, 2026
This was referenced Nov 1, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 9, 2026

This branch was previously deployed

1 inactive deployment
copilot-pat-pool — bfb106ad Deployed Oct 9, 2026 by kubaflo via conclusion #2296
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-ai-agents Copilot CLI agents, agent skills, AI-assisted development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants