IGNORE — DO NOT MERGE: dependency review evidence pilot by radical · Pull Request #20741 · microsoft/aspire

Ankit Jain (radical) · 2026-10-05T19:38:22Z

Description

IGNORE — DO NOT MERGE. Detection-only dependency-review experiment, not a production rollout or compatibility/safety certification. Original PRs and seeded defects remain untouched; nothing merged.

The published pilot adds concise dependency review guidance, small objective constraints/tests, and CI preparation wiring. It reuses the repository's existing pr-testing skill and selective-CI infrastructure; it does not modify unrelated dependencies, global.json or NuGet.config.

Important scope correction: the user's desired CI-confidence report is a separate skill installed only on their local machine. Its contents were not accessible in this remote session; the attempted folder attachment returned ENOENT. This pilot did not invoke or evaluate that CI-confidence skill, and did not test the combination of that skill with Copilot review. The pr-testing assessments below are a distinct, repo-accessible experiment—not a substitute or proxy. The requested CI-confidence comparison remains pending until its actual skill files are made available; conclusions here are provisional with respect to that request.

Comparison and evidence

Experiment	Observed
Repository `pr-testing` baseline vs candidate skill, same blind 15 synthetic cases/19 targets	Both sessions invoked `pr-testing` successfully. Fresh arm-blinded judge: finding TP/miss 19/0 each, recognition19/19, substantive FP/unsupported/duplicate0/0/0, row states15/15 and Redis/template2/2. This is only an assessment of the repo `pr-testing` skill.
Exploratory instructions/guide text arms and repeats	Full finding counts19/17/17/19; all recognize19/19. Three-case repeats vary; posthoc candidate text improves some aggregate labels but not consistently reviewer findings. Reviewer still mislabels skipped Redis `blocked`. One baseline review unexpectedly invoked `code-review`; guide-only calls did not invoke `pr-testing`. No causal/statistical claim.
Hosted instruction-only #20744	Exact-head Copilot review5420252289 completed. Three dependency comments ground four seeded changes; Kusto's distinct hold was not independently grounded. Two checker-missing comments duplicate one intentionally withheld-helper dependency in this manipulated control, not a production-pilot bug. The review's input-body snapshot is uncertain.
Hosted baseline control	Supported PR creation failed twice; live enumeration still finds no baseline PR/review. No hosted pair or uplift conclusion.
Full-pilot NuGet #20742, head `5f69aaf1599f600bcfbb87fe87293601c45ee102`	Five seeded objective violations remain; checker emits exactly five. Completed review5419936161 retains them. This is not the instruction-only control.
npm #20743, head `b6a02aa69c0bbadf23c3a42154bf557ff17188f0`	Exact two-file fixture and 962 resolved URLs preserved. Frozen install blocked on approved-feed HTTP401; compatibility/minimum-host scenarios remain unvalidated, not incompatible.

Persistent reports and limits

At report commit 7a75af15e7cb7200e29c2ceee18010081ec7648c: scope-corrected final comparison, raw pr-testing outputs/provenance/blind decisions/scores, initial study, candidate follow-up.

The actual pr-testing pair used matched requested default Auto/autopilot settings, but concrete backend/sampling/context is unknown; repository heads, inherited instructions and reference loads differ. This does not isolate skill content as the sole cause. The blind judge knew the frozen oracle, but not arm mapping; decisions were saved before mapping was disclosed. Synthetic fixtures, small samples and manual adjudication do not establish causal or statistical uplift. The provided consumer rows scored correctly but are not proof of exhaustive consumer recall.

The focused suite has 27 passing Python tests; constraint and canonical extension registry checks pass; stored scores reproduce. Earlier Infrastructure.Tests run passed237 tests (quarantine/outerloop excluded; no later C# edits). CI at the later comparison/report head was pending in the last observation, and that earlier run does not certify the latest report commit. Selection, prerequisite build success and relevant consumer execution remain distinct. Feed401, policy API403, queued/skipped jobs, platform/minimum-host/older-VS/deployment paths remain blocked, unexercised or unknown as documented—not inferred compatibility findings.

All experiment PRs remain draft and marked IGNORE / DO NOT MERGE. No original PR posts, reviews, branch edits or fixes; nothing merged. No user approval/push/kick is needed for this provisional report. The CI-confidence assessment cannot continue until the actual skill contents are made available.

Fixes # (issue)

Checklist

Is this feature complete?
- Yes. Ready to ship.
- No. Follow-up changes expected.
Are you including unit tests for the changes and scenario tests if relevant?
- Yes
- No
Did you add public API?
- Yes
  - If yes, did you have an API Review for it?
    - Yes
    - No
  - Did you add <remarks /> and <code /> elements on your triple slash comments?
    - Yes
    - No
- No
Does the change make any security assumptions or guarantees?
- Yes
  - If yes, have you done a threat model and had a security review?
    - Yes
    - No
- No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-10-05T19:46:08Z

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 20741

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 20741"

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…tions Clarify that per-package lockfile coherence remains the frozen restore's responsibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

Preserve blind inputs, prespecified outcomes, local model outputs, explicit adjudications and invocation limits without changing dependency implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Preserve the posthoc four-arm synthetic assessment, raw outputs, manual annotations and score distinctions separately from the original corpus run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Preserve actual skill return outputs, invocation provenance, sealed A/B adjudications and reproducible scores. Distinguish local ties from instruction-only hosted control artifacts; no detection uplift or compatibility claim. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Distinguish the repository pr-testing assessment from the user-local CI-confidence skill and explicitly leave the intended combined evaluation pending its contents. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot review overview

🟡 Changes recommended

The scorer accepts invalid adjudication indices that can silently corrupt reported metrics, and the PR is explicitly marked do not merge.

Review effort: Balanced
Findings: 1

Open (1)

Validate adjudication indices before computing scores · New

+            finding_notes = finding_adjudication(decision)
+            notes[row["id"]] = {
+                "findings": finding_notes,
+                "row_targets": decision.get("row_targets", []),
+                "citation_relevance": [i not in decision.get("irrelevant_citations", []) for i in range(len(citations))],
+                "execution_correct": len(row["execution"]) - len(decision.get("execution_errors", [])),
+                "execution_total": len(row["execution"]),
+                "extra_unsupported": decision.get("extra_unsupported", 0),
+                "selection_correct": decision.get("selection_correct", 0),
+                "selection_total": decision.get("selection_total", 0),
+            }


github-actions · 2026-10-05T21:29:21Z

Tests selector

1 / 98 PR test projects · 0 PR jobs, from 15 changed files.

Selected PR test projects (1 / 98)

Infrastructure.Tests

Selected PR jobs (0)

none

How these were chosen — grouped by what changed

📄 .github/workflows/ci.yml (changed)
→ 1 directly: Infrastructure.Tests

📄 eng/github-ci/test-trigger-map.yml (changed)
→ 1 directly: Infrastructure.Tests

🧪 tests/Infrastructure.Tests/TestTriggerMap/TestTriggerMapTests.cs (changed test)
→ 1 directly: Infrastructure.Tests

Job reasons

none

Selection computed for commit 7a75af1.

Add experimental dependency constraints and consumer evidence guidance

8735945

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Ankit Jain (radical) requested a balanced review from Copilot October 5, 2026 19:39

github-actions Bot added the area-codeflow for labeling automated codeflow. intentionally a different color! label Oct 5, 2026

Exercise constraint CLI failure exit codes

8641981

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Ankit Jain (radical) added the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Oct 5, 2026

Use consumer evidence instead of dependency compatibility verdicts

8aab096

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Ankit Jain (radical) and others added 2 commits October 5, 2026 19:47

Reuse canonical extension feed validation alongside URL boundary checks

e5ad9f0

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Reject source namespace traversal and empty extension lockfile inspec…

f5d9a46

…tions Clarify that per-package lockfile coherence remains the frozen restore's responsibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI reviewed Oct 5, 2026

Ankit Jain (radical) and others added 5 commits October 5, 2026 20:34

Clarify consumer execution states and preserve explicit study annotat…

2bf727a

…ions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Record candidate guidance follow-up study

cece044

Preserve the posthoc four-arm synthetic assessment, raw outputs, manual annotations and score distinctions separately from the original corpus run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Clarify CI-confidence skill was unavailable

7a75af1

Distinguish the repository pr-testing assessment from the user-local CI-confidence skill and explicitly leave the intended combined evaluation pending its contents. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

aspire-repo-bot Bot requested a balanced review from Copilot October 5, 2026 21:14

Copilot started reviewing on behalf of Ankit Jain (radical) October 5, 2026 21:15

View session

Copilot AI reviewed Oct 5, 2026

View reviewed changes

github-actions Bot mentioned this pull request Oct 6, 2026

[repo-pulse] Daily Report #16404

Open

Ankit Jain (radical) closed this Oct 22, 2026

Navigation Menu

Repository navigation

IGNORE — DO NOT MERGE: dependency review evidence pilot - #20741

IGNORE — DO NOT MERGE: dependency review evidence pilot#20741
Ankit Jain (radical) wants to merge 10 commits into
mainfrom
shiny-pancake

Ankit Jain (radical) commented Oct 5, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Oct 5, 2026

Job reasons

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Ankit Jain (radical) commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Comparison and evidence

Persistent reports and limits

Checklist

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

github-actions Bot commented Oct 5, 2026

Tests selector

Selected PR test projects (1 / 98)

Selected PR jobs (0)

Job reasons

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Ankit Jain (radical) commented Oct 5, 2026 •

edited

Loading