Add task analyzer performance benchmarks

[baronfel]

Fixes #

Context

The task-analyzer package did not have a persistent way to measure analyzer overhead or compare a new rule with the analyzers already shipped in the package.

This is the foundation PR in a two-PR sequence. Follow-up #14388 is based on this branch so its measured changes can use the harness introduced here.

Changes Made

• Add a BenchmarkDotNet project covering analyzer earliest-exit cost, compliant-task zero-diagnostic cost, 1/10/100 diagnostic scaling, and diagnostic suppressor scaling.
• Validate benchmark setup against the exact expected diagnostic count and fail immediately when an analyzer throws.
• Separate stable benchmark classes and infrastructure from analyzer registrations and generated scenario sources so ordinary analyzer additions change only AnalyzerScenarios.cs.
• Mark the benchmark project as non-shipping and include it in MSBuild.slnx for repository builds.
• Add a dedicated task-analyzer authoring skill based on the PRs that introduced the existing analyzers.
• Require new analyzer PRs to compare time, allocations, and scaling with existing analyzers, then explicitly optimize clear outliers or retain the straightforward implementation when results are comparable.
• Route task-author source diagnostics from the engine error-and-warning guidance to the analyzer skill.

Testing

• Built the benchmark project in Release configuration.
• Completed the original full 35-scenario BenchmarkDotNet suite, including no-op, diagnostic, and suppressor benchmarks.
• Completed all five compliant-task scenarios with the BenchmarkDotNet short job.
• Passed all 234 task-analyzer and code-fix tests.

Notes

This PR intentionally adds measurement and authoring guidance without changing analyzer runtime behavior. Follow-up #14388 applies the documented benchmark-driven decision process to MSBuildTask0005.

[github-actions]

🔍 Skill Validator Results

⚠️ Warnings or advisories found

Scope	Checked
Skills	2
Agents	0
Total	2
Severity	Count
---	---:
❌ Errors	0
⚠️ Warnings	1
ℹ️ Advisories	0

Summary

Level	Finding
ℹ️	Found 2 skill(s)
ℹ️	[authoring-task-analyzers] 📊 authoring-task-analyzers: 3,045 BPE tokens [chars/4: 3,790] (standard ~), 11 sections, 3 code blocks
ℹ️	[authoring-task-analyzers] ⚠ Skill is 3,045 BPE tokens (chars/4 estimate: 3,790) — approaching "comprehensive" range where gains diminish.
ℹ️	[authoring-errors-and-warnings] 📊 authoring-errors-and-warnings: 1,522 BPE tokens [chars/4: 1,684] (detailed ✓), 14 sections, 4 code blocks
ℹ️	✅ All checks passed (2 skill(s))

Full validator output

```text Found 2 skill(s) [authoring-task-analyzers] 📊 authoring-task-analyzers: 3,045 BPE tokens [chars/4: 3,790] (standard ~), 11 sections, 3 code blocks [authoring-task-analyzers] ⚠ Skill is 3,045 BPE tokens (chars/4 estimate: 3,790) — approaching "comprehensive" range where gains diminish. [authoring-errors-and-warnings] 📊 authoring-errors-and-warnings: 1,522 BPE tokens [chars/4: 1,684] (detailed ✓), 14 sections, 4 code blocks ✅ All checks passed (2 skill(s)) ```

[Copilot]

Pull request overview

This PR adds an in-repo BenchmarkDotNet harness for src/TaskAnalyzer so analyzer authors can measure baseline overhead, diagnostic scaling (1/10/100), and suppressor scaling in a consistent way, alongside new authoring guidance for task analyzers and clearer routing between engine diagnostics and task-author analyzers.

Changes:

• Introduce src/TaskAnalyzer.Benchmarks with BenchmarkDotNet entrypoint + analyzer/no-op/diagnostic/suppressor benchmarks and scenario validation.
• Add an authoring-task-analyzers skill and update the existing authoring-errors-and-warnings skill to explicitly route task-author source diagnostics to analyzers/suppressors.
• Add the new benchmark project to MSBuild.slnx.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/TaskAnalyzer.Benchmarks/TaskAnalyzer.Benchmarks.csproj	New benchmark project referencing BenchmarkDotNet + Roslyn and the TaskAnalyzer project.
src/TaskAnalyzer.Benchmarks/Program.cs	Benchmark entrypoint with exit code based on summary validation/build/measurement success.
src/TaskAnalyzer.Benchmarks/AnalyzerBenchmarks.cs	Benchmark implementations, scenarios, compilation+runner helpers, and exact diagnostic-count validation.
MSBuild.slnx	Adds the new benchmark project to the solution.
.github/skills/authoring-task-analyzers/SKILL.md	New guidance for implementing task analyzers/code fixes/suppressors and required project surfaces.
.github/skills/authoring-errors-and-warnings/SKILL.md	Updates guidance to distinguish build end-user diagnostics from task-author analyzer diagnostics.

[github-actions]

Review Summary

Good addition — the benchmark harness is well-structured, validations are sound, and the SKILL.md is thorough and actionable. No blocking issues found.

Key findings (all non-blocking suggestions):

1. Silent analyzer exceptions (line 213): onAnalyzerException: null swallows errors. A throwing handler would catch broken analyzers that silently produce 0 diagnostics.

2. No-op measures early-exit only (line 44): CreateWithoutMSBuildReferences() means analyzers bail at type resolution. A "types present, no violations" variant would be more representative of real-world no-op cost.

3. Program.cs exit code (line 14): Returns 1 for --list/--help invocations.

4. csproj: Consider <IsShipping>false</IsShipping> and verifying CI coverage.

5. SKILL.md step 7: Could be clearer about suppressor-specific benchmark steps.

Positive notes:

• Validation-in-GlobalSetup is correct for BenchmarkDotNet's per-parameter-combo lifecycle.
• .GetAwaiter().GetResult() is safe here (no SynchronizationContext).
• Source factories scale cleanly to 1/10/100.
• Internal visibility for non-BDN-consumed members is appropriate.
• csproj is minimal and correct (IsPackable=false, explicit usings).

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by Expert Code Review (on open) for #14387 · 75.6 AIC · ⌖ 5.23 AIC · ⊞ 4.8K

[github-actions]

Minor: If BenchmarkSwitcher.Run returns an empty array (e.g. --list or --help), this returns exit code 1. That may be surprising for informational invocations.

[github-actions]

Documentation: Step 7 covers benchmarks generically, but for suppressors the specific steps (extend AnalyzerSuppressorBenchmark, use includeCompilerDiagnostics: true, set requireSuppressed: true) are only mentioned later in the Performance Harness section. Consider splitting step 7 into sub-steps for analyzers vs suppressors.