Skip to content

Optimize daily-syntax-error-quality: add deterministic pre-step, tighten turn budget #28839

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pelikhan merged 2 commits into from
Apr 28, 2026
Merged

Optimize daily-syntax-error-quality: add deterministic pre-step, tighten turn budget #28839

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
14d50b3
Initial plan
Copilot Apr 28, 2026
4cf9cee
optimize daily-syntax-error-quality: add pre-step for workflow discov…
Copilot Apr 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 15 additions & 13 deletions .github/workflows/daily-syntax-error-quality.lock.yml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 10 additions & 4 deletions .github/workflows/daily-syntax-error-quality.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,12 @@ safe-outputs:
timeout-minutes: 20
strict: true
steps:
- name: Find candidate workflows
run: |
mkdir -p /tmp/gh-aw/agent
find .github/workflows -name '*.md' -type f ! -name 'daily-*.md' ! -name '*-test.md' \
> /tmp/gh-aw/agent/candidates.txt
echo "Found $(wc -l < /tmp/gh-aw/agent/candidates.txt) candidate workflows"
Comment on lines +35 to +39

Copilot AI Apr 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The candidate find filters don’t fully match the stated goal of avoiding “test workflows” and will also include template files under .github/workflows/shared/ (since it searches the whole tree and only excludes daily-*.md and *-test.md). This can lead to selecting non-workflow/shared fragments or test-*.md workflows for syntax-error testing. Consider excluding .github/workflows/shared/** and test-*.md (or adjusting the selection criteria text to match the actual candidate set).

Copilot uses AI. Check for mistakes.
- name: Install gh-aw CLI
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Expand Down Expand Up @@ -68,7 +74,7 @@ Test the quality of compiler error messages by:

## Token Budget Guidelines

**Target**: Complete the full analysis in ≤ 40 turns.
**Target**: Complete the full analysis in ≤ 30 turns.

- Test **2 workflows** (not 3) — one simple, one complex.
- One error category per workflow (Category A for workflow 1, Category B for workflow 2).
Expand All @@ -86,11 +92,11 @@ Test the quality of compiler error messages by:

## Phase 1: Select Test Workflows

Select 2 diverse workflows for testing (avoid daily-* and test workflows):
Select 2 diverse workflows for testing from the pre-computed candidate list:

```bash
# Find candidate workflows
find .github/workflows -name '*.md' -type f ! -name 'daily-*.md' ! -name '*-test.md' | head -10
# Candidate workflows have already been discovered by the pre-step
cat /tmp/gh-aw/agent/candidates.txt
```
Comment on lines +95 to 100

Copilot AI Apr 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Phase 1 instructs the agent to cat /tmp/gh-aw/agent/candidates.txt, which is likely a very large list in this repo and will inflate the prompt/output token usage (working against the new ≤30 turn budget and the “avoid printing full file contents” guideline). Consider showing only a small window (e.g., head -n N, plus wc -l for total) and/or providing a deterministic sampling strategy for picking 2 workflows without printing the entire list.

See below for a potential fix:

Select 2 diverse workflows for testing from the pre-computed candidate list without printing the full file:

```bash
# Candidate workflows have already been discovered by the pre-step
wc -l /tmp/gh-aw/agent/candidates.txt
head -n 20 /tmp/gh-aw/agent/candidates.txt

# Select two workflows deterministically without expanding the entire list:
# - first candidate as the "simple" starting point
# - last candidate as the "complex/different" starting point
head -n 1 /tmp/gh-aw/agent/candidates.txt
tail -n 1 /tmp/gh-aw/agent/candidates.txt

Copilot uses AI. Check for mistakes.

**Selection Criteria**:
Expand Down
Loading