Repository navigation
E2E performance report — 2026-09-14 (gh-aw timing) #15409
Description
Run: https://github.com/githubnext/gh-aw-test/actions/runs/34801861826 · Generated: 2026-09-14T05:05:51Z
Performance history for copilot-create-issue.md
Run, job & step times (main, using inference)
84 successful runs. Regressions shown below are limited to the last six weeks.
| Run or job | Samples | Median | P90 |
|---|---|---|---|
| Workflow complete | 84 | 339.5s | 440.4s |
| Workflow start to proxy step | 84 | 112.0s | 142.7s |
| Proxy step to first reasoning/sample | 84 | 21.5s | 28.2s |
| Copilot phase — AWF startup | 84 | 14.2s | 19.2s |
| Copilot phase — harness startup | 84 | 2.1s | 4.9s |
| Copilot phase — Copilot process | 84 | 7.6s | 10.5s |
Job activation |
84 | 47.5s | 73.4s |
Job agent |
84 | 89.0s | 169.0s |
Job detection |
84 | 73.5s | 96.0s |
Job safe_outputs |
84 | 38.5s | 67.0s |
Job conclusion |
84 | 44.0s | 59.4s |
Major step Execute GitHub Copilot CLI |
84 | 28.0s | 113.0s |
Major step Set up job |
84 | 17.0s | 21.0s |
Major step Install ripgrep |
6 | 14.0s | 19.0s |
Major step Download container images |
84 | 11.0s | 18.0s |
Major step Start MCP Gateway |
84 | 6.0s | 10.7s |
Major step Install GitHub Copilot CLI |
84 | 4.0s | 10.0s |
Major step Setup Scripts |
83 | 3.0s | 5.0s |
Major step Download activation artifact |
40 | 2.0s | 2.0s |
Major step Upload agent artifacts |
27 | 2.0s | 2.0s |
Major step Checkout repository |
17 | 2.0s | 2.0s |
Major step Install AWF binary |
5 | 2.0s | 2.0s |
Major step Stop MCP Gateway |
10 | 2.0s | 2.0s |
Major step Print firewall logs |
1 | 2.0s | 2.0s |
Major step Audit pre-agent workspace |
1 | 2.0s | 2.0s |
Major step Upload agent output fallback artifact |
2 | 2.0s | 2.0s |
Major step times for job activation (main, using inference)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-08 | Set up job | 63.0s | 34.5s | 83% | #341 | v0.86.1-57-gba0a9f9589 / ba0a9f958976 |
| R2 | 2026-08-08 | Setup Scripts | 18.0s | 3.0s | 500% | #341 | v0.86.1-57-gba0a9f9589 / ba0a9f958976 |
| R3 | 2026-08-17 | Set up job | 48.0s | 31.5s | 52% | #352 | v0.87.0-133-g2b2cf3fb01 / 2b2cf3fb01ee |
| R4 | 2026-09-12 | Set up job | 70.0s | 35.5s | 97% | #506 | 76f0291fa1 / 76f0291fa192 |
Major step times for job agent (main, using inference)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-07 | Download container images | 24.0s | 11.0s | 118% | #340 | v0.86.1-3-ge1e298d64b / e1e298d64bfa |
| R2 | 2026-08-07 | Execute GitHub Copilot CLI | 38.0s | 22.5s | 69% | #340 | v0.86.1-3-ge1e298d64b / e1e298d64bfa |
| R3 | 2026-08-14 | Copilot phase — AWF startup | 23.2s | 12.9s | 80% | #349 | v0.86.2-73-gc35faf436c / c35faf436c79 |
| R4 | 2026-08-14 | Execute GitHub Copilot CLI | 38.0s | 23.0s | 65% | #349 | v0.86.2-73-gc35faf436c / c35faf436c79 |
| R5 | 2026-09-02 | Copilot phase — AWF startup | 27.9s | 14.9s | 87% | #470 | bde2c79aa0 / bde2c79aa0d6 |
| R6 | 2026-09-02 | Download container images | 24.0s | 9.5s | 153% | #470 | bde2c79aa0 / bde2c79aa0d6 |
| R7 | 2026-09-02 | Execute GitHub Copilot CLI | 47.0s | 24.0s | 96% | #470 | bde2c79aa0 / bde2c79aa0d6 |
| R8 | 2026-09-05 | Install GitHub Copilot CLI | 20.0s | 9.0s | 122% | #481 | 5473143ca3 / 5473143ca3ae |
Major step times for job detection (main, using inference)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-18 | Execute GitHub Copilot CLI | 50.0s | 31.5s | 59% | #353 | v0.87.1-4-g4845f00caf / 4845f00caf46 |
| R2 | 2026-08-24 | Download container images | 12.0s | 1.0s | 1100% | #437 | 5f0cc8dcc8 / 5f0cc8dcc819 |
Major step times for job safe_outputs (main, using inference)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-01 to 2026-08-03 | Set up job | 89.0s | 34.5s | 158% | #334 | v0.84.3-58-g53baccde53 / 53baccde5390 |
| R2 | 2026-08-03 | Setup Scripts | 41.0s | 4.5s | 811% | #334 | v0.84.3-58-g53baccde53 / 53baccde5390 |
| R3 | 2026-08-14 to 2026-08-16 | Set up job | 69.0s | 29.0s | 138% | #349 | v0.86.2-73-gc35faf436c / c35faf436c79 |
| R4 | 2026-08-25 | Download agent output artifact | 12.0s | 1.0s | 1100% | #445 | 818e3a3863 / 818e3a386376 |
| R5 | 2026-09-07 | Set up job | 70.0s | 34.0s | 106% | #486 | e79e3685c1 / e79e3685c126 |
| R6 | 2026-09-11 | Set up job | 70.0s | 35.5s | 97% | #502 | 1787de5150 / 1787de5150ac |
| R7 | 2026-09-14 | Set up job | 57.0s | 31.5s | 81% | #513 | aa0cabfb25 / aa0cabfb259e |
Major step times for job conclusion (main, using inference)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-03 | Setup Scripts | 14.0s | 4.0s | 250% | #334 | v0.84.3-58-g53baccde53 / 53baccde5390 |
| R2 | 2026-08-17 | Upload usage artifact | 11.0s | 1.0s | 1000% | #352 | v0.87.0-133-g2b2cf3fb01 / 2b2cf3fb01ee |
| R3 | 2026-08-19 | Set up job | 48.0s | 28.0s | 71% | #354 | v0.87.1-44-g5d5e0af5c4 / 5d5e0af5c46c |
Run, job & step times (released, using inference)
37 successful runs. Regressions shown below are limited to the last six weeks.
| Run or job | Samples | Median | P90 |
|---|---|---|---|
| Workflow complete | 37 | 220.0s | 332.6s |
| Workflow start to proxy step | 37 | 63.0s | 90.0s |
| Proxy step to first reasoning/sample | 34 | 21.4s | 28.8s |
| Copilot phase — AWF startup | 34 | 13.5s | 18.5s |
| Copilot phase — harness startup | 34 | 1.9s | 5.0s |
| Copilot phase — Copilot process | 34 | 7.8s | 8.8s |
Job activation |
37 | 17.0s | 43.2s |
Job agent |
37 | 74.0s | 157.4s |
Job detection |
37 | 54.0s | 78.2s |
Job safe_outputs |
37 | 12.0s | 17.6s |
Job conclusion |
37 | 15.0s | 21.4s |
Major step Execute GitHub Copilot CLI |
37 | 30.0s | 115.4s |
Major step Download container images |
37 | 12.0s | 17.4s |
Major step Install ripgrep |
3 | 12.0s | 14.4s |
Major step Start MCP Gateway |
37 | 6.0s | 9.4s |
Major step Install GitHub Copilot CLI |
37 | 4.0s | 9.4s |
Major step Set up job |
35 | 3.0s | 4.0s |
Major step Setup Scripts |
34 | 3.0s | 4.0s |
Major step Stop MCP Gateway |
6 | 2.0s | 2.0s |
Major step Install AWF binary |
3 | 2.0s | 2.0s |
Major step Download activation artifact |
10 | 2.0s | 2.0s |
Major step Upload agent artifacts |
7 | 2.0s | 2.0s |
Major step Checkout repository |
10 | 2.0s | 2.1s |
Major step times for job activation (released, using inference)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-31 | Checkout .github and .agents folders | 17.0s | 2.0s | 750% | #462 | v0.87.10 / ff62cdbec362 |
| R2 | 2026-08-31 | Set up job | 83.0s | 3.0s | 2667% | #462 | v0.87.10 / ff62cdbec362 |
| R3 | 2026-08-31 | Setup Scripts | 16.0s | 3.0s | 433% | #462 | v0.87.10 / ff62cdbec362 |
Major step times for job agent (released, using inference)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-15 | Copilot phase — AWF startup | 23.5s | 13.2s | 79% | #432 | v0.86.3 / 6062cd2238b6 |
| R2 | 2026-08-31 | Copilot phase — AWF startup | 35.6s | 14.7s | 142% | #462 | v0.87.10 / ff62cdbec362 |
| R3 | 2026-08-31 | Copilot phase — harness startup | 13.0s | 2.2s | 495% | #462 | v0.87.10 / ff62cdbec362 |
| R4 | 2026-08-31 | Download container images | 46.0s | 9.5s | 384% | #462 | v0.87.10 / ff62cdbec362 |
| R5 | 2026-08-31 | Execute GitHub Copilot CLI | 62.0s | 25.0s | 148% | #462 | v0.87.10 / ff62cdbec362 |
| R6 | 2026-08-31 | Set up job | 29.0s | 3.0s | 867% | #462 | v0.87.10 / ff62cdbec362 |
| R7 | 2026-08-31 | Start MCP Gateway | 23.0s | 6.5s | 254% | #462 | v0.87.10 / ff62cdbec362 |
Major step times for job detection (released, using inference)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-07 | Install GitHub Copilot CLI | 15.0s | 4.0s | 275% | #430 | v0.86.1 / 475927dfc6d1 |
| R2 | 2026-08-15 | Execute GitHub Copilot CLI | 36.0s | 23.5s | 53% | #432 | v0.86.3 / 6062cd2238b6 |
| R3 | 2026-08-31 | Set up job | 18.0s | 2.0s | 800% | #462 | v0.87.10 / ff62cdbec362 |
Major step times for job safe_outputs (released, using inference)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-07 | Setup Scripts | 20.0s | 2.0s | 900% | #430 | v0.86.1 / 475927dfc6d1 |
| R2 | 2026-08-31 | Set up job | 37.0s | 3.0s | 1133% | #462 | v0.87.10 / ff62cdbec362 |
Major step times for job conclusion (released, using inference)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-31 | Set up job | 27.0s | 2.5s | 980% | #462 | v0.87.10 / ff62cdbec362 |
Run, job & step times (main, using samples)
77 successful runs. Regressions shown below are limited to the last six weeks.
| Run or job | Samples | Median | P90 |
|---|---|---|---|
| Workflow complete | 77 | 199.0s | 259.4s |
| Workflow start to proxy step | 77 | 99.0s | 128.2s |
| Proxy step to first reasoning/sample | 77 | 0.0s | 1.0s |
Job activation |
77 | 46.0s | 67.0s |
Job agent |
77 | 45.0s | 56.4s |
Job detection |
0 | n/a | n/a |
Job safe_outputs |
77 | 32.0s | 48.4s |
Job conclusion |
77 | 37.0s | 63.2s |
Major step Download container images |
77 | 9.0s | 13.0s |
Major step Set up job |
77 | 7.0s | 12.0s |
Major step Start MCP Gateway |
77 | 6.0s | 11.0s |
Major step Install GitHub Copilot CLI |
77 | 4.0s | 9.4s |
Major step Setup Scripts |
76 | 2.5s | 4.0s |
Major step Checkout repository |
13 | 2.0s | 2.0s |
Major step Install AWF binary |
3 | 2.0s | 2.0s |
Major step Download activation artifact |
24 | 2.0s | 2.0s |
Major step Upload agent artifacts |
14 | 2.0s | 2.0s |
Major step Stop MCP Gateway |
8 | 2.0s | 2.0s |
Major step Upload agent output fallback artifact |
1 | 2.0s | 2.0s |
Major step times for job activation (main, using samples)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-09-01 | Set up job | 71.0s | 32.0s | 122% | #467 | 4a88fd99c3 / 4a88fd99c35f |
| R2 | 2026-09-05 | Set up job | 73.0s | 33.0s | 121% | #482 | 5473143ca3 / 5473143ca3ae |
| R3 | 2026-09-10 | Setup Scripts | 15.0s | 4.5s | 233% | #499 | 099efdda60 / 099efdda60fe |
Major step times for job agent (main, using samples)
Candidate regressions (last six weeks)
No candidate regressions in the last six weeks.
Major step times for job detection (main, using samples)
Candidate regressions (last six weeks)
No candidate regressions in the last six weeks.
Major step times for job safe_outputs (main, using samples)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-24 | Set up job | 54.0s | 27.5s | 96% | #438 | 5f0cc8dcc8 / 5f0cc8dcc819 |
| R2 | 2026-09-04 to 2026-09-10 | Set up job | 98.0s | 24.5s | 300% | #491 | 4fbd3efbb3 / 4fbd3efbb3c2 |
| R3 | 2026-09-10 | Setup Scripts | 16.0s | 6.0s | 167% | #499 | 099efdda60 / 099efdda60fe |
Major step times for job conclusion (main, using samples)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-28 | Setup Scripts | 15.0s | 3.0s | 400% | #454 | 76f6ea7c22 / 76f6ea7c2220 |
| R2 | 2026-09-04 | Set up job | 42.0s | 24.5s | 71% | #478 | 76182db3ee / 76182db3eedf |
| R3 | 2026-09-08 to 2026-09-09 | Set up job | 98.0s | 27.0s | 263% | #491 | 4fbd3efbb3 / 4fbd3efbb3c2 |
| R4 | 2026-09-13 | Set up job | 46.0s | 30.5s | 51% | #510 | 2713d4a5a6 / 2713d4a5a6c0 |
Run, job & step times (released, using samples)
129 successful runs. Regressions shown below are limited to the last six weeks.
| Run or job | Samples | Median | P90 |
|---|---|---|---|
| Workflow complete | 129 | 125.0s | 168.2s |
| Workflow start to proxy step | 129 | 65.0s | 97.2s |
| Proxy step to first reasoning/sample | 129 | 0.0s | 1.0s |
Job activation |
129 | 18.0s | 39.4s |
Job agent |
129 | 41.0s | 58.2s |
Job detection |
0 | n/a | n/a |
Job safe_outputs |
129 | 13.0s | 23.0s |
Job conclusion |
129 | 16.0s | 24.4s |
Major step Download container images |
129 | 9.0s | 17.0s |
Major step Install ripgrep |
8 | 9.0s | 18.0s |
Major step Start MCP Gateway |
129 | 7.0s | 12.0s |
Major step Install GitHub Copilot CLI |
129 | 4.0s | 10.4s |
Major step Set up job |
108 | 3.0s | 5.0s |
Major step Setup Scripts |
116 | 2.0s | 4.0s |
Major step Install AWF binary |
10 | 2.0s | 2.1s |
Major step Download activation artifact |
43 | 2.0s | 2.0s |
Major step Upload agent artifacts |
23 | 2.0s | 2.0s |
Major step Stop MCP Gateway |
17 | 2.0s | 2.0s |
Major step Checkout repository |
9 | 2.0s | 2.0s |
Major step Upload agent output fallback artifact |
2 | 2.0s | 2.0s |
Major step times for job activation (released, using samples)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-25 | Setup Scripts | 14.0s | 2.5s | 460% | #452 | v0.87.5 / 654cf351a595 |
| R2 | 2026-08-31 | Set up job | 58.0s | 3.0s | 1833% | #463 | v0.87.10 / ff62cdbec362 |
| R3 | 2026-08-31 | Setup Scripts | 21.0s | 2.0s | 950% | #463 | v0.87.10 / ff62cdbec362 |
| R4 | 2026-09-03 | Set up job | 14.0s | 3.0s | 367% | #475 | v0.88.2 / 8e30bcd8897f |
| R5 | 2026-09-08 | Checkout .github and .agents folders | 18.0s | 2.0s | 800% | #500 | v0.88.7 / bde367913ade |
| R6 | 2026-09-08 | Setup Scripts | 18.0s | 2.0s | 800% | #504 | v0.88.7 / bde367913ade |
| R7 | 2026-09-08 | Upload activation artifact | 17.0s | 2.0s | 750% | #515 | v0.88.7 / bde367913ade |
Major step times for job agent (released, using samples)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-11 | Download container images | 27.0s | 12.0s | 125% | #398 | v0.86.2 / 48e5fa3ff522 |
| R2 | 2026-08-11 | Install GitHub Copilot CLI | 15.0s | 4.0s | 275% | #398 | v0.86.2 / 48e5fa3ff522 |
| R3 | 2026-08-22 | Install GitHub Copilot CLI | 20.0s | 9.0s | 122% | #399 | v0.87.4 / 83d6315352f7 |
| R4 | 2026-08-31 | Download container images | 20.0s | 9.0s | 122% | #465 | v0.87.10 / ff62cdbec362 |
| R5 | 2026-08-31 | Install GitHub Copilot CLI | 28.0s | 9.5s | 195% | #468 | v0.87.10 / ff62cdbec362 |
| R6 | 2026-08-31 | Set up job | 13.0s | 2.0s | 550% | #463 | v0.87.10 / ff62cdbec362 |
| R7 | 2026-09-03 | Download container images | 22.0s | 11.5s | 91% | #492 | v0.88.2 / 8e30bcd8897f |
| R8 | 2026-09-04 | Install GitHub Copilot CLI | 29.0s | 10.0s | 190% | #489 | v0.88.4 / 82239c030d6a |
| R9 | 2026-09-11 | Download container images | 18.0s | 7.0s | 157% | #508 | v0.89.7 / 93c3fc498dbd |
Major step times for job detection (released, using samples)
Candidate regressions (last six weeks)
No candidate regressions in the last six weeks.
Major step times for job safe_outputs (released, using samples)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-31 | Set up job | 14.0s | 2.0s | 600% | #463 | v0.87.10 / ff62cdbec362 |
| R2 | 2026-09-03 | Set up job | 18.0s | 2.5s | 620% | #475 | v0.88.2 / 8e30bcd8897f |
| R3 | 2026-09-07 to 2026-09-08 | Set up job | 18.0s | 6.5s | 177% | #493 | v0.88.6 / b52dd75307b4 |
| R4 | 2026-09-13 | Set up job | 18.0s | 6.0s | 200% | #516 | v0.89.12 / 3c53578b6037 |
Major step times for job conclusion (released, using samples)
Candidate regressions (last six weeks)
| Label | Episode | Step | Peak | Prior median | Increase | Run | gh-aw version / commit |
|---|---|---|---|---|---|---|---|
| R1 | 2026-08-31 | Set up job | 67.0s | 2.0s | 3250% | #463 | v0.87.10 / ff62cdbec362 |
| R2 | 2026-09-03 | Set up job | 34.0s | 2.0s | 1600% | #475 | v0.88.2 / 8e30bcd8897f |
| R3 | 2026-09-08 | Set up job | 28.0s | 6.0s | 367% | #496 | v0.88.7 / bde367913ade |
Method
Each section fixes both independent dimensions: gh-aw source (main or combined stable/pre-release released) and execution mode (inference or samples). Only overall-successful workflow_dispatch runs with a successful agent job are included. Candidate regression baselines use up to ten preceding observations from the same section and step; displayed regression episodes are limited to the six weeks before report generation. A step is graphed when it has a sustained cost, recent slowdown, or recent regression. Runs with missing compiler metadata remain in CSV/JSON but are excluded from graphs.
For inference runs, Execute GitHub Copilot CLI is additionally split using timestamped runtime markers: AWF startup is step start to the AWF agent-container entrypoint, harness startup is that entrypoint to the first Copilot process start, and Copilot process is the first process start through the final process close (including retries and retry delays). Cleanup after process close remains visible only in the full step duration, while unavailable markers produce no phase value.
Activity
dsyme commentedon Sep 14, 2026
Superseded by #15601 — #15601