Skip to content

ci(e2e): surface rate-limit state in harness-wait timeouts - #8809

Merged
jflowers merged 9 commits into
fullsend-ai:mainfrom
jflowers:opsx/instrument-github-rate-limits
Nov 4, 2026
Merged

jflowers merged 9 commits into
fullsend-ai:mainfrom
jflowers:opsx/instrument-github-rate-limits

Conversation

@jflowers

Copy link
Copy Markdown
Contributor

Summary

Closes the instrumentation loop for #6702: the pool-org installation token
exhausting the GitHub API budget mid-run left behaviour harness waits failing
blind. Every harness-wait timeout error now carries the polling client's
last-seen GitHub rate-limit state, and a mechanical scanner makes #6702's
closure criterion (zero blind-wait markers across recent e2e runs) checkable
on demand.

Related Issue

Refs #6702 — the missing instrumentation delta of the instrument-then-fix
program (#6705, #6698, #6797/#7879, #8578/#8622, #8036). Deliberately not
Closes: #6702 is disposed of by evidence — the post-merge operational
checklist (run the 20-run scan on main, resolve the two closure-rule
questions, then close-or-escalate) is task group 4 and stays out of this PR
by design.

Changes

  • Timeout state line — describeRateLimitState in the GitHub Actions CI
    behaviour driver type-asserts the existing forge.RateLimitReporter and
    appends github rate limit: <state>, observed <age> ago (or
    unknown (no observation yet)) as the final section of every harness-wait
    timeout error — including timeouts with no poll failures, which previously
    carried no quota information at all. Non-implementing clients (GitLab,
    plain fakes) are unchanged; the read sits outside the lookup-budget
    envelope since RateLimit() is in-memory client state.
  • Evidence scanner — skills/e2e-health/scripts/scan-ratelimit-evidence.sh:
    enumerates completed e2e.yml runs on main, walks every attempt via the
    actions attempts API, fetches behaviour-job logs, detects the ci(#6697): make harness-wait timeouts report why the wait failed #6698
    blind-wait markers, resolves conclusions from run/attempt metadata (never
    log parsing), and emits a JSON report plus human table with verdicts at
    both signal levels (any occurrence; retry-masked).
  • Paired hermetic test — scan-ratelimit-evidence_test.sh (70
    assertions, zero network — passes under env -i with gh removed from
    PATH), registered in make script-test.
  • Docs — e2e-health skill documents the scan procedure (when to run,
    invocation, reading the two signal levels) with path-scoped allowed-tools;
    docs/guides/dev/behaviour-testing.md gains a rate-limit observability
    section (lease-boundary/teardown quota logging, timeout state line,
    poll-failure markers as the scanner contract).

Testing

  • make lint passes (stage changes first, then run)

  • Tests added/updated for new or modified logic

  • go test -race -count=1 ./pkg/behaviourtest/drivers/ci/githubactions/ —
    5 new rate-limit tests (exact state-line tail shape plus the four timeout
    scenarios: no-poll-failure, stale-observation age, non-implementing client
    unchanged, unknown-marker); package at 83.6% coverage.

  • bash skills/e2e-health/scripts/scan-ratelimit-evidence_test.sh — 70/70
    assertions, exit 0, hermetic.

  • skills/e2e-health/scripts/scan-ratelimit-evidence.sh --runs 2 and the
    default 20-run window — well-formed JSON + verdict table (current main:
    0 markers at both signal levels).

  • make lint PASS; openspec validate instrument-github-rate-limits valid.

  • Pre-existing main failures (make go-test: 17; make script-test: macOS
    BSD wc -c) reproduced byte-identically in a baseline worktree at the
    base SHA (c0695aa) — not branch-caused.

Checklist

  • PR title follows Conventional Commits (correct type, ! for breaking changes)
  • Commits are signed off (DCO) — human and human-directed agent sessions only
  • I wrote this contribution myself and can explain all changes in it

I wrote this contribution with AI assistance and cannot explain all changes in it.

Known Issues

Review councils approved with non-blocking advisories (spec review: 20 —
9 MEDIUM / 11 LOW; code review: 8 LOW). Themes: --runs lacks validation
bounds; hardcoded marker strings (drift mitigated by the paired test);
OpenSpec artifact sections deviating from canonical templates; specs
codifying already-landed behavior as ADDED requirements; broad
Bash(jq:*)-style allowed-tools grants in SKILL.md; awk marker detection
not looping for multiple same-line occurrences; testify usage vs the
convention pack's stdlib-only rule (matches the pre-existing package
pattern).

Reported enforcement gaps (from the requirements mapping, deliberately not
patched here):

  1. No test asserts the allocation log line carries the org's observed quota
    (quotaSummary's observed branch untested).
  2. No test asserts the release log line at all (slotpool.go/composed.go).
  3. Composed driver's Finalize per-org listing content/reset/resource fields
    not asserted within pkg/behaviourtest.
  4. No test captures the "allocation before any observation" emitted log line.
  5. GitLab CI driver carries the same pollErrors.describe shape with no
    gitlabci test asserting any poll-failure timeout tail — flagged for
    review to pick a reading (forge-neutral vs GitHub-scoped requirement).

This PR was generated by /uf.finale (AI-assisted).

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Surface rate-limit state in harness timeouts and scan e2e evidence

🐞 Bug fix ✨ Enhancement 🧪 Tests 📝 Documentation 🕐 40+ Minutes

Grey Divider

AI Description

• Append last-observed GitHub rate-limit state to harness-wait timeouts, including waits without
 poll failures.
• Scan recent e2e attempts for blind-wait markers and report strict and retry-masked counts.
• Add hermetic tests and document how to collect and interpret the evidence.
Diagram

graph TD
  Client["GitHub client"] --> Timeout["Timeout diagnostics"] --> Errors["Harness errors"]
  API["GitHub Actions API"] --> Attempts["Run attempts"] --> Logs["Behaviour logs"] --> Report["Evidence report"]
  Errors --> Logs
  Attempts --> Report
Loading
High-Level Assessment

Keep the optional RateLimitReporter assertion and cached-state read: requiring every forge client to report quota or fetching quota at timeout would add compatibility costs or API traffic to a diagnostic path. Scanning every attempt, rather than only a run's final logs, is necessary to expose retry-masked occurrences.

Files changed (7) +1476 / -10

Enhancement (1) +411 / -0
scan-ratelimit-evidence.shScan completed e2e attempts for blind-wait markers +411/-0

Scan completed e2e attempts for blind-wait markers

• Fetches each behaviour-job log across attempts of recent completed main-branch runs and detects the established timeout markers. Produces per-attempt JSON evidence, metadata-based conclusions, two verdict counts, and a human table; unfetchable logs mark the report incomplete.

skills/e2e-health/scripts/scan-ratelimit-evidence.sh

Bug fix (1) +49 / -6
githubactions.goAppend cached rate-limit state to harness timeout diagnostics +49/-6

Append cached rate-limit state to harness timeout diagnostics

• Adds a final rate-limit section to timeout diagnostics when the polling client implements RateLimitReporter, including an explicit unknown before its first observation. Non-reporting clients retain their existing errors, and the in-memory read consumes no diagnostics lookup budget.

pkg/behaviourtest/drivers/ci/githubactions/githubactions.go

Documentation (2) +196 / -4
behaviour-testing.mdDocument rate-limit diagnostics and evidence checks +106/-1

Document rate-limit diagnostics and evidence checks

• Explains quota observations at lease boundaries, teardown, and harness-wait timeouts. Documents the scanner's marker contract, invocation, incomplete reports, and two verdict levels.

docs/guides/dev/behaviour-testing.md

SKILL.mdAdd the rate-limit evidence scan procedure +90/-3

Add the rate-limit evidence scan procedure

• Adds path-scoped tool permissions and instructions for running the scanner when investigating e2e rate limits. Explains its window, output channels, incomplete-report warning, and strict versus retry-masked counts.

skills/e2e-health/SKILL.md

Other (3) +820 / -0
MakefileRegister the evidence scanner test +1/-0

Register the evidence scanner test

• Adds the scanner's hermetic shell test to the script-test target so it runs with the repository's existing script checks.

Makefile

githubactions_test.goCover rate-limit state across timeout cases +228/-0

Cover rate-limit state across timeout cases

• Adds a reporting fake and tests the diagnostic tail and full harness-wait timeout path. Cases cover observed and stale state, missing observation timestamps, no poll failures, explicit unknown state, and unchanged errors for non-reporting clients.

pkg/behaviourtest/drivers/ci/githubactions/githubactions_test.go

scan-ratelimit-evidence_test.shTest scanner analysis and windowing without network access +591/-0

Test scanner analysis and windowing without network access

• Exercises marker matches and near-misses, evidence assembly, retry masking, verdicts, and report rendering with fixture logs. A stubbed gh also checks end-to-end window bounds and stdout/stderr separation.

skills/e2e-health/scripts/scan-ratelimit-evidence_test.sh

@fullsend-ai-review

fullsend-ai-review Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 6:07 PM UTC · Completed 6:19 PM UTC

Commit: 2930d96 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $3.63

@qodo-code-review

qodo-code-review Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (1) 📎 Requirement gaps (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (1)

Grey Divider


Remediation recommended

1. Two harness waits omit quota on timeout ✓ Resolved

Resolving commit: 6138777

Description
harnessTimeoutDiagnostics appends the new rate-limit line, but WaitForHarnessEmptyMatrix and
CountHarnessDispatches construct timeout errors without calling it or describeRateLimitState.
When either wait fails to settle, its error still lacks the polling client's last-seen quota,
including when the client has observed a rate limit.

2. Broken job metadata can hide recent runs ✓ Resolved

Resolving commit: 6138777

Description
The jobs page check accepts a successful JSON response with no jobs field because both sides of
its fallback comparison evaluate to zero. If an attempt returns an unexpected response such as
{"message":"temporary upstream failure"}, the subsequent job extraction fails inside a
conditionally invoked scan_run, and the run can be excluded as having no behaviour job while an
older run fills the window.

3. The scan bypasses GitHub wrappers 📘 Rule violation ⌂ Architecture
Description
scan-ratelimit-evidence.sh invokes gh api directly to list attempt jobs and fetch behaviour logs
instead of using the repository's GitHub wrapper. Every scan execution therefore performs GitHub
operations from a package outside internal/forge/github, bypassing the approved abstraction
boundary.
Code

skills/e2e-health/scripts/scan-ratelimit-evidence.sh[R242-243]

+    if ! gh api "repos/${repo}/actions/runs/${run_id}/attempts/${n}/jobs?per_page=100" \
+        > "${dir}/jobs-${nn}.json" 2>"${ERRFILE}"; then
Relevance

●● Moderate

Architecture concern is subjective; no close precedent rejects direct gh usage in this shell-skill
context.

PR-#2301

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The changed scanner directly executes gh api for per-attempt jobs and behaviour-job logs. The
compliance checklist permits gh execution only under internal/forge/github/, while this script
is under skills/e2e-health/scripts/.

Rule 1062053: Restrict gh CLI exec.Command usage to internal/forge/github
skills/e2e-health/scripts/scan-ratelimit-evidence.sh[242-243]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The evidence scanner invokes `gh api` directly outside `internal/forge/github`, violating the repository boundary for GitHub CLI usage.

## Fix Focus Areas
- skills/e2e-health/scripts/scan-ratelimit-evidence.sh[242-243]
- skills/e2e-health/scripts/scan-ratelimit-evidence.sh[273-274]

## Recommended Fix
Move the GitHub-specific CLI/API operations into an approved wrapper under `internal/forge/github`, expose the required operations through that wrapper or an injected interface, and update the scanner to call the abstraction instead of invoking `gh` directly. Preserve the scanner's JSON output, attempt traversal, log-fetch error handling, and read-only behavior.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗



Informational

4. Developers lack scan prerequisites 📜 Skill insight ✧ Quality
Description
The new evidence-scan procedure gives invocation commands without adding a clearly labeled
Prerequisites section to the guide. Readers can reach the commands without the required checkout,
authenticated GitHub CLI, and scanner-tool setup being collected before the procedure begins.
Code

docs/guides/dev/behaviour-testing.md[R612-620]

+#### Running the evidence check
+
+The evidence check is `skills/e2e-health/scripts/scan-ratelimit-evidence.sh`.
+It is read-only (`gh` GETs, on your own token rather than the installation
+token it measures) and runs from any checkout with ordinary `gh` auth:
+
+```bash
+bash skills/e2e-health/scripts/scan-ratelimit-evidence.sh          # default: 20 runs
+bash skills/e2e-health/scripts/scan-ratelimit-evidence.sh --runs 2 # window size, 1-100
Relevance

● Weak

Recent precedent rejected adding a prerequisites section before command documentation.

PR-#8699

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The changed guide adds a procedure titled Running the evidence check and immediately presents
commands, but the modified guide contains no prerequisites subsection for this procedure. The
checklist requires every guide procedure to state prerequisites before step 1.

docs/guides/dev/behaviour-testing.md[612-620]
Skill: writing-user-docs

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new evidence-scan procedure lacks a clearly labeled prerequisites section before its invocation commands.

## Fix Focus Areas
- docs/guides/dev/behaviour-testing.md[612-620]

## Recommended Fix
Add a `## Prerequisites` or procedure-scoped `### Prerequisites` subsection before `#### Running the evidence check`. State that the reader must run from a repository checkout, have authenticated `gh` access to the target repository, and have the scanner's required tools available before showing the invocation commands.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


Grey Divider

Context sources
✅ Compliance rules (platform): 95 rules
✅ Skills: 5 invoked
  writing-user-docs
  writing-adrs
  check-patch-coverage
  cutting-releases
  adr-corner
✅ Cross-repo context — repo relationships
  Explored: repo: fullsend-ai/agents (sha: 15e2d79a) — View relationship
Review mode: Auto: ⚖️ Balanced: Downgraded extended -> standard: change is below the extended eligibility bar (hunks 11/18, lines 1486/200; both must reach the floor). Router rationale: Substantial Go and shell logic spans multiple independent API and parsing paths.

Grey Divider

Tip of the day
💡 Did you know, you can commit Qodo's fix in one click with committable suggestions (GitHub & GitLab)

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Site preview

Preview: https://5af1ba32-site.fullsend-ai.workers.dev

Commit: 885dc309cb024faa59d7c58dab6bf7f367310482

@fullsend-ai-review fullsend-ai-review Bot added the risk/elevated PR risk: elevated label Oct 25, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff across 8 files touching protected skills paths and CI tooling with high git history churn, offset by extensive test coverage and alignment with linked rate-limit instrumentation issue #6702.

Previous run

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff across 8 files touching protected skills paths and build tooling with high git history churn, offset by extensive test coverage and alignment with linked rate-limit instrumentation issue #6702.

Previous run (2)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics, preserved from prior review as signals remain unchanged.

Previous run (3)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics, preserved from prior review as signals remain unchanged.

Previous run (4)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (5)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (6)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (7)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (8)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff touching protected skill paths and CI build files combined with high churn and change coupling in touched driver and build files warrants an elevated risk rating despite strong test coverage.

Previous run (9)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff touching protected skill paths and CI build files combined with high churn and change coupling in touched driver and build files warrants an elevated risk rating despite strong test coverage.

Previous run (10)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff touching protected skill paths and high-churn CI behaviour test files yields an elevated risk score despite thorough test additions and additive scanner logic.

Previous run (11)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff touching protected skill paths and high-churn CI behaviour test files yields an elevated risk score despite thorough test additions and additive scanner logic.

Previous run (12)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff touching protected skill paths and high-churn CI behaviour test files yields an elevated risk score despite thorough test additions and additive scanner logic.

Previous run (13)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (14)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (15)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (16)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (17)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (18)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (19)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (20)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (21)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (22)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (23)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics, preserved from prior review as signals remain unchanged.

Previous run (24)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics, preserved from prior review as signals remain unchanged.

Previous run (25)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics, preserved from prior review as signals remain unchanged.

Previous run (26)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics, preserved from prior review as signals remain unchanged.

Previous run (27)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics, preserved from prior review as signals remain unchanged.

Previous run (28)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics, preserved from prior review as signals remain unchanged.

Previous run (29)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics, preserved from prior review as signals remain unchanged.

Previous run (30)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (31)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (32)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (33)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (34)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (35)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (36)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff modifying protected skill paths and high-churn CI behaviour test components yields an elevated risk score despite extensive test additions and additive diagnostics.

Previous run (37)

<!-- fullsend:risk-assessment -->
Risk Assessment: elevated (3/5)

Details

Large diff touching protected skill paths and high-churn CI behaviour test files yields an elevated risk score despite thorough test additions.

@fullsend-ai-review

fullsend-ai-review Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Review

Findings

Medium

  • [protected-path] skills/e2e-health/SKILL.md, skills/e2e-health/scripts/scan-ratelimit-evidence.sh, skills/e2e-health/scripts/scan-ratelimit-evidence_test.sh — This PR modifies files under the protected skills/ path. The PR references behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 and explains why the scanner and skill documentation are being added, so the context is sufficient. Human approval is still required for protected-path changes, whatever the context.

Earlier findings

  • skills/e2e-health/scripts/scan-ratelimit-evidence.sh:360, edge-case — resolved. The --runs check now matches the bounded decimal string ^([1-9]|[1-9][0-9]|100)$, so an oversized integer can no longer wrap in Bash arithmetic. The test file adds regression cases for 0, 101, 007, -1, abc, an empty value, and two oversized integers.
  • skills/e2e-health/scripts/scan-ratelimit-evidence.sh:2, scope-exceeded — reclassified to info. The scanner automates behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702's own stated validation criterion, and its marker contract depends on the diagnostic strings this PR adds. It is within the issue's intent. Its size is not a blocker.

1 finding(s) below the low severity threshold were not posted.

Previous run

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJyZXF1ZXN0LWNoYW5nZXMiLCJmaW5kaW5ncyI6W3sic2V2ZXJpdHkiOiJsb3ciLCJjYXRlZ29yeSI6InNjb3BlLWV4Y2VlZGVkIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL3NjcmlwdHMvc2Nhbi1yYXRlbGltaXQtZXZpZGVuY2Uuc2giLCJpZCI6ImZfYTUyNTk4YTVjMGY1YjUzYSIsImxpbmUiOjIsImFjdGlvbmFibGUiOmZhbHNlLCJkaW1lbnNpb24iOiJpbnRlbnQtY29oZXJlbmNlIn0seyJzZXZlcml0eSI6ImxvdyIsImNhdGVnb3J5IjoiZWRnZS1jYXNlIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL3NjcmlwdHMvc2Nhbi1yYXRlbGltaXQtZXZpZGVuY2Uuc2giLCJpZCI6ImZfMzlmZmNiZDkyYzQ3NzBhNSIsImxpbmUiOjM2MCwiYWN0aW9uYWJsZSI6dHJ1ZSwiZGltZW5zaW9uIjoiY29ycmVjdG5lc3MifV19 -->

Review

Findings

Medium

  • [protected-path] skills/e2e-health/SKILL.md — The PR modifies files under protected path skills/ (SKILL.md, scan-ratelimit-evidence.sh, scan-ratelimit-evidence_test.sh). It links behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 and explains the rationale, but human approval is always required for protected-path changes, regardless of context. The SKILL.md change also adds auto-execution grants for the new scanner and its test, narrowly scoped to explicit script paths.

Low

  • [edge-case] skills/e2e-health/scripts/scan-ratelimit-evidence.sh:360 — The --runs upper-bound check uses Bash fixed-width arithmetic after only a digits regex, so oversized decimals wrap and pass validation. For example --runs 18446744073709551616 wraps to 0, so 0 > 100 is false and the kept >= runs check (line 403) stops on the first iteration. The script then exits 0 with zero runs scanned and no markers. It needs a deliberately absurd input, hence low severity.
    Remediation: Validate the bounded decimal representation before any arithmetic (e.g. match ^([1-9]|[1-9][0-9]|100)$) and add an argument-validation test for an oversized integer.
  • [scope-exceeded] skills/e2e-health/scripts/scan-ratelimit-evidence.sh:2 — Issue behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 authorizes rate-limit instrumentation and a later evidence-based fix, and its Validation section names the acceptance check (zero ci(#6697): make harness-wait timeouts report why the wait failed #6698 blind-wait diagnostics over 20 runs). Besides the in-scope Go driver instrumentation, this PR adds durable tooling for that check: a ~450-line log scanner, a ~660-line hermetic test, a Makefile entry, a skill procedure and a guide subsection that makes the driver's timeout strings a scanner contract. The tooling serves the stated validation criterion, but the issue does not explicitly ask for durable automation. The marker-string contract also couples future wording changes in the driver to the scanner.
    Remediation: A maintainer should confirm this tooling belongs under behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702; if not, move it to a dedicated issue or PR.

Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run (2)

Review

Reason: stale-head

The review agent reviewed commit cd1c83924f8bc57b460b2b2598d49fb70ce1a246 but the PR HEAD is now a673ce82f1a994e9f1da5d39d184d66c6bb50730. This review was discarded to avoid approving unreviewed code.

Previous run (3)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJkb2Mtc3R5bGUiLCJmaWxlIjoiZG9jcy9ndWlkZXMvZGV2L2JlaGF2aW91ci10ZXN0aW5nLm1kIiwiaWQiOiJmXzhlZmQ0Njk2ODhlYTk1OWYiLCJsaW5lIjozMzgsImFjdGlvbmFibGUiOmZhbHNlLCJkaW1lbnNpb24iOiJzdHlsZS1jb252ZW50aW9ucyJ9LHsic2V2ZXJpdHkiOiJpbmZvIiwiY2F0ZWdvcnkiOiJsb2dpYy1lcnJvciIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl8xM2Q1YTNlZjMzZDBhMjVjIiwibGluZSI6MTIyLCJhY3Rpb25hYmxlIjpmYWxzZSwiZGltZW5zaW9uIjoiY29ycmVjdG5lc3MifSx7InNldmVyaXR5IjoiaW5mbyIsImNhdGVnb3J5Ijoic2NvcGUtY3JlZXAiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvc2NyaXB0cy9zY2FuLXJhdGVsaW1pdC1ldmlkZW5jZS5zaCIsImlkIjoiZl9iMTVlMTNlNWI1NzUzOTRmIiwibGluZSI6MiwiYWN0aW9uYWJsZSI6ZmFsc2UsImRpbWVuc2lvbiI6ImludGVudC1jb2hlcmVuY2UifSx7InNldmVyaXR5IjoiaW5mbyIsImNhdGVnb3J5IjoiZGVzaWduLXNtZWxsIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL1NLSUxMLm1kIiwiaWQiOiJmXzQxN2FiZmE5MjEyYmE5Y2MiLCJsaW5lIjo2LCJhY3Rpb25hYmxlIjpmYWxzZSwiZGltZW5zaW9uIjoiaW50ZW50LWNvaGVyZW5jZSJ9XSwiZGlzcG9zaXRpb25zIjpbeyJpZCI6ImZfOGVmZDQ2OTY4OGVhOTU5ZiIsInN0YXR1cyI6Im9wZW4ifSx7ImlkIjoiZl8xM2Q1YTNlZjMzZDBhMjVjIiwic3RhdHVzIjoib3BlbiJ9LHsiaWQiOiJmX2IxNWUxM2U1YjU3NTM5NGYiLCJzdGF0dXMiOiJvcGVuIn0seyJpZCI6ImZfNDE3YWJmYTkyMTJiYTljYyIsInN0YXR1cyI6InJlc29sdmVkX2J5X2NoYW5nZSJ9XX0= -->

Review

Findings

Medium

  • [protected-path] skills/e2e-health/SKILL.md, skills/e2e-health/scripts/scan-ratelimit-evidence.sh, skills/e2e-health/scripts/scan-ratelimit-evidence_test.sh — This PR modifies files under the protected skills/ path. The PR links behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 and explains the rationale, but human approval is always required for protected-path changes, regardless of context.
    Remediation: A human maintainer with authority over skills/ must review and approve these files.

Low

  • [doc-style] docs/guides/dev/behaviour-testing.md:338 — The paragraph was edited to link to the new Rate-limit observability section but is still one 572-character line. docs/contributing/documentation.md says to rewrap an existing paragraph when you are already editing it.
    Remediation: Wrap the edited paragraph at line 338 to ~80 columns.
Previous run (4)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJkb2Mtc3R5bGUiLCJmaWxlIjoiZG9jcy9ndWlkZXMvZGV2L2JlaGF2aW91ci10ZXN0aW5nLm1kIiwiaWQiOiJmXzhlZmQ0Njk2ODhlYTk1OWYiLCJsaW5lIjozMzgsImFjdGlvbmFibGUiOmZhbHNlLCJkaW1lbnNpb24iOiJzdHlsZS1jb252ZW50aW9ucyJ9LHsic2V2ZXJpdHkiOiJpbmZvIiwiY2F0ZWdvcnkiOiJsb2dpYy1lcnJvciIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl8xM2Q1YTNlZjMzZDBhMjVjIiwibGluZSI6MTIyLCJhY3Rpb25hYmxlIjpmYWxzZSwiZGltZW5zaW9uIjoiY29ycmVjdG5lc3MifSx7InNldmVyaXR5IjoiaW5mbyIsImNhdGVnb3J5Ijoic2NvcGUtY3JlZXAiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvc2NyaXB0cy9zY2FuLXJhdGVsaW1pdC1ldmlkZW5jZS5zaCIsImlkIjoiZl9iMTVlMTNlNWI1NzUzOTRmIiwibGluZSI6MiwiYWN0aW9uYWJsZSI6ZmFsc2UsImRpbWVuc2lvbiI6ImludGVudC1jb2hlcmVuY2UifSx7InNldmVyaXR5IjoiaW5mbyIsImNhdGVnb3J5IjoiZGVzaWduLXNtZWxsIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL1NLSUxMLm1kIiwiaWQiOiJmXzQxN2FiZmE5MjEyYmE5Y2MiLCJsaW5lIjo2LCJhY3Rpb25hYmxlIjpmYWxzZSwiZGltZW5zaW9uIjoiaW50ZW50LWNvaGVyZW5jZSJ9XSwiZGlzcG9zaXRpb25zIjpbeyJpZCI6ImZfOGVmZDQ2OTY4OGVhOTU5ZiIsInN0YXR1cyI6Im9wZW4ifSx7ImlkIjoiZl80MTdhYmZhOTIxMmJhOWNjIiwic3RhdHVzIjoicmVzb2x2ZWRfYnlfY2hhbmdlIn0seyJpZCI6ImZfMTNkNWEzZWYzM2QwYTI1YyIsInN0YXR1cyI6Im9wZW4ifSx7ImlkIjoiZl9iMTVlMTNlNWI1NzUzOTRmIiwic3RhdHVzIjoib3BlbiJ9XX0= -->

Review

Findings

Medium

Low

  • [doc-style] docs/guides/dev/behaviour-testing.md:338 — The paragraph edited at this line is a single ~619-character line. docs/contributing/documentation.md (Line wrapping) asks for prose wrapped at about 80 columns and says to rewrap an existing paragraph when you are already editing it.
    Remediation: Rewrap the paragraph to about 80 columns.

Earlier findings

  • skills/e2e-health/SKILL.md:6, design-smell, resolved — allowed-tools is now limited to path-scoped script invocations (no standalone gh api/jq grants).
Previous run (5)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJkb2Mtc3R5bGUiLCJmaWxlIjoiZG9jcy9ndWlkZXMvZGV2L2JlaGF2aW91ci10ZXN0aW5nLm1kIiwiaWQiOiJmXzhlZmQ0Njk2ODhlYTk1OWYiLCJsaW5lIjozMzgsImFjdGlvbmFibGUiOmZhbHNlLCJkaW1lbnNpb24iOiJzdHlsZS1jb252ZW50aW9ucyJ9LHsic2V2ZXJpdHkiOiJpbmZvIiwiY2F0ZWdvcnkiOiJkZXNpZ24tc21lbGwiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvU0tJTEwubWQiLCJpZCI6ImZfNDE3YWJmYTkyMTJiYTljYyIsImxpbmUiOjYsImFjdGlvbmFibGUiOmZhbHNlLCJkaW1lbnNpb24iOiJpbnRlbnQtY29oZXJlbmNlIn0seyJzZXZlcml0eSI6ImluZm8iLCJjYXRlZ29yeSI6ImxvZ2ljLWVycm9yIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL1NLSUxMLm1kIiwiaWQiOiJmXzEzZDVhM2VmMzNkMGEyNWMiLCJsaW5lIjoxMjIsImFjdGlvbmFibGUiOmZhbHNlLCJkaW1lbnNpb24iOiJjb3JyZWN0bmVzcyJ9LHsic2V2ZXJpdHkiOiJpbmZvIiwiY2F0ZWdvcnkiOiJzY29wZS1jcmVlcCIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9zY3JpcHRzL3NjYW4tcmF0ZWxpbWl0LWV2aWRlbmNlLnNoIiwiaWQiOiJmX2IxNWUxM2U1YjU3NTM5NGYiLCJsaW5lIjoyLCJhY3Rpb25hYmxlIjpmYWxzZSwiZGltZW5zaW9uIjoiaW50ZW50LWNvaGVyZW5jZSJ9XSwiZGlzcG9zaXRpb25zIjpbeyJpZCI6ImZfOGVmZDQ2OTY4OGVhOTU5ZiIsInN0YXR1cyI6Im9wZW4ifSx7ImlkIjoiZl80MTdhYmZhOTIxMmJhOWNjIiwic3RhdHVzIjoicmVjbGFzc2lmaWVkIn0seyJpZCI6ImZfMTNkNWEzZWYzM2QwYTI1YyIsInN0YXR1cyI6InJlY2xhc3NpZmllZCJ9LHsiaWQiOiJmX2IxNWUxM2U1YjU3NTM5NGYiLCJzdGF0dXMiOiJyZWNsYXNzaWZpZWQifV19 -->

Review

Findings

Medium

Low

  • [doc-style] docs/guides/dev/behaviour-testing.md:338 — The paragraph on line 338 ("Godog concurrency stays independent of slot capacity...") was edited in this PR but is still a single ~600-character line. docs/contributing/documentation.md asks for Markdown prose wrapped at about 80 columns and for an edited paragraph to be rewrapped.
    Remediation: Rewrap the edited paragraph at about 80 columns.

Earlier findings

  • skills/e2e-health/SKILL.md:6, design-smell, reclassified from low to info — the test script is hermetic (no network or gh calls), so the allow-list widening has negligible impact.
  • skills/e2e-health/SKILL.md:122, logic-error, stays at info — the "Lenient" label and the closing sentence are ambiguous, not contradictory.
  • skills/e2e-health/scripts/scan-ratelimit-evidence.sh:2, scope-creep, stays at info — the scanner serves behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702's own validation criterion.

3 finding(s) below the low severity threshold were not posted.

Previous run (6)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJkb2Mtc3R5bGUiLCJmaWxlIjoiZG9jcy9ndWlkZXMvZGV2L2JlaGF2aW91ci10ZXN0aW5nLm1kIiwiaWQiOiJmXzhlZmQ0Njk2ODhlYTk1OWYiLCJsaW5lIjozMzgsImFjdGlvbmFibGUiOmZhbHNlLCJkaW1lbnNpb24iOiJzdHlsZS1jb252ZW50aW9ucyJ9LHsic2V2ZXJpdHkiOiJsb3ciLCJjYXRlZ29yeSI6ImRlc2lnbi1zbWVsbCIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl80MTdhYmZhOTIxMmJhOWNjIiwibGluZSI6NiwiYWN0aW9uYWJsZSI6ZmFsc2UsImRpbWVuc2lvbiI6ImludGVudC1jb2hlcmVuY2UifSx7InNldmVyaXR5IjoiaW5mbyIsImNhdGVnb3J5IjoibG9naWMtZXJyb3IiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvU0tJTEwubWQiLCJpZCI6ImZfMTNkNWEzZWYzM2QwYTI1YyIsImxpbmUiOjEyMiwiYWN0aW9uYWJsZSI6ZmFsc2UsImRpbWVuc2lvbiI6ImNvcnJlY3RuZXNzIn0seyJzZXZlcml0eSI6ImluZm8iLCJjYXRlZ29yeSI6InNjb3BlLWNyZWVwIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL3NjcmlwdHMvc2Nhbi1yYXRlbGltaXQtZXZpZGVuY2Uuc2giLCJpZCI6ImZfYjE1ZTEzZTViNTc1Mzk0ZiIsImxpbmUiOjIsImFjdGlvbmFibGUiOmZhbHNlLCJkaW1lbnNpb24iOiJpbnRlbnQtY29oZXJlbmNlIn1dLCJkaXNwb3NpdGlvbnMiOlt7ImlkIjoiZl84ZWZkNDY5Njg4ZWE5NTlmIiwic3RhdHVzIjoib3BlbiJ9LHsiaWQiOiJmXzEzZDVhM2VmMzNkMGEyNWMiLCJzdGF0dXMiOiJyZWNsYXNzaWZpZWQifSx7ImlkIjoiZl9iMTVlMTNlNWI1NzUzOTRmIiwic3RhdHVzIjoicmVjbGFzc2lmaWVkIn1dfQ== -->

Review

Findings

Medium

  • [protected-path] skills/e2e-health/SKILL.md and skills/e2e-health/scripts/ — This PR modifies files under the protected skills/ path (skills/e2e-health/SKILL.md, skills/e2e-health/scripts/scan-ratelimit-evidence.sh, skills/e2e-health/scripts/scan-ratelimit-evidence_test.sh). The PR links behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 and explains the rationale, but human approval is always required for protected-path changes, regardless of context.

Low

  • [doc-style] docs/guides/dev/behaviour-testing.md:338 — The paragraph edited by this PR is still a single 528-character line. docs/contributing/documentation.md asks that an existing paragraph be rewrapped (~80 columns) when it is edited.
    Remediation: Rewrap the paragraph at line 338 to about 80 columns.
  • [design-smell] skills/e2e-health/SKILL.md:6 — The skill's frontmatter description, which agents use for skill discovery, now includes the transient issue reference (the #6702 validation). The reference goes stale once behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 closes.
    Remediation: Drop (the #6702 validation) from the frontmatter description so it stays generic.

2 finding(s) below the low severity threshold were not posted.

Previous run (7)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJkb2Mtc3R5bGUiLCJmaWxlIjoiZG9jcy9ndWlkZXMvZGV2L2JlaGF2aW91ci10ZXN0aW5nLm1kIiwiaWQiOiJmXzhlZmQ0Njk2ODhlYTk1OWYiLCJsaW5lIjozMzgsImFjdGlvbmFibGUiOmZhbHNlLCJkaW1lbnNpb24iOiJzdHlsZS1jb252ZW50aW9ucyJ9LHsic2V2ZXJpdHkiOiJpbmZvIiwiY2F0ZWdvcnkiOiJsb2dpYy1lcnJvciIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl8xM2Q1YTNlZjMzZDBhMjVjIiwibGluZSI6MTIyLCJhY3Rpb25hYmxlIjpmYWxzZSwiZGltZW5zaW9uIjoiY29ycmVjdG5lc3MifSx7InNldmVyaXR5IjoiaW5mbyIsImNhdGVnb3J5Ijoic2NvcGUtY3JlZXAiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvc2NyaXB0cy9zY2FuLXJhdGVsaW1pdC1ldmlkZW5jZS5zaCIsImlkIjoiZl9iMTVlMTNlNWI1NzUzOTRmIiwibGluZSI6MiwiYWN0aW9uYWJsZSI6ZmFsc2UsImRpbWVuc2lvbiI6ImludGVudC1jb2hlcmVuY2UifV0sImRpc3Bvc2l0aW9ucyI6W3siaWQiOiJmXzhlZmQ0Njk2ODhlYTk1OWYiLCJzdGF0dXMiOiJvcGVuIn0seyJpZCI6ImZfMTNkNWEzZWYzM2QwYTI1YyIsInN0YXR1cyI6InJlY2xhc3NpZmllZCJ9LHsiaWQiOiJmX2IxNWUxM2U1YjU3NTM5NGYiLCJzdGF0dXMiOiJyZWNsYXNzaWZpZWQifV19 -->

Review

Findings

Medium

  • [protected-path] skills/e2e-health/SKILL.md — This PR modifies protected paths: skills/e2e-health/SKILL.md, skills/e2e-health/scripts/scan-ratelimit-evidence.sh, and skills/e2e-health/scripts/scan-ratelimit-evidence_test.sh (under skills/). The PR references behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 and explains the rationale (evidence scanner for the rate-limit closure criterion), but human approval is always required for protected-path changes, regardless of context.

Low

  • [doc-style] docs/guides/dev/behaviour-testing.md:338 — An existing paragraph was edited to append a cross-reference to the new Rate-limit observability section and remains a single ~620-character line. docs/contributing/documentation.md (Line wrapping) asks that prose be wrapped at about 80 columns and that an existing paragraph be rewrapped when it is being edited.
    Remediation: Rewrap the edited paragraph at line 338 to about 80 columns.

Earlier findings


2 finding(s) below the low severity threshold were not posted.

Previous run (8)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJkb2Mtc3R5bGUiLCJmaWxlIjoiZG9jcy9ndWlkZXMvZGV2L2JlaGF2aW91ci10ZXN0aW5nLm1kIiwiaWQiOiJmXzhlZmQ0Njk2ODhlYTk1OWYiLCJsaW5lIjozMzgsImFjdGlvbmFibGUiOmZhbHNlLCJkaW1lbnNpb24iOiJzdHlsZS1jb252ZW50aW9ucyJ9LHsic2V2ZXJpdHkiOiJpbmZvIiwiY2F0ZWdvcnkiOiJsb2dpYy1lcnJvciIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl8xM2Q1YTNlZjMzZDBhMjVjIiwibGluZSI6MTIyLCJhY3Rpb25hYmxlIjpmYWxzZSwiZGltZW5zaW9uIjoiY29ycmVjdG5lc3MifSx7InNldmVyaXR5IjoiaW5mbyIsImNhdGVnb3J5Ijoic2NvcGUtY3JlZXAiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvc2NyaXB0cy9zY2FuLXJhdGVsaW1pdC1ldmlkZW5jZS5zaCIsImlkIjoiZl9iMTVlMTNlNWI1NzUzOTRmIiwibGluZSI6MiwiYWN0aW9uYWJsZSI6ZmFsc2UsImRpbWVuc2lvbiI6ImludGVudC1jb2hlcmVuY2UifV0sImRpc3Bvc2l0aW9ucyI6W3siaWQiOiJmXzEzZDVhM2VmMzNkMGEyNWMiLCJzdGF0dXMiOiJyZWNsYXNzaWZpZWQifSx7ImlkIjoiZl84ZWZkNDY5Njg4ZWE5NTlmIiwic3RhdHVzIjoib3BlbiJ9LHsiaWQiOiJmX2IxNWUxM2U1YjU3NTM5NGYiLCJzdGF0dXMiOiJyZWNsYXNzaWZpZWQifV19 -->

Review

Findings

Medium

Low

  • [doc-style] docs/guides/dev/behaviour-testing.md:338 — The edited paragraph is still one unwrapped line. docs/contributing/documentation.md asks for about 80 columns when an existing paragraph is edited.
    Remediation: Rewrap the paragraph to about 80 columns.

Earlier findings


2 finding(s) below the low severity threshold were not posted.

Previous run (9)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJsb2dpYy1lcnJvciIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl8xM2Q1YTNlZjMzZDBhMjVjIiwibGluZSI6MTIyLCJhY3Rpb25hYmxlIjpmYWxzZSwiZGltZW5zaW9uIjoiY29ycmVjdG5lc3MifSx7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJkb2Mtc3R5bGUiLCJmaWxlIjoiZG9jcy9ndWlkZXMvZGV2L2JlaGF2aW91ci10ZXN0aW5nLm1kIiwiaWQiOiJmXzhlZmQ0Njk2ODhlYTk1OWYiLCJsaW5lIjozMzgsImFjdGlvbmFibGUiOmZhbHNlLCJkaW1lbnNpb24iOiJzdHlsZS1jb252ZW50aW9ucyJ9LHsic2V2ZXJpdHkiOiJpbmZvIiwiY2F0ZWdvcnkiOiJzY29wZS1jcmVlcCIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9zY3JpcHRzL3NjYW4tcmF0ZWxpbWl0LWV2aWRlbmNlLnNoIiwiaWQiOiJmX2IxNWUxM2U1YjU3NTM5NGYiLCJsaW5lIjoyLCJhY3Rpb25hYmxlIjpmYWxzZSwiZGltZW5zaW9uIjoiaW50ZW50LWNvaGVyZW5jZSJ9XSwiZGlzcG9zaXRpb25zIjpbeyJpZCI6ImZfMTNkNWEzZWYzM2QwYTI1YyIsInN0YXR1cyI6Im9wZW4ifSx7ImlkIjoiZl84ZWZkNDY5Njg4ZWE5NTlmIiwic3RhdHVzIjoib3BlbiJ9LHsiaWQiOiJmX2IxNWUxM2U1YjU3NTM5NGYiLCJzdGF0dXMiOiJyZWNsYXNzaWZpZWQifV19 -->

Review

Findings

Medium

Low

  • [logic-error] skills/e2e-health/SKILL.md:122 — The lenient-reading bullet defines runs_with_markers_that_passed as marker-bearing runs that ultimately succeeded, then calls it the input to the looser reading that counts only failures that never recover. Read literally this inverts the metric; unrecovered runs are runs_with_markers minus runs_with_markers_that_passed.
    Remediation: State the subtraction explicitly, or describe the lenient reading only as counting retry-masked occurrences.
  • [doc-style] docs/guides/dev/behaviour-testing.md:338 — The paragraph edited in this PR remains a single ~560-character line; docs/contributing/documentation.md asks for ~80-column wrapping and rewrapping of paragraphs when edited.
    Remediation: Rewrap the edited paragraph at about 80 columns.

Earlier findings


1 finding(s) below the low severity threshold were not posted.

Previous run (10)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJsb2dpYy1lcnJvciIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl8xM2Q1YTNlZjMzZDBhMjVjIiwibGluZSI6MTIyLCJhY3Rpb25hYmxlIjpmYWxzZX0seyJzZXZlcml0eSI6ImxvdyIsImNhdGVnb3J5IjoiZG9jLXN0eWxlIiwiZmlsZSI6ImRvY3MvZ3VpZGVzL2Rldi9iZWhhdmlvdXItdGVzdGluZy5tZCIsImlkIjoiZl84ZWZkNDY5Njg4ZWE5NTlmIiwibGluZSI6MzM0LCJhY3Rpb25hYmxlIjpmYWxzZX0seyJzZXZlcml0eSI6ImluZm8iLCJjYXRlZ29yeSI6InNjb3BlLWNyZWVwIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL3NjcmlwdHMvc2Nhbi1yYXRlbGltaXQtZXZpZGVuY2Uuc2giLCJpZCI6ImZfYjE1ZTEzZTViNTc1Mzk0ZiIsImxpbmUiOjIsImFjdGlvbmFibGUiOmZhbHNlfV0sImRpc3Bvc2l0aW9ucyI6W3siaWQiOiJmXzEzZDVhM2VmMzNkMGEyNWMiLCJzdGF0dXMiOiJvcGVuIn0seyJpZCI6ImZfYjE1ZTEzZTViNTc1Mzk0ZiIsInN0YXR1cyI6InJlY2xhc3NpZmllZCJ9XX0= -->

Review

Findings

Medium

Low

  • [logic-error] skills/e2e-health/SKILL.md:122 — The last sentence of the Lenient bullet ("The input to the looser reading that counts only failures that never recover.") is unclear about what runs_with_markers_that_passed measures. The scanner counts select(.has_markers and .ultimately_passed) (scan-ratelimit-evidence.sh:176), i.e. marker-bearing runs that recovered; the sentence can be read as saying this field is the never-recovered failures, which would apply the closure criterion backwards. The doc never states the subtraction (runs_with_markers minus runs_with_markers_that_passed).
    Remediation: State exactly how the count is used, e.g. "The looser reading counts only failures that never recover: runs_with_markers minus runs_with_markers_that_passed."
  • [doc-style] docs/guides/dev/behaviour-testing.md:334 — docs/contributing/documentation.md ("Line wrapping") says an existing paragraph should be rewrapped to about 80 columns when you are already editing it. This PR edits the "Godog concurrency stays independent of slot capacity" paragraph (adds the Rate-limit observability cross-reference) but leaves it as one very long line.
    Remediation: Rewrap the paragraph at word boundaries close to 80 columns.

Earlier findings


1 finding(s) below the low severity threshold were not posted.

Previous run (11)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJsb2dpYy1lcnJvciIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl8xM2Q1YTNlZjMzZDBhMjVjIiwibGluZSI6MTIyLCJhY3Rpb25hYmxlIjpmYWxzZX0seyJzZXZlcml0eSI6ImluZm8iLCJjYXRlZ29yeSI6InNjb3BlLWNyZWVwIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL3NjcmlwdHMvc2Nhbi1yYXRlbGltaXQtZXZpZGVuY2Uuc2giLCJpZCI6ImZfYjE1ZTEzZTViNTc1Mzk0ZiIsImxpbmUiOjIsImFjdGlvbmFibGUiOmZhbHNlfV0sImRpc3Bvc2l0aW9ucyI6W3siaWQiOiJmXzEzZDVhM2VmMzNkMGEyNWMiLCJzdGF0dXMiOiJvcGVuIn0seyJpZCI6ImZfYjE1ZTEzZTViNTc1Mzk0ZiIsInN0YXR1cyI6InJlY2xhc3NpZmllZCJ9XX0= -->

Review

Findings

Medium

Low

  • [logic-error] skills/e2e-health/SKILL.md:122 — The lenient-signal description is inverted or ambiguous. runs_with_markers_that_passed counts marker-bearing runs that recovered, but the text calls it "the input to the looser reading that counts only failures that never recover". Following that guidance can produce the opposite closure decision.
    Remediation: Describe the counter as recovered, retry-masked occurrences. If the lenient criterion should count only unrecovered failures, state that its count is runs_with_markers - runs_with_markers_that_passed.

1 finding(s) below the low severity threshold were not posted.

Previous run (12)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJsb2dpYy1lcnJvciIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl8xM2Q1YTNlZjMzZDBhMjVjIiwibGluZSI6MTIyLCJhY3Rpb25hYmxlIjpmYWxzZX0seyJzZXZlcml0eSI6ImluZm8iLCJjYXRlZ29yeSI6InNjb3BlLWNyZWVwIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL3NjcmlwdHMvc2Nhbi1yYXRlbGltaXQtZXZpZGVuY2Uuc2giLCJpZCI6ImZfYjE1ZTEzZTViNTc1Mzk0ZiIsImxpbmUiOjIsImFjdGlvbmFibGUiOmZhbHNlfV0sImRpc3Bvc2l0aW9ucyI6W3siaWQiOiJmX2IxNWUxM2U1YjU3NTM5NGYiLCJzdGF0dXMiOiJyZWNsYXNzaWZpZWQifV19 -->

Review

Findings

Medium

Low

  • [logic-error] skills/e2e-health/SKILL.md:122 — The lenient-signal text calls runs_with_markers_that_passed "the input to the looser reading that counts only failures that never recover". The scanner computes it as marker-bearing runs that ultimately passed (scan-ratelimit-evidence.sh:176), the population behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702's "pass on re-run" criterion targets. The wording invites the opposite reading.
    Remediation: Say the lenient signal counts marker-bearing runs that ultimately passed. If a never-recover count is wanted, state that it equals runs_with_markers minus runs_with_markers_that_passed.

Earlier findings


1 finding(s) below the low severity threshold were not posted.

Previous run (13)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJzY29wZS1jcmVlcCIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9zY3JpcHRzL3NjYW4tcmF0ZWxpbWl0LWV2aWRlbmNlLnNoIiwiaWQiOiJmX2IxNWUxM2U1YjU3NTM5NGYiLCJsaW5lIjoyLCJhY3Rpb25hYmxlIjpmYWxzZX1dfQ== -->

Review

Findings

Medium

Low

Previous run (14)

Review

Findings

Medium

Low

  • [logic-error] skills/e2e-health/SKILL.md:122 — The lenient-verdict sentence describes runs_with_markers_that_passed as "the input to the looser reading that counts only failures that never recover", while the metric counts marker-bearing runs that ultimately recovered to success. The wording can invert how the evidence is read against behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702's validation criterion.
    Remediation: Describe the metric as counting retry-masked failures that ultimately recovered; define an unrecovered-failure count separately if wanted.
  • [design-smell] skills/e2e-health/SKILL.md:6 — The skill frontmatter description embeds the transient issue number (the #6702 validation) in permanent discovery metadata.
    Remediation: Keep the description capability-focused and leave the issue context in the skill body.
  • [edge-case] skills/e2e-health/scripts/scan-ratelimit-evidence.sh:323 — Earlier low-severity edge-case concern in the behaviour-job log fetch/analysis path remains open; this push does not show it resolved.

Earlier findings

  • skills/e2e-health/SKILL.md:7, misplaced-abstraction — reclassified; remains info-level and below the posting threshold.
  • skills/e2e-health/SKILL.md:7, permission-expansion — reclassified; remains info-level (grants are path-scoped to the scanner and its test).

3 finding(s) below the low severity threshold were not posted.

Previous run (15)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJlZGdlLWNhc2UiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvc2NyaXB0cy9zY2FuLXJhdGVsaW1pdC1ldmlkZW5jZS5zaCIsImlkIjoiZl9jMWI5ZmRhZjc1ZGJhMzcxIiwibGluZSI6MzIzLCJhY3Rpb25hYmxlIjpmYWxzZX0seyJzZXZlcml0eSI6ImxvdyIsImNhdGVnb3J5IjoiZGVzaWduLXNtZWxsIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL1NLSUxMLm1kIiwiaWQiOiJmXzVlODE1NzI4YjI4ZTFlNDIiLCJsaW5lIjo2LCJhY3Rpb25hYmxlIjpmYWxzZX0seyJzZXZlcml0eSI6ImluZm8iLCJjYXRlZ29yeSI6Im1pc3BsYWNlZC1hYnN0cmFjdGlvbiIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl85ZTc5ZmQzZWM4YTI4ZmJiIiwibGluZSI6NywiYWN0aW9uYWJsZSI6ZmFsc2V9LHsic2V2ZXJpdHkiOiJpbmZvIiwiY2F0ZWdvcnkiOiJwZXJtaXNzaW9uLWV4cGFuc2lvbiIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl85ZDlkNWQwNTkxNTY4Yzk2IiwibGluZSI6NywiYWN0aW9uYWJsZSI6ZmFsc2V9LHsic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6ImVkZ2UtY2FzZSIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9zY3JpcHRzL3NjYW4tcmF0ZWxpbWl0LWV2aWRlbmNlLnNoIiwiaWQiOiJmXzg4MTE2ZDkzZjBkMmQ3YTIiLCJsaW5lIjoxNDIsImFjdGlvbmFibGUiOnRydWV9LHsic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6ImVkZ2UtY2FzZSIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9zY3JpcHRzL3NjYW4tcmF0ZWxpbWl0LWV2aWRlbmNlLnNoIiwiaWQiOiJmX2NkZWFmMDQzYjRkYzYwYTEiLCJsaW5lIjozMjMsImFjdGlvbmFibGUiOnRydWV9XSwiZGlzcG9zaXRpb25zIjpbeyJpZCI6ImZfYzFiOWZkYWY3NWRiYTM3MSIsInN0YXR1cyI6Im9wZW4ifSx7ImlkIjoiZl81ZTgxNTcyOGIyOGUxZTQyIiwic3RhdHVzIjoib3BlbiJ9LHsiaWQiOiJmXzllNzlmZDNlYzhhMjhmYmIiLCJzdGF0dXMiOiJyZWNsYXNzaWZpZWQifSx7ImlkIjoiZl85ZDlkNWQwNTkxNTY4Yzk2Iiwic3RhdHVzIjoicmVjbGFzc2lmaWVkIn0seyJpZCI6ImZfODgxMTZkOTNmMGQyZDdhMiIsInN0YXR1cyI6InJlc29sdmVkX2J5X2NoYW5nZSJ9LHsiaWQiOiJmX2NkZWFmMDQzYjRkYzYwYTEiLCJzdGF0dXMiOiJyZXNvbHZlZF9ieV9jaGFuZ2UifV19 -->

Review

Findings

Medium

Low

  • [edge-case] skills/e2e-health/scripts/scan-ratelimit-evidence.sh:323 — scan_run runs on the left of || in main, so errexit is suspended inside it. The analyze_log call is guarded explicitly, but the build_attempt_record ... > rec-NN.json call and the following jq -s are not. If build_attempt_record fails (for example, a malformed markers string makes jq --argjson fail), the record file is empty, jq -s skips it, and the attempt is silently omitted from a report that still says incomplete=false. The failure is unlikely in practice.
    Remediation: check build_attempt_record and the aggregation jq calls explicitly (record the failure and set INCOMPLETE=1, or die), and add a hermetic test that makes build_attempt_record fail.

  • [design-smell] skills/e2e-health/SKILL.md:6 — The generic skill description (the trigger text) embeds a one-off issue reference, "(the behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 validation)". The description should state the durable trigger condition, not a specific issue's closure procedure.
    Remediation: describe the trigger generically (for example, "scanning behaviour runs for GitHub API rate-limit exhaustion evidence") and keep the behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 pointer in the body section.


2 finding(s) below the low severity threshold were not posted.

Previous run (16)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJlZGdlLWNhc2UiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvc2NyaXB0cy9zY2FuLXJhdGVsaW1pdC1ldmlkZW5jZS5zaCIsImlkIjoiZl9jMWI5ZmRhZjc1ZGJhMzcxIiwibGluZSI6MzIzLCJhY3Rpb25hYmxlIjpmYWxzZX0seyJzZXZlcml0eSI6ImxvdyIsImNhdGVnb3J5IjoiZGVzaWduLXNtZWxsIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL1NLSUxMLm1kIiwiaWQiOiJmXzVlODE1NzI4YjI4ZTFlNDIiLCJsaW5lIjo2LCJhY3Rpb25hYmxlIjpmYWxzZX0seyJzZXZlcml0eSI6ImluZm8iLCJjYXRlZ29yeSI6Im1pc3BsYWNlZC1hYnN0cmFjdGlvbiIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl85ZTc5ZmQzZWM4YTI4ZmJiIiwibGluZSI6NywiYWN0aW9uYWJsZSI6ZmFsc2V9LHsic2V2ZXJpdHkiOiJpbmZvIiwiY2F0ZWdvcnkiOiJwZXJtaXNzaW9uLWV4cGFuc2lvbiIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl85ZDlkNWQwNTkxNTY4Yzk2IiwibGluZSI6NywiYWN0aW9uYWJsZSI6ZmFsc2V9LHsic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6ImVkZ2UtY2FzZSIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9zY3JpcHRzL3NjYW4tcmF0ZWxpbWl0LWV2aWRlbmNlLnNoIiwiaWQiOiJmXzg4MTE2ZDkzZjBkMmQ3YTIiLCJsaW5lIjoxNDIsImFjdGlvbmFibGUiOnRydWV9LHsic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6ImVkZ2UtY2FzZSIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9zY3JpcHRzL3NjYW4tcmF0ZWxpbWl0LWV2aWRlbmNlLnNoIiwiaWQiOiJmX2NkZWFmMDQzYjRkYzYwYTEiLCJsaW5lIjozMjMsImFjdGlvbmFibGUiOnRydWV9XSwiZGlzcG9zaXRpb25zIjpbeyJpZCI6ImZfYzFiOWZkYWY3NWRiYTM3MSIsInN0YXR1cyI6Im9wZW4ifSx7ImlkIjoiZl81ZTgxNTcyOGIyOGUxZTQyIiwic3RhdHVzIjoib3BlbiJ9LHsiaWQiOiJmXzllNzlmZDNlYzhhMjhmYmIiLCJzdGF0dXMiOiJyZWNsYXNzaWZpZWQifSx7ImlkIjoiZl85ZDlkNWQwNTkxNTY4Yzk2Iiwic3RhdHVzIjoicmVjbGFzc2lmaWVkIn0seyJpZCI6ImZfODgxMTZkOTNmMGQyZDdhMiIsInN0YXR1cyI6InJlc29sdmVkX2J5X2NoYW5nZSJ9LHsiaWQiOiJmX2NkZWFmMDQzYjRkYzYwYTEiLCJzdGF0dXMiOiJyZXNvbHZlZF9ieV9jaGFuZ2UifV19 -->

Review

Findings

Medium

Low

  • [edge-case] skills/e2e-health/scripts/scan-ratelimit-evidence.sh:323 — build_attempt_record receives the whole markers JSON as one jq --argjson argument and its exit status is not checked. Linux caps a single argument at 128 KiB, and scan_run runs with errexit suspended (left side of || in main), so an oversized marker set would leave an empty rec file that jq -s accepts, dropping the attempt while the scan reports incomplete=false. Triggering this needs several hundred marker hits in one behaviour-job log, so it is unlikely in practice.
    Remediation: Pass the markers via --slurpfile or stdin, check the exit status of build_attempt_record (or that the rec file is non-empty) and treat failure as fatal; optionally add a >128 KiB regression test.
  • [design-smell] skills/e2e-health/SKILL.md:6 — The frontmatter description hardcodes (the #6702 validation). Frontmatter is long-lived skill-routing text, so tying it to one issue's lifecycle makes it go stale once behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 is resolved.
    Remediation: Drop the issue reference so the description states the capability generically.

Earlier findings

  • skills/e2e-health/SKILL.md:7, misplaced-abstraction — reclassified (info, advisory): the allowed-tools grant still includes the scanner's test script.
  • skills/e2e-health/SKILL.md:7, permission-expansion — reclassified (info): grants are path-scoped to named scripts; the scanner issues only GitHub read calls.
Previous run (17)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJlZGdlLWNhc2UiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvc2NyaXB0cy9zY2FuLXJhdGVsaW1pdC1ldmlkZW5jZS5zaCIsImlkIjoiZl9jMWI5ZmRhZjc1ZGJhMzcxIiwibGluZSI6MzIzLCJhY3Rpb25hYmxlIjpmYWxzZX0seyJzZXZlcml0eSI6ImxvdyIsImNhdGVnb3J5IjoiZGVzaWduLXNtZWxsIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL1NLSUxMLm1kIiwiaWQiOiJmXzVlODE1NzI4YjI4ZTFlNDIiLCJsaW5lIjo2LCJhY3Rpb25hYmxlIjpmYWxzZX0seyJzZXZlcml0eSI6ImluZm8iLCJjYXRlZ29yeSI6Im1pc3BsYWNlZC1hYnN0cmFjdGlvbiIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl85ZTc5ZmQzZWM4YTI4ZmJiIiwibGluZSI6NywiYWN0aW9uYWJsZSI6ZmFsc2V9LHsic2V2ZXJpdHkiOiJpbmZvIiwiY2F0ZWdvcnkiOiJwZXJtaXNzaW9uLWV4cGFuc2lvbiIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl85ZDlkNWQwNTkxNTY4Yzk2IiwibGluZSI6NywiYWN0aW9uYWJsZSI6ZmFsc2V9LHsic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6ImVkZ2UtY2FzZSIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9zY3JpcHRzL3NjYW4tcmF0ZWxpbWl0LWV2aWRlbmNlLnNoIiwiaWQiOiJmXzg4MTE2ZDkzZjBkMmQ3YTIiLCJsaW5lIjoxNDIsImFjdGlvbmFibGUiOnRydWV9LHsic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6ImVkZ2UtY2FzZSIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9zY3JpcHRzL3NjYW4tcmF0ZWxpbWl0LWV2aWRlbmNlLnNoIiwiaWQiOiJmX2NkZWFmMDQzYjRkYzYwYTEiLCJsaW5lIjozMjMsImFjdGlvbmFibGUiOnRydWV9XSwiZGlzcG9zaXRpb25zIjpbeyJpZCI6ImZfNWU4MTU3MjhiMjhlMWU0MiIsInN0YXR1cyI6Im9wZW4ifSx7ImlkIjoiZl85ZTc5ZmQzZWM4YTI4ZmJiIiwic3RhdHVzIjoicmVjbGFzc2lmaWVkIn0seyJpZCI6ImZfOWQ5ZDVkMDU5MTU2OGM5NiIsInN0YXR1cyI6InJlY2xhc3NpZmllZCJ9LHsiaWQiOiJmXzg4MTE2ZDkzZjBkMmQ3YTIiLCJzdGF0dXMiOiJyZXNvbHZlZF9ieV9jaGFuZ2UifSx7ImlkIjoiZl9jZGVhZjA0M2I0ZGM2MGExIiwic3RhdHVzIjoicmVzb2x2ZWRfYnlfY2hhbmdlIn1dfQ== -->

Review

Findings

Medium

  • [protected-path] skills/e2e-health/SKILL.md — This PR modifies protected paths: skills/e2e-health/SKILL.md, skills/e2e-health/scripts/scan-ratelimit-evidence.sh, and skills/e2e-health/scripts/scan-ratelimit-evidence_test.sh. The PR links behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 and explains the rationale (an evidence scanner for the rate-limit closure criterion), but human approval is always required for protected-path changes regardless of context.

Low

  • [edge-case] skills/e2e-health/scripts/scan-ratelimit-evidence.sh:323 — Marker evidence can be lost without a warning. build_attempt_record passes the whole markers array as one jq argument (--argjson markers "$4"); Linux caps a single argument at 128 KiB, so a large enough array makes jq fail with E2BIG. scan_run runs on the left of || in main, so errexit is off and the failure goes unchecked: the empty rec-NN.json is accepted by jq -s, the attempt is dropped, and the run can report has_markers=false with incomplete=false. Reaching the limit takes roughly 600+ marker hits in one log, which is unlikely given the harness emits aggregated per-timeout marker lines.
    Remediation: Feed the markers to jq via stdin or --slurpfile from a temp file, check the exit status of build_attempt_record and the jq -s aggregation, and mark the scan incomplete if either fails.
  • [design-smell] skills/e2e-health/SKILL.md:6 — The description frontmatter hardcodes an issue number ("the behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 validation"). Harnesses use the description for semantic routing, so a short-lived ticket reference ties a general capability to that issue's lifecycle.
    Remediation: Remove "(the behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 validation)" and describe the capability generically.
Previous run (18)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJkZXNpZ24tc21lbGwiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvU0tJTEwubWQiLCJpZCI6ImZfNWU4MTU3MjhiMjhlMWU0MiIsImxpbmUiOjYsImFjdGlvbmFibGUiOmZhbHNlfSx7InNldmVyaXR5IjoiaW5mbyIsImNhdGVnb3J5IjoibWlzcGxhY2VkLWFic3RyYWN0aW9uIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL1NLSUxMLm1kIiwiaWQiOiJmXzllNzlmZDNlYzhhMjhmYmIiLCJsaW5lIjo3LCJhY3Rpb25hYmxlIjpmYWxzZX0seyJzZXZlcml0eSI6ImluZm8iLCJjYXRlZ29yeSI6InBlcm1pc3Npb24tZXhwYW5zaW9uIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL1NLSUxMLm1kIiwiaWQiOiJmXzlkOWQ1ZDA1OTE1NjhjOTYiLCJsaW5lIjo3LCJhY3Rpb25hYmxlIjpmYWxzZX0seyJzZXZlcml0eSI6Im1lZGl1bSIsImNhdGVnb3J5IjoiZWRnZS1jYXNlIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL3NjcmlwdHMvc2Nhbi1yYXRlbGltaXQtZXZpZGVuY2Uuc2giLCJpZCI6ImZfODgxMTZkOTNmMGQyZDdhMiIsImxpbmUiOjE0MiwiYWN0aW9uYWJsZSI6dHJ1ZX0seyJzZXZlcml0eSI6Im1lZGl1bSIsImNhdGVnb3J5IjoiZWRnZS1jYXNlIiwiZmlsZSI6InNraWxscy9lMmUtaGVhbHRoL3NjcmlwdHMvc2Nhbi1yYXRlbGltaXQtZXZpZGVuY2Uuc2giLCJpZCI6ImZfY2RlYWYwNDNiNGRjNjBhMSIsImxpbmUiOjMyMywiYWN0aW9uYWJsZSI6dHJ1ZX1dLCJkaXNwb3NpdGlvbnMiOlt7ImlkIjoiZl81ZTgxNTcyOGIyOGUxZTQyIiwic3RhdHVzIjoib3BlbiJ9LHsiaWQiOiJmXzllNzlmZDNlYzhhMjhmYmIiLCJzdGF0dXMiOiJyZWNsYXNzaWZpZWQifSx7ImlkIjoiZl85ZDlkNWQwNTkxNTY4Yzk2Iiwic3RhdHVzIjoicmVjbGFzc2lmaWVkIn0seyJpZCI6ImZfODgxMTZkOTNmMGQyZDdhMiIsInN0YXR1cyI6InJlc29sdmVkX2J5X2NoYW5nZSJ9LHsiaWQiOiJmX2NkZWFmMDQzYjRkYzYwYTEiLCJzdGF0dXMiOiJyZXNvbHZlZF9ieV9jaGFuZ2UifV19 -->

Review

Findings

Medium

Low

Earlier findings

  • skills/e2e-health/SKILL.md:7, misplaced-abstraction — reclassified (lowered). The allowed-tools grant for the scanner's hermetic test is limited to one script path. skills/ here holds maintainer-facing skills, so this is a stylistic choice and not a defect.
Previous run (19)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJkZXNpZ24tc21lbGwiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvU0tJTEwubWQiLCJpZCI6ImZfNWU4MTU3MjhiMjhlMWU0MiIsImxpbmUiOjYsImFjdGlvbmFibGUiOmZhbHNlfSx7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJtaXNwbGFjZWQtYWJzdHJhY3Rpb24iLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvU0tJTEwubWQiLCJpZCI6ImZfOWU3OWZkM2VjOGEyOGZiYiIsImxpbmUiOjcsImFjdGlvbmFibGUiOmZhbHNlfSx7InNldmVyaXR5IjoiaW5mbyIsImNhdGVnb3J5IjoicGVybWlzc2lvbi1leHBhbnNpb24iLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvU0tJTEwubWQiLCJpZCI6ImZfOWQ5ZDVkMDU5MTU2OGM5NiIsImxpbmUiOjcsImFjdGlvbmFibGUiOmZhbHNlfSx7InNldmVyaXR5IjoibWVkaXVtIiwiY2F0ZWdvcnkiOiJlZGdlLWNhc2UiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvc2NyaXB0cy9zY2FuLXJhdGVsaW1pdC1ldmlkZW5jZS5zaCIsImlkIjoiZl84ODExNmQ5M2YwZDJkN2EyIiwibGluZSI6MTQyLCJhY3Rpb25hYmxlIjp0cnVlfSx7InNldmVyaXR5IjoibWVkaXVtIiwiY2F0ZWdvcnkiOiJlZGdlLWNhc2UiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvc2NyaXB0cy9zY2FuLXJhdGVsaW1pdC1ldmlkZW5jZS5zaCIsImlkIjoiZl9jZGVhZjA0M2I0ZGM2MGExIiwibGluZSI6MzIzLCJhY3Rpb25hYmxlIjp0cnVlfV0sImRpc3Bvc2l0aW9ucyI6W3siaWQiOiJmXzVlODE1NzI4YjI4ZTFlNDIiLCJzdGF0dXMiOiJvcGVuIn0seyJpZCI6ImZfOWU3OWZkM2VjOGEyOGZiYiIsInN0YXR1cyI6Im9wZW4ifSx7ImlkIjoiZl85ZDlkNWQwNTkxNTY4Yzk2Iiwic3RhdHVzIjoicmVjbGFzc2lmaWVkIn0seyJpZCI6ImZfODgxMTZkOTNmMGQyZDdhMiIsInN0YXR1cyI6InJlc29sdmVkX2J5X2NoYW5nZSJ9LHsiaWQiOiJmX2NkZWFmMDQzYjRkYzYwYTEiLCJzdGF0dXMiOiJyZXNvbHZlZF9ieV9jaGFuZ2UifV19 -->

Review

Findings

Medium

Low

Earlier findings

  • skills/e2e-health/SKILL.md:7, permission-expansion, reclassified (info) — the grants are scoped to single script paths and the scanner uses read-only gh calls.
Previous run (20)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJkZXNpZ24tc21lbGwiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvU0tJTEwubWQiLCJpZCI6ImZfNWU4MTU3MjhiMjhlMWU0MiIsImxpbmUiOjYsImFjdGlvbmFibGUiOmZhbHNlfSx7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJtaXNwbGFjZWQtYWJzdHJhY3Rpb24iLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvU0tJTEwubWQiLCJpZCI6ImZfOWU3OWZkM2VjOGEyOGZiYiIsImxpbmUiOjcsImFjdGlvbmFibGUiOmZhbHNlfSx7InNldmVyaXR5IjoiaW5mbyIsImNhdGVnb3J5IjoicGVybWlzc2lvbi1leHBhbnNpb24iLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvU0tJTEwubWQiLCJpZCI6ImZfOWQ5ZDVkMDU5MTU2OGM5NiIsImxpbmUiOjcsImFjdGlvbmFibGUiOmZhbHNlfSx7InNldmVyaXR5IjoibWVkaXVtIiwiY2F0ZWdvcnkiOiJlZGdlLWNhc2UiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvc2NyaXB0cy9zY2FuLXJhdGVsaW1pdC1ldmlkZW5jZS5zaCIsImlkIjoiZl84ODExNmQ5M2YwZDJkN2EyIiwibGluZSI6MTQyLCJhY3Rpb25hYmxlIjp0cnVlfSx7InNldmVyaXR5IjoibWVkaXVtIiwiY2F0ZWdvcnkiOiJlZGdlLWNhc2UiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvc2NyaXB0cy9zY2FuLXJhdGVsaW1pdC1ldmlkZW5jZS5zaCIsImlkIjoiZl9jZGVhZjA0M2I0ZGM2MGExIiwibGluZSI6MzIzLCJhY3Rpb25hYmxlIjp0cnVlfV0sImRpc3Bvc2l0aW9ucyI6W3siaWQiOiJmX2NkZWFmMDQzYjRkYzYwYTEiLCJzdGF0dXMiOiJyZXNvbHZlZF9ieV9jaGFuZ2UifSx7ImlkIjoiZl81ZTgxNTcyOGIyOGUxZTQyIiwic3RhdHVzIjoib3BlbiJ9LHsiaWQiOiJmXzllNzlmZDNlYzhhMjhmYmIiLCJzdGF0dXMiOiJvcGVuIn0seyJpZCI6ImZfOWQ5ZDVkMDU5MTU2OGM5NiIsInN0YXR1cyI6InJlY2xhc3NpZmllZCJ9LHsiaWQiOiJmXzg4MTE2ZDkzZjBkMmQ3YTIiLCJzdGF0dXMiOiJyZXNvbHZlZF9ieV9jaGFuZ2UifV19 -->

Review

Findings

Medium

Low

  • [design-smell] skills/e2e-health/SKILL.md:6 — The skill frontmatter description embeds a transient issue reference (the #6702 validation). Frontmatter descriptions are durable discovery metadata; coupling them to one issue degrades longevity once that validation completes.
    Remediation: Phrase the capability generally and keep the issue reference in the skill body.
  • [misplaced-abstraction] skills/e2e-health/SKILL.md:7 — allowed-tools grants scan-ratelimit-evidence_test.sh, and the skill body has a "Verifying the scanner contract" section. Running the hermetic test is a contributor/CI responsibility (already in make script-test and the behaviour-testing dev guide), not an operational capability for agents diagnosing e2e health.
    Remediation: Drop the test script from allowed-tools and leave verification guidance in developer docs.

Earlier findings

  • skills/e2e-health/scripts/scan-ratelimit-evidence.sh:323, edge-case — resolved: scan_run now handles analyzer failure explicitly (discards partial output, records log_error, marks the scan incomplete, warns the operator), and the paired test covers that path.
  • skills/e2e-health/SKILL.md:7, permission-expansion — reclassified to info: the grants are scoped to specific script paths, the scanner only issues read-only gh calls, and no token scopes or write operations are added.
Previous run (21)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJyZXF1ZXN0LWNoYW5nZXMiLCJmaW5kaW5ncyI6W3sic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6ImVkZ2UtY2FzZSIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9zY3JpcHRzL3NjYW4tcmF0ZWxpbWl0LWV2aWRlbmNlLnNoIiwiaWQiOiJmX2NkZWFmMDQzYjRkYzYwYTEiLCJsaW5lIjozMjMsImFjdGlvbmFibGUiOnRydWV9LHsic2V2ZXJpdHkiOiJsb3ciLCJjYXRlZ29yeSI6ImRlc2lnbi1zbWVsbCIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl81ZTgxNTcyOGIyOGUxZTQyIiwibGluZSI6NiwiYWN0aW9uYWJsZSI6ZmFsc2V9LHsic2V2ZXJpdHkiOiJsb3ciLCJjYXRlZ29yeSI6Im1pc3BsYWNlZC1hYnN0cmFjdGlvbiIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl85ZTc5ZmQzZWM4YTI4ZmJiIiwibGluZSI6NywiYWN0aW9uYWJsZSI6ZmFsc2V9LHsic2V2ZXJpdHkiOiJpbmZvIiwiY2F0ZWdvcnkiOiJwZXJtaXNzaW9uLWV4cGFuc2lvbiIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl85ZDlkNWQwNTkxNTY4Yzk2IiwibGluZSI6NywiYWN0aW9uYWJsZSI6ZmFsc2V9LHsic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6ImVkZ2UtY2FzZSIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9zY3JpcHRzL3NjYW4tcmF0ZWxpbWl0LWV2aWRlbmNlLnNoIiwiaWQiOiJmXzg4MTE2ZDkzZjBkMmQ3YTIiLCJsaW5lIjoxNDIsImFjdGlvbmFibGUiOnRydWV9XSwiZGlzcG9zaXRpb25zIjpbeyJpZCI6ImZfY2RlYWYwNDNiNGRjNjBhMSIsInN0YXR1cyI6Im9wZW4ifSx7ImlkIjoiZl85ZDlkNWQwNTkxNTY4Yzk2Iiwic3RhdHVzIjoicmVjbGFzc2lmaWVkIn0seyJpZCI6ImZfODgxMTZkOTNmMGQyZDdhMiIsInN0YXR1cyI6InJlc29sdmVkX2J5X2NoYW5nZSJ9XX0= -->

Review

Findings

Medium

  • [edge-case] skills/e2e-health/scripts/scan-ratelimit-evidence.sh:323 — A marker-heavy log can silently turn into clean evidence. build_attempt_record (lines 140–149) passes the whole markers array to jq as a single --argjson argument, and Linux caps one argument at 128 KiB (MAX_ARG_STRLEN). Once the array exceeds that, jq fails and rec-NN.json is left empty. Line 323 does not check the result, and scan_run runs on the left of || so errexit is off. scan_run's status comes from its final jq ... | build_run_evidence pipeline, so main sees success. The attempt then drops out of the report, which can still show incomplete=false. This contradicts the script's fail-closed design.
    Remediation: Pass the marker JSON through a file or stdin (--slurpfile/--rawfile) instead of argv. Check build_attempt_record's exit status explicitly and set log_error/INCOMPLETE or die on failure. Verify recs.json holds attempt_count records. Add a regression test with more than 128 KiB of marker output.
  • [protected-path] skills/e2e-health/SKILL.md — This PR modifies files under the protected skills/ path (skills/e2e-health/SKILL.md, skills/e2e-health/scripts/scan-ratelimit-evidence.sh, skills/e2e-health/scripts/scan-ratelimit-evidence_test.sh). The PR links behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 and explains the rationale. Human approval is always required for protected-path changes, regardless of context.

Low

  • [design-smell] skills/e2e-health/SKILL.md:6 — The skill description and procedure hardcode the transient issue behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 (the #6702 validation, #6702's closure-rule decision), which ties a reusable skill to one issue's lifecycle.
    Remediation: Describe the rate-limit scan as an ongoing diagnostic and keep behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702-specific criteria in the issue or PR.
  • [misplaced-abstraction] skills/e2e-health/SKILL.md:7 — allowed-tools grants Bash(bash skills/e2e-health/scripts/scan-ratelimit-evidence_test.sh:*), and the "Verifying the scanner contract" section tells the agent to run the test suite. This mixes a contributor test workflow into an operational diagnostic skill. The grant is narrowly scoped, so the risk is small.
    Remediation: Consider moving the test-suite instructions to contributor docs or the Makefile target.

Earlier findings

  • skills/e2e-health/SKILL.md:7, permission-expansion — reclassified (info): the allowed-tools grants are scoped to the two specific script paths and do not widen the underlying token's permissions.

Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run (22)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJyZXF1ZXN0LWNoYW5nZXMiLCJmaW5kaW5ncyI6W3sic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6ImVkZ2UtY2FzZSIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9zY3JpcHRzL3NjYW4tcmF0ZWxpbWl0LWV2aWRlbmNlLnNoIiwiaWQiOiJmX2NkZWFmMDQzYjRkYzYwYTEiLCJsaW5lIjozMjMsImFjdGlvbmFibGUiOnRydWV9LHsic2V2ZXJpdHkiOiJpbmZvIiwiY2F0ZWdvcnkiOiJwZXJtaXNzaW9uLWV4cGFuc2lvbiIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9TS0lMTC5tZCIsImlkIjoiZl85ZDlkNWQwNTkxNTY4Yzk2IiwibGluZSI6NywiYWN0aW9uYWJsZSI6ZmFsc2V9LHsic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6ImVkZ2UtY2FzZSIsImZpbGUiOiJza2lsbHMvZTJlLWhlYWx0aC9zY3JpcHRzL3NjYW4tcmF0ZWxpbWl0LWV2aWRlbmNlLnNoIiwiaWQiOiJmXzg4MTE2ZDkzZjBkMmQ3YTIiLCJsaW5lIjoxNDIsImFjdGlvbmFibGUiOnRydWV9XSwiZGlzcG9zaXRpb25zIjpbeyJpZCI6ImZfOWQ5ZDVkMDU5MTU2OGM5NiIsInN0YXR1cyI6InJlY2xhc3NpZmllZCJ9LHsiaWQiOiJmXzg4MTE2ZDkzZjBkMmQ3YTIiLCJzdGF0dXMiOiJyZXNvbHZlZF9ieV9jaGFuZ2UifV19 -->

Review

Findings

Medium

  • [edge-case] skills/e2e-health/scripts/scan-ratelimit-evidence.sh:323 — A log with many markers can silently turn into a clean run. build_attempt_record (line 140) passes the whole markers array to jq as a single --argjson argument, and analyze_log caps snippet length but not marker count. A single argument over Linux's 128 KiB limit (MAX_ARG_STRLEN) fails with "Argument list too long" (reported reproduction: 1,000 diagnostic lines, about 348 KB of marker JSON, exit 126). scan_run runs on the left of || in main, so errexit is suspended and the failure is not checked. The empty rec-NN.json is skipped by jq -s, and scan_run returns 0 because its status comes from the final pipeline. The run can then be reported with zero markers, has_markers=false and no incomplete flag, which is a false negative for the behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 closure criterion.
    Remediation: Pass the marker JSON via --slurpfile/--rawfile or stdin instead of a command-line argument. Check build_attempt_record success explicitly (set INCOMPLETE=1 and record a log error on failure) so a failure cannot disappear during aggregation. Add a marker-heavy-log regression test.

  • [protected-path] skills/e2e-health/SKILL.md — This PR modifies files under the protected skills/ path (skills/e2e-health/SKILL.md, skills/e2e-health/scripts/scan-ratelimit-evidence.sh, skills/e2e-health/scripts/scan-ratelimit-evidence_test.sh). The PR links behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 and explains the rationale, but human approval is always required for protected-path changes, regardless of context.

Earlier findings


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run (23)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJjb21tZW50IiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoiaW5mbyIsImNhdGVnb3J5IjoicGVybWlzc2lvbi1leHBhbnNpb24iLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvU0tJTEwubWQiLCJpZCI6ImZfOWQ5ZDVkMDU5MTU2OGM5NiIsImxpbmUiOjcsImFjdGlvbmFibGUiOmZhbHNlfSx7InNldmVyaXR5IjoibWVkaXVtIiwiY2F0ZWdvcnkiOiJlZGdlLWNhc2UiLCJmaWxlIjoic2tpbGxzL2UyZS1oZWFsdGgvc2NyaXB0cy9zY2FuLXJhdGVsaW1pdC1ldmlkZW5jZS5zaCIsImlkIjoiZl84ODExNmQ5M2YwZDJkN2EyIiwibGluZSI6MTQyLCJhY3Rpb25hYmxlIjp0cnVlfV0sImRpc3Bvc2l0aW9ucyI6W3siaWQiOiJmXzg4MTE2ZDkzZjBkMmQ3YTIiLCJzdGF0dXMiOiJyZXNvbHZlZF9ieV9jaGFuZ2UifSx7ImlkIjoiZl85ZDlkNWQwNTkxNTY4Yzk2Iiwic3RhdHVzIjoicmVjbGFzc2lmaWVkIn1dfQ== -->

Review

Findings

Medium

  • [protected-path] skills/e2e-health/SKILL.md — This PR modifies files under the protected skills/ path (skills/e2e-health/SKILL.md, skills/e2e-health/scripts/scan-ratelimit-evidence.sh, skills/e2e-health/scripts/scan-ratelimit-evidence_test.sh). The PR links behaviourtest: pool-org installation token exhausts GitHub API budget ~10 min into the suite #6702 and explains the rationale (rate-limit instrumentation and scan procedure), but human approval is always required for protected-path changes regardless of context.
    Remediation: A human maintainer with authority over skills/ must review and approve these changes.

Earlier findings

  • skills/e2e-health/scripts/scan-ratelimit-evidence.sh:142, edge-case, resolved by change — the scanner now loops over every same-line marker occurrence in its awk detection, validates the jobs-page envelope before excluding a run, and records a log_error and marks the report incomplete when the log analyzer fails.
  • skills/e2e-health/SKILL.md:7, permission-expansion, reclassified to info — the new allowed-tools grants are scoped to exact script paths and the scanner only performs read-only gh calls (gh api GETs, gh repo view, gh run list).

@codecov

codecov Bot commented Oct 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review

fullsend-ai-review Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 8:01 PM UTC · Completed 8:11 PM UTC

Commit: 16ed5f2 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.95

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review

fullsend-ai-review Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 10:31 PM UTC · Completed 10:39 PM UTC

Commit: 3a7240c · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.39

fullsend-ai-review[bot]

This comment was marked as outdated.

@jflowers
jflowers force-pushed the opsx/instrument-github-rate-limits branch from 3a7240c to d64fe86 Compare October 26, 2026 11:26
@fullsend-ai-review

fullsend-ai-review Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 9:14 AM UTC · Completed 9:27 AM UTC

Commit: 4c1eefe · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.33

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review

fullsend-ai-review Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 7:36 PM UTC · Completed 7:47 PM UTC

Commit: 80db8b2 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.26

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review

fullsend-ai-review Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 9:20 AM UTC · Completed 9:37 AM UTC

Commit: 7cbf9bb · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.17

fullsend-ai-review[bot]

This comment was marked as outdated.

@rh-hemartin rh-hemartin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me, next step is to somehow analyze these reports to decide what to do, right?

@jflowers

jflowers commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Sounds good to me, next step is to somehow analyze these reports to decide what to do, right?

I think we may need @ifireball to weigh in. I was told he has overlapping work. @ifireball what do you think about this PR?

@fullsend-ai-review

fullsend-ai-review Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 5:17 PM UTC · Completed 5:28 PM UTC

Commit: 5d6ccc5 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.62

fullsend-ai-review[bot]

This comment was marked as outdated.

Closes the instrumentation loop for the pool-org token exhausting the
GitHub API budget mid-run (fullsend-ai#6702): harness waits that time out no
longer fail blind.

- githubactions: append the polling client's last-seen GitHub
  rate-limit state (remaining/reset, observation age, or an explicit
  unknown) to every harness-wait timeout tail - including timeouts
  with no poll failures, which previously carried no quota
  information at all. Non-implementing clients (GitLab, plain
  fakes) are unchanged; 5 new tests pin the state line and each
  timeout scenario.
- e2e-health: add scan-ratelimit-evidence.sh, which walks completed
  e2e.yml runs and attempts via the actions API, detects the fullsend-ai#6698
  blind-wait markers in behaviour-job logs, and emits a JSON report
  with verdicts at both signal levels (any occurrence;
  retry-masked), making fullsend-ai#6702's closure criterion mechanically
  checkable. Paired 70-assertion hermetic test registered in
  make script-test.
- docs: rate-limit observability section in the behaviour-testing
  guide and the scan procedure in the e2e-health skill.

Assisted-by: glm-5
Generated with AI assistance (glm-5)
AI-Assisted: true

Signed-off-by: Jay Flowers <jay.flowers@gmail.com>
… timeouts

Both harness waits built their timeout errors without the client's
last-seen GitHub rate-limit state, contradicting the documented claim
that every harness-wait timeout error carries it. Append
describeRateLimitState to the CountHarnessDispatches and
WaitForHarnessEmptyMatrix timeout paths, and align the count wait with
the other waits' injectable clock and timers so its timeout path is
deterministically testable.

Addresses PR fullsend-ai#8809 review feedback from @fullsend-ai-review[bot] and
@qodo-code-review[bot].

AI-Assisted: true
Assisted-by: glm-5
Signed-off-by: Jay Flowers <jay.flowers@gmail.com>
The scanner counted a same-error-lookups marker the driver can never
emit: the timeout diagnostics pass no current lookup error, so the
lookup tail always renders the counted-failures shape. Count that real
shape instead (failed-lookups), keep the poll near-miss excluded, and
rework the fixtures to the producer's verbatim rendering, including
the rate-limit state line. The docs now describe the real contract
too. Also validate the jobs-page envelope before extracting from it —
a 200 error body is not a jobs page and must fail the scan rather than
silently excluding the run — die on any scan_run exit status other
than the no-behaviour-job sentinel, and use die for main's argument
validation so every failure carries the script prefix.

Addresses PR fullsend-ai#8809 review feedback from @fullsend-ai-review[bot] and
@qodo-code-review[bot].

AI-Assisted: true
Assisted-by: glm-5
Signed-off-by: Jay Flowers <jay.flowers@gmail.com>
The five standalone grants (gh repo view, gh run list, two gh api
path-prefix grants, and blanket jq) widened the skill beyond its
documented procedure: the scanner performs those calls itself, gh api
path-prefix grants accept any --method, and jq can read arbitrary
files via --rawfile. Keep only the path-scoped script grants, and
reword the one documented jq step to read the verdict object from the
saved report instead.

Addresses PR fullsend-ai#8809 review feedback from @fullsend-ai-review[bot].

AI-Assisted: true
Assisted-by: glm-5
Signed-off-by: Jay Flowers <jay.flowers@gmail.com>
Addresses PR fullsend-ai#8809 review feedback from @fullsend-ai-review[bot].

scan_run runs on the left of || in main, so errexit is suspended inside
it: a failing analyze_log was masked, jq emitted [] for a dead awk, no
log_error or INCOMPLETE was recorded, and the function returned its
last pipeline's status, so the rc check in main could not catch it. A
scan could report a clean, marker-free run on an analyzer failure.

Handle the analyzer like a failed log fetch: discard partial output,
record a per-attempt log_error, set INCOMPLETE, and warn the operator.
Add awk to the tool preflight and a hermetic test that overrides
analyze_log to die mid-stream.

Signed-off-by: Jay Flowers <jay.flowers@gmail.com>
AI-Assisted: true
Assisted-by: glm-5
Signed-off-by: Jay Flowers <jay.flowers@gmail.com>
Addresses PR fullsend-ai#8809 review feedback from @fullsend-ai-review[bot].

The failure-path assignment inside scan_run leaked analyzer_rc into
global scope; every other function-scoped variable in the script is
declared local. Add it to the existing local declaration. No behavior
change; the analyzer-failure test still passes.

Signed-off-by: Jay Flowers <jay.flowers@gmail.com>
AI-Assisted: true
Assisted-by: glm-5
Signed-off-by: Jay Flowers <jay.flowers@gmail.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 2:40 AM UTC · Completed 2:52 AM UTC

Commit: aaa3a0e · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.62

fullsend-ai-review[bot]

This comment was marked as outdated.

@ifireball ifireball left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@fullsend-ai-review

fullsend-ai-review Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 1:10 PM UTC · Completed 1:18 PM UTC

Commit: cd1c839 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $1.64

…umns

The paragraph was edited in this PR to link the Rate-limit observability
section but stayed one ~570-character line. Rewrap it per
docs/contributing/documentation.md; wording is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jay Flowers <jay.flowers@gmail.com>
@fullsend-ai-review

Copy link
Copy Markdown

/fs-review

@fullsend-ai-review

fullsend-ai-review Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 1:20 PM UTC · Completed 1:31 PM UTC

Commit: a673ce8 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $3.17

fullsend-ai-review[bot]

This comment was marked as outdated.

jflowers and others added 2 commits November 4, 2026 08:40
The --runs upper-bound check compared with Bash fixed-width arithmetic
after a digits-only regex, so an oversized value such as
18446744073709551616 wrapped to 0, passed validation, and produced a
zero-run scan that exited 0. Match ^([1-9]|[1-9][0-9]|100)$ directly
and add argument-validation tests, including oversized integers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jay Flowers <jay.flowers@gmail.com>
The frontmatter description drives skill discovery; the (the fullsend-ai#6702
validation) reference goes stale once the issue closes. The body keeps
the fullsend-ai#6702 context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jay Flowers <jay.flowers@gmail.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 1:44 PM UTC · Completed 1:53 PM UTC

Commit: 885dc30 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $1.73

This branch was successfully deployed

1 active and 1 inactive deployments
site-preview — 885dc309 Deployed Nov 4, 2026 by github-actions[bot]
dev — 885dc309 Deployed Nov 4, 2026 by jflowers via playback #18860
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/e2e End-to-end tests component/skills requires-manual-review Review requires human judgment risk/elevated PR risk: elevated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants