Skip to content

fix(pre-commit): scope ty hook to staged files - #61

Merged
waynesun09 merged 2 commits into
mainfrom
agent/60-scope-precommit-hooks
Oct 16, 2026
Merged

waynesun09 merged 2 commits into
mainfrom
agent/60-scope-precommit-hooks

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor

Summary

Makes pre-commit check only the files in a change, so contributors who touch one experiment get a passing run and a PR containing only their files.

  • .pre-commit-config.yaml: the local ty hook ran uvx ty check . with pass_filenames: false, so any commit that staged a Python file type-checked the entire repo and failed on pre-existing errors in other experiments. Removed the . target and the pass_filenames: false override, so pre-commit now passes only the staged Python files to ty. types: [python] is unchanged.
  • AGENTS.md: replaced the pre-commit run --all-files recommendation with pre-commit run --files <paths>, and noted that --all-files runs ruff, ruff-format, ty and bandit across every experiment and rewrites or flags code outside your change.

Testing

  • Reproduced on main with ty 0.0.87: ty check . --ignore unresolved-import --ignore unresolved-attribute reports 8 diagnostics, all in 0008-guardrails-eval/eval-model-armor.py and 0024-codeowners-app-matrix/run_matrix.py.
  • Ran the updated ty hook through pre-commit run -c <ty-only config> --files ...:
    • a new file with a deliberate type error → fails and reports only that file
    • 0026-eval-statistical-significance/significance.py + Markdown → passes
    • a Markdown-only change → ty skipped
  • Checked directly with ty: 0025-.../.fullsend/scripts/post-emit-cost.py, in a hidden directory, is checked when passed as a path, and passes.
  • The full pre-commit run could not fetch remote hook repos in the sandbox (TLS verification failure). The pre-commit-hooks v6.0.0 checks (check-yaml, end-of-file-fixer, trailing-whitespace, detect-private-key, large files, merge conflict, mixed line endings) were run directly on the changed files and passed. Secrets were scanned with gitleaks via scan-secrets.

Notes

  • If you stage a Python file that already has ty errors (e.g. 0024-.../run_matrix.py), the hook still flags them. That is expected, since the file is part of the change.
  • Out of scope: a one-time cleanup of the existing ty, ruff-format and bandit findings so that --all-files passes again. It touches other people's experiments, so it is better as a separate change.

🤖 Generated with Claude Code


Closes #60

Post-script verification

  • Branch is not main/master (agent/60-scope-precommit-hooks)
  • Secret scan passed (gitleaks — 3f1614fa8adcf731c6d415b9e0af8b4a2556e9ed..HEAD)
  • PR body secret scan passed (gitleaks — no-git)

The local ty hook ran `ty check .` with pass_filenames: false, so any
commit staging a Python file type-checked the whole repository and
failed on pre-existing errors in other experiments (currently 0008 and
0024). Drop the `.` target and the pass_filenames override so pre-commit
passes only the staged Python files to ty.

AGENTS.md recommended `pre-commit run --all-files`, which runs ruff,
ruff-format, ty and bandit across every experiment and rewrites or
flags code outside the contributor's change. Point contributors at
`pre-commit run --files <paths>` instead and explain why --all-files
should be avoided when preparing a change.

Cleaning up the pre-existing ty, ruff-format and bandit findings is
left out of scope, as it touches other experiments.

Note: `pre-commit run` could not fetch remote hook repos in the sandbox
(TLS verification failure). The pre-commit-hooks checks were run
directly at rev v6.0.0 and passed; gitleaks was covered by
scan-secrets. The ty hook was exercised via pre-commit with a ty-only
config: it fails on a staged file with a type error, reports only that
file, passes on clean files, and is skipped for Markdown-only changes.

Closes #60

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fullsend-ai-coder
fullsend-ai-coder Bot requested a review from a team October 15, 2026 06:22
@fullsend-ai-coder fullsend-ai-coder Bot added the ready-for-review Triggers review agent dispatch label Oct 15, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 6:24 AM UTC · Completed 6:32 AM UTC

Commit: fe218d1 · View workflow run →

Runtime: claude · Model: opus → claude-opus-5-5 · Effort: high · Cost: $2.72

@fullsend-ai-review fullsend-ai-review Bot added the risk/low PR risk: low label Oct 15, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Risk Assessment: moderate (2/5)

Details

Small 7-line, 2-file bot-authored config/docs change with low churn and a well-scoped linked issue; moderate only because both changed files are protected paths.

Previous run

Risk Assessment: low (1/5)

Details

Tiny 7-line config-only fix by a bot addressing a well-scoped issue; the only elevated signal is PROTECTED_PATH_COUNT=2 (touching AGENTS.md and .pre-commit-config.yaml), which is offset by minimal churn, no security or dependency concerns, stable file history, and complete acceptance-criteria coverage.

@fullsend-ai-review

fullsend-ai-review Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Review

Findings

Medium / Low / Info

  • [protected-path] .pre-commit-config.yaml:29: This PR modifies protected paths: .pre-commit-config.yaml and AGENTS.md. The change is justified: it closes pre-commit fails or rewrites files outside a change set, so clean PRs need --no-verify or drag in other experiments #60, the issue's triage recommends exactly these two edits, and the PR description explains why. Human approval is still always required for protected-path changes.
    Remediation: A maintainer with write access should review and approve the protected-path changes.

  • [edge-case] .pre-commit-config.yaml:29 (low): ty now type-checks only the staged Python files it is given. A staged change can break other, unstaged files (for example, a changed function signature or a removed symbol imported elsewhere in the same experiment), and those files are not checked, so the commit passes. No CI workflow runs ty or pre-commit to catch it later. Issue pre-commit fails or rewrites files outside a change set, so clean PRs need --no-verify or drag in other experiments #60 asked for this trade-off, so it is not a defect, but contributors may not expect it.
    Remediation: Optional: note in AGENTS.md that ty only checks the files passed to it, so contributors should include dependent files from their experiment (e.g. pre-commit run --files <experiment-dir>/**/*.py), or add a CI job that runs ty per changed experiment directory.

Previous run

Review

Findings

Medium / Low / Info

  • [protected-path] .pre-commit-config.yaml:29 — This PR modifies protected governance/infrastructure files: .pre-commit-config.yaml and AGENTS.md. The PR links issue pre-commit fails or rewrites files outside a change set, so clean PRs need --no-verify or drag in other experiments #60 and explains the rationale (scoping the ty hook to staged files and updating the manual-run guidance), but human approval is always required for protected-path changes, regardless of context.

  • [technical-accuracy] AGENTS.md:48 — The new guidance says pre-commit run --all-files "runs ruff, ruff-format, ty and bandit across every experiment, rewriting and flagging pre-existing code". The list is incomplete: end-of-file-fixer, trailing-whitespace and mixed-line-ending (pre-commit-hooks v6.0.0) are configured in .pre-commit-config.yaml without file filters, so under --all-files they also rewrite files in other experiments — the same "drags in other experiments" problem issue pre-commit fails or rewrites files outside a change set, so clean PRs need --no-verify or drag in other experiments #60 describes. A contributor relying on the enumerated list could be surprised by whitespace/EOF rewrites.
    Remediation: Generalize the wording instead of enumerating a partial list, e.g. "it runs every hook across every experiment — including file-rewriting hooks such as ruff --fix, ruff-format, end-of-file-fixer and trailing-whitespace, plus ty and bandit — rewriting and flagging pre-existing code outside your change."


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-coder

fullsend-ai-coder Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Finished Fix · ✅ Success · Started 6:33 AM UTC · Completed 6:37 AM UTC

Commit: fe218d1 · View workflow run →

Runtime: claude · Model: opus → claude-opus-5-5 · Effort: high · Cost: $0.61

Address review feedback on PR #61: the AGENTS.md warning against
`pre-commit run --all-files` listed only ruff, ruff-format, ty and
bandit. end-of-file-fixer, trailing-whitespace and mixed-line-ending
also run without file filters and rewrite files in other experiments.
Reword to say every hook runs, naming the file-rewriting hooks as
examples.

Not changed: the protected-path finding is the merge gate requiring
human approval and does not call for a content edit.

Verification: `pre-commit run` could not fetch remote hook repos in
the sandbox (TLS verification failure). The applicable hooks were
replicated directly on AGENTS.md (trailing whitespace, EOF newline,
mixed line endings, merge-conflict markers, private keys, file size)
and passed; hack/lint-experiment-* scripts passed; gitleaks scan
passed.

Addresses #61

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fullsend-ai-coder

Copy link
Copy Markdown
Contributor Author

🔧 Fix agent — iteration 1 (bot-triggered)

Addressed the technical-accuracy finding: the AGENTS.md --all-files warning now says every hook runs repo-wide and names the hooks that rewrite files. The protected-path finding needs human approval and has no edit to make. No merge conflict was found. The project CI checks (Build Doc Site, DCO) pass. pre-commit could not fetch remote hook repos in the sandbox, so the hooks that apply to AGENTS.md were reproduced by hand and passed, along with the repo lint scripts and the secret scan.

Fixed (2):

  1. Conflict update: inspected forge mergeability before applying fixes: GitHub reports mergeable=MERGEABLE (no conflict), so no merge or rebase was performed.
  2. [technical-accuracy] AGENTS.md:48 — the --all-files warning lists only ruff, ruff-format, ty and bandit, omitting unfiltered file-rewriting hooks (AGENTS.md): Checked against .pre-commit-config.yaml: end-of-file-fixer, trailing-whitespace and mixed-line-ending have no file filters, so they rewrite files repo-wide under --all-files. Used the reviewer's suggested wording: --all-files runs every hook across every experiment, with ruff --fix, ruff-format, end-of-file-fixer and trailing-whitespace named as examples of hooks that rewrite files, plus ty and bandit.

Disagreed (1):

  1. [protected-path] .pre-commit-config.yaml:29 — PR modifies protected files .pre-commit-config.yaml and AGENTS.md: This finding is the mandatory merge gate. It asks for human approval and does not prescribe a content change, so there is nothing for the fix agent to edit. Reverting the protected-path changes would undo the purpose of issue pre-commit fails or rewrites files outside a change set, so clean PRs need --no-verify or drag in other experiments #60. A human reviewer must still approve this PR.

Tests: passed

CI inspections (2):

  1. Build Doc Site / build (passing, success): Doc site build passed on the PR head.
  2. DCO (passing, success): DCO check passed.

Conflict update: skipped; strategy none; forge MERGEABLE; target main@3f1614fa8adcf731c6d415b9e0af8b4a2556e9ed

Decision points
  • How to make the --all-files warning accurate (alternatives: Append end-of-file-fixer, trailing-whitespace and mixed-line-ending to the enumerated list, Generalize to 'every hook' and name the file-rewriting hooks as examples (reviewer's suggestion); rationale: Generalizing stays correct if hooks are added later, and it matches the reviewer's suggested remediation.)

Next steps:

  • /fs-review — request a re-review of the changes
  • /fs-fix <your instruction> — run another fix pass with specific guidance
  • Push commits directly — review re-runs automatically on push
    Updated by fullsend fix agent

@fullsend-ai-review

fullsend-ai-review Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 6:39 AM UTC · Completed 6:48 AM UTC

Commit: 1db22ec · View workflow run →

Runtime: claude · Model: opus → claude-opus-5-5 · Effort: high · Cost: $3.31

@fullsend-ai-review fullsend-ai-review Bot added risk/moderate PR risk: moderate and removed risk/low PR risk: low labels Oct 15, 2026
@fullsend-ai-review
fullsend-ai-review Bot dismissed their stale review October 15, 2026 06:48

Superseded by updated review

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review comment for full details.

Comment thread .pre-commit-config.yaml
- id: ty
name: ty check
entry: uvx ty check . --ignore unresolved-import --ignore unresolved-attribute
entry: uvx ty check --ignore unresolved-import --ignore unresolved-attribute

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] protected-path

This PR modifies protected paths: .pre-commit-config.yaml and AGENTS.md (both listed in REVIEW_PROTECTED_PATHS). The change is justified: it closes #60, the issue's triage recommends exactly these two edits, and the PR description explains why. Human approval is still always required for protected-path changes, whatever the justification.

Suggested fix: A maintainer with write access should review and approve the protected-path changes.

Comment thread .pre-commit-config.yaml
- id: ty
name: ty check
entry: uvx ty check . --ignore unresolved-import --ignore unresolved-attribute
entry: uvx ty check --ignore unresolved-import --ignore unresolved-attribute

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] edge-case

ty now type-checks only the staged Python files it is given. A staged change can break other, unstaged files, for example by changing a function signature or removing a symbol imported elsewhere in the same experiment. Those files are not checked, so the commit passes. No CI workflow runs ty or pre-commit to catch it later. Issue #60 asked for this trade-off, so it is not a defect, but contributors may not expect it.

Suggested fix: Optional: note in AGENTS.md that ty only checks the files passed to it, so contributors should include dependent files from their experiment (e.g. pre-commit run --files <experiment-dir>/**/*.py). Alternatively, add a CI job that runs ty on each changed experiment directory.

@fullsend-ai-review fullsend-ai-review Bot added the requires-manual-review Review requires human judgment label Oct 15, 2026

@jwm4 jwm4 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I filed #60 and tested this branch (1db22ec) locally.

With the config on main, staging a one-line change to 0017-reasoning-monitor/monitor/__init__.py fails the ty hook on pre-existing errors in 0008-guardrails-eval/eval-model-armor.py, which the change does not touch. With this branch, the same change passes ty and every other hook, and no other file is modified. Staging a change to 0024-codeowners-app-matrix/run_matrix.py, which has its own ty errors, still fails as it should.

On the low-severity finding that ty now checks only staged files: I accept that trade-off. Experiments are largely independent of each other, and the scoping is what #60 asked for.

This fixes the problem from the reporter's side. It still needs a maintainer's approval for the protected-path change.

Comment from Bill Murdock with assistance from Claude Code.

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, including the protected-path change. I tested the hook with ty 0.0.89 and pre-commit 4.7.0: staged-only checking works, and --all-files still reports the same diagnostics as ty check ..

@waynesun09
waynesun09 added this pull request to the merge queue Oct 16, 2026
Merged via the queue into main with commit 5f6c966 Oct 16, 2026
68 of 72 checks passed
@waynesun09
waynesun09 deleted the agent/60-scope-precommit-hooks branch October 16, 2026 18:56
@fullsend-ai-retro

fullsend-ai-retro Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 6:58 PM UTC · Completed 7:12 PM UTC

Commit: 1db22ec · View workflow run →

Runtime: claude · Model: opus → claude-opus-5-5 · Effort: high · Cost: $2.48

@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #61 (scope pre-commit ty hook to staged files, closes #60)

Overall: the workflow went well. Triage → code → review → one fix iteration → re-review → human approval → merge. Total agent spend was about $7.90 (triage $0.60, code $0.66, review $2.72, fix $0.61, re-review $3.31) on a 7-line, 2-file change.

Review quality

  • First review caught a real problem: the new AGENTS.md text listed only some of the hooks that --all-files runs repo-wide. The fix agent corrected it in one pass ($0.61).
  • The protected-path finding was correct and routed the PR to a maintainer, as designed.
  • The re-review added a low-severity edge-case finding: ty now checks only staged files, and no CI runs ty. The issue reporter explicitly accepted that trade-off.
  • Both humans (the issue reporter, then a maintainer) tested locally. Their testing agreed with the agent review and found nothing the agents missed.

Autonomy readiness: no review gap between humans and agents on this PR. Human approval was needed only because of the protected-path policy, which is the intended gate for .pre-commit-config.yaml and AGENTS.md.

Evidence for existing issues (no new proposals filed):

  • fullsend#6950 / agents#1092: the re-review after a one-line AGENTS.md fix re-ran all 7 review sub-agents on the whole PR. It cost $3.31, more than the first review ($2.72), even though pr-incremental-diff.txt was built. It also raised a new finding on a file the fix never touched (see also fullsend#3687 and agents#1208).
  • experiments#55 / open PR experiments#56: the triage, code and fix runs, plus later runs on this PR, show a red ❌ even though the agents succeeded. The cause is the "Harness dispatch" job failing with a qualityflow ... 404. PR fix(#55): remove broken qualityflow agent reference #56 fixes it and has been idle since 2026-08-22, so merging it would clear the false failures.

New proposal: in the code and fix sandboxes, pre-commit hook-repo fetches now fail because git rejects the server certificate. The sandbox policy allows the connection, so this is a trust problem, not the egress block tracked in fullsend#7298. The same failure appeared in three other runs across the org. It caused little lost time here, but the repo's real pre-commit config is never exercised before push.

Agents repo discovered from run logs: fullsend-ai/agents@main (a159f388).

Proposals filed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-review Triggers review agent dispatch requires-manual-review Review requires human judgment risk/moderate PR risk: moderate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pre-commit fails or rewrites files outside a change set, so clean PRs need --no-verify or drag in other experiments

2 participants