Repository navigation
fix(list-recent-runs): anchor completion window to intended cron tick - #526
Conversation
…lack review-reviewers' hourly schedule (cron `47 * * * *`) is delayed 20–40 min during peak hours, producing 80–100 min gaps between consecutive cycles. The 1h `updatedAt` cutoff in list-recent-runs.sh dropped everything older than the previous cycle's expected start, silently hiding tend runs that landed in the delay slack. Bump the cutoff to 2h. Covers the largest observed gap (99 min) with margin; the small re-analysis overlap between cycles is cheap because gist entries are keyed by run ID. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
do we get an input of the intended cron time, which we can use to look over exactly one hour of time? |
|
Not directly — GHA doesn't expose an intended-fire timestamp on the schedule event. The only related field is But the intended time is derivable from that expression + Sketch: THIS_MIN_TS=$(date -u -d "$(date -u +%Y-%m-%dT%H:00:00) 47 minutes" +%s)
if [ "$(date -u +%s)" -lt "$THIS_MIN_TS" ]; then
INTENDED=$((THIS_MIN_TS - 3600))
else
INTENDED=$THIS_MIN_TS
fi
CREATED_SINCE=$(date -u -d "@$((INTENDED - 3600))" +%Y-%m-%dT%H:%M:%S)
COMPLETED_AFTER=$((INTENDED - 3600))Caveats:
Want me to switch the PR to the intended-time approach? Happy to either replace the 2h bump or layer the intended-time anchor on top of a smaller (e.g. 75 min) safety margin. |
|
yes please, and remove the 2H window change |
Replaces the 2h widening with an intended-tick anchor: when the script runs under a scheduled workflow with a simple hourly cron (`MM * * * *`), read the cron from $GITHUB_EVENT_PATH and anchor the window to the most recent matching tick instead of `now`. Consecutive cycles tile exactly ([intended-1h, intended]) regardless of GHA scheduler delay, with no overlap. Non-schedule events and non-hourly crons fall back to the original now-anchored 1h window. Per maintainer feedback on PR #526 — strictly cleaner than the 2h widening it replaces. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
Done — pushed fdc947a. Auto-detects the cron from |
## Summary Release 0.0.25. 8 PRs since 0.0.24: - #566 review: push fixes to bot PRs by default, even when the fix needs a judgment call - #563 skills(running-tend): drop stale session-log count - #562 skills: drop dollar-amount cost asides corrupted by arg substitution - #526 fix(list-recent-runs): anchor completion window to the intended cron tick - #559 fix(skills): hard-code /tmp instead of $TMPDIR in worktree recipes - #560 fix(enrich-tend-outage-issues): batch into one comment per issue - #557 config: switch review-reviewers back to claude, demote codex-auth-refresh to reference - #556 config: switch tend's own harness back to claude #566 is the reason for cutting now: it flips the bot-PR review default from "push only mechanical fixes" to "push the fix whenever the review can articulate it, defer only when no defensible default exists." The behavior is release-gated, so it only reaches tend's own dogfooded workflows (and adopters like PRQL) once 0.0.25 is published and the nightly regen repins the action ref. No new breaking changes since 0.0.24. ## Test plan - [ ] CI green (test, test-worker, lint) - [ ] After merge: tag `0.0.25`, PyPI publish (`pypi-release.yaml` on the tag push), `uvx tend@0.0.25 --help` works - [ ] Regenerate tend's own workflows to `@0.0.25` (separate `chore: regenerate` PR, after the tag exists) Co-authored-by: Claude <noreply@anthropic.com>
…chor (#753) ## Problem `list-recent-runs.sh` anchors its completion window to the most recent intended cron tick and floors it at `intended - 3600`. This tiles exactly — `[intended-1h, intended]` then `[intended, intended+1h]` — **only when consecutive cron ticks actually fire.** GHA doesn't just *delay* scheduled ticks (which the tick-anchoring in #526 handles well); it *drops* them. A dropped tick leaves that hour's completions in the gap between the previous and next cycle's windows, and no run ever analyzes them. [#526](#526) named this exact "skipped-tick edge case" and deferred it as "rare under sustained outage," because the only fix on the table then was the 2h-overlap approach the maintainer had rejected for re-analyzing runs. This run is direct evidence it is **not** rare and the miss is **invisible** — a dropped tick produces no error, so the skipped outcomes just never get audited. ## Evidence (this run) Analyzing `max-sixty/worktrunk`, `review-reviewers` fired at `07:50Z` (07:47 tick) then `09:46Z` (09:47 tick) — **GHA dropped the 08:47 tick.** The window for the 09:47 cycle floored at `08:47`, leaving `[07:47, 08:47]` covered by no cycle. `list-recent-runs.sh` returned **3** of the **10** tend-* runs actually in `[07:47, 09:47]`. The orphaned hour contained real, substantive bot output: - **[worktrunk#3355](max-sixty/worktrunk#3355 — a well-sourced answer on Windows Defender code-signing (comment `4874219842`, `08:16Z`). - **[worktrunk#3357](max-sixty/worktrunk#3357 — a full review cycle: `COMMENTED` flagging `codecov/patch`, author pushed coverage, bot `APPROVED` on the fixed commit (`08:15Z`–`08:23Z`). Both were healthy outcomes this time — but had either gone wrong, no `review-reviewers` cycle would have caught it. (I covered the gap manually for this run's audit; the gist records the full window.) ## Fix Floor the window at the previous **actual** completed run's intended tick instead of assuming the previous tick fired. #526's discussion weighed only two options — the *intended* fire time (not exposed by GHA) vs 2h overlap (rejected) — and missed a third: the previous run's own start time **is** queryable, and rounding it down to its cron tick recovers exactly where the last cycle's window ended. - **Every tick fires** → previous intended tick `== intended - 3600` → byte-identical to today's behavior, **no overlap** (the constraint #526 set). - **A tick was dropped** → floor reaches back to the last actual run's tick, covering the orphaned hour with no double-analysis. - Capped at **6h** reach-back so a sustained outage can't create an unbounded window. Falls back cleanly (`gh` failure, no `GITHUB_WORKFLOW`) to the current floor. Verified in this CI session: the patched script returns all **10** in-window runs (was 3); with consecutive ticks it resolves the floor to `intended - 3600` unchanged. ## Gate assessment - **Type:** targeted fix to one branch of the window logic (not a structural rewrite). - **Classification:** structural + deterministic — a dropped tick *always* produces the gap; not a stochastic model lapse. - **Occurrences:** 1 confirmed gap-with-missed-outcomes this run. Below the normal 2–3 bar, but acted on because the failure mode is **self-concealing**: future cycles use the same window and won't re-detect the gap, so evidence cannot accumulate — "wait for occurrence 2" would sit at 1 while silently under-covering. This is tend's own audit tooling, the fix reduces to a no-op in the common case, and human review is the backstop. Evidence log: https://gist.github.com/b923bbc52f3c6606931d59abdae616a3 Co-authored-by: tend-agent <tend-agent@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
… and stop the fetch limit re-truncating it (#838) `list-recent-runs.sh`'s dropped-tick recovery anchors the window floor on the previous **completed** run of the analyzing workflow. `completed` is a status, not a conclusion, so a run that failed before its agent produced any analysis advances the anchor exactly as if it had covered its hour. After an outage the next green run therefore resumes one tick back and reports an all-clear for a window nothing ever looked at — and the gap is unrecoverable, because the following run's floor advances past it too. This is not hypothetical: it just consumed ~15 hours of `review-reviewers` coverage. ## What happened Every `review-reviewers` run from [30897445507](https://github.com/max-sixty/tend/actions/runs/30897445507) (09:41Z) through [30959115609](https://github.com/max-sixty/tend/actions/runs/30959115609) (23:10Z) failed — 16 consecutive runs, all five matrix legs, `claude -p` exiting 1 after ~4s. The `result` event in every leg's session log is identical: `is_error: true`, `num_turns: 1`, `result: "You've hit your weekly limit · resets 12am (UTC)"`, with `token-usage.json` all zeros. The agent never started, so none of those runs analyzed anything. (The outage itself is already tracked by #831 and the annotation-legibility work in #816 / #818 — this PR is only about the window the outage swallowed.) The last run that did analyze anything is [30893072876](https://github.com/max-sixty/tend/actions/runs/30893072876) at 08:42Z, which is also the last entry in the `numbagg/numbagg` evidence gist. At this run's tick (23:47Z intended), the recovery query picked `{"conclusion":"failure","createdAt":"2026-08-04T23:10:57Z","databaseId":30959115609}` and floored the window at 22:47Z, so `list-recent-runs.sh` returned exactly one row: a `tend-notifications` pre-check no-op. Everything `numbagg-bot` actually did in the swallowed window sat outside that floor — PRs [#719](numbagg/numbagg#719) and [#720](numbagg/numbagg#720), issue [#721](numbagg/numbagg#721), three reviews and four inline comments on #719, and comments on #716, #721 and #1. That is the densest window numbagg has had in days, and per the skill's Step 1 ("If empty, record the run as all-clear … then skip to Step 6") it would have been recorded as a quiet hour. I only found it by diffing the run list against the gist's last recorded boundary by hand. ## The change Anchor on the previous **successful** run rather than any completed one, filtering server-side with `--status success` (the flag takes conclusions as well as statuses), and when the existing 6h cap clamps the recovered floor, say so on stderr so the caller records a coverage gap instead of a false all-clear. A partially-failed matrix run counts as a failure here, which only ever widens the window — overlap is re-offered work the caller dedups against its own evidence log, whereas a gap is silently unanalyzed. Widening the window then exposed a second truncation one layer down, so the fetch loop moves with it. It passed `--limit 50` while a recovered window spans up to 8h, and `gh run list` returns newest-first — so a workflow over the limit silently drops its *oldest* runs, which are exactly the ones in the gap the anchor just reached back for. On `max-sixty/tend`, `tend-mention` alone produces 57 runs in an 8h window. The limit is now 200, and a workflow returning exactly the limit warns rather than truncating in silence. Two smaller pieces: the anchor query routes through `gh_retry` and exits non-zero instead of `2>/dev/null || true`, so a transient API error can no longer masquerade as "no successful run" and emit a confident warning naming a cause that didn't happen; and `review-reviewers/SKILL.md`, which still said "if empty, record all-clear" with nothing about the stderr warnings, now treats any `WARNING:` as a coverage gap — without that, an agent following the skill literally would print the warning and record an all-clear anyway. When every tick fires and succeeds the anchor is the previous tick, `prev_intended == intended - 3600 == COMPLETED_AFTER`, and the comparison is a no-op — output is byte-identical to today's on the healthy path. <details><summary>Verification against live data</summary> Same tick, same repo, before vs. after: ``` # anchor query — before (--limit 10, no conclusion filter) {"conclusion":"failure","createdAt":"2026-08-04T23:10:57Z","databaseId":30959115609} # anchor query — after (--limit 50, conclusion == "success") {"conclusion":"success","createdAt":"2026-08-04T08:42:12Z","databaseId":30893072876} ``` `TARGET_REPO=numbagg/numbagg ./list-recent-runs.sh` after the change returns 5 runs instead of 1, preceded by: ``` WARNING: the last successful 'review-reviewers' run started 2026-08-04T08:42:12Z, more than 6h back. Window floored at 2026-08-04T17:47:00Z; runs that completed before it are NOT in this list. Record a coverage gap, not an all-clear. ``` The recovered anchor (08:42:12Z) matches the gist's last recorded coverage boundary exactly. Also exercised: the no-successful-run branch (warns, floors at the cap, exits 0) and the unset-`GITHUB_WORKFLOW` branch (recovery skipped, unchanged). `shellcheck` clean; `bash -n` clean. Also exercised after the review follow-ups: the fetch-limit truncation warning (via a copy with `RUN_LIMIT=5`) and the anchor-query failure path (exits 1 rather than degrading). See [the follow-up comment](#838 (comment)) for the full branch table. </details> ## Gates - **Evidence level**: Critical — a clearly wrong outcome (the analysis records an all-clear for a window it never examined, and the window is then unreachable), which acts on 1 occurrence. Occurrences: 1 this run, 0 historical (no prior gist entry records an outage of a length that would trigger it). - **Structural, not stochastic**: no decision point. The anchor query is deterministic; replayed ten times it picks the failed run ten times. - **Change type**: targeted fix — one predicate, one query limit, two stderr warnings. Normal evidence bar under Gate 2, met. - **Dedup**: checked open issues #831, #830, #829, #828, #827, #824, #822, #817, #816, #801, #799, #750, #624 and open PRs #826, #825, #823, #821, #819, #818, #809. #816 / #818 / #823 / #809 all concern outage *reporting* — naming the cause in the annotation, the stranded trigger, per-run comment dedup — none touch window recovery. Searching issues and PRs for `list-recent-runs` surfaces only the three merged predecessors #526, #753 and #784. Evidence log: https://gist.github.com/19b5ab297bb7ac7e1e9a44d595ccde0f --------- Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
Summary
Anchor
list-recent-runs.sh's completion window to the most recent intended cron tick (read from$GITHUB_EVENT_PATH) instead ofnow. Consecutivereview-reviewerscycles then tile exactly:[intended-1h, intended]→[intended, intended+1h], regardless of GHA scheduler delay.Replaces an earlier 2h-widening approach per maintainer feedback (#issuecomment-4472767685). Strictly cleaner: no overlap between consecutive cycles, no slack between them.
Background
The hourly
review-reviewersschedule (cron47 * * * *) is delayed 20–40 min during peak hours, producing back-to-back cycle gaps that systematically exceed the script's 1hnow-anchored window. Last 11review-reviewersruns onmax-sixty/tend(oldest → newest), interval to next:The four most recent intervals exceed 1h. In the analysis run 25954184316 (gap 99 min from the prior
review-reviewers), the script returned 4 of 17 in-window tend runs, missing PR #523 (release 0.0.21), PR #524 (regen workflows), and the 25952710024tend-review-runsfailure oncutover-to-codex— substantive activity recovered only via a directgh run listsweep.How it works
Non-schedule events (
workflow_dispatch, ad-hoc CLI use) and non-hourly crons fall back to the originalnow - 1hwindow — same behavior as before the change.Trade-offs
0 4 * * *,0,30 * * * *) keep the now-anchored fallback. Could extend later if needed.🤖 Generated with Claude Code