Skip to content

fix(list-recent-runs): recover dropped cron ticks via previous-run anchor - #753

Merged
max-sixty merged 1 commit into
mainfrom
hourly/review-28652500581
Jul 10, 2026
Merged

max-sixty merged 1 commit into
mainfrom
hourly/review-28652500581

Conversation

@tend-agent

Copy link
Copy Markdown
Collaborator

Problem

list-recent-runs.sh anchors its completion window to the most recent intended cron tick and floors it at intended - 3600. This tiles exactly — [intended-1h, intended] then [intended, intended+1h] — only when consecutive cron ticks actually fire. GHA doesn't just delay scheduled ticks (which the tick-anchoring in #526 handles well); it drops them. A dropped tick leaves that hour's completions in the gap between the previous and next cycle's windows, and no run ever analyzes them.

#526 named this exact "skipped-tick edge case" and deferred it as "rare under sustained outage," because the only fix on the table then was the 2h-overlap approach the maintainer had rejected for re-analyzing runs. This run is direct evidence it is not rare and the miss is invisible — a dropped tick produces no error, so the skipped outcomes just never get audited.

Evidence (this run)

Analyzing max-sixty/worktrunk, review-reviewers fired at 07:50Z (07:47 tick) then 09:46Z (09:47 tick) — GHA dropped the 08:47 tick. The window for the 09:47 cycle floored at 08:47, leaving [07:47, 08:47] covered by no cycle. list-recent-runs.sh returned 3 of the 10 tend-* runs actually in [07:47, 09:47]. The orphaned hour contained real, substantive bot output:

  • worktrunk#3355 — a well-sourced answer on Windows Defender code-signing (comment 4874219842, 08:16Z).
  • worktrunk#3357 — a full review cycle: COMMENTED flagging codecov/patch, author pushed coverage, bot APPROVED on the fixed commit (08:15Z–08:23Z).

Both were healthy outcomes this time — but had either gone wrong, no review-reviewers cycle would have caught it. (I covered the gap manually for this run's audit; the gist records the full window.)

Fix

Floor the window at the previous actual completed run's intended tick instead of assuming the previous tick fired. #526's discussion weighed only two options — the intended fire time (not exposed by GHA) vs 2h overlap (rejected) — and missed a third: the previous run's own start time is queryable, and rounding it down to its cron tick recovers exactly where the last cycle's window ended.

  • Every tick fires → previous intended tick == intended - 3600 → byte-identical to today's behavior, no overlap (the constraint fix(list-recent-runs): anchor completion window to intended cron tick #526 set).
  • A tick was dropped → floor reaches back to the last actual run's tick, covering the orphaned hour with no double-analysis.
  • Capped at 6h reach-back so a sustained outage can't create an unbounded window. Falls back cleanly (gh failure, no GITHUB_WORKFLOW) to the current floor.

Verified in this CI session: the patched script returns all 10 in-window runs (was 3); with consecutive ticks it resolves the floor to intended - 3600 unchanged.

Gate assessment

  • Type: targeted fix to one branch of the window logic (not a structural rewrite).
  • Classification: structural + deterministic — a dropped tick always produces the gap; not a stochastic model lapse.
  • Occurrences: 1 confirmed gap-with-missed-outcomes this run. Below the normal 2–3 bar, but acted on because the failure mode is self-concealing: future cycles use the same window and won't re-detect the gap, so evidence cannot accumulate — "wait for occurrence 2" would sit at 1 while silently under-covering. This is tend's own audit tooling, the fix reduces to a no-op in the common case, and human review is the backstop.

Evidence log: https://gist.github.com/b923bbc52f3c6606931d59abdae616a3

…chor

The hourly completion window floors at intended-1h, which tiles exactly only
when consecutive cron ticks fire. GHA drops ticks, not just delays them; a
dropped tick leaves its hour uncovered by any cycle — the skipped-tick case
#526 deferred as "rare under sustained outage." It is not rare, and it silently
drops outcome audits.

Floor the window at the previous *actual* completed run's intended tick instead
of assuming the previous tick fired. When every tick fires this is byte-identical
(no overlap, as #526 required); when one is dropped it reaches back to cover the
orphaned hour. Capped at 6h.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@tend-agent tend-agent added the claude-behavior Behavioral issues found by review-reviewers label Jul 3, 2026
@max-sixty
max-sixty merged commit 8a3d73d into main Jul 10, 2026
5 checks passed
@max-sixty
max-sixty deleted the hourly/review-28652500581 branch July 10, 2026 23:26
@max-sixty max-sixty mentioned this pull request Jul 16, 2026
2 of 3 tasks
max-sixty added a commit that referenced this pull request Jul 16, 2026
## Summary

Version bump + CHANGELOG for 0.1.11. See
[CHANGELOG.md](https://github.com/max-sixty/tend/blob/release/CHANGELOG.md#0111)
for the full notes.

13 commits since 0.1.10:
- Adopter sandbox levers (`sandbox_path`/`sandbox_env`/`sandbox_setup`,
#768)
- Sandbox `PATH` derived from the runner's own `PATH` (#767)
- Invited upstream contributions (#770)
- `tend-mention` self-comment no-op skip (#751)
- Actionable self-review handling (#762, #775)
- `tend-outage` dedup self-heal (#744)
- Dropped-cron-tick recovery in `list-recent-runs` (#753)
- Drop self-authored attribution sign-offs (#773)
- Docs fixes (#776, #771)
- `claude_version` bump (#774)

## Test plan

- [x] `wt test` (275 passed)
- [x] `pre-commit run --all-files` (all hooks passed)
- [ ] CI green on this PR

> _This was written by Claude Code on behalf of Maximilian Roos_
max-sixty pushed a commit to max-sixty/worktrunk that referenced this pull request Jul 17, 2026
Automated nightly regeneration of tend's workflow files, picking up the
upstream release.

**tend version:** 0.1.10 → 0.1.11

**Notable changes** (consumer-relevant):

- Skip no-op `tend-mention` sessions triggered by the bot's own comments
— a new pre-mention guard in the generated `tend-mention.yaml`
short-circuits before the engagement heuristics (max-sixty/tend#751).
- Respond to actionable bot self-reviews instead of exiting as a
self-loop: a review the review workflow leaves on its own PR is the
bot's reviewer role, so it gets actioned rather than silently skipped
(max-sixty/tend#762).
- Sandbox: adopter levers to reach inside the Claude-family sandbox, and
derive the sandbox `PATH` from the runner's `PATH` (max-sixty/tend#768,
max-sixty/tend#767).
- `running-in-ci`: permit maintainer-invited upstream contributions, and
prohibit self-authored attribution sign-offs (max-sixty/tend#770,
max-sixty/tend#773).
- Action reliability: self-heal duplicate `tend-outage` issues from
concurrent leg failures, and recover dropped cron ticks in
list-recent-runs via a previous-run anchor (max-sixty/tend#744,
max-sixty/tend#753).

Compare: max-sixty/tend@0.1.10...0.1.11

Co-authored-by: worktrunk-bot <254187624+worktrunk-bot@users.noreply.github.com>
max-sixty pushed a commit that referenced this pull request Aug 12, 2026
… and stop the fetch limit re-truncating it (#838)

`list-recent-runs.sh`'s dropped-tick recovery anchors the window floor
on the previous **completed** run of the analyzing workflow. `completed`
is a status, not a conclusion, so a run that failed before its agent
produced any analysis advances the anchor exactly as if it had covered
its hour. After an outage the next green run therefore resumes one tick
back and reports an all-clear for a window nothing ever looked at — and
the gap is unrecoverable, because the following run's floor advances
past it too.

This is not hypothetical: it just consumed ~15 hours of
`review-reviewers` coverage.

## What happened

Every `review-reviewers` run from
[30897445507](https://github.com/max-sixty/tend/actions/runs/30897445507)
(09:41Z) through
[30959115609](https://github.com/max-sixty/tend/actions/runs/30959115609)
(23:10Z) failed — 16 consecutive runs, all five matrix legs, `claude -p`
exiting 1 after ~4s. The `result` event in every leg's session log is
identical: `is_error: true`, `num_turns: 1`, `result: "You've hit your
weekly limit · resets 12am (UTC)"`, with `token-usage.json` all zeros.
The agent never started, so none of those runs analyzed anything. (The
outage itself is already tracked by #831 and the annotation-legibility
work in #816 / #818 — this PR is only about the window the outage
swallowed.)

The last run that did analyze anything is
[30893072876](https://github.com/max-sixty/tend/actions/runs/30893072876)
at 08:42Z, which is also the last entry in the `numbagg/numbagg`
evidence gist. At this run's tick (23:47Z intended), the recovery query
picked
`{"conclusion":"failure","createdAt":"2026-08-04T23:10:57Z","databaseId":30959115609}`
and floored the window at 22:47Z, so `list-recent-runs.sh` returned
exactly one row: a `tend-notifications` pre-check no-op.

Everything `numbagg-bot` actually did in the swallowed window sat
outside that floor — PRs
[#719](numbagg/numbagg#719) and
[#720](numbagg/numbagg#720), issue
[#721](numbagg/numbagg#721), three reviews and
four inline comments on #719, and comments on #716, #721 and #1. That is
the densest window numbagg has had in days, and per the skill's Step 1
("If empty, record the run as all-clear … then skip to Step 6") it would
have been recorded as a quiet hour. I only found it by diffing the run
list against the gist's last recorded boundary by hand.

## The change

Anchor on the previous **successful** run rather than any completed one,
filtering server-side with `--status success` (the flag takes
conclusions as well as statuses), and when the existing 6h cap clamps
the recovered floor, say so on stderr so the caller records a coverage
gap instead of a false all-clear. A partially-failed matrix run counts
as a failure here, which only ever widens the window — overlap is
re-offered work the caller dedups against its own evidence log, whereas
a gap is silently unanalyzed.

Widening the window then exposed a second truncation one layer down, so
the fetch loop moves with it. It passed `--limit 50` while a recovered
window spans up to 8h, and `gh run list` returns newest-first — so a
workflow over the limit silently drops its *oldest* runs, which are
exactly the ones in the gap the anchor just reached back for. On
`max-sixty/tend`, `tend-mention` alone produces 57 runs in an 8h window.
The limit is now 200, and a workflow returning exactly the limit warns
rather than truncating in silence.

Two smaller pieces: the anchor query routes through `gh_retry` and exits
non-zero instead of `2>/dev/null || true`, so a transient API error can
no longer masquerade as "no successful run" and emit a confident warning
naming a cause that didn't happen; and `review-reviewers/SKILL.md`,
which still said "if empty, record all-clear" with nothing about the
stderr warnings, now treats any `WARNING:` as a coverage gap — without
that, an agent following the skill literally would print the warning and
record an all-clear anyway.

When every tick fires and succeeds the anchor is the previous tick,
`prev_intended == intended - 3600 == COMPLETED_AFTER`, and the
comparison is a no-op — output is byte-identical to today's on the
healthy path.

<details><summary>Verification against live data</summary>

Same tick, same repo, before vs. after:

```
# anchor query — before (--limit 10, no conclusion filter)
{"conclusion":"failure","createdAt":"2026-08-04T23:10:57Z","databaseId":30959115609}

# anchor query — after (--limit 50, conclusion == "success")
{"conclusion":"success","createdAt":"2026-08-04T08:42:12Z","databaseId":30893072876}
```

`TARGET_REPO=numbagg/numbagg ./list-recent-runs.sh` after the change
returns 5 runs instead of 1, preceded by:

```
WARNING: the last successful 'review-reviewers' run started 2026-08-04T08:42:12Z, more than 6h back. Window floored at 2026-08-04T17:47:00Z; runs that completed before it are NOT in this list. Record a coverage gap, not an all-clear.
```

The recovered anchor (08:42:12Z) matches the gist's last recorded
coverage boundary exactly. Also exercised: the no-successful-run branch
(warns, floors at the cap, exits 0) and the unset-`GITHUB_WORKFLOW`
branch (recovery skipped, unchanged). `shellcheck` clean; `bash -n`
clean. Also exercised after the review follow-ups: the fetch-limit
truncation warning (via a copy with `RUN_LIMIT=5`) and the anchor-query
failure path (exits 1 rather than degrading). See [the follow-up
comment](#838 (comment))
for the full branch table.

</details>

## Gates

- **Evidence level**: Critical — a clearly wrong outcome (the analysis
records an all-clear for a window it never examined, and the window is
then unreachable), which acts on 1 occurrence. Occurrences: 1 this run,
0 historical (no prior gist entry records an outage of a length that
would trigger it).
- **Structural, not stochastic**: no decision point. The anchor query is
deterministic; replayed ten times it picks the failed run ten times.
- **Change type**: targeted fix — one predicate, one query limit, two
stderr warnings. Normal evidence bar under Gate 2, met.
- **Dedup**: checked open issues #831, #830, #829, #828, #827, #824,
#822, #817, #816, #801, #799, #750, #624 and open PRs #826, #825, #823,
#821, #819, #818, #809. #816 / #818 / #823 / #809 all concern outage
*reporting* — naming the cause in the annotation, the stranded trigger,
per-run comment dedup — none touch window recovery. Searching issues and
PRs for `list-recent-runs` surfaces only the three merged predecessors
#526, #753 and #784.

Evidence log: https://gist.github.com/19b5ab297bb7ac7e1e9a44d595ccde0f

---------

Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-behavior Behavioral issues found by review-reviewers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants