The recurrence bot reopened ~130 issues on evidence 7–23 days old — 'happening again' for faults last seen in August #4577

Activity

1. rbuergi commented on Sep 17, 2026

   

   rbuergi

   on Sep 17, 2026

   ContributorAuthor

   More actions

   Root cause, and the fix

   Systemorph/MeshWeaver.Plugins#2014.

   LogIncidentFiler.Reopen reopened a closed issue on exactly one condition — that the issue was
   closed. Nothing compared against when the fault was last seen. So the line the comment prints,
   "this fault is happening again", was never a statement about the fault. It was a statement about
   the ticket: "this was closed, and I am commenting now."

   That was safe while the only thing that could produce a comment was a fresh report. The
   comment-debt clock (Plugins#1836) removed that implication on purpose: an incident whose fault has
   stopped still owes its issue a final count, so the clock pays old debt on a timer, stamped in
   fault time, precisely for incidents where nothing new has happened. Each late comment then met
   "closed ⇒ reopen" — and that is the whole wave.

   Neither side's tests could see it: the clock's tests assert the comment lands (it did), and the
   reopen test asserted a reopen on a fixture whose incident bore no relation to the close time at all.
   The defect lived in the seam.

   Reopening now requires positive evidence — issue.ClosedAt is known and incident.LastSeen
   is after it. GitHub's closed_at was already mapped and the reopen path was already reading the
   issue, so it costs no extra call. Both unprovable cases leave the issue closed and say which one they
   are, in the comment and in the log.

   The comment still posts in every case: the occurrences are real history, and the comment is what
   settles the debt — suppressing it would leave the clock re-asking for that incident forever.

   The ~130 already reopened

   Untouched by that PR, and still open. Re-closing them is an outward-facing act on tickets other
   people can see, so it is a call for @rbuergi, not something to fold into a code fix — and it is not
   the bulk-close I argued against above: the proposal is to revert the bot's action only where the
   same evidence the new rule uses says the fault predates the close, leaving every issue a human
   touched, and every one whose fault really did recur, exactly as it is. I am measuring that set now
   and will post the list before anything is changed.
2. rbuergi commented on Sep 17, 2026

   

   rbuergi

   on Sep 17, 2026

   ContributorAuthor

   More actions

   Correction: the wave was 125 reopens, and 53 of them were wrong — not all of them

   I measured it rather than leaving my earlier estimate standing. Over all 166 open issues, reading
   per issue the most recent reopened event and actor, the last closed event before it, and the
   Last seen: line of the newest recurrence comment — the same two timestamps the new rule compares:

   	count
   last reopened by systemorph-com[bot]	136
   …of those, reopened on 2026-09-17	128 (125 inside the 07:4x–07:5xZ sweep)
   newest occurrence PREDATES the close it reverted	53
   newest occurrence postdates the close — a real recurrence	75
   undecidable from the ticket (no close event, or no last-seen line)	the remainder, reported as undecidable, never folded into either side

   So my "~130 reopened over stale faults" above was an over-claim: the sweep was that size, the wrong
   subset is 53. The oldest of those was last seen 2026-08-10 — five weeks before the close it
   reverted.

   That makes the defect sharper rather than smaller. It is not "the bot reopens too much" — 75 of these
   reopens were correct, if belated. It is that the bot could not tell the two cases apart and printed
   the same sentence about both, which is what made the whole batch unreadable: every one had to be
   opened to find out which kind it was.

   On remediation: the same per-issue read is what would make reverting an evidence-based undo of
   the bot's action rather than the bulk close I argued against — it names which of the two cases each
   issue is, and leaves alone every issue a human last touched and every one whose fault genuinely
   recurred. The candidate set is those 53. I am not changing any issue state; that is @rbuergi's call,
   and the list is ready when wanted.
3. rbuergi commented on Sep 17, 2026

   

   rbuergi

   on Sep 17, 2026

   ContributorAuthor

   More actions

   The 53 are re-closed — independently re-measured first, then reverted one by one

   Re-measured from GitHub rather than taken from the count above, 2026-09-17 11:1xZ:

   	count
   reopened events by systemorph-com[bot] on 2026-09-17	129 (125 inside 07:51–07:53Z; 4 outside)
   newest occurrence predates the close it reverted	53
   newest occurrence postdates the close — a real, belated recurrence	76
   undecidable (no close event, or no Last seen: line)	0
   of the 129: any event or human comment after the reopen	0

   Independent reproduction of the 53/76 split above — per issue, the last reopened event, the last
   closed event before it, and the maximum Last seen: over every comment the bot posted today (not
   just the reopen comment, so a genuine recurrence posted later could not be missed).

   Re-closed, with the original close reason preserved (49 × completed, 3 × duplicate, 1 × not planned),
   each carrying a comment naming this issue, its own two timestamps and the rule from
   Systemorph/MeshWeaver.Plugins#2014:

   #1116, #1120, #1130, #1194, #1195, #1245, #1312, #1313, #1316, #1317, #1542, #1544, #1563, #1642, #1657, #1658, #1677, #1678, #1684, #1693, #1748, #1799, #1800, #1801, #2131, #2139, #2163, #2172, #2176, #2177, #2320, #2406, #2510, #2532, #2621, #2638, #2885, #2897, #2918, #3046, #3121, #3518, #3656, #3780, #3781, #3783, #3784, #3876, #3983, #3984, #4067, #4068, #4462

   Conservative by construction: no issue was touched whose last-seen postdates its close, none that a person
   had touched since the reopen (there were none), and none closed in the 2026-09-10 bulk sweep — the
   oldest close restored is 2026-08-10T12:19:37Z, the newest 2026-09-16T22:40:03Z. Nothing is asserted about
   any of the underlying defects; if one occurs again, the next occurrence postdates this close and reopens
   the issue on evidence that means what it says. Core's open-issue count is 97 (PRs excluded), from 164
   after the wave.

   🚨 What is still owed: the fix is NOT merged, so the wave can repeat

   Systemorph/MeshWeaver.Plugins#2014 is open, not merged (created 09:12Z, mergeable_state: blocked,
   auto-merge armed). It is red on the Plugins platform-ceiling deadlock, not on anything in its own diff:

   resolve-gate-platform: the platform volume at '/opt/platform' does not carry set 3.0.0-ci.8767
   (core CD #8767) … and never will: core CD #8767 is below the OLDEST set kept (#8802)
   

   (run 35209513859, gate shards 1–4, 10:34:48Z — the #4491 / Systemorph/Memex#329 ceiling problem.)

   So the deployed LogIncidentFiler.Reopen still reopens on "the issue is closed" alone, and the next late
   comment from the debt clock on any closed incident reopens it again. This issue closes when #2014 merges
   and the log watcher runs it — not when the tickets look tidy.

   _{Measured and remediated 2026-09-17 by Claude Opus 5 (1M context) on the maintainer's triage pass.}
4. rbuergi commented on Sep 17, 2026

   

   rbuergi

   on Sep 17, 2026

   Last edited by rbuergi

   ContributorAuthor

   More actions

   The wave's residue measured, one issue at a time — and it does not partition the way unwinding it assumed.

   Full write-up, method and a 51-row evidence table: Doc/Architecture/ReadingARecurrenceReopen, in #4637.

   Denominator. Of core's 102 open issues, 51 carry a reopened event inside the 07:51:00Z–07:53:59Z burst, all closed before 2026-09-17 (50 by rbuergi, 1 by github-actions[bot]). Eight further 2026-09-17 reopens sit outside the burst — one of them by a human (#1198) — and were left alone. The rest of the ~130 had already been unwound by earlier sessions.

   First axis — the predicate that authorises a re-close (Plugins#2014, LastSeen > ClosedAt), taken from the bot's own Last seen: in the comment it posted alongside each reopen:

   	count
   newest occurrence at or before the close — the reopen was wrong	0
   newest occurrence after the close	51

   So zero qualified and zero were closed. The residue is already clean by that rule.

   Second axis — recency, the predicate that is missing. 3 were seen today, 7 within a week, and 41 had been quiet for 8 to 37.4 days when the bot said "happening again". That is the dishonesty in the count, and no proposed close predicate reaches it: those faults were seen after their close, and then stopped.

   The three that are live — #1549, #2299, Systemorph/MeshWeaver.Plugins#2155 — each have an occurrence on 2026-09-17 (11:11Z, 11:28Z, 14:25Z) from memex-portal-deployment-6c967c7c4c-*, the running replica. They stay open; they are the wave's entire signal. ⚠️ What each close asserted differs, and that decides what the recurrence means: #1549 closed with "the defect this issue converged on is fixed and deployed" — so a delivered fix did not hold; Systemorph/MeshWeaver.Plugins#2155 closed as "a single transient with no code defect identified" — no fix exists to recur against, and what today falsifies is the "single transient" premise (the record now reads 1,069); #2299 closed with no comment and no linked fix at all — today is the first evidence either way. Only one of the three is evidence that a fix failed.

   Awaiting delivery is empty in this wave, by construction: the newest close in the population is 2026-09-08, while the portals run afde4ea (09-15) and c84c6c0 (09-12). Every fix that closed a wave issue is in both running images by at least four days, so no wave issue can be excused as delivery lag — which establishes only that if a close carried a fix, it is running; whether it did is answered per issue.

   🚨 A third predicate this wave points at, stated as inferred. 15 of the 51 have their ONLY post-close evidence within 17 hours of the close and nothing in the weeks since. The likely mechanism is the close racing the roll — a fix closes its issue at MERGE while the old replicas keep emitting until DELIVERY — which would make the right test "was it seen on an image containing the fix", not "was it seen after the close". That would clear those 15 without weakening the rule for the three that are live. Confirming it needs a pod→image mapping the incident comments do not carry.

   The honest triple: 0 wrongly reopened and re-closed · 3 genuinely recurring (+7 within a week) · 0 awaiting delivery in the wave — plus 41 reopened on post-close but 8-to-37-day-old evidence, which is the number the second predicate has to remove.

   _{Read-only over the GitHub REST API. No issue in the wave was closed, reopened, commented on or labelled.}

   _{Edited to correct an over-claim in the first version: I had written that all three live issues recur "against their own fix". Checking each closing comment, only #1549's close carried a stated, deployed fix.}
5. rbuergi

   added a commit that references this issue on Sep 17, 2026

   The two wave tickets are ONE burst measured twice — say so, so nobody…

   ...

   585ec1a
6. meshweaver-cloud

   closed this as completedon Sep 18, 2026