Skip to content

A failed plan-force state read journals blocked rows that hold the plans for a day #4769

Closed

Description

@erikdarlingdata

The plan-force bot's state read has a 5-second deadline. Any fault returns a null state with a reason, and each candidate (up to 10) gets a blocked row. Those rows feed last_journaled, so the 24-hour cooldown then hides the real verdict for a day.

The row itself is honest. The cooldown ranks above the state blockers on purpose, and a test pins that order.

Where (as of 30cbbcf)

  • The deadline: Darling/PerformanceMonitor.Darling.Service/ServiceCommandDeadlines.cs:157.
  • The blocked rows: Darling/PerformanceMonitor.Darling.Service/PlanForceBot.cs:153 and :187-212.
  • last_journaled: Darling/PerformanceMonitor.Darling.Service/PgPlanForceActionStore.cs:236-243 and :317-324.
  • The order: PerformanceMonitor.Analysis/ForcePlanBotPolicy.cs:391-399, pinned by TheCooldown_StillOutranksTheStateBlockers (Darling/Darling.Tests/PlanForceBotTests.cs:479).

Fix

Keep that order, but give state_unavailable its own short window of about 1 hour instead of the 24-hour cooldown. If it is dropped from the cooldown altogether, a long outage floods the journal.

Test that must fail first

  • After a state-read failure, the candidate is judged again after the short window, not after 24 hours.
  • Repeated failures inside the window journal once.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions