Skip to content

Restore Hindsight consolidation worker #4

Description

@linear-code

Problem

Hindsight retention succeeds, but observation consolidation has stopped. The omp bank has 19,933+ unconsolidated facts; the latest consolidation has been pending since 2026-07-20 and the preceding operation is stuck processing at 24/14,536.

Diagnosis

  • API and PostgreSQL health are green.
  • The deployed ai-dev-env Helm values use Hindsight chart 0.8.5 with worker.enabled: false, leaving the single API pod's internal worker responsible for async jobs.
  • The API pod has restarted since the stalled operation began.
  • Hindsight 0.8.5 only recovers processing tasks owned by the current worker ID, so an operation owned by the previous API process remains orphaned.
  • A second consolidation remains pending even while retain tasks complete, so consolidation has no reliable dedicated capacity.
  • Hindsight 0.8.6 includes the Kubernetes worker-port fix and newer worker scheduling safeguards.

Scope

Upgrade the chart to 0.8.6, enable a dedicated worker with explicit resources, safely recover the orphaned consolidation operation, and retain automatic consolidation.

Acceptance criteria

  • Dedicated Hindsight worker is deployed and healthy.
  • The orphaned processing operation no longer blocks consolidation.
  • A consolidation operation advances its progress timestamp and processed count.
  • pending_consolidation decreases from the pre-repair baseline.
  • Retain, recall, and API health continue to work.
  • Deployment evidence and recovery procedure are recorded.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions