Keep pipeline freshness independent of large article recovery scans #337

Problem

Production Articles and Sources boards still display “Stale data · Verified through Oct 30, 2026, 10:47 AM” after the 0.1.8 rollout. The finite historical import is complete; ongoing current-card verification is a separate bottleneck. Ready pods and advancing cursors have not established fresh boards.

Confirmed evidence

• The displayed 05:17:50 UTC boundary comes from one shared cards progress row. It advances only after source and reverse-target passes finish for all five populations. Sources is therefore held behind Articles.
• At 14:41 UTC, canonical current targets included 14,807 Articles, 1,321 Sources and 1,276 relationship proposals. The batch is 100, producing at least 350 current pages before weighted competition.
• _card_targets omits the immutable shard from its (policy, kind, shard, entity_id) key lookup. Same-ID production probes returned identical complete rows: 100 IDs improved from 414 ms to 192 ms, and 200 from 584 ms to 231 ms. The 100-ID primary-key plan reduced selected granules from 28 to 6. All 196,712 canonical FINAL rows matched the UUID-derived shard invariant.
• Fingerprint churn also exceeds semantic card changes; its exact components are under investigation. Global catalog revisions did not change during the measured window, so catalog churn is not yet a confirmed cause.

Required behavior and validation

• Each card population has an independent source-plus-target cycle and conservative coverage boundary. A Source board combines Source and Candidate coverage; Articles uses Article coverage; aggregate operations views combine their actual populations.
• Preserve reverse-target deletion detection, interrupted-page acknowledgement, dirty-owner delivery across upgrades and queue priority. Never substitute request/completion time for the verified source boundary or widen freshness limits.
• Bound canonical lookups by derived immutable shards while preserving FINAL and existing safety checks.
• Test realistic retained/current populations, restart and continuous dirty-work behavior, cold and steady recovery, and native database query plans. Verify sustained live freshness before calling the production issue resolved.

No production history/checkpoints have been reset and no jobs or AI calls have been replayed for these probes.

Additional confirmed builder cost

With the production catalog counts (4,450 topics and 53,555 tags), native PostgreSQL 18.6 profiling found repeated catalog loading and ranking-index construction on every successful-analysis card page. A revision-validated cache owned by the serialized reconciler builder reduces warm 100-card builds from approximately six seconds to three, while recomputing article/job readiness and invalidating before a changed catalog is loaded. Native tests cover changed job inputs, catalog edits, loader failure and cancellation without concurrent builders.

These are local AMD64 measurements with explicit CPU/resource limits, not production ARM64 or whole-cycle freshness proof. Batch size and freshness limits remain unchanged. Live sustained coverage remains the acceptance gate.