Repository navigation
X discovery projects past the accepted $40 monthly spend line #179
Description
Activity
🤖 ISSUE TRIAGE
Harlan Agent Kit posted this automated triage. It is not Harlan's personal assessment or commitment. AI open source policy.
- Route: Wait to implement
- Difficulty: 3/5
- Impact: 4/5
- Reproduction: Yes
- Codebase review: Not needed
- Summary: The issue's premise is obsolete. It was auto-opened 2026-09-07 when the check-in projected $42.60/mo because discovery saturated the 2000-read budget (1992/2000) via repeated first-page reads. That defect was fixed by merged fix(x-ingest): resume unread pages across discovery runs #409 (resume un
- Next action: Harlan compares X Developer Console actual spend and resource counts for the complete UTC days 2026-10-07 through 2026-10-10 against the $40/month line (repo projections: $13.20, $21.75, $7.25, $30.90, $28.95). If actuals are at or under the line, close X discovery projects past the accepted $40 monthly spend line #179 as resolved by fix(x-ingest): resume unread pages across discovery runs #409/fix(checkin): detect unusable X discovery read budgets #410, giving that reason so the daily-checkin routine stops offering the budget-lowering candidate. If actuals exceed $40, open the narrower-discovery-coverage decision and measure each X_DISCOVERY_QUERY arm's useful posts per read before changing the query or the budget.
- Related pull requests: fix(x-ingest): resume unread pages across discovery runs #409: Merged 2026-10-05; resumed unread pages across discovery runs, removing the repeated first-page reads that saturated the 2000-read budget and drove the over-line projection this issue reports.; fix(checkin): detect unusable X discovery read budgets #410: Merged 2026-10-05; shipped from this issue's investigation, treats fewer than ten remaining reads as exhausted so the check-in no longer misreads an eight-read remainder as usable budget.
Agent triage by Codex, 2026-10-05. This is an agent assessment.
Keep as the canonical X budget and discovery-scope decision. Difficulty 3/5, impact 4/5.
Production reads show 1992/2000 discovery reads spent today and 32 hot posts. The current collector formula projects $303.60/month. This is an estimate, not a billing statement.
A lower read ceiling alone cannot solve the issue. Current discovery saturates the ceiling. Reducing it grows the backlog beyond X's seven-day retention window.
Next step: choose narrower discovery coverage within the accepted $40/month line, or explicitly revise that line. Narrower coverage is the recommended direction under VISION.md's cost constraint. Do not implement an arbitrary 250-read ceiling.
Telemetry follow-up: eight remaining reads cannot satisfy X's ten-result minimum. The exhaustion signal should use the minimum request size.
Harlan Agent Kit wrote this investigation. It is an agent assessment.
The current projection does not establish actual X spend.
X pricing lists $0.005 per Post and $0.010 per User.
X usually deduplicates each resource within its UTC day. This guarantee has exceptions.
Our 2,000 ceiling counts returned Posts, including repeat IDs. It does not count invoice charges.
Both discovery and refresh request author expansions. Their billable User IDs are not recorded separately.Read-only production measurements on 2026-10-05, filtered to platform X:
UTC day New stored X Posts Distinct observed X Posts New-Post authors New X Posts linked to Skills 2026-10-04 416 498 334 67 2026-10-05, partial 33 499 32 7 Discovery and refresh share observations. These counts cannot separate their billed usage.
This table corrects earlier measurements that included Bluesky in the shared tables.
The earlier 463 new Posts covered a rolling 24 hours and both platforms.
It cannot be compared directly with today's 1,992 returned X reads.
Text extracts do not identify which query arm returned a Post.
Literal GitHub links in those extracts cannot establish query selectivity.#409 addresses repeated first-page reads. Measure complete UTC days after that fix.
A lower ceiling alone can hide unread results until X's seven-day search retention expires.Recommended sequence:
- Ship fix(x-ingest): resume unread pages across discovery runs #409 and fix(checkin): detect unusable X discovery read budgets #410. Preserve the current query during measurement.
- Compare three complete UTC days with the Developer Console's actual spend and resource counts.
- If actual spend exceeds $40 monthly, measure each query arm's useful Posts before narrowing it.
#410 treats fewer than ten remaining returned reads as exhausted.
It records distinct observed X Posts separately from the Post-only spend estimate.
That exhaustion flag warns of possible backlog. It does not prove unread results remain.The remaining decision is query coverage if actual billed spend exceeds $40 after continuation recovery.
Changing refresh timing alone cannot remove discovery charges. A provider spending limit pauses API access when reached.- added and removed
on Oct 5, 2026 Codex wrote this agent investigation update, October 5 at 08:15 UTC.
#409 and #410 are merged. #410's final pull request checks passed.
At 08:31 UTC, the telemetry change was verified live on descendant commit48b0590d.
Deploy 37283498548 and production smoke passed.
Cloudflare serves version7d52a867at 100% traffic.
A complete measurement period is still needed.At 08:11 UTC, the returned-read budget remained 1,992 of 2,000.
Eight remaining reads cannot satisfy the minimum request size.
Distinct observed X Posts remain 498 for October 4 and 499 for October 5.
No completed post-pagination UTC day is available yet.The next budget reset is October 6 at 00:00 UTC, 11:00 Melbourne.
The first complete day can be sampled October 7 at 00:15 UTC, 11:15 Melbourne.
Check cursor and window exhaustion to distinguish catch-up from steady discovery.
Compare three complete days with actual Developer Console resource counts and spend.The budget and query remain unchanged pending those measurements.
X discovery projects $42.60/mo, over the $40 accepted line, because the 2000-read
DAILY_DISCOVERY_READ_BUDGETnever binds at the observed 284 reads/day and needs lowering to about 250.Target:
shared/server/x-ingest.tsVerify with:
pnpm vitest run test/unit/x-ingest.test.tsEstimated to change 1 file.