Discard vendor outlier bars in intraday data

[AlexCatarino]

Problem

EODHD sporadically returns junk prints in the 1-minute intraday endpoint. The raw response for USDKRW on 2024-01-02 opens with:

{"timestamp":1704155040,"datetime":"2024-01-02 00:24:00","open":2.4,"high":2.4,"low":2.4,"close":2.4,"volume":1}
{"timestamp":1704161460,"datetime":"2024-01-02 02:11:00","open":1299.84,"high":1300.05,...,"volume":28}

An isolated bar with all four OHLC fields at 2.4, 1h47m before the first genuine bar of the session, on a day the pair traded around 1300. It is the only one of the day's 1182 bars deviating more than 10% from the median. The downloader only filtered nulls, so the bar was written verbatim into forex/oanda/minute/usdkrw/20240102_quote.zip.

The vendor volume is not a usable signal: that day has 30 volume-1 bars and the other 29 are legitimate quiet minutes.

Fix

Bars further than 25% from the median close of their own day are discarded, and each drop is logged.

• The reference is the daily median, so a genuine trend across the 120-day download window never trips the filter — each day is judged against itself.
• Days with fewer than 10 bars have no meaningful median and fall back to the median of the whole window. This is what catches a junk print landing alone on a day.
• The check spans all of O/H/L/C, since open and close are not guaranteed to sit inside the high/low of a malformed bar.
• A non-positive reference leaves the day untouched; dropping a whole day on degenerate input would be worse.

Threshold

Measured over the full USDKRW minute history, 5323 daily files / 4.8M bars:

Measurement	Value
Widest deviation of any bar from its daily median	10.5%
Widest deviation of a sparse day (<10 bars, 81 such days) from its window median	3.1%
The junk bar	99.8%

25% clears every real move with no false positives across the whole history.

Verification

Re-running the downloader for 20240102:

ERROR:: RemoveOutliers(): Discarding USDKRW outlier bar 20240102 05:24:00 O=2.4 H=2.4 L=2.4 C=2.4. Reference price level: 1309.48

The regenerated file is byte-identical to the previous output minus that row. 9 new unit tests cover the real 2024-01-02 case, zero and negative prices, the sparse-day fallback, chronological ordering, empty input, and two no-false-positive guards (the widest genuine intraday move, and a 120-day window drifting 1000 to 1595).

Six other days (20231124, 20231201, 20231214, 20231229, 20240312, 20240607) carry 5-10% spikes that look like the same family of vendor glitch but sit below this threshold. Catching them safely needs a different rule (deviation from neighboring bars rather than the daily median), so they are left alone here.

🤖 Generated with Claude Code