fix: report stored calendar gaps before targeted source acquisition #110

Reproduced public workflows

The acquisition range planner in update_manager is separate from the intraday classification utility repaired in PR100. Its undeclared intraday path still uses elapsed-day arithmetic, and declared calendars/feeds explicitly raise NotImplementedError outside daily ranges.

Using real SyntheticProvider minute bars at Friday July11,2025 19:59UTC and Monday July14 13:30UTC, ml4t.data.update_manager.GapDetector().detect_gaps(frame, frequency="minute") reports a gap from Saturday July12 19:59UTC through Sunday July13 13:30UTC with size_days=1. Expected no missing NYSE bars between these adjacent valid session boundaries.

With real FlatStorage and HiveStorage imports containing July7 13:31/13:32UTC observations, call detect_gaps_in_storage(storage,key,datetime(2025,7,7,13,30,tzinfo=UTC),datetime(2025,7,7,13,33,tzinfo=UTC),frequency="minute"). Expected two one-minute boundary ranges,13:30 and13:33. Actual first range ends July6 13:31, before its July7 start, and the trailing range begins a day after its end. The implementation subtracts/adds a day around intraday observations.

Three assertions fail cleanly under ordinary repository fixtures with no acquisition/storage/calendar mocks. Source be323a0, Python3.14.3, Polars1.43.2. Affected update_manager.py and utils/gaps.py are byte-identical to merged main74f4e483dfe3e57b259c65de751f6f2afa6e93b2. Existing validation changes are not this fix. DataManager import is used only to produce actual storage; validation is explicitly disabled for this planner reproduction.

Revised maintainer requirement: detection first, targeted acquisition second

The maintainer clarified the scope on2026-10-17. First implement one provider-independent mechanism: apply a declared trading calendar, frequency and timestamp/bar convention to a finite requested interval; compare expected observations with each dataset already stored on disk; report missing timestamps/ranges and exact observation counts without modifying data or contacting providers. Share authoritative schedules and the existing expected-grid arithmetic with daily/intraday detection, including leading/trailing gaps and wholly absent sessions/datasets. Preserve explicit argument > feed/stored metadata > NYSE and explicitNone disabled versus24/7 real schedule. Do not infer holidays from observed rows, count elapsed days as minute bars, generate synthetic prices or require provider support just to inspect a stored dataset.

Second, consume that report in a separately invoked targeted acquisition step for sources that support instrument/market and timestamp-range requests. Prioritize CME futures and cryptocurrency data, including Coin Metrics. Confirm the maintainer's vendor identity (Databento assumed pending confirmation), preserve dataset/contract/market/schema identity and query only the requested source/ranges. Qualify actual source-specific boundaries, pagination, historical availability and returned-row filtering. Keep unsupported capabilities and remaining absences explicit; do not promise all providers or arbitrary data types.

Databento OHLCV and Coin Metrics market candles legitimately omit intervals without trades, according to their official schema/API docs. Calendar-derived absence is therefore a missing scheduled observation for investigation, not proof of a recoverable vendor outage. An empty response does not authorize interpolation or a loop that keeps downloading indefinitely. Preserve original rows and metadata, and retain a post-download residual report. No new release/deployment or paid data call is implied by this issue.

Official references: https://databento.com/docs/schemas-and-data-formats/ohlcv and https://docs.coinmetrics.io/api/v4/ . Actual supported gates and failing-base/passing-fix disk/provider workflows govern completion; do not close after the report-only step while accepted targeted acquisition remains.