Derivative symbols trigger repeated 10-year daily history requests, causing sustained HTTP 429 rate limiting

[marketcalls]

Reported downstream in marketcalls/openalgo#1746: AmiBroker requests through the plugin generate repeated HTTP 429 responses when charting NIFTY options and index futures, with requests spanning 2016 to 2026. Root cause is in this plugin, not in the OpenAlgo server - the server components only forward the dates they are given.

Symptom

For derivative symbols the plugin issues, per symbol, per refresh:

symbol=NIFTY... start=2026-07-05 end=2026-08-04   (30-day 1m)
symbol=NIFTY... start=2016-08-06 end=2026-08-04   (10-year daily)

Result: sustained rate limiting, long download times, repeated retry cycles and unnecessary broker API load. Equity symbols are unaffected.

Reported on OpenAlgo 2.0.1.8, Fyers broker, AmiBroker bar count 10,000.

Root cause

1. MIN_DAILY_BARS = 250 can never be reached by a derivative contract

OpenAlgoHistory.inc, daily gap-detection branch:

const int MIN_DAILY_BARS = 250;               // ~1 year of trading days
// CHECK 1: Do we have enough bars for proper analysis?
if (lastMatchingBarIndex < MIN_DAILY_BARS)
    startTime = todayDate - CTimeSpan(3650, 0, 0, 0);   // 10 years

The heuristic reads "fewer than 250 daily bars means fresh symbol, do a full initial load." Valid for equities. Invalid for derivatives: a NIFTY weekly option exists for weeks and a futures contract for months, so the bar count never reaches 250. The branch therefore fires on every single refresh and the requested range is permanently pinned to today - 3650 days.

2016-08-06 to 2026-08-04 is exactly 3650 days, which identifies this branch as the source.

The plugin has no notion of contract start date or expiry, so nothing else bounds the range.

2. Intraday requests also queue the daily fetch

OpenAlgoAmiBroker.inc, GetQuotesEx 1-minute branch:

if (bDailyStale  && !pCache->bDailyFetchInProgress)  { ... bNeedDaily  = TRUE; }
if (bOneMinStale && !pCache->bOneMinFetchInProgress) { ... bNeedOneMin = TRUE; }
...
if (bNeedOneMin) QueueHttpFetch(ticker, 60, 0);
if (bNeedDaily)  QueueHttpFetch(ticker, 86400, 0);

A 1-minute chart therefore emits both the 30-day 1m request and the 10-year daily request. The two log lines above are these two jobs, not a duplicated request.

3. Failure paths never mark the cache as attempted, producing an instant-retry loop

OpenAlgoWorkers.inc, HttpWorkerThreadProc:

if (nResult > 0) { ...; pCache->lastDailyFetch = now; }

GetOpenAlgoHistory returns nLastValid + 1 on every failure path: non-200 status, dataArray.GetLength() < 10, timestamp parse miss, CInternetException. When the symbol has no cached bars nLastValid is -1, so the return value is 0.

nResult == 0 leaves the timestamp at 0, and GetQuotesEx treats that as stale:

BOOL bDailyStale = (pCache->lastDailyFetch == 0) || ((now - pCache->lastDailyFetch) > DAILY_CACHE_LIFETIME_MS);

so the symbol is re-queued on the next call with no delay and no backoff.

This is self-sustaining: OpenAlgo's /api/v1/history is limited to 10 per second, the plugin reads the resulting 429 as "no data", and retries immediately. Rate limiting causes faster retries, which cause more rate limiting.

The queue dedup in QueueHttpFetch only merges against jobs still waiting in the queue, so it gives no protection against this loop.

4. Amplification

Each 10-year daily request is split server-side into 300-day chunks (broker/fyers/api/data.py), so one plugin request becomes roughly 13 Fyers calls against a shared ~8 req/sec budget (Fyers documents 10/sec, 200/min, 100k/day per API key, and blocks the user for the rest of the day after exceeding the per-minute limit more than 3 times).

g_SymbolBarCache is never evicted during a session (CleanupSymbolBarCache runs only at shutdown), so every symbol charted once continues to be refreshed for the whole session. Browsing an option chain permanently enrols every strike visited.

Refresh cadence

Path	Interval
Daily cache (DAILY_CACHE_LIFETIME_MS)	1 hour
1-minute cache (g_nBackfillRefreshIntervalSec)	30s default, 5s minimum
After a failed or empty fetch	No TTL applies; repeats as fast as GetQuotesEx is called

Equity vs derivative

	Equity (e.g. SBIN)	NIFTY option / future
Daily bars available	Thousands	Tens
Passes the 250-bar check	Yes, after first load	Never
Subsequent daily requests	Incremental gap fill (startTime = lastBarDate)	Full 10 years, every hour
Broker returns no data	Rare	Common (expired/illiquid strikes)

Suggested direction for the fix

1. Bound the history range by the instrument's actual life rather than a fixed bar-count heuristic, or exempt derivative segments (NFO, BFO, CDS, MCX) from the MIN_DAILY_BARS initial-load branch.
2. Record fetch attempts on failure as well as success, so an error or empty response starts a backoff instead of an immediate retry. Distinguish "no data exists for this symbol" from "the request failed" and treat 429 with an explicit backoff.
3. Do not queue the 10-year daily fetch from an intraday-only request unless daily bars are actually needed.
4. Consider evicting symbols from g_SymbolBarCache once they are no longer charted or subscribed.

Affected files: OpenAlgoHistory.inc, OpenAlgoAmiBroker.inc, OpenAlgoWorkers.inc, Plugin.cpp.

Target: next plugin release.