Expand disclosure latency providers - #120
Conversation
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_78d8df1b-d8f2-42de-8a43-c18fc4fb9e17) |
4400274 to
1bb938e
Compare
1bb938e to
2eb6e6c
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4400274851
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ); | ||
| } catch (err) { | ||
| if (!storageMissing(err)) console.warn('disclosure latency candidate write failed:', (err as Error).message); | ||
| for (const provider of DIRECT_PROVIDER_IDS) { |
There was a problem hiding this comment.
When DISCLOSURE_LATENCY_PROVIDERS excludes a provider or that provider's key is not configured, this still creates pending rows for every direct provider. Those rows are not part of a simultaneous race; if Unusual Whales or Quiver is enabled later, the first match will use the later first-poll time and can generate false Congress.Trade-ahead deltas/alerts, while summary totals are inflated even for fmp-only runs. Candidate creation should be gated to the providers actually selected/configured at discovery time.
Useful? React with 👍 / 👎.
| } | ||
|
|
||
| async function fetchUnusualWhalesRows(apiKey: string, max: number, fetchImpl: typeof fetch): Promise<DisclosureProviderRow[]> { | ||
| const url = `https://api.unusualwhales.com/api/congress/recent-trades?limit=${Math.min(max, 200)}`; |
There was a problem hiding this comment.
This polls /api/congress/recent-trades with only limit, but the Unusual Whales docs describe that endpoint's date filter as a Market Date defaulting to the last trading date, while PTR filings commonly report transactions from days or weeks earlier. In that case newly filed disclosures with older transaction_date values never enter disclosure_provider_observations, so the provider looks pending/missing even though filed_at_date may be current; use a filing/report-recency query or page recent reports instead of the default last-trading-date slice.
Useful? React with 👍 / 👎.
| return { ...base, configured: false, enabled: false, fetchedRows: 0, pending: 0, matched: 0, errors, reason: `${provider.secretNames[0]} missing` }; | ||
| } | ||
|
|
||
| const nowIso = now.toISOString(); |
There was a problem hiding this comment.
When multiple providers are configured, every provider run receives the same now from the scheduled/probe call, and this line records that value as first_observed_at even for providers fetched after earlier network calls complete. A slow earlier provider can therefore make later providers appear to have been observed seconds or minutes before they actually were, corrupting the latency comparison; capture a fresh timestamp around each provider response or run the probes in parallel with per-provider observation times.
Useful? React with 👍 / 👎.
| return parseUnusualWhalesDisclosureRows(await fetchJson(url, { authorization: `Bearer ${apiKey}` }, fetchImpl)); | ||
| } | ||
|
|
||
| async function fetchQuiverRows(apiKey: string, _max: number, fetchImpl: typeof fetch): Promise<DisclosureProviderRow[]> { |
There was a problem hiding this comment.
When a Quiver key is configured, the scheduled probe ignores DISCLOSURE_LATENCY_WATCH_LIMIT/FMP_DISCLOSURE_WATCH_LIMIT for this provider and writes however many rows the live House and Senate endpoints return each minute. If those feeds return more than the intended latest slice, this can burn API/D1 work and make probes much heavier than configured; apply the limit in the request if supported or slice the parsed rows before upserting.
Useful? React with 👍 / 👎.
1 participant
Summary
Verification
Note
Medium Risk
Touches cron ingestion probes and external API credentials; schema migration adds columns but matching logic changes could affect latency metrics accuracy.
Overview
Disclosure latency monitoring is generalized from an FMP-only race to a multi-provider benchmark against FMP, Unusual Whales, and Quiver, with Finnhub, AInvest, and Capitol Trades listed in admin status but not auto-probed.
New filings now create pending candidates per direct provider; the scheduled probe fetches each provider’s latest feed, matches rows (including Quiver upload timestamps when present), and stores
provider_published_atalongside monitor first-seen times. Admin APIs addGET /disclosure-latency/summary(aggregates plus apublicSummarywithout doc/member IDs),providerfiltering on the list endpoint, and?providers=on the manual probe. Config moves toDISCLOSURE_LATENCY_*env vars while keeping legacy FMP watch flags.Also adds a CodeQL workflow for JS/TS, documents new API keys in
.dev.vars.example, migration 0023 for provider timestamps, and removes stray Xcode workspace metadata files.Reviewed by Cursor Bugbot for commit 4400274. Configure here.