Live GPU price ingestion from RunPod + Vast.ai#6
Draft
aurph wants to merge 15 commits into
Draft
Conversation
… copying curation The 'daily recorder' was copying the hand-curated current price into the history file once a day - a flat line wearing a recorded-daily badge. It now records real observations from providers with public, keyless APIs, verified live before building: - RunPod GraphQL: secure + community on-demand pricing (community rows under 40% of secure are placeholder floors and are dropped) - Vast.ai marketplace: median dph_total of the cheapest 50 verified on-demand single-GPU offers Blend = median across observations; low/high = observed span; guard rail drops anything 4x off the curated price as a bad SKU match (real rentals do not 4x in a day). Models with no live source that day record NOTHING - anchors + est. flags carry them honestly. Total fetch failure records nothing and fabricates nothing. Serving: a fresh live observation (<=2 Eastern days) becomes the served current price and DROPS the est. flag - the number is observed, not estimated. Rows carry liveSources/liveDate; the row tooltip shows 'live price · runpod-secure + vast · <date>' and the methodology footnote explains the live/curated split. Sweep is request-driven (first metrics hit of the Eastern day, single-flight, response never waits on providers) so it works on autoscale hosting. E2E verified against the real APIs: 6 of 10 models recorded live (H100 $2.69 n=3, B200 $5.89 n=3, H200, B300, A100, MI300X), fleet avg moved 4.47 -> 4.31, est. flags dropped on covered models and kept on GB200/GH200/MI325X/MI355X. Suite 195 tests.
GPU price history grid panels:
- Observed marketplace range renders as a shaded band in EVERY panel
('mkt $1.38-$6.98'), so each panel carries real spread context even
where the line is thin; band included in the log domain.
- Sparse series (<=5 points) annotate every real point with its value -
panels read as data, not gaps. Overlay does the same when <=3 series
are visible and a series has <=4 points.
- Panel headers now distinguish live (latest point observed from
provider APIs) from est. - six models tag live today.
- Panel height +18px for the labels.
Buildout history chart: end-value labels on both series ('4.1 GW
online', '18.8 GW committed') so the chart's two headline numbers read
without hovering.
Recorder fixes surfaced while verifying:
- The sweep now gates on a LIVE snapshot for today; a same-day legacy
static row is superseded by real observations instead of blocking
them (earlier broad commits had accidentally shipped two
machine-written static rows, which blocked the first live sweep).
- gpu-price-history.json committed clean ([]) - production accrues
live rows from day one; anchors in gpu-rental-prices.json carry the
pre-recorder history.
Verified live: 6 models tagged live, 4 est., bands in all 10 panels,
fleet avg $4.31 (-14.2% 1Y real). Suite 195.
…orange hexes The /api/catalysts/all payload embeds a legacy stageColor hex per earnings item - the old five-orange family (#F0A500/#F07800/#C87533/ #D4A843/#B8860B) that Lake 1 replaced client-side. Both calendars preferred the server value, silently overriding the migration, so every earnings dot still rendered near-identical orange. Clients now derive stage colors exclusively from the token palette (by stage name, ticker map fallback) and ignore the payload field; the server keeps sending it so the API shape is unchanged. Verified live: calendar dots render six distinct series colors.
Removes the lowercase editorial taglines ('the buildout, measured -
source on every card', 'Where the money moved') and colloquial
phrasing ('press releases are not steel', 'hit All to reset',
'interconnection gets hard') from user-facing copy. Labels now state
what things are; functional legend text is unchanged.
The CCJ/CEG correlation scatter was Math.random dots generated around a hardcoded target r (0.82) on every request - the top item in the known-debt list and the one surface contradicting the honest-data doctrine. It now computes from real weekly closes: SRUUF (Sprott Physical Uranium Trust, a physical-uranium fund and the closest freely quotable spot proxy) vs CCJ and CEG, trailing year, paired by calendar week, real Pearson r, 24h cache. The real numbers correct the record: CCJ vs uranium r = 0.87 (the fiction said 0.82), but CEG came back NEGATIVE (-0.31) where the fiction claimed +0.65 - CEG actually moved against uranium over the past year, which strengthens the card's own miner-vs-utility point. Honest degradation end to end: fetch failure serves an empty set and null coefficients (briefly cached so an outage does not hammer retries); the client renders an unavailable state with retry, never invented dots. Axis and copy now say SRUUF weekly close - the old '$/lb' spot labeling is gone since we chart the proxy, not spot. Also: ETN added to STATIC_MARKET_DATA (it vanished instead of going stale when Yahoo failed - known debt), so Eaton keeps its name and row under throttle. New pure module server/uranium-correlation.ts (alignment, Pearson, thresholds, cache/failure) fully covered. Suite 207 tests.
…, adapted to current code SEC-1: /api/newsletter/preview requires admin (was public, leaked the subscriber count); the internal send-path call forwards the server's own key. SEC-2: /api/social/generate requires admin (was public, burned Yahoo quota). SEC-3: /api/stack timeframe allowlisted to 1D/5D/1M and cached on the normalized value, killing the ~200-call Yahoo fan-out an arbitrary query string caused. SEC-4: the news scanners (which write curated JSON) no longer run from public GET /api/news - only from authenticated /api/admin/scan-news-now. SEC-5: datacenter names HTML-escaped before Leaflet divIcon interpolation (stored XSS via the admin/ingester pipeline). Ported the auth-boundary test suite: every admin route rejects a missing key, the correct key is accepted, junk timeframes coerce gracefully. Suite 209 tests. Operational note: cron-job.org needs a second job POSTing /api/admin/scan-news-now (x-admin-key header) to restore the automated news-driven dataset updates that SEC-4 removed from the public path.
…tatic values The Stack's Rev Growth column served only hand-curated STATIC_MARKET_DATA values - months old, rendered identically to live data (NVDA showed a stale 122.4% vs the real 85.2%; INTC showed -2.1% vs the real +7.2%). New server/fundamentals.ts: Yahoo quoteSummary financialData revenue growth for all 100 tickers, bounded concurrency, per-ticker failure isolation, daily in-memory cache, single-flight non-blocking refresh (same request-driven pattern as the GPU sweep, autoscale-safe). Fraction-to-percent conversion guards absurd magnitudes. Until the first sweep lands the field serves null - the UI's defined em-dash - never a stale number pretending to be current. Fundamentals overlay at response time so the 10-minute stock cache cannot serve pre-sweep rows. E2E: 82/100 tickers refreshed live on first sweep. Suite 217 tests.
README no longer describes phantom architecture (Drizzle, Postgres, shared/, storage.ts) or stale counts; it now states the 8 consolidated tools, the data-honesty stance (live GPU recorder, real uranium correlation, labeled degradation), the real stack, and the deploy gotcha. replit.md cut from a March fossil to a short accurate pointer (Replit's agent reads it). CLAUDE.md known-debt updated to match.
date-fns, @tailwindcss/vite, @radix-ui/react-toggle uninstalled (zero importers, grep-verified; toggle was the radix dep's only consumer). lib/dates.ts, ui/dialog.tsx, ui/toggle.tsx deleted (unreferenced). ui/sheet.tsx keeps @radix-ui/react-dialog. Package name rest-express (Replit template artifact) is now gridtilt.
The raw-materials bottleneck copy hardcoded spot prices that drift (copper $10,200/ton; uranium $80+ while market-constants.json says $92). The copy now describes the constraint without quoting spot prices that go stale; the sourced 475k tons/yr copper projection stays. Keys and structure unchanged.
/api/newsletter/preview (and therefore send) now renders a full weekly digest from the same composed Brief the site shows: summary, the three measured gauges (tracked GW, GPU fleet avg, tightest reserve margin), brief sections, the day's top movers (labeled as-of send time), the takeaway, and a per-recipient unsubscribe hook. Pure renderer in server/weekly-digest.ts - table layout, inline styles, no external resources - with tests covering structure, escaping of untrusted names, the personalization hook, and honest omission of missing gauges. Brief input assembly extracted to buildBriefInput(), shared with /api/brief. Send route: subject is now 'The GridTilt Weekly · Week of ...' and a cadence guard refuses a second send within 6 days unless force:true - a misfired cron cannot double-send. To automate: cron-job.org weekly POST /api/newsletter/send with the x-admin-key header.
No cookies, no personal data, nothing loads unless VITE_GOATCOUNTER_CODE is set at build time. SPA-aware: automatic counting disabled, every route change counted from the router. Activation is a two-minute GoatCounter signup plus one deployment secret.
…y tool page Every tool opened with a splash hero - oversized mono title with a slash subtitle, a paragraph of description, and the orange-tinted grid-bg pattern. All of it is replaced by a shared compact PageHeader (components/PageHeader.tsx): small title, the page description demoted to an info popover, one to three real inline numbers, freshness and badges on the right, tabs/toggles in a controls row. Data starts at the top of every page, matching the Overview strip. Converted: The Stack, GPU Prices, Analyze, Compute Frontier + its three subpages, Catalyst Tracker, Analysis. Stock/sector/region/ operator pages had no heroes and are untouched. Unreachable standalone heroes in embedded-only pages lose the grid-bg class only. Decorative classes removed app-wide and from index.css: the grid-bg pattern, glow-amber/glow-orange, the logo entrance animation, and the count-up keyframes. Suite 225 tests, build green.
The AsOf chip escalated to an amber 'stale' badge past 2x a refresh interval - an ops signal pointed at visitors. It misfired by design: an earnings calendar is fine being hours old, and quotes are normally a night old while markets are closed, so pages would flip to 'stale' overnight for no reason a reader could act on. AsOf now stays a quiet 'as of Xm ago' with the exact timestamp on hover, always. Genuinely broken sources keep their honest signals: per-item delayed badges (server-driven) and the in-place error state with retry. Snapshot updated; the old stale branch is gone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The GPU price recorder now observes the market daily (RunPod GraphQL + Vast.ai marketplace, both keyless public APIs) instead of copying the hand-curated price. Median blend, observed low/high, 4x outlier guard, single-flight request-driven sweep (autoscale-safe). Fresh observations become the served price and drop the est. flag; uncovered models stay curated and flagged. E2E verified live: 6/10 models recorded real prices on first run. 195 tests.
Chain: base is feat/real-gauges (PR #5); merge order #4 -> #5 -> this.
🤖 Generated with Claude Code
https://claude.ai/code/session_011dDxxKQpH3o4CA7qnDFGdN