Stonks Love API

A real-time stock market intelligence backend that fuses live market data with Reddit social sentiment to surface the stocks people are talking about and the stocks that are actually moving. It exposes everything through a single GraphQL API and runs continuous background workers that ingest price/volume ticks from Polygon.io and scrape retail-investor chatter from Reddit's finance subreddits.

Think of it as the engine behind a "what's hot on WallStreetBets right now, and is it backed by real volume?" dashboard.

What It Does

At a high level, the system does four things continuously:

Maintains a universe of tradeable symbols — syncs the full NASDAQ + NYSE ticker lists into the database daily.
Streams live intraday quotes — subscribes to Polygon.io's delayed WebSocket feed and records per-minute aggregate bars (price, volume) for every symbol it knows about.
Backfills daily OHLC history — pulls the previous trading day's open/high/low/ close/volume for each symbol so it has a historical baseline to compare against.
Mines Reddit for stock mentions — polls the hot/new posts and latest comments across eight finance subreddits every 10 minutes, extracts ticker symbols from the text, scores sentiment, and stores each mention.

On top of that data it computes derived signals — most importantly relative volume and price change, so a stock's activity is measured against its own recent history rather than in raw absolute terms.

Unique Features

What makes this project interesting rather than just another market-data wrapper:

1. Time-of-day-aware relative volume ("volume at percent through the trading day")

Absolute volume is misleading — of course a stock has more volume at 3pm than at 10am. This system computes how far through the trading day the current moment is, then compares today's cumulative volume against the same point in a previous trading day.

getPercentThroughOpenAtDate() in src/services/market.ts computes the fraction of the 13:30–21:00 UTC trading window that has elapsed.
getVolumeAtPercentOnClosestDate() in src/tables/dailyQuote.ts scales the most recent daily volume by that fraction to estimate "what volume should look like by now."
The difference becomes volumeChangePreviousDayRelativeHour — the core "unusual volume" signal that powers the volume-movers query.

There's even a standalone CLI (src/bin/getVolumeAtPercentageOpen.ts) to inspect this calculation for a single symbol.

2. Ticker extraction from free-form Reddit text

src/services/symbol.ts parses raw post titles, self-text, and comment bodies to pull out stock tickers using a carefully crafted regular expression that handles:

Optional $ cashtags ($TSLA)
Exchange prefixes (NYSE:GME) and suffixes (SHOP.TO)
Plain uppercase runs (GME) — while filtering out common English words

It uses a multi-source stopword list (src/data/words.json) to avoid false positives, plus a requireDollarSign list (src/data/reddit.json) for tickers that are also common words (DD, ALL, PM, AM, GO…) — these are only counted if explicitly written as cashtags. Extracted symbols are then validated against the real symbol universe before being persisted.

3. Finance-tuned sentiment analysis

src/services/sentiment.ts wraps the AFINN-based sentiment library but injects a custom lexicon of WallStreetBets vernacular: moon, tendies, bagholder, stonk, rocket, bullish/bearish, calls/puts, pump, rip, sus, undervalued, etc. This makes sentiment scoring meaningful for retail-investor slang that a generic model would miss entirely.

4. "Reddit Trending" cross-referenced with real stocks

The API can rank symbols by number of Reddit mentions in the last 24 hours and join those back to the actual stock records — so trending chatter is always tied to a real, tradeable ticker with live quotes attached.

5. Mention-velocity signal

Beyond raw mention counts, getMentionChangePercentLast24Hours() compares the most recent 24-hour mention count against the prior 24 hours (derived from a 48-hour window) to measure whether buzz is accelerating or decelerating — a leading indicator, not just a snapshot.

Architecture Overview

                          ┌─────────────────────────────────────────┐
                          │              External Sources            │
                          │                                          │
   Polygon.io WebSocket ──┤  AM.* aggregate bars (delayed feed)      │
   Polygon.io REST      ──┤  daily open/close backfill               │
   Reddit API           ──┤  hot / new posts + latest comments       │
   NASDAQ/NYSE lists    ──┤  bundled JSON ticker files               │
                          └───────────────┬──────────────────────────┘
                                          │
        ┌─────────────────────────────────┼─────────────────────────────────┐
        │                                 │                                  │
  ┌─────▼──────┐                   ┌──────▼───────┐                   ┌───────▼──────┐
  │  socket    │                   │    queue     │                   │     api      │
  │  worker    │                   │   worker     │                   │  (Apollo)    │
  │            │                   │  (Bull/cron) │                   │              │
  │ streams    │                   │ • sync       │                   │ GraphQL over │
  │ live ticks │                   │   tickers    │                   │ HTTP         │
  │ → quotes   │                   │ • fetch      │                   │              │
  │            │                   │   reddit     │                   │ reads        │
  │            │                   │ • backfill   │                   │ derived data │
  │            │                   │   dailies    │                   │              │
  └─────┬──────┘                   └──────┬───────┘                   └──────┬───────┘
        │                                 │                                  │
        └───────────────┬─────────────────┴──────────────┬───────────────────┘
                        │                                 │
                  ┌─────▼──────┐                    ┌──────▼──────┐
                  │   MySQL    │                    │    Redis    │
                  │ (via Knex) │                    │ (Bull jobs) │
                  └────────────┘                    └─────────────┘

Three independent Node processes share one MySQL database. Redis backs the job queue used by the queue worker. Apollo Server exposes the read API.

Tech Stack

Concern	Choice
Language	TypeScript (compiled to `lib/` via `tsc`)
API layer	Apollo Server 2 (`apollo-server`) + GraphQL 15
Custom scalars	`graphql-scalars` (`Date`, `DateTime`, `BigInt`)
Database	MySQL 8 (`mysql2`) via Knex query builder
Case mapping	`knex-stringcase` (camelCase in code ↔ snake_case in DB)
Batching	`dataloader` (per-symbol / per-entity loaders)
Job queue	Bull (backed by Redis)
Scheduling	Bull repeatable jobs with cron expressions
Market data	Polygon.io (`@polygon.io/client-js`) — REST + WebSocket
Reddit	`snoowrap` + `got` for OAuth token fetch
Sentiment	`sentiment` (AFINN) with a custom finance lexicon
Scraping	`jsdom` + `got` (Yahoo Finance / TipRanks — mostly legacy)
SMS alerts	ClickSend REST API (`src/services/clicksend.ts`)
Logging	`bunyan` structured logger with child loggers
Dates	`date-fns`
Process mgmt	PM2 (`ecosystem.config.js`)
Lint/format	ESLint + Prettier (with `@typescript-eslint`)
Tests	Mocha + Chai + Proxyquire + Timekeeper

Process Model (the three services)

The app is deliberately split into three long-running processes so that ingestion load never impacts API latency. All are defined as PM2 apps in ecosystem.config.js and all enforce TZ=UTC via src/util/timezoneGuard.ts (the process throws on startup if the timezone isn't UTC — see Gotchas).

`api` — `src/api/index.ts` → `src/api/server.ts`

Boots Apollo Server. On startup it loads the symbol whitelist from src/data/symbols.csv into memory (load() in src/services/symbol.ts), then starts listening. This is the only process clients talk to.

`queue` — `src/worker/queue.ts`

Registers and schedules the Bull background jobs:

Job	Schedule	Purpose
`sync tickers`	on boot + `0 1 * * *` (01:00 daily)	Load new NASDAQ/NYSE symbols
`fetch reddit`	on boot + `/10 * * *` (every 10 min)	Scrape & store Reddit mentions
`fetch yesterday daily quotes`	`0 2 * * *` (02:00 daily)	Backfill prior-day OHLC
`notify movers`	(commented out)	SMS alerts for volume movers

`socket` — `src/worker/socket.ts`

Opens the Polygon.io delayed WebSocket, subscribes to all minute aggregates (AM.*), buffers them, and flushes to the quotes table in batches of 20. It filters to symbols the system knows about and de-duplicates by keeping only the newest tick per symbol per flush.

Data Model

Schema is defined programmatically in src/services/database/schema.ts (via Knex schema builder). Code uses camelCase; knex-stringcase maps to snake_case columns automatically.

`stocks`

The symbol universe. Primary key is symbol.

Column	Notes
`symbol` (PK)	e.g. `TSLA`
`name`	Company name
`industry`	Nullable
`is_blacklisted`	Set when Polygon returns NOT_FOUND for a symbol, so it's skipped in future backfills
`created_at` / `updated_at`

`quotes` — intraday ticks

Composite PK (symbol, date). Written by the socket worker. date is rounded to the nearest 15 minutes on insert (roundingMinutes = 15) so upserts collapse onto stable buckets.

Column	Notes
`symbol`, `date` (PK)
`price`
`volume`	This tick's volume
`total_volume`	Cumulative day volume (from Polygon `av`)
`volume_change_previous_day_relative_hour`	Derived — the relative-volume signal
`price_change_previous_day_close`	Derived — % change vs. previous close

Note: the derived columns are computed and merged in src/tables/quote.ts at insert time even though the base schema in schema.ts only declares price/volume — the production schema has drifted ahead of the checked-in createQuotes() definition.

`daily_quotes` — historical OHLC

Composite PK (symbol, date). One row per symbol per trading day. Backfilled from Polygon's dailyOpenClose. Provides the baseline used to compute relative volume/price.

`reddit_entities` — individual posts/comments

Primary key is the Reddit id. A "reddit entity" is either a submission (post) or a comment (post_id is null for posts, set for comments). Stores excerpt, upvotes, sentiment, subreddit, permalink, created_at.

`reddit_entity_symbols` — mention join table

Many-to-many between reddit entities and stock symbols. One row per (entity, symbol) pair. This is what makes "how many times was $GME mentioned in the last 24h" a simple COUNT(...) with a join.

Relationships:

stocks 1───∞ quotes
stocks 1───∞ daily_quotes
stocks 1───∞ reddit_entity_symbols ∞───1 reddit_entities

The GraphQL API

Schema and resolvers live in src/api/resolvers.ts. There are exactly two root queries.

`stock(symbol: String!): Stock`

Fetch a single stock and drill into its quotes and Reddit activity.

{
  stock(symbol: "TSLA") {
    symbol
    name
    industry
    quotes {
      latest { price volume percentPriceChangeLastClose percentVolumeChangeRelativeHour date }
      today  { price volume date }
      weekly { open high low close volume date }
    }
    reddit {
      numberOfMentions
      mentionChangePercentage
      latest(limit: 5) { title link subreddit upvotes sentiment }
      hot(limit: 5)    { title link subreddit upvotes sentiment }
    }
  }
}

`today: TodayMarket!`

The "what's happening right now" entry point.

{
  today {
    date
    isOpen
    volumeMovers(minimumVolume: 1000000, minimumPercentChange: 20, minimumPrice: 1, maxPrice: 100, limit: 10) {
      minimumVolume minimumPercentChange stocks { symbol name }
    }
    priceMovers(minimumPercentChange: 20, minimumPrice: 1, maxPrice: 100, limit: 10) {
      stocks { symbol name }
    }
    redditTrending {
      date
      stocks { symbol name reddit { numberOfMentions } }
    }
  }
}

Resolver structure:

Scalar resolvers wire up Date, DateTime, BigInt.
Stock type resolvers lazily resolve quotes and reddit sub-fields only when requested — each field maps to a focused function in the tables/ layer.
TodayMarket resolvers compute date (last open date) and isOpen, and delegate movers/trending to shared resolverFunctions with sensible defaults (min volume 1M, min % change 20, price band $1–$100k, limit 10).
Movers are found via raw SQL (findBiggestVolumeMovers / findBiggestPriceMovers in src/tables/stock.ts) that selects each symbol's latest quote for the given day and filters/sorts on the derived relative-change columns.

There is a fair amount of commented-out schema (DatedMarket, day(date:), per-stock analysis) hinting at planned historical-replay and TipRanks-analysis features.

How Data Flows

Ingestion path (write side)

Live quotes:

Polygon WS "AM.*" ─→ socket worker buffers (20) ─→ filter to known symbols
  ─→ dedupe newest-per-symbol ─→ quote.createMany()
      ├─ round date to 15-min bucket
      ├─ compute volumeChangePreviousDayRelativeHour  (vs daily_quotes baseline)
      ├─ compute priceChangePreviousDayClose          (vs last week's close)
      └─ upsert into `quotes` (merge on conflict)

Reddit mentions (every 10 min):

fetchReddit job ─→ reddit service: hot + new posts + latest comments (8 subreddits)
  ─→ formatEntity(): extract symbols (symbol.fromText) + score sentiment
  ─→ uniqBy id, keep only entities with ≥1 symbol
  ─→ redditEntity.createMany()
      ├─ filter symbols to known universe
      ├─ upsert into `reddit_entities` (merge upvotes/excerpt)
      └─ insert (entity_id, symbol) rows into `reddit_entity_symbols`

Daily backfill (02:00):

fetchDailyQuotes job ─→ symbolsUnfilledForDate(lastOpenDate)
  ─→ Bluebird.map(concurrency 10): stocksClient.dailyOpenClose(symbol, date)
      ├─ success → dailyQuote.create()
      └─ NOT_FOUND → mark stock is_blacklisted = true

Ticker sync (01:00):

syncTickerSymbols job ─→ merge bundled nasdaq.json + nyse.json
  ─→ diff against existing symbols ─→ chunk(20) ─→ stock.createMany() (ignore conflicts)

Query path (read side)

GraphQL query ─→ Apollo resolver ─→ tables/* function ─→ Knex query / raw SQL ─→ MySQL
                                   └─ DataLoader batches per-symbol stock lookups

Key Subsystems

Symbol extraction (`src/services/symbol.ts`)

Loads symbols.csv into an in-memory Set at API boot; a whitelist set excludes stopwords.
fromText() strips subreddit references and vs. noise, runs the ticker regex, applies the requireDollarSign rule, normalizes to uppercase, and de-dupes.

Market clock (`src/services/market.ts`)

Encodes NYSE hours as 13:30–21:00 UTC (hence the strict TZ=UTC requirement).
isOpen(), isOpenOnDay(), getLastOpenDate() (skips weekends), getPercentThroughOpenAtDate().
This module is the single source of truth for "when is the market open" and underpins every relative calculation.

Relative signal computation (`src/tables/quote.ts` + `dailyQuote.ts`)

getPercentVolumeChangeRelativePreviousTradingDay() — today's cumulative volume vs. the prior day's volume scaled to the current time-of-day.
getPercentPriceChangeClosePreviousTradingDay() — current price vs. the most recent daily close.

Reddit ingestion (`src/services/reddit.ts`)

Fetches an OAuth client-credentials token from Reddit directly via got (bypassing snoowrap's own auth), caches it until expiry, and builds a snoowrap client per request.
formatEntity() normalizes both submissions and comments into a common Entity shape.

DataLoaders (`src/loaders/`)

stockBySymbol and redditEntityById batch and cache DB lookups within a request, avoiding N+1 queries when many resolvers ask for the same records. The stock loader is cache-invalidated on create/update in src/tables/stock.ts.

Job infrastructure (`src/services/queue.ts`)

A tiny factory wrapping Bull: create(name, handler) returns an add() function, attaches a child logger, and re-throws handler errors so Bull records the failure.

SMS alerting (`src/services/clicksend.ts`)

sendSymbolVolumes() formats a list of movers and sends SMS via ClickSend. Wired to the (currently commented-out) notify movers job — a partially built notification feature.

Scrapers (`src/services/scrapers/`)

yahooFinance.ts (profile + daily history via jsdom) and tipranks.ts (analyst sentiment) are legacy/experimental — largely superseded by Polygon and mostly commented out. Retained for reference and possible future analysis features.

Project Layout

src/
├── api/                    # GraphQL server
│   ├── index.ts            #   entrypoint (timezone guard + start)
│   ├── server.ts           #   Apollo Server setup
│   └── resolvers.ts        #   typeDefs + all resolvers
├── worker/
│   ├── queue.ts            # queue process: registers/schedules Bull jobs
│   └── socket.ts           # socket process: Polygon WS → quotes
├── queues/                 # Bull job definitions
│   ├── fetchReddit.ts
│   ├── fetchDailyQuotes.ts
│   ├── syncTickerSymbols.ts
│   └── notifyMovers.ts     #   (stub / disabled)
├── tables/                 # data-access layer (one file per table)
│   ├── stock.ts            #   incl. raw-SQL mover queries
│   ├── quote.ts            #   incl. derived-signal computation
│   ├── dailyQuote.ts
│   ├── redditEntity.ts     #   incl. mention counts & velocity
│   └── redditEntitySymbol.ts
├── loaders/                # DataLoader batchers
│   ├── stockBySymbol.ts
│   └── redditEntityById.ts
├── services/               # cross-cutting services
│   ├── market.ts           #   trading-hours math
│   ├── symbol.ts           #   ticker extraction from text
│   ├── sentiment.ts        #   finance-tuned sentiment
│   ├── reddit.ts           #   Reddit API client
│   ├── polygon.ts          #   Polygon REST + WS (polygonLegacy.ts = old impl)
│   ├── database.ts         #   Knex instance
│   ├── database/schema.ts  #   table definitions + dropCreate
│   ├── queue.ts            #   Bull factory
│   ├── clicksend.ts        #   SMS
│   ├── cache.ts            #   tiny in-memory cache helper
│   └── logger.ts           #   bunyan
├── caches/jobQueue.ts      # in-memory id/symbol sets
├── util/
│   ├── timezoneGuard.ts    # throws unless TZ=UTC
│   ├── camelCase.ts / snakeCase.ts
├── bin/
│   └── getVolumeAtPercentageOpen.ts  # CLI to inspect relative-volume math
├── data/                   # bundled reference data
│   ├── nasdaq.json / nyse.json       # symbol universes
│   ├── symbols.csv                   # symbol whitelist source
│   ├── words.json                    # stopword lists
│   └── reddit.json                   # subreddits + requireDollarSign list
└── types.d.ts

test/                       # Mocha tests (some reference an older schema)
ecosystem.config.js         # PM2 app definitions (api / queue / socket)
.github/workflows/          # CI: build + SFTP deploy + pm2 restart

Configuration & Conventions

TypeScript compiles src/ → lib/; the build also copies src/data/* into lib/data/ (see the build script) because the JSON/CSV are read at runtime.
Case convention: camelCase throughout the code, snake_case in the database, bridged automatically by knex-stringcase.
Data-access pattern: every table has its own module in src/tables/ exposing a getTable() plus focused query functions — resolvers never touch Knex directly.
In-memory sets track blacklisted symbols and known entity ids to avoid redundant DB round-trips during high-volume ingestion.
Structured logging via bunyan child loggers tagged per job/queue.

Running Locally

Prerequisites: Node.js, a MySQL server, and a Redis server (for Bull).

The Knex connection (src/services/database.ts) currently points at 127.0.0.1 / user root / db stonks_love, and the queue service uses redis://127.0.0.1:6379.

# install
npm install

# build TypeScript → lib/ (and copy data files)
npm run build

# (re)create the database schema
npm run database:reset

# run each process (each requires TZ=UTC — the scripts set it for you)
npm run start:api      # GraphQL server (Apollo playground on the printed URL)
npm run start:queue    # Bull jobs (reddit/daily/ticker sync)
npm run start:socket   # Polygon live-quote ingestion

# dev convenience: rebuild + restart the API on file change
npm run watch:api

# lint / autofix
npm run lint

# refresh bundled symbol lists from the US-Stock-Symbols repo
npm run symbols:nyse
npm run symbols:nasdaq

⚠️ Every process aborts immediately unless TZ=UTC is set — this is intentional (see below). The npm scripts already prefix TZ=UTC.

Build & Deployment

CI is defined in .github/workflows/ and triggers on push/PR to main:

npm install (cached)
npm run build
SFTP deploy the compiled lib/* to the production host
Restart the api, socket, and queue PM2 processes over SSH

Production runs under PM2 using ecosystem.config.js, which defines the same three processes with TZ=UTC and debug-level logging.

Notable Design Decisions & Gotchas

UTC is mandatory. All market-hours logic hardcodes NYSE hours in UTC (13:30–21:00), and every entrypoint imports timezoneGuard, which throws if TZ !== 'UTC'. This prevents subtle off-by-hours bugs in the relative-time-of-day volume math.
15-minute quote bucketing. Both quotes and daily_quotes round timestamps to 15-minute boundaries before upserting, so repeated ticks in a window collapse onto one row instead of exploding the table.
Self-healing symbol universe. When Polygon returns NOT_FOUND for a symbol during daily backfill, that stock is auto-blacklisted and excluded from future work — the universe prunes delisted/invalid tickers over time.
Relative, not absolute, signals. The headline metrics (volumeChange…RelativeHour, priceChange…Close, mention-velocity) are all comparative, which is what makes small movers detectable rather than being drowned out by mega-caps.
Schema drift. schema.ts::createQuotes() doesn't declare the derived columns (volume_change_previous_day_relative_hour, price_change_previous_day_close, total_volume) that the code writes to — the live DB is ahead of the checked-in schema definition. Keep this in mind when running database:reset.
Configuration is environment-driven. All credentials (Polygon API key, Reddit client id/secret, ClickSend auth, DB connection, Redis URL, Yahoo cookie) are read from environment variables — copy .env.example to .env and fill it in. No secrets are committed to the repository.
Legacy code retained. polygonLegacy.ts, the Yahoo/TipRanks scrapers, notifyMovers, and chunks of commented-out schema represent earlier iterations and half-built features (historical replay, analyst-sentiment analysis, SMS alerts) kept for reference.
Tests reference an older model. test/ uses table names (intraday_quotes, past_notifications) and paths (../../lib/jobs/..., ../models/stock) that predate the current tables/-based structure, so they won't run against the current schema without updating.

Name		Name	Last commit message	Last commit date
Latest commit History 161 Commits
.github/workflows		.github/workflows
.vscode		.vscode
src		src
test		test
.env.example		.env.example
.eslintrc.js		.eslintrc.js
.gitignore		.gitignore
.prettierrc.js		.prettierrc.js
README.md		README.md
ecosystem.config.js		ecosystem.config.js
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json

Folders and files

Latest commit

History

Repository files navigation

Stonks Love API

Table of Contents

What It Does

Unique Features

1. Time-of-day-aware relative volume ("volume at percent through the trading day")

2. Ticker extraction from free-form Reddit text

3. Finance-tuned sentiment analysis

4. "Reddit Trending" cross-referenced with real stocks

5. Mention-velocity signal

Architecture Overview

Tech Stack

Process Model (the three services)

api — src/api/index.ts → src/api/server.ts

queue — src/worker/queue.ts

socket — src/worker/socket.ts

Data Model

stocks

quotes — intraday ticks

daily_quotes — historical OHLC

reddit_entities — individual posts/comments

reddit_entity_symbols — mention join table

The GraphQL API

stock(symbol: String!): Stock

today: TodayMarket!

How Data Flows

Ingestion path (write side)

Query path (read side)

Key Subsystems

Symbol extraction (src/services/symbol.ts)

Market clock (src/services/market.ts)

Relative signal computation (src/tables/quote.ts + dailyQuote.ts)

Reddit ingestion (src/services/reddit.ts)

DataLoaders (src/loaders/)

Job infrastructure (src/services/queue.ts)

SMS alerting (src/services/clicksend.ts)

Scrapers (src/services/scrapers/)

Project Layout

Configuration & Conventions

Running Locally

Build & Deployment

Notable Design Decisions & Gotchas

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages