StockSense TW 📈

An agentic Taiwan-stock analytics platform that monetizes data by selling personalized, interpretable investment knowledge — and, as a premium layer, turns aggregated user interactions into a real-time market-sentiment signal.

Big Data Systems · Spring 2026 · National Taiwan University — Final Project Design a system that monetizes data.

• GitHub: https://github.com/muchi674/bda_spring2026_final

---

The idea in one paragraph

Most young Taiwanese investors use broker apps that show numbers but never explain them. StockSense asks the user three things — their investment objective, their risk tolerance, and their own belief about a stock — then combines those answers with the top metrics and an LLM (using the user's own API key) to produce a personalized, plain-language analysis that engages with the user's thesis instead of dumping more data. Every interaction is anonymized and aggregated into a real-time market-mood product: average sentiment per stock, bullish share, and a "crowd-vs-model" over-confidence gap — a signal that doesn't exist cleanly anywhere else.

See report/REPORT.md for the full business + technical write-up, and docs/architecture.md for component rationale and scalability notes.

Architecture

Why this isn't just an LLM wrapper: two refinement stages turn commodity inputs into things you can't get by pasting raw numbers into a chatbot — (1) a batch feature layer computes cross-sectional percentile ranks (vs the whole tracked universe), momentum, volatility and range position, which ground the analysis; (2) the aggregated interaction dataset (mood, momentum, crowd-vs-model gap, behavioral mix) is itself the saleable product (/sentiment/feed).

Course concepts exercised: message queue (Redpanda/Kafka API), stream processing (windowed sentiment aggregation), batch processing (the cross-sectional refinement layer), SQL store (Postgres + JSONB), API/dashboard delivery (FastAPI + Streamlit).

Repository layout

src/stocksense/
  config.py            settings from env / .env
  db.py                Postgres helpers (upserts, queries, windowed aggregation)
  kafka_io.py          confluent-kafka producer/consumer wrappers
  metrics_catalog.py   metric labels/units/importance ranks
  watchlist.py         tracked tickers + display names
  data/                schema.sql + committed offline sample dataset
  features.py          batch refinement layer (cross-sectional signals)
  ingestion/           FinMind client + producer (--sample / --live)
  stream/              metrics_consumer.py, sentiment_consumer.py
  api/                 FastAPI app + Claude agent + prompts
  ui/                  Streamlit dashboard
demand/                real demand evidence: public-data findings + sources,
                       App Store reviews/ratings, competitor pricing, survey instrument
scripts/               sample-data generator, demo seeder, run_local.sh
docs/architecture.md   architecture + scalability
report/REPORT.md       the project report
docker-compose.yml     Redpanda + Postgres + topic init
environment.yml        conda env (recommended)  |  requirements.txt for venv/uv

---

Quick start

0. Prerequisites

• Docker (for Redpanda + Postgres) — start Docker Desktop.
• conda (Miniconda/Anaconda) — recommended; a plain venv also works.

1. Create & activate the environment

conda (recommended):

conda env create -f environment.yml
conda activate stocksense

or plain venv / uv:

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt          # editable install of the project
# uv equivalent:  uv venv && source .venv/bin/activate && uv pip install -r requirements.txt

2. Configure

cp .env.example .env        # defaults work out of the box for local Docker

3. One command (does everything)

bash scripts/run_local.sh

This starts infra, ingests the offline sample data, launches both stream consumers + the API + the Streamlit UI, and seeds demo sentiment.

Then open:

• UI: http://localhost:8501
• API docs: http://localhost:8000/docs
• Redpanda console: http://localhost:8080

In the UI sidebar, paste your own Anthropic API key and run an analysis.

---

Run it manually (to see each stage)

# 1. Infrastructure
docker compose up -d

# 2. Stream consumers (separate terminals, or backgrounded)
stocksense-metrics-consumer        # stock.metrics.raw -> Postgres
stocksense-sentiment-consumer      # user.interactions -> rolling sentiment

# 3. Ingest data
stocksense-ingest --sample         # offline, deterministic (default)
# stocksense-ingest --live --loop 3600   # live FinMind, refresh hourly

# 4. Build the refined signal layer (cross-sectional ranks, momentum, vol, ...)
stocksense-build-features          # (the metrics consumer also rebuilds this on new data)

# 5. (optional) seed the premium sentiment tab without spending LLM tokens
python scripts/seed_demo_sentiment.py --per-stock 25

# 6. API + UI
stocksense-api
streamlit run src/stocksense/ui/app.py

Using live FinMind data

Set FINMIND_TOKEN in .env (free token from https://finmindtrade.com/; the free tier also works without one at a lower rate limit), then run stocksense-ingest --live. The bundled offline dataset (src/stocksense/data/sample_dataset.json) is a real FinMind snapshot (TWSE data, dated in the file's _snapshot_date) so the system runs accurately with no network or token; refresh it any time with python scripts/snapshot_finmind.py.

Bring-your-own LLM key

The platform never stores your Anthropic key. It's sent per request to the backend, used to call Claude (default model claude-opus-4-8, override in the sidebar), and discarded. This BYO-key model keeps the platform's variable cost near zero — a core part of the unit economics (see the report).

---

Reproduce the demand evidence

All evidence is real public data (no survey was run; see the honesty note in demand/README.md).

# Real App Store ratings + reviews (Apple public APIs), then analyze
python demand/reviews/fetch_app_ratings.py
python demand/reviews/fetch_appstore_reviews.py
python demand/reviews/analyze_reviews.py
# Verified competitor pricing benchmark
python demand/competitors/pricing_benchmark.py
# Official TWSE/TDCC statistics write-up (with source URLs):
#   demand/public_data/findings.md  +  demand/public_data/sources.csv

See demand/README.md for methodology, sources, and findings.

---

Notes & limitations

• The bundled offline dataset is a real FinMind snapshot committed to the repo, so the system runs accurately with no network or token. Refresh it with python scripts/snapshot_finmind.py, or use stocksense-ingest --live for always-fresh data.
• The analysis is educational, not investment advice — the agent is instructed never to issue a buy/sell directive or price target.
• confluent-kafka ships prebuilt wheels (bundled librdkafka) on macOS/Linux; if your platform lacks a wheel, install librdkafka first (brew install librdkafka).