Merchant Site Selection Data Analytics Platform

This directory provides a complete Dockerized demonstration version of the merchant site selection data analytics system, built on the following technology stack:

• Flume: collects signaling data and writes it to Kafka and HDFS
• Kafka: carries the real-time signaling stream
• HDFS: stores offline raw data
• Flink: consumes Kafka, assigns users to custom regions via point-in-polygon tests, writes the latest regional user profiles to HBase, and detects crowd surge alerts in real time
• Spark: offline, associates signaling with profiles via point-in-polygon spatial assignment and aggregates the results into ClickHouse
• Continuous Generator: continuously simulates realistic movement trajectories for 1,000 users (15 provinces, ages 16-58), who wander probabilistically across 36 independent hotspots, emitting a signaling record every second with laccell set to the nearest base station, including random pauses/reroutes/drift
• HBase: stores real-time regional user profile details, sequential path matching results, and crowd alert events (region IDs are dynamic user-defined values)
• ClickHouse: stores offline statistics and distribution results
• FastAPI: provides APIs for statistical analysis, detail analysis, distribution analysis, AI Q&A, sequence recognition, real-time location monitoring, region comparison, site selection analysis reports, region configuration management, effective customer profiles, site selection scoring, etc.
• Vue 3 frontend: map visualization, region management, AI assistant, site selection decision panel
• Agent Runtime: AI coding agent isolated with Docker, supporting autonomous tool invocation and deep analysis

Architecture

• Real-time processing pipeline: generator (spool) -> Flume -> Kafka -> Flink -> HBase (point-in-polygon region assignment + profile storage + crowd alerts + sequence recognition)
• Real-time display channel: generator → Flume → Kafka → Flink → HBase → API /api/realtime/positions → frontend 1s polling → moving map markers
• Offline pipeline: signaling + user profiles -> HDFS -> Spark (spatial UDF region assignment) -> ClickHouse (multi-dimensional aggregation analysis)
• Service pipeline: FastAPI -> ClickHouse / HBase
• Region determination: instead of a static laccell→region mapping table, users are assigned to regions with a point-in-polygon test (ray casting) on their (lon, lat); region polygons come from the user-defined regions.json configuration
• Hotspot system: 36 independent commercial-district POI hotspots (6 spatial clusters × 6 hotspots); the generator wanders between hotspots with distance-weighted probabilities (closer = more likely, low variance), not bound to any region
• Base station system: 22 simulated base stations evenly distributed; the generator fills the laccell field with the nearest base station, mimicking real signaling collection
• Key design principle: the generator only simulates a data collector writing to spool files; all data must pass through the full big data pipeline Flume→Kafka→Flink→HBase
• Replay feature: indexes historical data from the accumulated signal file, supporting time-based backtracking with a draggable progress bar

Business Positioning

• Business scenario: big-data-driven site selection decision services for merchants
• Pre-analysis: analyze candidate commercial districts in terms of foot traffic scale, target customers, and profile structure
• In-process monitoring: identify sequential paths through key regions in real time, observing movement flows and traffic changes
• Post-hoc review: summarize site selection effectiveness through AI Q&A and profile analysis, providing data-backed evidence for operational decisions

Directory Structure

bigdata-final-system-demo/
├── apps/                  # Standalone applications
│   ├── api-server/        # FastAPI HTTP service
│   ├── stream-job/        # Flink real-time streaming job
│   ├── batch-job/         # Spark offline batch job
│   ├── generator/         # Data generator
│   └── web/               # Vue 3 frontend
├── packages/              # Shared libraries
│   ├── domain/            # Business objects and rules (zero I/O)
│   └── infra/             # Infrastructure interfaces and adapters
├── platform/              # Middleware runtime platform
│   ├── hadoop/  hbase/  clickhouse/  flink/  flume/
├── data/                  # Static seed data
├── scripts/               # Startup/verification scripts
├── tests/                 # Tests
├── docs/                  # Documentation
└── docker-compose.yml

Getting Started

First enter the current directory before running the scripts:

cd .\final_system_demo

1. Environment Configuration

The project manages key API and Spark batch processing settings through a single .env file. On first run, copy the template:

Copy-Item .\.env.example .\.env

The default template already reflects the current reproducible configuration; no changes are usually needed:

CLICKHOUSE_URL=http://clickhouse:8123
CLICKHOUSE_DATABASE=portrait_demo
HBASE_HOST=hbase
HBASE_PORT=9090
HBASE_TABLE=region_user_portrait
HBASE_SEQUENCE_TABLE=region_sequence_match
HBASE_ALERT_TABLE=region_event_alert
OPENAI_API_KEY=
OPENAI_BASE_URL=https://api.deepseek.com
OPENAI_MODEL=deepseek-chat
SPARK_SIGNAL_PATH=/shared/static/signal/generated_signal.txt
SPARK_PROFILE_PATH=/shared/static/profile/user_profile.csv
SPARK_REGIONS_JSON_PATH=/data/static/regions.json

Notes:

• CLICKHOUSE_*: ClickHouse connection settings used by FastAPI and Spark batch processing
• HBASE_*: HBase settings used by FastAPI to read real-time profiles and sequence recognition results
• OPENAI_*: LLM configuration for the F04 agent-based visual Q&A; if only OPENAI_API_KEY and OPENAI_BASE_URL=https://api.deepseek.com are set, the system defaults to deepseek-chat
• SPARK_*: Spark offline batch processing reads static files written by the generator to the shared volume by default, making reproduction more stable and independent of in-container HDFS hostname resolution

.env supports quoted values, for example:

OPENAI_API_KEY="your-key"
OPENAI_BASE_URL="https://api.deepseek.com"
OPENAI_MODEL="deepseek-chat"

If .env only contains the AI configuration, you can still run directly; run_all.ps1 fills in the Spark and ClickHouse defaults automatically.

2. One-Click System Startup

It is recommended to run the full one-click workflow directly:

powershell -ExecutionPolicy Bypass -File .\scripts\run_all.ps1

run_all.ps1 performs, in order:

• Starts Hadoop, Kafka, ClickHouse, HBase
• Starts Flume, Flink, FastAPI
• Runs the one-shot data generator (seed data)
• Starts the continuous data generator (real-time simulation of 1,000 users)
• Uploads static files to HDFS
• Initializes HBase tables
• Submits the Flink streaming job
• Runs the Spark offline batch processing

If you only want the basic bootstrap first and then run the offline job separately, follow these two steps.

Run the basic bootstrap first:

powershell -ExecutionPolicy Bypass -File .\scripts\bootstrap.ps1

Then run the offline Spark batch processing:

Get-Content .\.env | ForEach-Object {
  if ($_ -and -not $_.StartsWith("#")) {
    $parts = $_ -split '=', 2
    Set-Item -Path ("Env:" + $parts[0]) -Value $parts[1]
  }
}
docker compose run --rm `
  -e CLICKHOUSE_URL=$env:CLICKHOUSE_URL `
  -e CLICKHOUSE_DATABASE=$env:CLICKHOUSE_DATABASE `
  -e SIGNAL_PATH=$env:SPARK_SIGNAL_PATH `
  -e PROFILE_PATH=$env:SPARK_PROFILE_PATH `
  -e REGIONS_JSON_PATH=$env:SPARK_REGIONS_JSON_PATH `
  spark /opt/spark/bin/spark-submit --master local[*] /opt/spark/jobs/batch_region_portrait.py

3. How to View the Web UI

Once the system is up, open the following addresses in your browser:

• Frontend main page: http://localhost:8088
• API backend: http://localhost:18080
• Health check: http://localhost:18080/health
• Flink Web UI: http://localhost:8081
• HDFS Web UI: http://localhost:9871
• HBase Master UI: http://localhost:16010

Recommended checking order for teammates:

1. Open http://localhost:18080/health; seeing {"status":"ok"} means the API is running
2. Open http://localhost:8088; the top navigation bar has 5 modules:
   • Location Profiles: real-time headcount per region plus gender/age pie charts per region; 1,000 user markers move on the map, refreshing every second; the "Effective Customer Profiles" section at the bottom shows seven-dimensional profiles of users who completed a full path (gender, age, resident/visitor, consumption activity, path dwell time, arrival time slot, province of origin), auto-refreshing every 30 seconds
   • Traffic Heatmap: region hotspots and activity rankings, auto-refreshing every 30 seconds
   • Dwell Characteristics: dwell duration and hourly rhythm per region, auto-refreshing every 30 seconds
   • Migration Profiles: movement directions between regions and hotspots, auto-refreshing every 30 seconds
   • Funnel Analysis: path conversion funnel for target regions
3. The "Regions" button on the right: opens the region management panel
   • Click "Add Region" → click on the map to add vertices → click "Finish" in the panel to save → the region appears in the list
   • ✓ Check/uncheck to control which regions participate in the computation
   • Double-click a region name to rename it (the display name is for the frontend only and does not affect the internal ID)
   • ▲▼ buttons drag to reorder = funnel order (affects the funnel conversion stages)
   • ✕ deletes a region immediately (takes effect at once, no confirmation needed)
   • Hovering over a list item highlights the corresponding region on the map with gray diagonal hatching
   • Click "Confirm" to activate all changes at once (clears HBase + ClickHouse; the pipeline recomputes with the new regions)
4. Offline modules can be manually refreshed immediately via the "Refresh" button in the panel
5. The AI Q&A floating window at the bottom right supports natural language queries and activity analysis report generation
6. The "Replay" button at the bottom left: opens the historical replay panel; drag the progress bar to review user location changes over time
7. Watch the status tag in the AI Q&A card:
   • AI status: connected to deepseek-chat means the LLM is connected
   • AI status: rule mode means the local rule-based fallback is currently in use

4. Example API Calls

• F01 statistical analysis of profiled user counts within a region:

Invoke-WebRequest -UseBasicParsing "http://localhost:18080/api/stats/user-count?region_id=region_001&gender=F&age_min=18&age_max=34&start_time=1718179200000&end_time=1718265600000" | Select-Object -ExpandProperty Content

• F02 detail analysis of profiled users within a region:

Invoke-WebRequest -UseBasicParsing "http://localhost:18080/api/details?region_id=region_001&gender=F&province=%E5%B9%BF%E4%B8%9C%E5%B9%BF%E5%B7%9E&limit=10&offset=0&start_time=1718179200000&end_time=1718265600000" | Select-Object -ExpandProperty Content

• F03 distribution statistics of user profiles within a region:

Invoke-WebRequest -UseBasicParsing "http://localhost:18080/api/distribution?region_id=region_001&dimension=province&start_time=1718179200000&end_time=1718265600000" | Select-Object -ExpandProperty Content

• F04 agent-based visual Q&A:

Invoke-WebRequest -UseBasicParsing "http://localhost:18080/api/ask?question=%E5%A4%A7%E5%AD%A6%E5%9F%8E%E5%8C%BA%E5%9F%9F2%E5%B9%BF%E4%B8%9C%E5%B9%BF%E5%B7%9E%E5%B9%B4%E8%BD%BB%E5%A5%B3%E6%80%A7%E6%9C%89%E5%A4%9A%E5%B0%91" | Select-Object -ExpandProperty Content

• Real-time user location monitoring:

Invoke-WebRequest -UseBasicParsing "http://localhost:18080/api/realtime/positions" | Select-Object -ExpandProperty Content

• F05 team extension analysis: sequence recognition for monitored target regions:

Invoke-WebRequest -UseBasicParsing "http://localhost:18080/api/realtime/sequence-matches?limit=10" | Select-Object -ExpandProperty Content

5. Performance Benchmark Script

The project ships a reproducible performance benchmark script that measures:

• Response times of the F01 / F02 / F03 / F04 APIs
• End-to-end latency of the F05 real-time sequence recognition pipeline from Kafka injection to API visibility
• Duration of one full Spark offline batch refresh

Run it with:

python scripts/performance_benchmark.py

Optional parameters:

python scripts/performance_benchmark.py `
  --base-url http://localhost:18080 `
  --repetitions 5 `
  --timeout-seconds 90

The script outputs:

• artifacts/performance_benchmark.json
• artifacts/performance_benchmark.md

Notes:

• The script requires the system to be running and http://localhost:18080/health reachable
• The script injects a unique test IMSI into Kafka to measure the F05 real-time pipeline latency
• The script runs one Spark offline batch to measure the offline refresh duration
• It is recommended to run it once before the defense presentation, and backfill the measured values into the requirements specification or presentation materials

Example of the latest benchmark results:

• F01_user_count: average 40.12ms, P95 108.48ms
• F02_details: average 16.22ms, P95 31.77ms
• F03_distribution: average 14.21ms, P95 26.94ms
• F04_ask: average 2825.77ms, P95 3549.98ms
• F05_sequence_latency: 1239.56ms
• Spark_batch_refresh: 10820.12ms

How Changes Take Effect

Different components load source code differently, so whether you need to rebuild the image varies.

Overview

Component	Source Path	Load Method	Rebuild Needed?	How It Takes Effect
API backend + frontend	apps/api-server/ (includes frontend static/)	bind mount + hot reload	No	Takes effect automatically on save
Flink streaming job	apps/stream-job/	bind mount (:ro)	No	Cancel the old job and resubmit
Spark offline job	apps/batch-job/	bind mount (:ro)	No	Re-run spark-submit
One-shot generator	apps/generator/	COPY into image	Yes	Rebuild and rerun
Continuous generator	apps/generator/	COPY into image	Yes	Rebuild and restart
Infrastructure	hadoop / kafka / hbase / clickhouse / flume	image	Only when Dockerfile changes	Rebuild and restart

Tip: even when a rebuild is needed, the slow apt/pip layers are cached; only the COPY layer is redone, usually finishing in seconds. A slow first build is normal.

API Backend (no rebuild, hot reload)

API source is bind-mounted; after editing Python code, uvicorn reloads automatically — save and it takes effect.

Frontend (rebuild required)

The frontend lives in apps/web/ and is a Vue 3 + TypeScript SPA; rebuild after changes:

docker compose up -d --build web

If you modified apps/api-server/pyproject.toml (new dependencies), rebuild as well:

docker compose up -d --build api

Flink Streaming Job (no rebuild, resubmit required)

# Cancel the running job
docker compose exec flink-jobmanager flink list --running
docker compose exec flink-jobmanager flink cancel <jobId>
# Resubmit
docker compose exec flink-jobmanager flink run -d -py /opt/flink/jobs/stream_region_portrait.py

Spark Offline Job (no rebuild, resubmit required)

docker compose run --rm \
  -e CLICKHOUSE_URL=http://clickhouse:8123 \
  -e CLICKHOUSE_DATABASE=portrait_demo \
  -e SIGNAL_PATH=/shared/static/signal/generated_signal.txt \
  -e PROFILE_PATH=/shared/static/profile/user_profile.csv \
  -e REGIONS_JSON_PATH=/data/static/regions.json \
  spark /opt/spark/bin/spark-submit --master 'local[*]' /opt/spark/jobs/batch_region_portrait.py

Generators (rebuild required)

# One-shot generator
docker compose build generator && docker compose run --rm generator
# Continuous generator
docker compose up -d --build continuous-generator

Verification Commands

Run the unified verification script:

powershell -ExecutionPolicy Bypass -File .\scripts\verify.ps1

Or verify individually:

# Continuous generator output
docker compose exec continuous-generator wc -l /data/static/signal/generated_signal.txt
# Real-time positions API (1,000 users)
Invoke-WebRequest -UseBasicParsing "http://localhost:18080/api/realtime/positions" | Select-Object -ExpandProperty Content
# Replay time range
Invoke-WebRequest -UseBasicParsing "http://localhost:18080/api/replay/time-range" | Select-Object -ExpandProperty Content
# HDFS files
docker compose exec hadoop bash -lc "hdfs dfs -ls -R /demo/raw"
# Kafka messages
docker compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic region-signal-realtime --from-beginning --max-messages 15
# ClickHouse statistics
docker compose exec clickhouse clickhouse-client --query "SELECT region_id, user_count, male_count, female_count, unknown_count FROM portrait_demo.region_crowd_stats ORDER BY region_id"
# HBase real-time profiles
docker compose exec hbase python3 -c "import happybase; c=happybase.Connection('localhost',9090); c.open(); t=c.table('region_user_portrait'); rows=list(t.scan(limit=5)); print(len(rows)); [print(r[0].decode('utf-8')) for r in rows]"
docker compose exec hbase python3 -c "import happybase; c=happybase.Connection('localhost',9090); c.open(); t=c.table('region_sequence_match'); rows=list(t.scan(limit=5)); print(len(rows)); [print(r[0].decode('utf-8')) for r in rows]"
Invoke-WebRequest -UseBasicParsing http://localhost:18080/health | Select-Object -ExpandProperty Content
Invoke-WebRequest -UseBasicParsing "http://localhost:18080/api/stats/user-count?region_id=region_001" | Select-Object -ExpandProperty Content
Invoke-WebRequest -UseBasicParsing "http://localhost:18080/api/realtime/sequence-matches?limit=20" | Select-Object -ExpandProperty Content
Invoke-WebRequest -UseBasicParsing http://localhost:8088 | Select-Object -ExpandProperty StatusCode

Public API Endpoints

• GET /health
• GET /api/stats/user-count
• GET /api/details
• GET /api/distribution
• GET /api/compare
• GET /api/realtime/portraits
• GET /api/realtime/positions
• GET /api/realtime/sequence-matches
• GET /api/realtime/sequence-dashboard
• GET /api/realtime/alerts
• GET /api/realtime/effective-portrait
• GET /api/replay/time-range
• GET /api/replay/positions?timestamp=XXX
• GET /api/regions
• POST /api/regions
• PUT /api/regions/{id}
• DELETE /api/regions/{id}
• PUT /api/regions/reorder
• POST /api/regions/confirm
• GET /api/ask
• GET /api/ask-status
• GET /api/ai-report
• GET /sequence-dashboard

New Capabilities in This Release

• User-defined regions: managed from the right-hand panel, draw polygons on the map, dynamic point-in-polygon determination
  • Region management APIs: GET/POST/PUT/DELETE /api/regions, POST /api/regions/confirm, PUT /api/regions/reorder
  • Region configuration persistence: data/static/regions.json (concise single-list structure)
  • Map drawing: click to add vertices, click "Finish" in the panel to close and save
  • Draft editing: checking/reordering/renaming are all local operations; click "Confirm" to commit and activate in one shot
  • Deletion: calls the API directly and takes effect immediately, no confirmation needed
  • Custom funnel: ▲▼ drag ordering = funnel path order
  • Hover highlight: hovering over a panel region item shows the corresponding polygon on the map with gray diagonal hatching
  • Activation automatically clears the HBase + ClickHouse tables, and the pipeline regenerates outputs with the new regions
  • Region names can be edited by double-clicking (does not affect internal IDs)
• Real-time user location monitoring: real-time headcount per region + gender/age pie charts per region; 1,000 user markers wander probabilistically across 36 independent hotspots, refreshing every second
  • Hotspot system: 36 independent commercial-district POIs, not bound to any region (packages/domain/hotspot.py)
  • Movement model: distance-weighted probability selects the next hotspot (closer = more likely, low variance), with Bézier curve path smoothing
  • Base station system: 22 simulated base stations; laccell is set to the nearest one (data/base_stations.csv)
  • Real-time positions API: GET /api/realtime/positions (reads the last 50k lines of the signal file + point-in-polygon region assignment)
  • Map markers: Leaflet CircleMarker + Canvas rendering, blue markers, hover shows masked IMSI / gender / age / place of origin
• F05 team extension analysis: sequence recognition extension analysis for monitored target regions
• Funnel path: determined by the custom ordering in the region panel, no longer a static config file
• Crowd surge alerts: Flink detects crowd surges in real time over a 30-minute window (baseline ×1.6 or increment ≥5), with a 10-minute cooldown
• HBase result tables: region_user_portrait (user profiles), region_sequence_match (sequential path matches), region_event_alert (crowd alerts)
• Sequence recognition API: GET /api/realtime/sequence-matches
• Alerts API: GET /api/realtime/alerts
• Visualization pages: 5 navigation modules (location profiles / traffic heatmap / dwell characteristics / migration profiles / funnel analysis)
• Historical replay feature: review user location changes on the map over time
  • Replay APIs: GET /api/replay/time-range, GET /api/replay/positions?timestamp=XXX
  • Frontend interaction: the "Replay" button at the bottom left opens the control panel
  • Data source: read directly from the accumulated signal file, decoupled from the real-time pipeline
• Effective customer profiles: seven-dimensional profiling of users who completed a full target path and met the dwell conditions
  • Definition: users who hit a full path and are already written into region_sequence_match; the total path duration can be filtered via the min_path_minutes parameter
  • Seven profile dimensions:
    • Gender distribution (male/female/unknown)
    • Age distribution (seven buckets: youth 1-4, middle age 1-2, senior)
    • Resident/visitor identification (local resident / in-province visitor / out-of-province visitor, based on the province field)
    • Consumption activity (high/medium/low, based primarily on signaling event frequency; when no data is available, path duration serves as a proxy)
    • Path dwell duration distribution (<10 min / 10-30 min / 30-60 min / >1 hour)
    • Arrival time slot distribution (by hour)
    • Top 8 provinces of origin
  • Backend: packages/domain/portrait.py adds pure functions residency_type / path_duration_bucket / arrival_hour_label; hbase_service.py adds the effective_portrait() aggregation function
  • API: GET /api/realtime/effective-portrait?min_path_minutes=0
  • Frontend: a new "Effective Customer Profiles" SectionCard at the bottom of the location profiles panel, featuring ring charts (gender/age/resident-visitor/consumption activity), bar charts (path duration/arrival time slot), and a horizontal ranking chart (province of origin), auto-refreshing every 30 seconds

Shutdown and Cleanup

Stop all services:

docker compose down

Clean up together with data volumes:

docker compose down -v

Known Implementation Notes

• ClickHouse is built from a local Dockerfile in the current environment rather than pulling the official runtime image directly
• The Flink image includes kafka-clients and a python -> python3 compatibility setup
• This directory is independent of the parent-level docker-hadoop demo directory and can run standalone
• run_all.ps1 now reads .env; Spark offline batch processing refreshes from static files on the shared volume by default, avoiding HDFS hostname resolution issues when running standalone