Status: ✅ Operational - Fully containerized and deployed
A Docker-based system for collecting real-time power consumption data from TP-Link Tapo P110 smart plugs distributed globally, storing in TimescaleDB, and exploring through the GOS REM Data Exploration Tool.
- Simple Deployment Guide - Step-by-step setup for non-technical users
- User Guide - Complete guide to using the GOS REM Data Exploration Tool
- Changelog - Version history and release notes
- Documentation - All development notes and detailed guides
- LEM (companion project) - Local Energy Measurement: a lab/field tool that measures plugs over the LAN (no TP-Link cloud) and streams into REM experiments. See the Field API.
For non-technical users - Get the system running in under 10 minutes!
You'll need:
- A computer with Docker Desktop installed (Download here)
- Your TP-Link Cloud API credentials (Client ID, Client Secret, and Refresh Token)
- About 10 minutes
- Download and install Docker Desktop from docker.com
- Open Docker Desktop and make sure it's running (you'll see a Docker icon in your system tray)
- Open a terminal/command prompt
- Run:
(Or download the ZIP file from GitHub and extract it, then open a terminal in that folder)
git clone https://github.com/Greening-of-Streaming/rem.git cd rem
-
Copy the template file:
cp ENV_TEMPLATE .env
-
Open the
.envfile in a text editor and fill in your TP-Link credentials:TPLINK_CLIENT_ID=your-client-id-here TPLINK_CLIENT_SECRET=your-secret-here TPLINK_REFRESH_TOKEN=your-refresh-token-here POSTGRES_PASSWORD=choose-a-secure-password
Run this command:
docker-compose up -dWait about 30 seconds, then check if everything started:
docker-compose psAll services should show "Up" status.
Open your web browser and go to:
- Main Interface: http://localhost:7001
- The system will automatically start collecting data from your TP-Link devices
- Click "Exploration" in the top menu to see charts
- Click "Groups" to organize your devices into experiment groups
- Data will start appearing within 30 seconds
To stop everything:
docker-compose downTo stop but keep your data:
docker-compose stop- Check the logs:
docker-compose logs collector - See Troubleshooting section below
- Check the detailed deployment guide
The Greening of Streaming (GOS) organization uses this system to:
- Collect real-time power measurements from Tapo P110 wireless smart plugs
- Store time-series data in TimescaleDB (PostgreSQL extension)
- Explore data through the interactive GOS REM Data Exploration Tool
- Analyze energy usage during streaming experiments with advanced filtering, grouping, and statistical analysis
- Remote Energy Measurement (REM): Monitor power consumption of computers/equipment globally
- Streaming Experiments: Measure energy impact of different streaming configurations
- Cost Analysis: Calculate energy costs across different compute loads
- Sustainability Research: Support GOS's mission of understanding streaming energy consumption
- ✅ Fully Containerized: Docker Compose stack for collector, TimescaleDB, and admin UI
- ✅ TimescaleDB: PostgreSQL-based time-series database
- ✅ Data Collector: Polling TP-Link API every 30 seconds
- ✅ GOS REM Data Exploration Tool: Interactive web UI for data analysis
- ✅ Collector Control: Web-based start/stop/pause and polling interval control
- ✅ Experiment Management: Device grouping and A/B testing support
- ✅ Snapshot Gallery: Save and archive chart snapshots with annotations
- ✅ Database Export/Import: Full backup and migration capabilities
- ✅ Production Deployment: Primary instance running on Akamai Linode (
rem.greeningofstreaming.org) - ✅ Pi400 Dev/Staging: Original Pi400 stack retained as development and backup environment
- 📊 Interactive Charts: Overlay multiple experiments, toggle device visibility, statistical overlays
- 🔬 Experiment Groups: Create device groups for A/B testing and comparisons
- 📈 Statistical Analysis: Mean, median, total, and average calculations with legend-based filtering
- 📸 Snapshot Archive: Save chart snapshots with annotations and download as ZIP (image + CSV + metadata)
- ⏱️ Dynamic Time Ranges: Zoom, pan (xy mode), and select time ranges for detailed analysis
- 🖱️ Grafana-like Selection: Single-click to select only one device, shift-click to toggle multiple devices
- 💾 Data Export/Import: Full database backup and migration support (ZIP format)
- 🎨 GoS Branding: Consistent branding with Greening of Streaming logo and colors
┌─────────────────────────────────────────────────────────────┐
│ Docker Compose Stack │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ ┌──────────────────┐ ┌──────────────┐ │
│ │ Data Collector │ │ TimescaleDB │ │ Admin UI │ │
│ │ (Python) │──│ (PostgreSQL) │──│ (FastAPI) │ │
│ └─────────────────┘ └──────────────────┘ └──────────────┘ │
│ │ │
│ └──────────────┬───────────────┘
│ │
│ ┌──────────────────┐
│ │ Data Exploration│
│ │ Tool │
│ └──────────────────┘
└───────────────────────────────┬───────────────────────────────┘
│
┌───────────────────────┐
│ TP-Link Cloud API │
│ (Tapo P110 Devices) │
└───────────────────────┘
- Python Data Collector: Polls TP-Link Cloud API, collects power readings from all P110 devices
- TimescaleDB: PostgreSQL-based time-series database for storing measurements
- Admin UI (FastAPI): Web interface for system management and data exploration
- GOS REM Data Exploration Tool: Interactive charts, experiment grouping, snapshot gallery
- Docker & Docker Compose
- TP-Link Cloud API credentials (client ID, secret)
- OAuth authorization code (obtained via browser)
- Tapo P110 smart plugs registered to TP-Link account
See Simple Deployment Guide above for step-by-step instructions.
For developers familiar with Docker:
git clone https://github.com/Greening-of-Streaming/rem.git
cd rem
cp ENV_TEMPLATE .env
# Edit .env with your credentials
docker-compose up -dAccess:
- Data Exploration Tool: http://localhost:7001
- Manage Groups: http://localhost:7001/manage
- Snapshot Gallery: http://localhost:7001/gallery
- Admin (Export/Import): http://localhost:7001/admin
# TP-Link Cloud API
TPLINK_CLIENT_ID=your-client-id
TPLINK_CLIENT_SECRET=your-secret
TPLINK_REFRESH_TOKEN=initial-token
# PostgreSQL/TimescaleDB
POSTGRES_HOST=timescaledb
POSTGRES_PORT=5432
POSTGRES_DB=gos_rem
POSTGRES_USER=gos
POSTGRES_PASSWORD=your-secure-password
# Collector
POLL_INTERVAL=30 # seconds (configurable via UI)
LOG_LEVEL=INFO
# Admin UI Basic Auth (optional, recommended for production)
ADMIN_BASIC_USER=your-admin-username
ADMIN_BASIC_PASSWORD=your-strong-password- Current: 10 seconds (Pi400)
- Recommended: 30 seconds (safer for API limits)
- Range: 10-300 seconds
CREATE TABLE gos_rem (
time TIMESTAMPTZ NOT NULL,
alias TEXT NOT NULL,
power_watts FLOAT NOT NULL
);
SELECT create_hypertable('gos_rem', 'time');SELECT time_bucket('1 minute', time) AS time,
alias,
AVG(power_watts) AS avg_power
FROM gos_rem
WHERE time > NOW() - INTERVAL '1 hour'
AND alias = 'London-Office-PC'
GROUP BY time_bucket('1 minute', time), alias
ORDER BY time;- Interactive Charts: Overlay multiple experiments with smooth curves and zoom/pan
- Device Grouping: Create experiment groups (A/B testing support)
- Statistical Analysis: Mean, median, total, and average with dynamic recalculation
- Legend-based Filtering: Show/hide devices and recalculate statistics
- Time Range Selection: Zoom and pan to focus on specific time periods
- Annotation System: Add timeline markers with notes
- Snapshot Gallery: Save, archive, and search historical charts
- Collector Control: Start/stop polling and adjust polling frequency via UI
- Database Export/Import: Full backup and migration support for all data, experiments, groups, and snapshots
# Default ports
7001: Admin UI / Data Exploration Tool
5432: TimescaleDB (PostgreSQL)- Role: Primary production deployment
- Instance: Nanode 1GB ($5/month)
- Region: Closest to team
- Access:
https://rem.greeningofstreaming.org(Caddy + Let's Encrypt) - Auth: HTTP Basic Auth via admin middleware (
ADMIN_BASIC_USER/ADMIN_BASIC_PASSWORD)
- Status: ✅ Raspberry Pi 400, used for development and staging
- URL: https://rem.greeningofstreaming.org
- Access: Protected by Authelia authentication
- Services: All services running in Docker containers
- Base URL:
https://aps1-openapi.tplinknbu.com/v1/ - Authentication: OAuth 2.0
- Rate Limits: Unknown (being conservative)
- Devices Supported: Tapo P110, P110M
/oauth/token- Get/refresh access token/getDeviceList- List all devices/device/deviceControl- Get real-time power
stats/
├── app/
│ ├── collector.py # Main data collector (polls TP-Link API)
│ ├── config/ # Configuration files
│ └── Dockerfile # Collector container
├── admin/ # Admin web UI and Data Exploration Tool
│ ├── app.py # FastAPI backend
│ ├── templates/ # HTML templates
│ ├── static/ # CSS, JavaScript, images
│ └── Dockerfile # Admin UI container
├── scripts/ # Database initialization scripts
├── docker-compose.yml # Full stack orchestration
├── ENV_TEMPLATE # Environment variables template
└── README.md
# Unit tests
pytest tests/
# Integration test (requires API credentials)
pytest tests/integration/
# Load test
python tests/load_test.pyThe easiest way to backup or migrate your data is through the web interface:
- Navigate to Admin in the menu (http://localhost:7001/admin)
- Click Export Database to download a complete backup ZIP file
- To restore, select the ZIP file and click Import Database
The export includes:
- All TimescaleDB power measurement data
- All experiments and configurations
- All device groups
- All snapshots (images and metadata)
- All annotations
For manual backups using pg_dump:
# Export database
docker exec stats-timescaledb pg_dump -U gos gos_rem > backup_$(date +%Y%m%d).sql
# Copy JSON files
cp admin/data/device_groups.json backup/
cp admin/data/experiments.json backup/
cp admin/data/snapshots.json backup/
cp -r admin/data/snapshots/ backup/# Restore database
cat backup_*.sql | docker exec -i stats-timescaledb psql -U gos gos_rem
# Restore JSON files
cp backup/*.json admin/data/
cp -r backup/snapshots/* admin/data/snapshots/# Check logs
docker-compose logs collector
# Common issues:
# - Missing refresh token
# - Invalid credentials
# - API rate limit# Check collector is writing
docker-compose logs collector | grep "Wrote"
# Check TimescaleDB
docker exec stats-timescaledb psql -U gos -d gos_rem -c \
"SELECT COUNT(*), MAX(time) FROM gos_rem WHERE time > NOW() - INTERVAL '1 hour';"# Get new authorization code from browser
# Update .env with new code
# Restart collector
docker-compose restart collectorThis project is part of the Greening of Streaming organization's research initiative.
- Fork the repository
- Create feature branch
- Make changes
- Test locally
- Submit pull request
- Python: PEP 8
- Docstrings: Google style
- Type hints: Required for public functions
- Assess current Pi400 setup
- Create project plan
- Improve Python collector
- Create Docker Compose stack
- Test local deployment
- Configurable polling
- Data retention policies
- Backup automation
- Deploy to Akamai Linode
- Set up monitoring
- SSL/TLS configuration
- Team access setup
- Web-based control panel
- Start/stop/pause controls
- Device management
- Log viewer
- Compute: $5/month (Nanode 1GB)
- Backup: $2/month
- Total: ~$7/month
- TimescaleDB Cloud: $0-50/month
- Cloud Run: $0-5/month
- Total: $0-55/month
Recommendation: Self-hosted for cost control
- OAuth credentials in environment variables
- PostgreSQL/TimescaleDB password in environment variables
- No secrets in Git repository
- TimescaleDB not exposed publicly
- Admin UI behind authentication:
- Production (Linode): Built-in HTTP Basic Auth + Caddy HTTPS
- Dev (Pi400): Traefik/Authelia in front of the admin UI
- HTTPS for public interfaces
- bentasker/tplink_to_influxdb - Original inspiration
- Greening of Streaming - Organization
Report issues on GitHub Issues page
- Project Lead: Dom Robinson
- Organization: Greening of Streaming
- Web: https://www.greeningofstreaming.org
Apache License 2.0 — see LICENSE.
Last Updated: 2026-03-25
Version: 1.5.0
Status: ✅ Operational – Production on Linode, Pi400 as dev/staging
Documentation: All development notes and guides are in the docs/ folder
Added
- ghcr.io publishing:
docker-compose.ymlcarries bothbuild:andimage:blocks forcollectorandadmin.docker compose build && pushfrom a dev box,docker compose pull && up -don the host. Same file in both places. Images atghcr.io/nebul2/rem-collector/rem-admin(public). enabledtoggle is now real. Whencollector_control.jsonsays"enabled": false, the collector skips the cycle (no TP-Link call, no DB write). Previously the toggle was cosmetic — the admin UI showed it but the collector ignored it.
Changed
- Collector defaults to disabled on a fresh deploy. Operators must flip the admin-UI toggle to begin polling. Stops accidental TP-Link traffic on redeploys / new installs.
- TimescaleDB pinned to
2.25.2-pg16(was a moving:latest-pg16tag).
Fixed
- Production
gos_remwas a plain table, not a hypertable —create_hypertablehad silently failed at first init three months ago. Migrated in place (55s, 13 chunks); admin healthcheck has gone from "unhealthy for 5 weeks" to "healthy" without code change. Init script was already correct for fresh deploys.
Added
- Adaptive backoff when the TP-Link cloud signals overload (HTTP 429/503, rate-limit style messages): the collector calms down (longer sleep, fewer parallel workers, more chunk delay) and recovers after clean cycles. Status is written to
collector_status.jsonand shown under Exploration → TP-Link cloud / throttling (effective settings, health, reset button). Toggleadaptive_backoffin the UI or JSON.
Added
benchmark_poll_cycle.py(in the collector image): time full TP-Link rounds and compare effective cadence (round + poll interval) to your target (e.g. 10s)../scripts/run_benchmark_poll.shordocker compose exec collector python /app/benchmark_poll_cycle.py --sweep --rounds 3.
Added
- Parallel device polls: the collector can run up to N concurrent
getDeviceRealTimeEnergycalls per chunk (default 8, range 1–32). Devices are processed in waves; Device query delay applies between chunks when N is greater than 1, or between each device when N is 1 (sequential). - Exploration UI control for parallel workers;
COLLECTOR_PARALLEL_WORKERSenv can override the JSON setting for ops. - Substantially shorter round duration, so per-device sample spacing can approach round time + sleep with a much smaller round time (e.g. ~10–20s instead of ~60s+ for large fleets when parallel is enabled).
Documentation / UX
- “Polling frequency” in Exploration is the sleep after one full sequential pass over all devices, not the time between samples for a single device. Per-device spacing ≈ round duration + sleep (often ~60–80s with many plugs). Export README and Exploration copy updated so “10s + 60s” confusion is explained.
Fixed
- Poll interval from the Exploration UI (e.g. Ben’s 10s) was saved under the admin container’s data volume, while the collector read
collector_control.jsonfrom a separate collector volume — so the running collector never applied UI changes and kept sleeping at yaml/env interval (~30–60s). The collector now mounts the admin data volume read-only and readsCOLLECTOR_CONTROL_FILE=/app/data/admin/collector_control.json, and appliespoll_intervalfrom that file every cycle (takes precedence overPOLL_INTERVAL/ yaml).
Note: Past experiment windows cannot be re-sampled; only new data reflects the corrected behavior.
Changed
- Collector now honors
POLL_INTERVALfrom the environment (e.g. Docker.env), matching what Compose already passed; previously onlyapp/config/config.yamlpoller.intervalwas used. - Experiment export README inside the ZIP explains that row spacing reflects poll cadence, not UI chart aggregation.
Fixed
- “Download all data” experiment export failed when experiment
time_rangestored naive ISO timestamps (no timezone) while “now” end time was UTC-aware. All bounds are normalized to UTC-aware before comparison and SQL.
Fixed
- 500 / Internal Server Error on all HTML pages after dependency upgrades: Starlette 1.x requires
Jinja2Templates.TemplateResponse(request, name, context)instead of(name, context). All admin page renders were updated accordingly.
What's New
- Experiment “Download all data” — ZIP export of every stored power reading (one row per collector poll) for the experiment time range and devices from linked groups, plus
experiment_metadata.json,annotations.json, andREADME_export.txtexplaining the difference vs chart aggregation. - API —
GET /api/experiments/{experiment_id}/exportreturns that ZIP (same auth rules as the rest of the admin UI). - Routing —
GET /experimentredirects to/experiments(307) for common bookmark typos. - Copy — Experiments and Exploration pages clarify that Exploration charts use aggregated series; this export is full-resolution history.
See docs/CHANGELOG.md for the full changelog.
Fixed
- Collector no longer stalls silently when the TP-Link cloud returns token invalid (
-10902): it refreshes the OAuth token and retries the polling cycle, and prefers the persisted rotating refresh token on disk.
What's New
- ✅ First production deployment on Akamai Linode (
rem.greeningofstreaming.org) - ✅ Built‑in HTTP Basic Auth in the admin UI, controlled via
ADMIN_BASIC_USER/ADMIN_BASIC_PASSWORD - ✅ Caddy reverse proxy on Linode for HTTPS termination (Let's Encrypt) and HTTP→HTTPS redirects
- ✅ End‑to‑end data migration from Pi400 to Linode TimescaleDB (historic + live data)
- ✅ Documented disaster‑recovery path from GitHub + DB backup
Cleanup Release - Removed unused services:
- Removed Grafana (replaced by custom admin UI)
- Removed InfluxDB (migrated to TimescaleDB)
- Freed ~6GB disk space on production server
Stability Update - API rate limiting fix:
- Added configurable Device Query Delay setting in UI
- Prevents TP-Link API rate limiting (429 errors)
- Fixed collector hanging after rate limit errors
User Feedback Release - Based on feedback from Ben:
- Added horizontal scrollbar for zoomed charts
- Snapshot ZIP downloads now include CSV data export
- Improved chart navigation and data export capabilities
- ✅ Database Export/Import: Full backup and migration support - export all data, experiments, groups, and snapshots as ZIP
- ✅ Grafana-like Device Selection: Single-click legend to select one device, shift-click to toggle multiple devices
- ✅ Snapshot ZIP Downloads: Download snapshots as ZIP containing image, CSV data, and metadata
- ✅ Chart B Statistical Overlays: Statistical overlays now work correctly in split chart mode
- ✅ Experiment Reactivation: Clear end dates to reactivate "current" experiments
- ✅ Improved Pan/Zoom: Click and drag to pan in both horizontal and vertical directions
- ✅ Gallery Layout Fixes: Better handling of long experiment details in snapshot gallery
- Fixed Chart B statistical overlays not working in split mode
- Fixed gallery download button not working
- Fixed gallery layout breaking with long experiment details
- Fixed experiment reactivation (clearing end dates)
- Fixed pan mode (now supports both x and y axes)
- Fixed alert spam during auto-refresh failures
- Added export/import endpoints with pg_dump/CSV fallback support
- Improved error handling for consecutive API failures
- Enhanced UI with progress indicators for export/import
- Better validation and user feedback for destructive operations
- ✅ Improved Error Handling: Better timeout management and error messages for large queries
- ✅ Smart Aggregation: Automatic interval adjustment based on time range and device count
- ✅ Query Optimization: Faster queries for large datasets (>50k points)
- ✅ Enhanced Time Range: Added 7-day and 30-day lookback options
- ✅ Live Updates: Auto-refreshing charts with pause/resume controls
- ✅ Fixed Time Range & Aggregation: Dropdowns now properly reload charts
- ✅ Better Error Messages: Clear feedback for timeouts and API errors
- Fixed 502/504 timeout errors for large time ranges
- Fixed time range dropdown not updating charts
- Fixed aggregation dropdown not applying changes
- Fixed JSON parsing errors on failed requests
- Fixed MutationObserver errors in Chart.js
- Fixed database password configuration issues
- Added 5-minute query timeout handling
- Optimized time bucket generation for large datasets
- Improved fetch error handling with proper JSON parsing
- Added request abort controllers for timeout management
- ✅ Complete Docker-based containerization
- ✅ Migration from InfluxDB to TimescaleDB (PostgreSQL extension)
- ✅ GOS REM Data Exploration Tool with interactive charts
- ✅ Experiment management system with device grouping
- ✅ Snapshot gallery with annotations
- ✅ Collector control via web UI (start/stop/polling frequency)
- ✅ Default chart loading with all devices
- ✅ Full GoS branding with logo and colors
- Legacy InfluxDB data can be migrated using forward-fill script
- Old native InfluxDB/Grafana installation on Pi400 has been decommissioned