Medical D2 — Data & Intelligence

Subgroup D2 for the healthcare monitoring program: vitals feature engineering, PyTorch models, Kafka streaming, PostgreSQL persistence, and a FastAPI layer for D3 dashboards.

🚀 Complete System Deployment Status

All Services Running on Server 139.59.66.194 (K3s/ArgoCD)

Service	Port	Namespace	Status	Image
D2 API	8000	d2	✅ Running	d2-data-intelligence:latest
D2 Worker	-	d2	✅ Running	d2-data-intelligence:latest
Backend API	8000	d2	✅ Running	backend:latest
Frontend (D3)	3000	d2	✅ Running	d3-frontend-dashboard:main
PostgreSQL	5432	d2	✅ Running	Stateful Set
InfluxDB	8086	d2	✅ Running	Stateful Set

Quick Verification

# SSH to server
ssh root@139.59.66.194

# View all running services
kubectl get all -n d2

# Check specific services
kubectl get pods -n d2 -o wide
kubectl get svc -n d2

# View logs for any service
kubectl logs -n d2 deployment/d2-api --tail=20
kubectl logs -n d2 deployment/backend-api --tail=20
kubectl logs -n d2 deployment/d3-frontend --tail=20

For detailed service-specific verification:

• D2 Data Intelligence: See docs/ARCHITECTURE.md
• Backend API: See backend/README.md
• Frontend Dashboard: See d3-frontend-dashboard/DEPLOY.md

Securely Accessing Internal UIs (Port Forwarding)

For internal infrastructure tools like ArgoCD and InfluxDB, it is highly recommended to use SSH port forwarding to access them locally, rather than exposing them directly to the public internet.

1. ArgoCD UI Run this command in a terminal window on your local machine:

ssh -L 8080:localhost:8080 root@139.59.66.194 "kubectl port-forward svc/argocd-server -n argocd 8080:443"

Then navigate to https://localhost:8080 in your browser. (Note: You can safely bypass the self-signed certificate warning).

2. InfluxDB UI Run this command in another terminal window:

ssh -L 8086:localhost:8086 root@139.59.66.194 "kubectl port-forward svc/d2-influxdb -n d2 8086:8086"

Then navigate to http://localhost:8086 in your browser.

System architecture (D2 in full Group D context)

This diagram shows how D2 (Data & Intelligence) sits between D1 (devices and edge), the event bus, D3 (dashboards and real-time UX), and D4 (platform, security, and observability). Arrows follow the main path: raw vitals → stream processing → features and models → persistence and APIs → clinicians and operators.

Zone	Responsibility
D1 — Device & edge	Wearables / gateway, optional Raspberry Pi, MQTT; validates and forwards vitals.
Kafka	vitals.raw → D2 worker → vitals.scored + alerts; decouples producers from consumers.
D2 — This repo	Stream worker, feature engine, PyTorch inference, PostgreSQL, FastAPI; hybrid batch path from the database for retraining and analytics.
D3 — Sys eng & interaction	Doctor and patient UIs (Next.js / Flutter), Socket.IO or gateway for live alerts.
D4 — Platform & security	Docker / Kubernetes, Keycloak (OAuth2/OIDC), Vault, Prometheus / Grafana, logs and tracing, GitOps (Argo CD), Terraform, CI/CD (GitHub Actions).

For a shorter text-only overview, see docs/ARCHITECTURE.md.

Quick start (local API)

cd d:\medical_d2
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
pip install -e ".[dev]"
copy .env.example .env

Edit repo-root .env: set DATABASE_URL for your Postgres host (Supabase pooler URI + PG_SSL_REQUIRED=true are typical). URL-encode special characters in passwords inside the URI.

$env:PYTHONPATH = "src"
uvicorn d2.api.app:app --reload --host 0.0.0.0 --port 8000

• Open http://127.0.0.1:8000/docs for OpenAPI.
• GET /live confirms the API process is running.
• GET /ready confirms the API can reach the database (503 → fix DATABASE_URL / TLS / network).
• GET /health checks broader dependencies and may return 503 until Kafka / optional services are configured.

Streaming stack (Kafka worker, certs): see Kafka stream and database/HOW_TO_RUN.md.

Tests

$env:PYTHONPATH = "src"
pytest

Admin Demo Seed

The admin dashboard demo data lives in src/d2/db/seeds/demo/medipulse_admin_demo_seed.sql. It is development/demo data only: reruns update deterministic demo rows, keep demo users at @demo.medipulse.local, keep demo devices under DEMO-DEVICE-*, and do not overwrite real Keycloak-synced users such as admin@test.com, doctor@test.com, or patient@test.com.

Core identity/domain schema for shared environments is owned by the backend repo. D2 should not redefine, drop, or overwrite backend-owned tables in production-like databases. D2 schema changes should be limited to analytics/model/vitals-specific tables unless coordinated through the backend migrations. The D2 Kubernetes base does not run D2 SQL migrations automatically; backend deploys own shared schema migration Job.

Run D2-owned analytics migrations with a database URL pointed at a local or disposable development database:

$env:PYTHONPATH = "src"
$env:DATABASE_URL = "postgresql://..."
python -m d2.db.migrations.run

The migration runner applies every SQL file in src/d2/db/migrations in lexicographic order. This folder must not contain backend-owned core auth/domain schema. To apply only the idempotent demo seed to an existing development database, run the SQL file directly with psql:

psql "$DATABASE_URL" -f src/d2/db/seeds/demo/medipulse_admin_demo_seed.sql

or:

DATABASE_URL="postgresql://..." scripts/run-demo-seed.sh

The seed prints verification counts at the end for demo users, doctors, patients, devices, alerts, and preserved real users. Expected demo counts are 21 users, 10 doctors, 10 patients, 10 devices, and 20 alerts.

Validation queries:

SELECT email, keycloak_sub, role
FROM users
WHERE email IN ('admin@test.com', 'doctor@test.com', 'patient@test.com')
ORDER BY email;

SELECT count(*) AS demo_users
FROM users
WHERE email LIKE '%@demo.medipulse.local';

SELECT count(*) AS demo_users_with_keycloak_sub
FROM users
WHERE email LIKE '%@demo.medipulse.local'
  AND keycloak_sub IS NOT NULL;

SELECT column_name
FROM information_schema.columns
WHERE table_name = 'users'
  AND column_name = 'keycloak_sub';

Repo layout

See docs/REPOSITORY_STRUCTURE.md for the full tree and docs/CONTRIBUTING.md for branch → PR → main workflow.

Path	Purpose
src/d2/	Shared Python package (features, models, streaming, api, db)
data/	Raw / processed datasets (large files gitignored)
scripts/	Dataset build and maintenance scripts
docs/	Architecture and D3 integration contracts
tests/	Unit + integration tests
infra/	Docker / K8s / Terraform (with D4)

Team workflow

1. Create a branch: feature/<area>-<short-description> (see docs/CONTRIBUTING.md).
2. Implement in the folder that matches your track; keep train/serve feature logic in src/d2/features/.
3. Open a Pull Request into main; ensure pytest and CI pass.
4. Update docs/INTEGRATION_D3.md whenever request/response JSON changes.

For the command-by-command Droplet setup using k3s and Argo CD, see docs/DROPLET_K3S_ARGOCD_RUNBOOK.md.

Who owns what (reminder)

Details: docs/MEMBER_ROLES.md (five peer roles).

• Member 1 — Features — src/d2/features/, scripts/, data/
• Member 2 — ML — src/d2/models/
• Member 3 — Kafka — src/d2/streaming/
• Member 4 — API + DB — src/d2/api/, src/d2/db/, docs/INTEGRATION_D3.md (with code changes)
• Member 5 — CI & observability — .github/workflows/, src/d2/observability/, tests/integration/

Each module file contains TODO comments describing what to implement there.