diff --git a/README.md b/README.md index 333ae48..9178104 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,14 @@ # SPECTRA-Lab -A semiconductor characterization and manufacturing-execution platform. Combines a Python/FastAPI backend, a Next.js 14 frontend, PostgreSQL persistence, and a Celery+Redis async layer into one repo. Covers metrology (electrical, optical, structural, chemical), process control (CVD, PVD, CMP, Diffusion, Oxidation, Ion Implant, RTP), SPC, ML/virtual-metrology, and a LIMS/ELN. +A semiconductor characterization and manufacturing-execution platform. Combines a Python/FastAPI backend, a Next.js 15 / React 19 frontend, PostgreSQL persistence, and a Celery+Redis async layer into one repo. Covers metrology (electrical, optical, structural, chemical), process control (CVD, PVD, CMP, Diffusion, Oxidation, Ion Implant, RTP), SPC, ML/virtual-metrology, and a LIMS/ELN. -> **Project status (2026-06-09):** the multi-phase remediation pass is **complete** — the platform is functionally whole and honest end-to-end: real DB-backed login authorizes all three services, every one of the ~68 sidebar routes fetches real backend data, every characterization method is backed by real physics, and there are **zero silently-fabricated values** anywhere in the UI. See [`docs/REMEDIATION_STATUS.md`](docs/REMEDIATION_STATUS.md) for the authoritative current state and the [Remediation pass](#remediation-pass-2026-05--2026-06) section below. +> **Project status (2026-08-04): production-ready.** The remediation pass, the production-completion plan (Phases 0–6), and the account-lifecycle phase are all complete and merged. [Release v0.1.0](https://github.com/alovladi007/SPECTRA-Lab/releases/tag/v0.1.0) exists with all four service images built, Trivy-scanned to **zero HIGH/CRITICAL findings**, and pushed to ghcr by the tag-triggered CD pipeline; the Kubernetes manifests deploy every workload (services, celery workers, MinIO, Postgres, redis, backups) and are render-validated in CI. What stands between this repo and a live deployment is provisioning only — a cluster, a domain, and secret values — with the exact sequence in [`docs/deployment/PRODUCTION_RUNBOOK.md`](docs/deployment/PRODUCTION_RUNBOOK.md) and one-command helpers in [`scripts/provision/`](scripts/provision/). ## Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ -│ apps/web Next.js 14 frontend (port 3012) │ +│ apps/web Next.js 15 frontend (port 3012) │ └────────────────────────────┬────────────────────────────────────┘ │ REST + WebSocket ┌────────────────────────────┴────────────────────────────────────┐ @@ -84,7 +84,7 @@ If Docker is broken or you want faster inner-loop iteration, the repo ships a co ## Repository layout ``` -apps/web/ Next.js 14 + React 18 + Tailwind + shadcn/ui +apps/web/ Next.js 15 + React 19 + Tailwind + shadcn/ui src/app/ Routes: cvd, dashboard, data, experiments, login, process-control, results, samples, system @@ -100,12 +100,15 @@ services/ shared/ SQLAlchemy models, JWT/RBAC, SPC engine, test fixtures -alembic/versions/ 15 migration files; chain head = - 20260604_1800_0015 (PVD/CMP tables) +alembic/versions/ 24 migration files; chain head = + 20260803_1000_0024 (password reset tokens) -infra/ Prometheus + Grafana provisioning -k8s/ Kustomize base + production overlay - (Helm chart land in Phase 9) +infra/ Prometheus + Grafana + Alertmanager + Keycloak + (dev realm) + MinIO lifecycle provisioning +k8s/ Kustomize base + production overlay: all four + services, celery worker/beat, pc-worker, MinIO, + Postgres, redis, backup CronJobs, ingress + TLS; + render-validated in CI on every PR docs/ Architecture, API specs, build plan, archived session reports @@ -123,6 +126,8 @@ tests/ Cross-service integration tests | [`docs/ROADMAP.md`](docs/ROADMAP.md) | High-level roadmap | | [`docs/DATA_MODEL_SPECIFICATION.md`](docs/DATA_MODEL_SPECIFICATION.md) | Database schema reference | | [`docs/api/`](docs/api/) | OpenAPI specs and per-service API references | +| [`docs/deployment/PRODUCTION_RUNBOOK.md`](docs/deployment/PRODUCTION_RUNBOOK.md) | Go-live sequence: cluster → add-ons → DNS/TLS → secrets → deploy → first admin | +| [`docs/deployment/`](docs/deployment/) | SECRETS (Sealed Secrets flow), AUTH (SSO), MIGRATIONS, ALERTING runbooks | | [`docs/sessions/`](docs/sessions/) | Per-session implementation notes | | [`docs/archive/sessions/`](docs/archive/sessions/) | Historical status reports (not authoritative) | @@ -152,92 +157,45 @@ The platform's domain coverage. Depth varies by area; details and current implem - SPC: X-bar/R, I-MR, EWMA, CUSUM, Western Electric rules, Cp/Cpk/Pp/Ppk, drift detection - ML/Virtual Metrology: AutoML (Optuna), forecasting (Prophet), explainability (SHAP/LIME), drift, A/B testing, model registry, online learning - LIMS/ELN: samples, recipes (with PI approval workflow), SOPs, ELN with e-signatures, exports -- Auth: JWT (HS256/RS256), 4-tier RBAC, multi-tenancy via `organization_id` +- Auth & accounts: JWT (HS256) + optional Keycloak OIDC/SSO (RS256), 5-role RBAC, Postgres row-level security on all 91 org-scoped tables, password reset + change-password + admin user invitations over env-configured SMTP (dev mode logs the emails) - Real-time: WebSocket telemetry; Celery + Redis for background jobs -- Observability: Prometheus + Grafana scaffolding (full wiring in Phase 5) - -## Phase progress - -The build plan is the authoritative status. Summary as of 2026-05-12: - -| Phase | Sessions | Status | -|---|---|---| -| 1 — Hygiene & consolidation | 6 | ✅ done | -| 2 — Finish half-built services | 8 | ✅ done | -| 3 — Data integrity & migrations | 3 | ✅ done | -| 4 — Auth, security, multi-tenancy | 4 | ✅ done | -| 5 — Observability | 3 | ✅ done | -| 6 — Storage & data lifecycle | 3 | ✅ done | -| 7 — Frontend hardening | 4 | 🚧 3/4 (7.1, 7.2, 7.3 done) | -| 8 — Testing, CI/CD, releases | 4 | ⏳ | -| 9 — Production deployment | 4 | ⏳ | - -### Phase A — Foundation re-audit (2026-05-11 → present) - -The "Phase 1–6 done" claim above was structural-only. A re-audit found -several load-bearing items left as stubs: audit-log table with zero -writers, multi-tenancy filters opt-in only, ~77 SQLAlchemy models -absent from alembic, and 35 F821 real bugs (renamed classes still -referenced in routers). Phase A is the foundation-fix that hardens -those before adding new product surface. - -| Step | What | Status | -|---|---|---| -| A1 | Stabilize CI (Black + Ruff baseline, migration 0006, test imports, coverage floor) | ✅ done (PRs #1–#3) | -| A2 | PdM orphan tables → alembic (`equipment_*`, `maintenance_events`) | ✅ done (PR #4) | -| A3 | Audit log writers via SQLAlchemy event listener + `AuditContext` contextvar | ✅ done (PR #5) | -| A4 | Postgres RLS for multi-tenancy (17/27 tables) + GUC plumbing | ✅ done (PR #6) | -| A.next.1 | Backfill `organization_id` on 10 drifted tables → RLS coverage 27/27 | ✅ done (PR #7) | -| A.next.2 | Fix 35 F821 real bugs + re-enable the Ruff rule | ✅ done (PR #8) | -| A6 | OpenTelemetry tracing across all 3 services + Celery | 🔄 in flight (PR #9) | -| A5 | This document refresh | 🔄 in flight | -| A7 | Wire dormant frontend WebSocket hook + `WaferMap.tsx` to real pages | ⏳ | - -Phase A is the gate to A7 (frontend) and the deferred sub-items of -Phase 7.4 (Playwright e2e tests) and Phase 8 (CI/CD release engineering). -After Phase A the codebase is honest with itself about its state — -schema matches model, audit trail captures every mutation, Postgres -enforces tenant isolation, every cross-service request emits a trace. - -### Remediation pass (2026-05 → 2026-06) - -A full four-dimension audit (backend / frontend / auth-db-infra / tests-ci) -found the platform booted but behaved like a convincing shell: auth couldn't -complete end-to-end, ~47/87 frontend pages rendered fabricated data, a tier -of "looks-real" endpoints returned hardcoded numbers, and CI exercised only a -sliver. The phased fix is in [`docs/REMEDIATION_PLAN.md`](docs/REMEDIATION_PLAN.md); -the authoritative end-state is [`docs/REMEDIATION_STATUS.md`](docs/REMEDIATION_STATUS.md). -**The pass is complete** — every phase landed as its own CI-green PR, one -branch (`main`) on origin throughout: - -| Phase | What | Status | -|---|---|---| -| 0 | Foundation cleanup: delete dead/damaged code, pin black+ruff (end lint vintage-skew) | ✅ done (#46) | -| 1 | **Auth keystone** — real DB-backed login; one login authorizes all three services | ✅ done (#47, #48, #49) | -| 1a | **Schema/model reconciliation** — migration 0014; fresh `upgrade head` now == the ORM models (was missing 73 cols / carrying 78 orphaned across 22 tables) | ✅ done (#47) | -| 2 | **Backend truth** — de-fake every "looks-real" endpoint: RTP/ion 500s, real Deal-Grove oxidation + least-squares calibration, VM predict → real physics engines, CVD tool-status/recipe, a real predictive-maintenance failure estimator, ML monitoring/explainability (removed forced-drift + seeded-random masquerading as live data), oxidation folded into the unified SPC roll-ups; fixed the audit-context bug that 500'd every authenticated write | ✅ done (#50–#54, #57, #58, #59) | -| 2.x | **Enum name-vs-value sweep** — audited every enum-typed column vs the live DB labels; fixed the `values_callable` bug (DB stores `'horizontal'`, ORM sent `'HORIZONTAL'`) that 500'd create endpoints across oxidation, diffusion, cvd and calibration | ✅ done (#59, #60) | -| 3 | **Frontend wiring + proxy fixes** — replaced the ~47 mock pages with real API calls; fixed the Next-14 path-remap + collection-root trailing-slash proxy bugs; PVD/CMP/ion/RTP/chemical wired | ✅ done (#70–#90) | -| 4 | **Missing characterization methods** — real physics backends + unit tests for 4PP, Hall, solar cell, PCD, DLTS, EBIC (electrical); UV-Vis Tauc, FTIR/Raman, PL, ellipsometry (optical); XRD Bragg/Scherrer, ISO-4287 roughness (structural) | ✅ done | -| 5 | **Test/CI/infra hardening** — all-service CI, frontend typecheck ratchet (now 512, only tightens), celery pinned, SQLite-compatible test models, numpy-2.0 sweep | ✅ done | -| 6 | **Honest-demo surfaces → real features** — the 9 remaining banner surfaces each got a real backend: LIMS users/signatures/custody/reports, FAIR export, data export, batch orchestrator, dashboard SPC, ML training | ✅ done (#95–#103) | -| 7 | **Frontend accuracy pass** — real per-service health indicator (replacing hardcoded "All Systems Operational"/"16/16 Sessions"), honest PVD/CMP MES KPIs, disclosed the two orphan UI-demo pages | ✅ done (#104) | +- Observability: Prometheus metrics on every service, Grafana dashboards, Alertmanager rules, OpenTelemetry tracing + +## How it got here (build history) + +Three major passes, every step a CI-green PR on a single `main` branch: + +1. **Build-out + foundation re-audit** (2026-03 → 2026-05) — the original + 37-session build plan, then Phase A which hardened what the plan had + left as stubs: audit-log writers, Postgres RLS, schema/model drift, + OpenTelemetry. History: [`docs/BUILD_PLAN.md`](docs/BUILD_PLAN.md). +2. **Remediation pass** (2026-05 → 2026-06, PRs #46–#104) — a + four-dimension audit found a convincing shell (auth couldn't complete, + ~47 pages rendered fabricated data, "looks-real" endpoints returned + hardcoded numbers). Every fake was replaced with real physics, real + queries, or an honest error. End state: + [`docs/REMEDIATION_STATUS.md`](docs/REMEDIATION_STATUS.md). +3. **Production-completion plan** (2026-06 → 2026-08, PRs #130–#174) — + security lockdown, broken execution paths, data-layer integrity + (RLS on all 91 org-scoped tables, CI-audited), frontend + deployability (runtime-env API proxy, standalone image), testing + depth (Playwright smoke + 26-page sweep in CI), deployability + (tag-triggered CD: build → Trivy gate → ghcr push → SBOM → release + → `kubectl apply`), product completion (global search, + notifications, localization, OIDC SSO), HIL-simulator physics + reconciliation (virtual-clock rework), and the account lifecycle + (password reset, change password, admin invitations over SMTP). + +## Production state (2026-08-04) + +- **Release**: [v0.1.0](https://github.com/alovladi007/SPECTRA-Lab/releases/tag/v0.1.0) — four images on ghcr (`analysis`, `lims`, `process-control`, `web`), each scanned to zero HIGH/CRITICAL CVEs (runtime images ship without pip/npm), SPDX SBOMs attached. +- **CI**: 17 required checks per PR — lint, typecheck ratchet, unit matrix (3.10/3.11/3.12), integration, LIMS suite, migration round-trip, Docker builds, security + secrets guardrails, k8s manifest rendering, Playwright e2e smoke against the real compose stack. +- **Tests**: ~950 backend tests across the four suites (analysis 304, lims 90, process-control 466 incl. accelerated-time soak, shared 88) plus 4-test e2e smoke and a 26-page dashboard sweep. +- **Deploy path**: `git tag v0.1.x && git push --tags` does everything through `kubectl rollout status` once the one `KUBE_CONFIG` repo secret is set. First-admin bootstrap: `bootstrap_admin.py` (refuses demo seeding in production). +- **Remaining before a live URL**: provisioning only — cluster, domain/DNS, TLS, sealed secret values, optional SMTP relay. Exact sequence: [`docs/deployment/PRODUCTION_RUNBOOK.md`](docs/deployment/PRODUCTION_RUNBOOK.md); automation: [`scripts/provision/`](scripts/provision/). Demo login after `make seed-db`: `admin@demo.lab` / `admin123` (also pi/engineer/tech/viewer `@demo.lab`). -Migration drift fixed in Session 3.1 + Phase A2/A.next + the remediation -0014 (below): `alembic upgrade head` against a fresh DB now runs cleanly -through 0001→0014, round-trips, AND the resulting schema matches the ORM -models exactly (verified — zero residual drift). `make smoke-test` is the -full-stack regression target. - -CD workflow is still disabled (`workflow_dispatch` only) until Phase 8.3 -rebuilds it correctly. CI's Lint, Migration Test, Unit Tests -(3.10/3.11/3.12), Integration Tests, Docker Build, Security, and the -**Frontend Typecheck** ratchet (added in Phase 5; fails if the `tsc` -error count regresses past the committed baseline) are all green on -`main` — 14 required checks per PR. - ## Contributing This is a private project; there is currently no `CONTRIBUTING.md`. If you have access and want to land work, the cross-cutting conventions in [`docs/BUILD_PLAN.md`](docs/BUILD_PLAN.md) apply: read first, plan-then-execute on >5-file changes, one concern per commit, tests with every behavior change, no new TODOs, docs updated in the same PR.