Debora — 4th Task Checklist

[MuhammadKhalilzadeh]

Debora — 4th Task Checklist · "Push All Layers to +90" Task Checklist

Baseline: Software layers analysis/VerifyWise Software Layers Benchmark.md. Your focus: make every score defensible with automated evidence — performance/load (Load balancing 55, Caching 45, APIs 62), reliability checks (Availability 60, Error tracking 58), and security-verification suites (Security 74, Rate limiting 72). Scores only move when a CI check proves the improvement — you own those checks. Every item below is yours to deliver end-to-end.

---

1. Build the k6 performance suite + CI regression gate

• Description: There is no load/perf testing. Every performance claim in this push (N+1 batching, pagination, indexes, caching, auth-middleware consolidation) currently rests on manual before/after claims.
• General guidance: Add a k6 suite in Servers/tests/perf/ covering the top-15 endpoints (from request-metrics p95 data): login, project list, dashboard aggregation, reporting, file list. Include a lightweight fixture/seed step in the suite itself (SQL inserts or API-driven setup) so it runs anywhere. Run it nightly (scheduled workflow) + on-demand for perf PRs; fail on p95 regression > 20% vs stored baseline; store baselines as versioned artifacts.
• Where to start: Servers/middleware/requestMetrics.middleware.ts (route list), Servers/tests/integration/ (setup patterns), .github/workflows/ (scheduled workflow pattern from zap-baseline.yml).
• What to expect: First baseline within 2 weeks; every perf PR carries a k6 before/after table. Target: APIs +3, Database +2.

2. Multi-replica load test for the scaling workstream

• Description: Scaling behavior (2 backend replicas, future autoscaling) is untested; the in-memory rate limiter only shows its flaw under multi-replica load.
• General guidance: Build a compose-based scenario (docker-compose.yml variant with 2 backend replicas behind nginx) and a k6 test verifying: (a) rate limits hold globally across replicas (this scenario doubles as the acceptance check for Redis-backed limiting — assert the cross-replica count; document the expected result for both in-memory and Redis-backed configs), (b) requests distribute across replicas, (c) nothing assumes local in-process state. Run it in the scheduled workflow once the Redis store lands; keep the scenario green as a permanent regression check.
• Where to start: docker-compose.yml, Servers/middleware/rateLimit.middleware.ts.
• What to expect: A repeatable multi-replica scenario = evidence for Load balancing & scaling +4.

3. Rate-limiter verification suite

• Description: 7 limiters exist but only middleware unit tests back them; no suite proves brute-force protection end-to-end (auth 5/15 min) or the RateLimit-* headers contract.
• General guidance: Integration tests in Servers/tests/integration/: each limiter — exceed limit → 429 → Retry-After/RateLimit-* headers correct → counter resets per window; IPv6-keyed requests; limiter state survives restart (Redis-backed config). Wire into backend-checks so it's blocking.
• Where to start: Servers/middleware/rateLimit.middleware.ts, existing middleware/__tests__/ patterns.
• What to expect: Blocking suite in CI. Target: Rate limiting +3.

4. Caching validation tests (contract-first)

• Description: The Caching & CDN layer is the only sub-baseline score — and nothing prevents silent regressions once caching lands (stale tenant data is a security issue here).
• General guidance: Author the contract tests now: (a) Cache-Control per route class (public static assets cached, authenticated API no-store unless explicitly declared); (b) cache-aside invalidation — write → read returns fresh data, with per-tenant isolation: org A's cached data never served to org B (belongs next to the existing tenant-isolation tests); (c) service worker smoke in Playwright — offline app-shell load, /api never cached. Wire them into CI; they validate whichever implementation is present.
• Where to start: Servers/tests/integration/tenant-isolation/ (patterns), Clients/e2e/ (SW smoke), Servers/middleware/requestMetrics.middleware.ts.
• What to expect: Blocking tests wired into CI. Target: Caching & CDN +4 (its score is only defensible with these).

5. RLS enforcement verification pack

• Description: RLS runs flag-gated (RLS_ENFORCEMENT_ENABLED); when it flips on, isolation must be proven continuously — not just at rollout. ~150 tables are still deferredScopedTables.
• General guidance: Extend the 30 tenant-isolation test files into a CI-blocking pack that also runs in a matrix job with RLS_ENFORCEMENT_ENABLED=true: cross-tenant read/write attempts on every scoped entity → 0 leaks; BullMQ worker context-propagation checks; raw-SQL path checks (AsyncLocalStorage scoping). Add the deferred-table count as an assertion that can only decrease (burndown metric in the monthly report).
• Where to start: Servers/tests/integration/tenant-isolation/, Servers/scripts/auditTenantIsolationCoverage.ts, Servers/middleware/rls.middleware.ts (flag).
• What to expect: Both matrix variants (RLS on/off) green and blocking; burndown metric reported monthly. Target: Security +4.

6. Synthetic uptime checks + Playwright canary

• Description: Availability (60) has no synthetic verification — health endpoints exist but nothing watches them.
• General guidance: (a) A canary Playwright project (login → dashboard render → project list) on a schedule against staging/prod, results surfaced as a CI badge/report; (b) deploy Uptime Kuma (or equivalent lightweight self-hosted checker) on the observability VM with checks for /health, frontend, and AI-gateway endpoints; document it in docs/deployment/; wire failure notifications to the alerting channel once Alertmanager lands.
• Where to start: Clients/playwright.config.ts (project pattern), observability/ (compose stack), .github/workflows/e2e-tests.yml.
• What to expect: Canary green baseline + uptime checks running with alerting. Target: Availability +3, Error tracking +1.

7. Error-tracking smoke tests

• Description: Exception tracking is being stood up from zero — the rollout needs automated proof that events actually arrive and are sanitized.
• General guidance: Deploy a self-hosted tracker (GlitchTip recommended — lightweight, Sentry-compatible) on the observability VM as part of this task, then build the smoke suite: forced frontend boundary error → event lands; forced backend 500 → event with org/request-id context; BullMQ worker failure → event. Assert PII sanitization (no emails/names in payloads — sample-payload assertions). Run the smoke as a staging job and after each deploy.
• Where to start: observability/docker-compose.observability.yml, the 4 FE TODO sites (secureLogger.utils.ts:74, DashboardErrorBoundary.tsx:40, WidgetErrorBoundary.tsx:41, MegaDropdownErrorBoundary.tsx:33), Servers/utils/logger/.
• What to expect: Deployed tracker + smoke suite passing end-to-end. Target: Error tracking & logs +4.

8. Alert rules as code + Alertmanager provisioning

• Description: Prometheus has no alert rules and Alertmanager isn't provisioned (the Loki ruler points at an unprovisioned localhost:9093) — dashboards exist but nothing pages anyone.
• General guidance: Provision Alertmanager in observability/docker-compose.observability.yml with a notification route (email/webhook), then define the initial rule set as code in observability/prometheus/alerts.yml: error-rate > 2%/5min, p95 latency SLO burn, queue depth, DB connections > 80%, backup-age > 26h, canary failure. Add a CI validation job (promtool check rules + a "rules reference existing metrics" check) and verify in staging that each rule can actually fire.
• Where to start: observability/prometheus.yml, observability/loki-config.yaml (ruler block), Servers/middleware/requestMetrics.middleware.ts (metric names).
• What to expect: Rules-as-code with promtool CI gate; one proven firing per rule in staging. Target: Error tracking +3, Availability +3.

9. DR restore-test workflow

• Description: Backups are being automated — but an untested backup is not a backup. Restore testing must be continuous and cheap, with duration measured against the DR guide's RTO targets.
• General guidance: A restore-test workflow (monthly schedule + manual trigger): spin a Postgres service container → produce a fresh pg_dump (or consume the committed backup script's output when present) → restore into it → run a minimal smoke query set + row counts → tear down; record duration vs the DR-guide RTO tiers as the job summary.
• Where to start: docs/deployment/DISASTER_RECOVERY_GUIDE.md (RTO/RPO tiers), .github/workflows/ service-container patterns (backend-checks.yml has them), Servers/database/migrations/ (latest migration to replay).
• What to expect: Monthly green restore job = Availability +4 becomes defensible.

10. Post-deploy smoke gate in the deploy pipeline

• Description: deploy-to-production.yml has no post-deploy verification or rollback step — a bad deploy sits in prod until someone notices.
• General guidance: Add a post-deploy job: /health 200 (incl. DB/Redis/AI-gateway sub-checks) + the Playwright canary against prod; on failure, automatic rollback to the previous compose project revision (keep the previous image tags as a workflow artifact so rollback is immediate). If the deployment shape changes (blue/green), the smoke gates the traffic switch instead — keep the gate, adapt the mechanism.
• Where to start: .github/workflows/deploy-to-production.yml, Servers/app.ts (/health route).
• What to expect: Every release either proves itself healthy or rolls back. Target: CI/CD +2, Hosting +2, Availability +3.

11. Security verification: CSP report aggregation + secrets hygiene

• Description: Two standing items block Security > 90: CSP is Report-Only (needs a violation-free evidence window before enforcement) and the committed-.env history purge/rotation is open.
• General guidance: (a) Automate CSP report collection — a Playwright capture of report-uri/Reporting-API payloads during e2e plus a weekly aggregation job that fails on any new violation directive; produce the evidence pack (violation-free window) that enforcement decisions are based on. (b) A recurring gitleaks full-history scan in CI (pre-commit only covers staged files) plus a tracking doc for the rotation status of the previously committed .env.prod/.env.dev secrets (docs/technical/security/platform-security-evaluation-2026-07.md:170).
• Where to start: Clients/nginx.conf (CSP header), .github/workflows/gitleaks.yml, .gitleaks.toml.
• What to expect: CSP evidence pack; history-scan job green or with documented exceptions. Target: Security +2.

12. Benchmark evidence binder

• Description: The benchmark doc is re-scored monthly; scores must be backed by runnable evidence, not anecdotes.
• General guidance: Maintain Software layers analysis/evidence/ — one short MD per layer per month linking: CI jobs, dashboard screenshots, k6 runs, test counts. Update right after each monthly score change. This is the artifact that makes the whole "+90" program auditable.
• Where to start: Software layers analysis/VerifyWise Software Layers Benchmark.md §5.
• What to expect: 12 monthly evidence packs; any layer claim traceable to a green check.