Close Week 4 with production readiness gates

.github/workflows/ci.yml

@@ -40,3 +40,7 @@ jobs:
       - name: Test ai-sre-assistant
         working-directory: apps/ai-sre-assistant
         run: pytest -q
+
+      - name: Evaluate ai-sre-assistant
+        working-directory: apps/ai-sre-assistant
+        run: python -m evals.run_evals

CONTRIBUTING.md

@@ -36,9 +36,11 @@ Run formatting and lint checks before opening a pull request:
 
 ```bash
 make format
-make lint
+make validate
 ```
 
+`make validate` builds both images, runs both test suites, runs lint checks, and executes the assistant evaluation corpus. The same evaluation gate runs in CI.
+
 ## Pull Request Notes
 
 In your PR, include:

Makefile

@@ -1,7 +1,7 @@
 COMPOSE ?= docker compose
 PYTHON ?= python
 
-.PHONY: up down logs test lint generate-traffic analyze-logs evaluate-assistant format
+.PHONY: up down logs build test lint generate-traffic analyze-logs evaluate-assistant validate format
 
 up:
 	$(COMPOSE) up --build -d
@@ -12,12 +12,14 @@ down:
 logs:
 	$(COMPOSE) logs -f demo-service ai-sre-assistant
 
-test:
+build:
 	$(COMPOSE) build demo-service ai-sre-assistant
+
+test: build
 	$(COMPOSE) run --rm --no-deps demo-service pytest -q
 	$(COMPOSE) run --rm --no-deps ai-sre-assistant pytest -q
 
-lint:
+lint: build
 	$(COMPOSE) run --rm --no-deps demo-service ruff check app tests
 	$(COMPOSE) run --rm --no-deps ai-sre-assistant ruff check app cli tests
 
@@ -27,10 +29,12 @@ generate-traffic:
 analyze-logs:
 	$(COMPOSE) run --rm --no-deps ai-sre-assistant python cli/sre.py analyze --max-lines 120
 
-evaluate-assistant:
-	$(COMPOSE) build ai-sre-assistant
+evaluate-assistant: build
 	$(COMPOSE) run --rm --no-deps ai-sre-assistant python -m evals.run_evals
 
+validate: test lint evaluate-assistant
+	@echo "Production-readiness validation passed."
+
 format:
 	$(COMPOSE) run --rm --no-deps demo-service ruff format app tests
 	$(COMPOSE) run --rm --no-deps ai-sre-assistant ruff format app cli tests

README.md

@@ -95,6 +95,8 @@ For the production observability architecture and staged migration path, see `do
 
 For the advanced model serving decision framework and commercialization path, see `docs/19-advanced-model-serving-roadmap.md`.
 
+For the Week 4 release gates, readiness verdict, and next-stage priorities, see `docs/20-production-readiness-review.md`.
+
 Ask the assistant directly:
 
 ```bash
@@ -138,6 +140,7 @@ ai-infra-starter-kit/
     17-assistant-evaluation.md
     18-production-observability.md
     19-advanced-model-serving-roadmap.md
+    20-production-readiness-review.md
   infra/                 # Docker, Kubernetes, and Terraform starter notes
     k8s/                 # kind-first Kubernetes manifests and walkthrough
   scripts/               # Local traffic and log helper scripts
@@ -167,7 +170,7 @@ You should see the assistant report intentionally generated 500s, latency spikes
 - Week 1: local demo-service, AI SRE Assistant, Docker Compose, sample logs, basic README.
 - Week 2: observability basics, metrics, dashboards, structured logging, incident examples.
 - Week 3: Kubernetes manifests, operations, config/secrets, probes/resources, incident debugging, production next steps.
-- Week 4: security hardening, cost optimization, evaluation basics, production observability, optional advanced serving path with vLLM, Triton, Ray, KServe, and GPU scheduling.
+- Week 4: security hardening, cost optimization, evaluation, production observability, advanced serving decisions, and a release-gated production-readiness review.
 
 ## Build In Public
 

docs/09-roadmap.md

@@ -33,6 +33,7 @@
 - Assistant evaluation basics.
 - Production observability upgrade path.
 - Optional advanced serving roadmap: vLLM, Triton, Ray, KServe, and GPU scheduling.
+- Production-readiness review with local and CI release gates.
 
 ## Later
 

docs/17-assistant-evaluation.md

@@ -18,7 +18,7 @@ Or run it directly from `apps/ai-sre-assistant`:
 python -m evals.run_evals
 ```
 
-The command exits with a non-zero status when any case fails, so it can become a simple CI release gate later.
+The command exits with a non-zero status when any case fails. CI and `make validate` run it as a release gate.
 
 ## Evaluation Corpus
 
@@ -73,7 +73,7 @@ That product direction reinforces the user outcome: teams should be able to adop
 
 ## Next Steps
 
-- Keep these deterministic cases in the normal test suite.
+- Keep these deterministic cases in the normal test suite and CI release gate.
 - Add sanitized real-world incidents as the assistant supports more failure modes.
 - Record provider usage metadata before comparing quality against cost.
 - Version the corpus and thresholds when evaluation results become a release gate.

docs/20-production-readiness-review.md

@@ -0,0 +1,88 @@
+# Production Readiness Review
+
+Week 4, Day 7 closes the first AIOps Lab learning cycle with an explicit release decision.
+
+The project now demonstrates a complete local path from application signals to evidence-grounded incident analysis. That makes it ready to publish as a learning lab. It does not make the current Docker Compose or kind configuration ready for internet-facing production use.
+
+## Release Decision
+
+| Target | Decision | Reason |
+| --- | --- | --- |
+| Local learning lab | **Go** | The setup is reproducible, deterministic without an LLM key, documented, tested, and covered by a runnable evaluation corpus. |
+| Public open-source learning release | **Go when `make validate` passes** | Tests, lint, and assistant quality checks are enforced locally and in pull-request CI. |
+| Internal pilot with sanitized data | **Conditional** | Add environment-specific access control, centralized telemetry, secret management, ownership, and a rollback plan first. |
+| Internet-facing or customer production | **No-go today** | The lab does not yet provide production authentication, durable telemetry backends, tenant isolation, managed secrets, SLO operations, or hardened deployment automation. |
+
+This distinction is intentional. A strong readiness review records what the evidence supports instead of relabeling a useful lab as a production platform.
+
+## One Release Gate
+
+Run the full local gate from the repository root:
+
+```bash
+make validate
+```
+
+It performs four checks:
+
+1. Builds both service images.
+2. Runs both pytest suites.
+3. Runs Ruff against both services.
+4. Runs the assistant evaluation corpus and fails on any grounding, usefulness, safety, privacy, or honesty regression.
+
+The CI workflow runs the same test, lint, and evaluation categories on every pull request and on pushes to `main`. No provider key or network model call is required for the evaluation gate.
+
+Formatting remains a deliberate author action:
+
+```bash
+make format
+```
+
+## Evidence Review
+
+| Readiness area | Evidence in this repository | Current boundary |
+| --- | --- | --- |
+| Reproducibility | Docker Compose, kind manifests, health checks, tests, and copy-paste workflows. | Local and learning environments only. |
+| Security | Threat model, safe defaults, secret guidance, and input/output redaction tests. | Pattern redaction is not a data-loss-prevention system; production auth and policy enforcement remain future work. |
+| Cost | Deterministic default, bounded evidence windows, prompt limits, and visible cost-control metadata. | Provider token accounting, quotas, budgets, and chargeback are not implemented. |
+| Quality | Seven deterministic incident cases and a five-dimension pass/fail rubric. | The corpus is small and does not replace sanitized real incidents, human review, or provider/model evaluation. |
+| Observability | Structured logs, request correlation, Prometheus-style metrics, incident walkthroughs, and a production signal contract. | The shared file is a teaching mechanism; no collector or durable logs, metrics, and traces backend is installed. |
+| Reliability | Health/readiness endpoints, Kubernetes probes and resources, runbooks, and rollback guidance. | No measured production SLO, alert delivery, autoscaling, multi-zone design, or disaster recovery exercise. |
+| Model serving | Provider-compatible configuration and a staged vLLM, Triton, Ray Serve, KServe, and GPU decision framework. | No GPU path is needed or claimed for the default project. |
+| Product direction | Open learning core plus a governed evaluation, private deployment, and team workflow direction. | Customer discovery and measured demand must precede platform investment. |
+
+## Promotion Gates For A Real Pilot
+
+Do not promote the current manifests by changing only an image tag. Before an internal pilot, require named owners and evidence for these gates:
+
+- **Identity and access:** authenticate users and services; authorize access to incident evidence; audit sensitive access.
+- **Data boundary:** classify allowed evidence, prevent secrets at the source, test redaction, define retention, and document provider handling.
+- **Secrets and supply chain:** use a managed secret workflow, pin and scan deployable artifacts, and define rotation and patch ownership.
+- **Reliability:** establish baseline SLIs, choose an initial SLO, route actionable alerts, test a runbook, and prove rollback.
+- **Observability:** replace shared files with controlled collection and storage; keep evidence queries bounded and attributable.
+- **Quality:** add sanitized representative incidents, preserve privacy and safety as hard gates, and version model, prompt, corpus, and thresholds together.
+- **Cost:** capture provider identity, latency, token usage, fallback outcome, and cost per successful evaluated analysis; add budgets and quotas.
+- **Deployment:** separate environments, use production storage and networking, define backups where state exists, and test recovery.
+
+Any failed privacy or safety gate is a no-go. Other exceptions need a named owner, explicit risk acceptance, a deadline, and a rollback path.
+
+## First 30 Days After This Milestone
+
+The next work should deepen measured contracts before adding infrastructure breadth:
+
+1. Capture provider model, latency, token usage, failure, and fallback metadata without storing prompt content.
+2. Expand the evaluation corpus with sanitized incidents and version its acceptance thresholds.
+3. Add optional OpenTelemetry collection after log, metric, and trace field contracts are stable.
+4. Turn one incident walkthrough into an alert-to-runbook exercise with an owner and measurable recovery result.
+5. Build a provider-versus-private-endpoint benchmark harness before adding a GPU deployment example.
+
+vLLM, Triton, Ray Serve, KServe, GPU scheduling, multi-tenancy, and paid platform features stay behind demonstrated demand and the adoption gates already documented in the advanced serving roadmap.
+
+## Day 7 Definition Of Done
+
+- One command runs build, tests, lint, and deterministic assistant evaluations.
+- Pull-request CI blocks an evaluation regression.
+- The project has an explicit go/no-go decision for each deployment target.
+- Local evidence and production gaps are visible in the same scorecard.
+- Pilot promotion gates have owners and measurable evidence requirements.
+- The next 30 days prioritize instrumentation, evaluation depth, and operational proof over tool accumulation.

docs/build-log.md

@@ -628,3 +628,41 @@ What comes next:
 - Build a provider-versus-self-host benchmark harness.
 - Add one optional single-GPU vLLM example only after the benchmark contract is ready.
 - Validate private deployment demand before building a broad serving platform.
+
+## Week 4, Day 7 - Production Readiness Review
+
+Today I closed Week 4 with a release decision and an enforced quality gate.
+
+Day 6 defined when advanced serving infrastructure is justified. Day 7 steps back and asks the more important question: what is this project ready for now, what evidence supports that decision, and what must be true before the next promotion?
+
+What changed:
+
+- Added `make validate` as one local gate for image builds, both test suites, lint, and deterministic assistant evaluations.
+- Added the assistant evaluation runner to pull-request CI so quality, safety, and privacy regressions block the build.
+- Added a production-readiness review with separate decisions for a local lab, public learning release, sanitized internal pilot, and customer production.
+- Mapped current evidence and remaining boundaries across security, cost, quality, observability, reliability, model serving, and product direction.
+- Defined promotion gates for identity, data handling, secrets, supply chain, reliability, observability, quality, cost, and deployment.
+- Prioritized a focused next 30 days around provider metadata, deeper evals, stable telemetry contracts, runbook exercises, and benchmark evidence.
+- Updated the roadmap, README, contribution checks, and evaluation guide to match the enforced release workflow.
+
+Why this matters:
+
+Production readiness is not a feeling and it is not a list of tools. It is a decision backed by tests, measurable gates, clear ownership, and an honest account of unresolved risk.
+
+AIOps Lab is ready to ship as an open learning project when the validation gate passes. It is not yet an internet-facing multi-tenant production service, and the review says so explicitly. That boundary makes the project more credible, not less ambitious.
+
+Lessons learned:
+
+- Documentation becomes stronger when its claims are enforced in CI.
+- Privacy and safety checks should block releases rather than disappear into an average score.
+- A single readiness label is misleading; local, pilot, and customer production targets have different gates.
+- Advanced infrastructure should follow measured requirements and operational ownership.
+- The strongest closeout is a clear decision, visible evidence, and a small next set of priorities.
+
+What comes next:
+
+- Capture provider usage, latency, model identity, and fallback outcomes.
+- Grow the evaluation corpus with sanitized representative incidents.
+- Add an optional telemetry collector only after signal contracts are stable.
+- Exercise one alert and runbook path end to end.
+- Benchmark provider and private endpoints before introducing GPU infrastructure.