diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8cc56e2..842e1ac 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -40,3 +40,7 @@ jobs: - name: Test ai-sre-assistant working-directory: apps/ai-sre-assistant run: pytest -q + + - name: Evaluate ai-sre-assistant + working-directory: apps/ai-sre-assistant + run: python -m evals.run_evals diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 907ccee..22f3592 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -36,9 +36,11 @@ Run formatting and lint checks before opening a pull request: ```bash make format -make lint +make validate ``` +`make validate` builds both images, runs both test suites, runs lint checks, and executes the assistant evaluation corpus. The same evaluation gate runs in CI. + ## Pull Request Notes In your PR, include: diff --git a/Makefile b/Makefile index da7f995..8bbb21a 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ COMPOSE ?= docker compose PYTHON ?= python -.PHONY: up down logs test lint generate-traffic analyze-logs evaluate-assistant format +.PHONY: up down logs build test lint generate-traffic analyze-logs evaluate-assistant validate format up: $(COMPOSE) up --build -d @@ -12,12 +12,14 @@ down: logs: $(COMPOSE) logs -f demo-service ai-sre-assistant -test: +build: $(COMPOSE) build demo-service ai-sre-assistant + +test: build $(COMPOSE) run --rm --no-deps demo-service pytest -q $(COMPOSE) run --rm --no-deps ai-sre-assistant pytest -q -lint: +lint: build $(COMPOSE) run --rm --no-deps demo-service ruff check app tests $(COMPOSE) run --rm --no-deps ai-sre-assistant ruff check app cli tests @@ -27,10 +29,12 @@ generate-traffic: analyze-logs: $(COMPOSE) run --rm --no-deps ai-sre-assistant python cli/sre.py analyze --max-lines 120 -evaluate-assistant: - $(COMPOSE) build ai-sre-assistant +evaluate-assistant: build $(COMPOSE) run --rm --no-deps ai-sre-assistant python -m evals.run_evals +validate: test lint evaluate-assistant + @echo "Production-readiness validation passed." + format: $(COMPOSE) run --rm --no-deps demo-service ruff format app tests $(COMPOSE) run --rm --no-deps ai-sre-assistant ruff format app cli tests diff --git a/README.md b/README.md index 565ad94..48a957e 100644 --- a/README.md +++ b/README.md @@ -95,6 +95,8 @@ For the production observability architecture and staged migration path, see `do For the advanced model serving decision framework and commercialization path, see `docs/19-advanced-model-serving-roadmap.md`. +For the Week 4 release gates, readiness verdict, and next-stage priorities, see `docs/20-production-readiness-review.md`. + Ask the assistant directly: ```bash @@ -138,6 +140,7 @@ ai-infra-starter-kit/ 17-assistant-evaluation.md 18-production-observability.md 19-advanced-model-serving-roadmap.md + 20-production-readiness-review.md infra/ # Docker, Kubernetes, and Terraform starter notes k8s/ # kind-first Kubernetes manifests and walkthrough scripts/ # Local traffic and log helper scripts @@ -167,7 +170,7 @@ You should see the assistant report intentionally generated 500s, latency spikes - Week 1: local demo-service, AI SRE Assistant, Docker Compose, sample logs, basic README. - Week 2: observability basics, metrics, dashboards, structured logging, incident examples. - Week 3: Kubernetes manifests, operations, config/secrets, probes/resources, incident debugging, production next steps. -- Week 4: security hardening, cost optimization, evaluation basics, production observability, optional advanced serving path with vLLM, Triton, Ray, KServe, and GPU scheduling. +- Week 4: security hardening, cost optimization, evaluation, production observability, advanced serving decisions, and a release-gated production-readiness review. ## Build In Public diff --git a/docs/09-roadmap.md b/docs/09-roadmap.md index d7bc88a..1714609 100644 --- a/docs/09-roadmap.md +++ b/docs/09-roadmap.md @@ -33,6 +33,7 @@ - Assistant evaluation basics. - Production observability upgrade path. - Optional advanced serving roadmap: vLLM, Triton, Ray, KServe, and GPU scheduling. +- Production-readiness review with local and CI release gates. ## Later diff --git a/docs/17-assistant-evaluation.md b/docs/17-assistant-evaluation.md index ad48cb3..ffbd0d2 100644 --- a/docs/17-assistant-evaluation.md +++ b/docs/17-assistant-evaluation.md @@ -18,7 +18,7 @@ Or run it directly from `apps/ai-sre-assistant`: python -m evals.run_evals ``` -The command exits with a non-zero status when any case fails, so it can become a simple CI release gate later. +The command exits with a non-zero status when any case fails. CI and `make validate` run it as a release gate. ## Evaluation Corpus @@ -73,7 +73,7 @@ That product direction reinforces the user outcome: teams should be able to adop ## Next Steps -- Keep these deterministic cases in the normal test suite. +- Keep these deterministic cases in the normal test suite and CI release gate. - Add sanitized real-world incidents as the assistant supports more failure modes. - Record provider usage metadata before comparing quality against cost. - Version the corpus and thresholds when evaluation results become a release gate. diff --git a/docs/20-production-readiness-review.md b/docs/20-production-readiness-review.md new file mode 100644 index 0000000..fc81d93 --- /dev/null +++ b/docs/20-production-readiness-review.md @@ -0,0 +1,88 @@ +# Production Readiness Review + +Week 4, Day 7 closes the first AIOps Lab learning cycle with an explicit release decision. + +The project now demonstrates a complete local path from application signals to evidence-grounded incident analysis. That makes it ready to publish as a learning lab. It does not make the current Docker Compose or kind configuration ready for internet-facing production use. + +## Release Decision + +| Target | Decision | Reason | +| --- | --- | --- | +| Local learning lab | **Go** | The setup is reproducible, deterministic without an LLM key, documented, tested, and covered by a runnable evaluation corpus. | +| Public open-source learning release | **Go when `make validate` passes** | Tests, lint, and assistant quality checks are enforced locally and in pull-request CI. | +| Internal pilot with sanitized data | **Conditional** | Add environment-specific access control, centralized telemetry, secret management, ownership, and a rollback plan first. | +| Internet-facing or customer production | **No-go today** | The lab does not yet provide production authentication, durable telemetry backends, tenant isolation, managed secrets, SLO operations, or hardened deployment automation. | + +This distinction is intentional. A strong readiness review records what the evidence supports instead of relabeling a useful lab as a production platform. + +## One Release Gate + +Run the full local gate from the repository root: + +```bash +make validate +``` + +It performs four checks: + +1. Builds both service images. +2. Runs both pytest suites. +3. Runs Ruff against both services. +4. Runs the assistant evaluation corpus and fails on any grounding, usefulness, safety, privacy, or honesty regression. + +The CI workflow runs the same test, lint, and evaluation categories on every pull request and on pushes to `main`. No provider key or network model call is required for the evaluation gate. + +Formatting remains a deliberate author action: + +```bash +make format +``` + +## Evidence Review + +| Readiness area | Evidence in this repository | Current boundary | +| --- | --- | --- | +| Reproducibility | Docker Compose, kind manifests, health checks, tests, and copy-paste workflows. | Local and learning environments only. | +| Security | Threat model, safe defaults, secret guidance, and input/output redaction tests. | Pattern redaction is not a data-loss-prevention system; production auth and policy enforcement remain future work. | +| Cost | Deterministic default, bounded evidence windows, prompt limits, and visible cost-control metadata. | Provider token accounting, quotas, budgets, and chargeback are not implemented. | +| Quality | Seven deterministic incident cases and a five-dimension pass/fail rubric. | The corpus is small and does not replace sanitized real incidents, human review, or provider/model evaluation. | +| Observability | Structured logs, request correlation, Prometheus-style metrics, incident walkthroughs, and a production signal contract. | The shared file is a teaching mechanism; no collector or durable logs, metrics, and traces backend is installed. | +| Reliability | Health/readiness endpoints, Kubernetes probes and resources, runbooks, and rollback guidance. | No measured production SLO, alert delivery, autoscaling, multi-zone design, or disaster recovery exercise. | +| Model serving | Provider-compatible configuration and a staged vLLM, Triton, Ray Serve, KServe, and GPU decision framework. | No GPU path is needed or claimed for the default project. | +| Product direction | Open learning core plus a governed evaluation, private deployment, and team workflow direction. | Customer discovery and measured demand must precede platform investment. | + +## Promotion Gates For A Real Pilot + +Do not promote the current manifests by changing only an image tag. Before an internal pilot, require named owners and evidence for these gates: + +- **Identity and access:** authenticate users and services; authorize access to incident evidence; audit sensitive access. +- **Data boundary:** classify allowed evidence, prevent secrets at the source, test redaction, define retention, and document provider handling. +- **Secrets and supply chain:** use a managed secret workflow, pin and scan deployable artifacts, and define rotation and patch ownership. +- **Reliability:** establish baseline SLIs, choose an initial SLO, route actionable alerts, test a runbook, and prove rollback. +- **Observability:** replace shared files with controlled collection and storage; keep evidence queries bounded and attributable. +- **Quality:** add sanitized representative incidents, preserve privacy and safety as hard gates, and version model, prompt, corpus, and thresholds together. +- **Cost:** capture provider identity, latency, token usage, fallback outcome, and cost per successful evaluated analysis; add budgets and quotas. +- **Deployment:** separate environments, use production storage and networking, define backups where state exists, and test recovery. + +Any failed privacy or safety gate is a no-go. Other exceptions need a named owner, explicit risk acceptance, a deadline, and a rollback path. + +## First 30 Days After This Milestone + +The next work should deepen measured contracts before adding infrastructure breadth: + +1. Capture provider model, latency, token usage, failure, and fallback metadata without storing prompt content. +2. Expand the evaluation corpus with sanitized incidents and version its acceptance thresholds. +3. Add optional OpenTelemetry collection after log, metric, and trace field contracts are stable. +4. Turn one incident walkthrough into an alert-to-runbook exercise with an owner and measurable recovery result. +5. Build a provider-versus-private-endpoint benchmark harness before adding a GPU deployment example. + +vLLM, Triton, Ray Serve, KServe, GPU scheduling, multi-tenancy, and paid platform features stay behind demonstrated demand and the adoption gates already documented in the advanced serving roadmap. + +## Day 7 Definition Of Done + +- One command runs build, tests, lint, and deterministic assistant evaluations. +- Pull-request CI blocks an evaluation regression. +- The project has an explicit go/no-go decision for each deployment target. +- Local evidence and production gaps are visible in the same scorecard. +- Pilot promotion gates have owners and measurable evidence requirements. +- The next 30 days prioritize instrumentation, evaluation depth, and operational proof over tool accumulation. diff --git a/docs/build-log.md b/docs/build-log.md index 43b1c9b..0a2a0ea 100644 --- a/docs/build-log.md +++ b/docs/build-log.md @@ -628,3 +628,41 @@ What comes next: - Build a provider-versus-self-host benchmark harness. - Add one optional single-GPU vLLM example only after the benchmark contract is ready. - Validate private deployment demand before building a broad serving platform. + +## Week 4, Day 7 - Production Readiness Review + +Today I closed Week 4 with a release decision and an enforced quality gate. + +Day 6 defined when advanced serving infrastructure is justified. Day 7 steps back and asks the more important question: what is this project ready for now, what evidence supports that decision, and what must be true before the next promotion? + +What changed: + +- Added `make validate` as one local gate for image builds, both test suites, lint, and deterministic assistant evaluations. +- Added the assistant evaluation runner to pull-request CI so quality, safety, and privacy regressions block the build. +- Added a production-readiness review with separate decisions for a local lab, public learning release, sanitized internal pilot, and customer production. +- Mapped current evidence and remaining boundaries across security, cost, quality, observability, reliability, model serving, and product direction. +- Defined promotion gates for identity, data handling, secrets, supply chain, reliability, observability, quality, cost, and deployment. +- Prioritized a focused next 30 days around provider metadata, deeper evals, stable telemetry contracts, runbook exercises, and benchmark evidence. +- Updated the roadmap, README, contribution checks, and evaluation guide to match the enforced release workflow. + +Why this matters: + +Production readiness is not a feeling and it is not a list of tools. It is a decision backed by tests, measurable gates, clear ownership, and an honest account of unresolved risk. + +AIOps Lab is ready to ship as an open learning project when the validation gate passes. It is not yet an internet-facing multi-tenant production service, and the review says so explicitly. That boundary makes the project more credible, not less ambitious. + +Lessons learned: + +- Documentation becomes stronger when its claims are enforced in CI. +- Privacy and safety checks should block releases rather than disappear into an average score. +- A single readiness label is misleading; local, pilot, and customer production targets have different gates. +- Advanced infrastructure should follow measured requirements and operational ownership. +- The strongest closeout is a clear decision, visible evidence, and a small next set of priorities. + +What comes next: + +- Capture provider usage, latency, model identity, and fallback outcomes. +- Grow the evaluation corpus with sanitized representative incidents. +- Add an optional telemetry collector only after signal contracts are stable. +- Exercise one alert and runbook path end to end. +- Benchmark provider and private endpoints before introducing GPU infrastructure.