Release Sentinel

Open-source DevOps/SRE reference project for safe releases, observability, supply-chain checks, and automated rollback.

Release Sentinel ships a small Go service with controlled failure injection, a hardened container image, Kubernetes and Helm manifests, Prometheus/Grafana observability, canary rollout analysis, load tests, CI security checks, and runbooks.

The core idea is simple: a release should not be trusted just because it builds. It should prove that it behaves correctly under real traffic signals before it is promoted.

Why it stands out

Most release examples stop at "deploy an app to Kubernetes." This project models the full release safety loop: build, scan, deploy, measure, analyze, abort, and document the incident path.

• Canary promotion is driven by Prometheus success-rate analysis, not manual judgment.
• CI produces an SBOM and blocks fixable high/critical image vulnerabilities.
• The runtime includes controlled fault injection for repeatable failure demos.
• Operational docs cover error-rate incidents, latency regressions, and rollback failures.
• The manifests use production-facing defaults: probes, resource boundaries, non-root containers, read-only root filesystem, NetworkPolicy, HPA, and PDB.

What this demonstrates

• reproducible local runtime for an HTTP service;
• Docker image built with a non-root runtime user;
• Kubernetes deployment with probes, security context, resource boundaries, and disruption controls;
• Helm chart with environment-specific values;
• Argo Rollouts canary strategy with Prometheus-backed analysis;
• Prometheus alerts, Grafana dashboard, and OpenTelemetry collector config;
• k6 smoke/load tests for release validation;
• CI pipeline with tests, Helm rendering, image build, SBOM generation, and vulnerability scanning;
• operational runbooks for high error rate, latency spikes, and failed rollbacks.

Architecture

flowchart LR
  Dev["Developer"] --> CI["GitHub Actions"]
  CI --> Image["Container image"]
  Image --> Rollout["Argo Rollouts canary"]
  Rollout --> Stable["Stable pods"]
  Rollout --> Canary["Canary pods"]
  Stable --> Metrics["Prometheus"]
  Canary --> Metrics
  Metrics --> Analysis["Success-rate analysis"]
  Analysis --> Rollout
  Metrics --> Grafana["Grafana dashboard"]
  AppLogs["Structured logs"] --> Loki["Loki"]
  AppTraces["OTLP traces"] --> OTel["OpenTelemetry Collector"]

Loading

Quick start

make test
make run

The service listens on http://127.0.0.1:8080.

Useful endpoints:

• GET /healthz - liveness check;
• GET /readyz - readiness check;
• GET /version - build and runtime metadata;
• GET /work - synthetic business endpoint with optional latency/error injection;
• GET /metrics - Prometheus metrics.

Smoke test:

make smoke

Local failure simulation

ERROR_RATE=0.35 LATENCY_MS=250 make run

This makes /work fail roughly 35% of the time and sleep for 250 ms. The same knobs are exposed as Kubernetes environment variables, which makes bad-release scenarios easy to reproduce.

Repository layout

apps/api/                 Go service used by the release controller
deploy/helm/              Helm chart for standard Kubernetes deployment
deploy/rollouts/          Argo Rollouts canary and analysis templates
observability/            Prometheus, Grafana, Loki, and OpenTelemetry config
tests/load/               k6 release validation scenarios
docs/runbooks/            Incident response runbooks
scripts/                  Local automation and validation scripts

Demo path

1. Build and test the service.
2. Deploy the stable version with the Helm chart.
3. Deploy a canary version with elevated ERROR_RATE.
4. Let Prometheus observe the degraded success rate.
5. Watch Argo Rollouts abort the canary before full promotion.

The manifests are designed so the demo can run on any Kubernetes cluster with Prometheus and Argo Rollouts installed.

Documentation

• Architecture
• Demo guide
• Roadmap
• Governance
• High error rate runbook
• Latency spike runbook
• Rollback failed runbook