Skip to content

implementation.md

Latest commit

 

History

History
372 lines (280 loc) · 12.8 KB

implementation.md

File metadata and controls

372 lines (280 loc) · 12.8 KB
Raw

Local Implementation Guide

Scope: Phases 1–4 — the complete BeyondSquare local platform.

This guide documents the implementation from the initial application container through Docker Compose, Kubernetes, CI/CD, and observability.

AWS deployment is documented separately in AWS_IMPLEMENTATION_GUIDE.md.

Implementation scope

This guide covers the complete local implementation:

Phase Implementation Status
1 Local Development — Docker Compose ✅ Complete
2 Local Kubernetes — Kind + Kustomize ✅ Complete
3 CI/CD — GitHub Actions + self-hosted runner ✅ Complete
4 Monitoring & Logging — Prometheus + Grafana + Loki ✅ Complete

What you will build

Developer
   │
   ▼
Docker Compose
   │
   ├── BeyondSquare API
   ├── PostgreSQL
   └── Redis
   │
   ▼
Kind Kubernetes cluster
   │
   ├── API Deployment
   ├── PostgreSQL StatefulSet
   ├── Redis Deployment
   └── NGINX Ingress
   │
   ▼
GitHub Actions
   │
   └── build → test → deploy
   │
   ▼
Observability
   ├── Prometheus
   ├── Grafana
   ├── Loki
   └── Promtail

Repository layout

mkdir -p beyondsquare/{app/src,app/tests,k8s/base,k8s/overlays/local,monitoring,.github/workflows,docs}
cd beyondsquare
git init

Phase 1 — Local Development with Docker Compose

Goal: establish the application inner loop before introducing Kubernetes: build the API, containerize it, connect PostgreSQL and Redis, validate health/readiness, and exercise the application endpoints.

1.1 The API

A small REST service (FastAPI/Express-equivalent) exposing:

  • GET /health — liveness: {"status":"alive"}. Answers one question only: is the process running?
  • GET /ready — readiness: {"status":"ready","checks":{"postgres":true,"redis":true}}. Verifies actual dependency connectivity — this is what Kubernetes and later the ALB will use to decide whether to route traffic to a pod.
  • POST /items / GET /items — CRUD against PostgreSQL.
  • GET /cache-demo — first call returns {"source":"generated"}, subsequent calls within the cache window return {"source":"cache"}, proving Redis is actually in the read path.
  • GET /metrics — Prometheus exposition format, via a client library (prom-client or equivalent). No code changes are needed later to start scraping it — this is intentional.

1.2 Container image

The API is containerized using the Dockerfile in app/Dockerfile.

The image uses a multi-stage build and runs the application as a non-root user.

See app/Dockerfile for the current implementation.

Multi-stage build → smaller final image (no build toolchain shipped). Non-root USER app → a security best practice worth naming explicitly in an interview.

1.3 docker-compose.yaml

See CONFIGURATION.md for the full file and notes on depends_on's limitations.

1.4 Run it

docker compose up --build
docker compose ps                      # confirm all three show (healthy)
curl http://localhost:3000/health
curl http://localhost:3000/ready
curl -X POST http://localhost:3000/items -H "Content-Type: application/json" -d '{"name":"first item"}'
curl http://localhost:3000/items
curl http://localhost:3000/cache-demo   # run twice — second call should say "source":"cache"
docker compose logs -f api
docker compose down -v                 # tear down + wipe volumes

Concepts this phase exercises: Docker layer caching, multi-stage builds, container networking (service-name DNS resolution), volumes vs. bind mounts, and the gap between "container started" and "container ready."

Phase 1 validation

The phase is complete when:

  • API container is healthy
  • PostgreSQL is reachable
  • Redis is reachable
  • /health returns alive
  • /ready reports both dependencies healthy
  • CRUD operations work
  • Redis cache behavior is observable

Phase 2 — Local Kubernetes (Kind)

Goal: the real K8s object model, service discovery, and networking, without any cloud cost.

2.1 Create the cluster

kind-config.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"
    extraPortMappings:
      - containerPort: 80
        hostPort: 80
      - containerPort: 443
        hostPort: 443
kind create cluster --name beyondsquare --config kind-config.yaml
kubectl cluster-info --context kind-beyondsquare

2.2 Install the NGINX Ingress Controller (Kind-flavored)

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
kubectl wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=120s

2.3 Load the locally built image into Kind

Kind cannot pull from your local Docker daemon automatically — the image has to be loaded explicitly. This trips up almost everyone the first time:

docker build -t beyondsquare-api:local ./app
kind load docker-image beyondsquare-api:local --name beyondsquare

2.4 Base manifests (k8s/base)

See CONFIGURATION.md for the full layout and probe wiring.

2.5 Deploy

kubectl apply -k k8s/overlays/local
kubectl get pods -w
kubectl get svc,ingress
echo "127.0.0.1 beyondsquare.local" | sudo tee -a /etc/hosts
curl http://beyondsquare.local/health

2.6 Daily operational commands (practice until they're muscle memory)

kubectl logs -f deploy/beyondsquare-api
kubectl exec -it deploy/beyondsquare-api -- sh
kubectl describe pod <pod>              # read the Events section first when debugging
kubectl rollout restart deploy/beyondsquare-api
kubectl rollout status deploy/beyondsquare-api
kubectl scale deploy/beyondsquare-api --replicas=3
kubectl top pod                          # needs metrics-server, see Phase 4

Concepts this phase exercises: Deployments vs. StatefulSets, Services (ClusterIP/NodePort), Ingress vs. Service, ConfigMaps/Secrets, liveness vs. readiness semantics, resource requests/limits and QoS classes, PersistentVolumeClaims.

Phase 2 validation

The phase is complete when:

  • Kind cluster is running
  • API pods are Ready
  • PostgreSQL StatefulSet is Ready
  • Redis is Ready
  • Ingress is reachable
  • /health and /ready work through the ingress

Phase 3 — CI/CD with GitHub Actions ✅ Complete

Goal: every push builds, tests, and deploys automatically to the local Kind cluster via a self-hosted runner.

3.1 Why a self-hosted runner

GitHub-hosted runners can't reach a laptop's Kind cluster. A self-hosted runner solves this at ₹0 (it uses existing hardware):

mkdir actions-runner && cd actions-runner
curl -o actions-runner.tar.gz -L https://github.com/actions/runner/releases/download/vX.Y.Z/actions-runner-linux-x64-vX.Y.Z.tar.gz
tar xzf actions-runner.tar.gz
./config.sh --url https://github.com/<you>/beyondsquare --token <token-from-github-ui>
sudo ./svc.sh install && sudo ./svc.sh start   # run as a service, not a foreground terminal

Running it as a systemd service rather than ./run.sh in a terminal matters: a suspended (Ctrl+Z) or closed terminal leaves the runner registered with GitHub but not actually listening — see Incident 003 in TROUBLESHOOTING.md.

3.2 .github/workflows/ci.yaml — runs on every push

name: CI
on:
  push:
    branches: [main]
  pull_request:

jobs:
  build-test:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4
      - name: Build image
        run: docker build -t beyondsquare-api:${{ github.sha }} ./app
      - name: Run unit tests
        run: docker run --rm beyondsquare-api:${{ github.sha }} npm test
      - name: Load image into Kind
        run: kind load docker-image beyondsquare-api:${{ github.sha }} --name beyondsquare
      - name: Deploy to Kind
        run: |
          kubectl set image deploy/beyondsquare-api api=beyondsquare-api:${{ github.sha }}
          kubectl rollout status deploy/beyondsquare-api

Build → test in isolation → load into the cluster's node → trigger a rolling update → wait for it to converge, failing the pipeline if it doesn't.

Phase 3 validation

The phase is complete when:

  • GitHub Actions receives a push
  • self-hosted runner picks up the job
  • image builds successfully
  • tests pass
  • image is loaded into Kind
  • deployment rolls out successfully

Phase 4 — Monitoring & Logging

Goal: observability is what separates "I deployed an app" from "I run production systems."

4.1 metrics-server (for kubectl top and HPA later)

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
# Kind needs --kubelet-insecure-tls; patch it in:
kubectl patch deployment metrics-server -n kube-system --type='json' \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

4.2 Prometheus + Grafana (kube-prometheus-stack)

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm install kube-prom prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  -f monitoring/prometheus-values.yaml

Add the ServiceMonitor shown in CONFIGURATION.md so Prometheus auto-discovers the API's /metrics endpoint.

4.3 Loki + Promtail (log aggregation)

helm install loki grafana/loki-stack -n monitoring \
  --set promtail.enabled=true \
  -f monitoring/loki-values.yaml

Watch for the datasource conflict described in CONFIGURATION.md — both charts try to provision a default Grafana datasource, which crash-loops Grafana if not configured around.

4.4 Access Grafana

kubectl port-forward svc/kube-prom-grafana -n monitoring 3001:80
kubectl get secret kube-prom-grafana -n monitoring -o jsonpath="{.data.admin-password}" | base64 -d

Username: admin. Open http://localhost:3001, confirm Loki is added as a datasource, and build one dashboard combining request rate/latency/error-rate (RED metrics) from Prometheus with a live log stream from Loki filtered by pod label.

4.5 Verification

kubectl get pods -n monitoring
kubectl get svc -n monitoring
curl http://localhost:3000/metrics
kubectl get servicemonitor -n monitoring

4.6 Troubleshooting drills — run these intentionally

This is where genuine operational muscle gets built, and where portfolio-worthy incident writeups come from:

  1. Kill the Postgres pod mid-traffic → watch the readiness probe fail → observe the Grafana error-rate spike → find the exact error in Loki → fix it.
  2. kubectl set image to a nonexistent tag → watch ImagePullBackOff → diagnose with kubectl describe pod → roll back with kubectl rollout undo.
  3. Set a CPU limit too low → induce throttling → find container_cpu_cfs_throttled_periods_total in Prometheus → correct it.
  4. Fill the Postgres PVC (or simulate it) → watch the app fail → diagnose from logs, not guesswork.
  5. Scale Redis to 0 → watch whether the app degrades gracefully or not — this is a real interview story about circuit breakers/fallbacks.

Document each drill using the Symptom → Investigation → Root Cause → Fix → Lesson format in TROUBLESHOOTING.md.

Concepts this phase exercises: the four golden signals (latency, traffic, errors, saturation), Prometheus's pull-based scrape model, basic PromQL (rate(), histogram_quantile()), logs vs. metrics vs. traces, ServiceMonitor CRDs, dashboards-as-code as a stretch goal.

Phase 4 validation

The phase is complete when:

  • Prometheus is scraping the API
  • Grafana is accessible
  • Loki receives application logs
  • Promtail is collecting logs
  • the API dashboard displays metrics
  • an intentional failure can be investigated through metrics and logs

Next step — AWS deployment

The local platform built in Phases 1–4 is the foundation for the AWS deployment.

The AWS implementation takes the same application and Kubernetes deployment model and extends it with:

  • Terraform-managed VPC
  • Amazon EKS
  • Amazon ECR
  • AWS Load Balancer Controller
  • Application Load Balancer
  • EBS CSI storage
  • CloudWatch
  • AWS-specific Kustomize overlays
  • GitHub Actions deployment

See AWS_IMPLEMENTATION_GUIDE.md for the complete AWS workflow