Skip to content
BranchesTags

Repository files navigation

Amazon EKS GitOps Governance Platform

AWS Terraform Kubernetes Argo CD Kyverno

A production-inspired Kubernetes governance platform built on Amazon EKS using Terraform, Argo CD, Helm, and Kyverno.

This project provisions AWS infrastructure with Terraform, deploys Kubernetes resources through GitOps, enforces container resource requirements with Kyverno, and automatically corrects configuration drift with Argo CD.

Project Highlights

  • Provisioned Amazon EKS and AWS networking with Terraform
  • Deployed worker nodes in private subnets
  • Installed Argo CD and Kyverno using Helm
  • Implemented GitOps synchronization, pruning, and self-healing
  • Enforced CPU and memory requirements with Kyverno
  • Tested valid and invalid Kubernetes workloads
  • Troubleshot Helm timeouts and EKS Pod-capacity limits
  • Added Terraform validation with GitHub Actions

Architecture

Developer
    |
    | git push
    v
GitHub Repository
    |
    v
Argo CD
    |
    +----------------------+
    |                      |
    v                      v
Kyverno Policies      Kubernetes Workloads
    |
    v
Admission Control
    |
    +----------------------+
    |                      |
Invalid Workload       Valid Workload
Rejected               Deployed

Terraform
    |
    v
AWS VPC
    |
    v
Amazon EKS
    |
    v
Managed Worker Nodes

Technology Stack

Tool Responsibility
Terraform Provisions AWS infrastructure and Amazon EKS
Amazon EKS Managed Kubernetes platform
Argo CD Deploys and reconciles resources from Git
Kyverno Enforces Kubernetes admission policies
Helm Installs Argo CD and Kyverno
GitHub Actions Validates Terraform configuration
AWS VPC CNI Provides networking for EKS Pods

Business Problem

Kubernetes workloads without resource requests and limits can cause:

  • Poor scheduling
  • Memory exhaustion
  • Noisy-neighbor issues
  • Unstable worker nodes
  • Inconsistent workload standards

Manual YAML review does not scale and cannot prevent configuration drift.

This project solves the problem by enforcing policies during Kubernetes admission and continuously reconciling approved resources from Git.

Policy Enforcement

The Kyverno policy requires every container to define:

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    memory: "256Mi"

A workload without these values is rejected before it is stored in the cluster.

kubectl apply -f workloads/invalid-deployment.yaml

Expected result:

admission webhook denied the request:
CPU and memory requests and a memory limit are required.

A compliant workload is successfully admitted:

kubectl apply -f workloads/valid-deployment.yaml

Project Screenshots

Amazon EKS Cluster

The EKS cluster was provisioned successfully with Terraform.

Amazon EKS Cluster

Worker Nodes

The managed worker nodes joined the cluster and reached the Ready state.

Worker Nodes

Argo CD Applications

Argo CD continuously synchronizes the governance policies and sample application from Git.

Argo CD Applications

Kyverno Policy

The Kyverno ClusterPolicy is active and running in Enforce mode.

Kyverno Policy

Invalid Workload Rejected

Kyverno rejected a workload that did not include the required CPU and memory configuration.

Invalid Workload Rejected

Valid Workload Running

A compliant workload was admitted and successfully deployed.

Valid Workload Running

GitHub Actions

GitHub Actions validates Terraform formatting and configuration.

GitHub Actions

Repository Structure

.
├── terraform/                  # AWS, EKS, Argo CD, and Kyverno
├── charts/                     # Kyverno governance policy Helm chart
├── gitops/                     # Argo CD Applications and workloads
├── policies/                   # Kubernetes policy definitions
├── workloads/                  # Valid and invalid test manifests
├── scripts/                    # Bootstrap and policy test scripts
├── docs/screenshots/           # Project screenshots
├── .github/workflows/          # Terraform CI validation
└── README.md

Deployment

1. Configure Terraform

cp terraform/terraform.tfvars.example terraform/terraform.tfvars

Example configuration:

aws_region          = "us-east-1"
project_name        = "eks-kyverno-demo"
environment         = "dev"
cluster_version     = "1.35"
node_instance_types = ["t3.medium"]

2. Deploy the Infrastructure

terraform -chdir=terraform init
terraform -chdir=terraform fmt -recursive
terraform -chdir=terraform validate
terraform -chdir=terraform plan
terraform -chdir=terraform apply

3. Connect to Amazon EKS

aws eks update-kubeconfig \
  --region us-east-1 \
  --name eks-kyverno-demo-dev

Verify the cluster:

kubectl get nodes
kubectl get pods -n argocd
kubectl get pods -n kyverno

4. Bootstrap Argo CD Applications

./scripts/bootstrap-argocd.sh \
  https://github.com/YOUR_USERNAME/eks-kyverno-policy-as-code.git

Verify the applications:

kubectl get applications -n argocd

Expected status:

NAME                          SYNC STATUS   HEALTH STATUS
kyverno-governance-policies   Synced        Healthy
policy-compliant-sample-app   Synced        Healthy

GitOps Self-Healing

Argo CD continuously compares the live cluster with the desired state stored in Git.

Create manual drift:

kubectl scale deployment policy-compliant-nginx \
  --replicas=5 \
  -n policy-demo

Argo CD detects the change and restores the replica count defined in Git.

This demonstrates:

  • Git as the source of truth
  • Continuous reconciliation
  • Drift detection
  • Automated recovery

Troubleshooting Experience

During deployment, Argo CD and Kyverno Pods remained in Pending status.

The Kubernetes scheduler reported:

0/2 nodes are available: 2 Too many pods

The issue was not caused by CPU or memory exhaustion. The worker nodes had reached their maximum Pod capacity under the AWS VPC CNI networking model.

The managed node group was scaled from two nodes to three:

min_size     = 2
desired_size = 3
max_size     = 4

After the additional node joined, the pending platform Pods were scheduled successfully.

This demonstrated practical experience with:

  • Kubernetes scheduler events
  • EKS Pod-density limits
  • AWS VPC CNI networking
  • Helm deployment troubleshooting
  • Managed node group scaling

Design Decisions

Terraform owns infrastructure

Terraform manages:

  • VPC and subnets
  • NAT gateway
  • IAM roles
  • Amazon EKS
  • Managed node groups
  • Argo CD installation
  • Kyverno installation

Argo CD owns Kubernetes configuration

Argo CD manages:

  • Kyverno policies
  • Application namespaces
  • Deployments
  • Services

This separation prevents Terraform and Argo CD from managing the same resources.

Kyverno runs in Enforce mode

Noncompliant workloads are blocked before deployment.

In a production environment, policies would normally begin in Audit mode before moving to Enforce.

CI Validation

GitHub Actions validates Terraform changes using:

terraform fmt -check
terraform init -backend=false
terraform validate

This catches formatting and configuration errors before changes are merged.

Lessons Learned

  • Provisioning Amazon EKS with Terraform
  • Designing ownership boundaries between Terraform and Argo CD
  • Implementing GitOps reconciliation and self-healing
  • Enforcing Kubernetes policies with Kyverno
  • Troubleshooting failed Helm releases
  • Investigating Kubernetes scheduler events
  • Understanding EKS Pod-capacity limitations
  • Scaling managed node groups
  • Testing valid and invalid workloads
  • Building Terraform CI validation

Future Improvements

  • Store Terraform state in Amazon S3
  • Add Terraform state locking
  • Add development, staging, and production environments
  • Add Karpenter or Cluster Autoscaler
  • Enable VPC CNI prefix delegation
  • Add Prometheus and Grafana
  • Add External Secrets Operator
  • Block privileged containers
  • Block the latest image tag
  • Require non-root containers
  • Restrict images to trusted registries

Cleanup

Delete the Argo CD Applications:

kubectl delete applications \
  kyverno-governance-policies \
  policy-compliant-sample-app \
  -n argocd

Destroy the AWS infrastructure:

terraform -chdir=terraform destroy

Amazon EKS, EC2 worker nodes, and NAT gateways generate AWS charges, so the environment should be destroyed when testing is complete.

Resume Bullet

Built a GitOps-based Kubernetes governance platform on Amazon EKS using Terraform, Argo CD, Helm, and Kyverno; automated infrastructure provisioning, enforced container resource requirements through admission control, implemented drift correction, and troubleshot EKS Pod-density and Helm deployment failures.

Author

Mina Bisa

DevOps | Cloud | Platform Engineering

GitHub: https://github.com/minabisa

LinkedIn: https://www.linkedin.com/in/mina-bisa

About

Enterprise Kubernetes governance with Terraform, Amazon EKS, Kyverno Policy as Code, Helm, and Argo CD GitOps.

Topics

argocdawsdevopseksgitopshelmiackuberneteskubernetes-securitykyvernopolicy-as-codeterraform

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages