EVPN-in-a-Box

One-command deployment of a full VXLAN/EVPN data-center fabric on a single Proxmox server.

Takes a YAML config file and builds everything: Proxmox networking, CML virtual Nexus switches, NDFC fabric controller configuration, underlay OSPF, BGP EVPN overlay, VRFs, L2 networks, and a GitLab CI/CD pipeline for Day 2 operations. No manual GUI clicks, no hand-crafted NX-OS configs, no guessing serial numbers.

cp config.example.yaml config.yaml   # fill in your environment values
python3 deploy.py --all               # go get coffee, come back to a working fabric

Total deployment time: ~25-40 minutes depending on N9Kv boot speed.

What It Does

The deployer ties together three Cisco products that normally require separate manual setup:

Cisco Modeling Labs (CML) - spins up N9Kv virtual switches that emulate physical data-center hardware
Cisco Nexus Dashboard + NDFC - the SDN controller that manages the fabric lifecycle
NetAsCode (cisco.nac_dc_vxlan) - an Ansible collection that treats NDFC configuration as declarative YAML, version-controlled and deployed through CI/CD

The three-phase orchestrator handles everything between "I have a Proxmox server" and "I have a working EVPN fabric":

Phase 0: Infrastructure     Phase 1: CML Lab              Phase 2: Fabric
========================    ========================      ========================
Create vmbr1 bridge         Generate topology YAML        Ansible validate (schema)
Move ND NIC to vmbr1        Import lab into CML           Ansible create (NDFC state)
Add CML second NIC          Start N9Kv switches           Ansible deploy (push to switches)
Create automation VM        Bootstrap switch configs
Install GitLab + Ansible    Collect serial numbers
                            Generate NAC YAML files

Architecture

Everything runs on a single Proxmox host. Two bridges provide network segmentation:

+-----------------------------------------------------------------------+
|  Proxmox VE Host                                                      |
|                                                                       |
|  +------------------+  +------------------+  +--------------------+   |
|  |   ND  (VM 100)   |  |   CML (VM 101)   |  | Automation (VM 102)|   |
|  |   64 GB RAM      |  |   48 GB RAM      |  | 8 GB RAM           |   |
|  |   Nexus Dashboard |  |   Virtual Switches|  | GitLab + Ansible   |   |
|  |                  |  |                  |  |                    |   |
|  |  mgmt0  -vmbr0-  |  |  bridge0 -vmbr0- |  | net0  --vmbr0--   |   |
|  |  fabric0 -vmbr1- |  |  bridge1 -vmbr1- |  | net1  --vmbr1--   |   |
|  +------------------+  +------------------+  +--------------------+   |
|           |                     |                       |             |
|      vmbr0 (LAN)           vmbr1 (internal)        vmbr0 + vmbr1     |
|      192.168.1.0/24        172.16.1.0/24                              |
+-----------------------------------------------------------------------+
                |
  Physical NIC (LAN uplink on vmbr0 only)

vmbr0 - your home LAN. Carries SSH, NDFC UI, GitLab web access.
vmbr1 - internal-only bridge (no physical uplink). Switch management traffic between CML and NDFC stays isolated from your LAN.

Fabric Topology

Standard 2-tier spine-leaf with OSPF underlay and BGP EVPN overlay:

          +----------+        +----------+
          | Spine-1  |        | Spine-2  |
          | (RR)     |        | (RR)     |
          +----+-----+        +-----+----+
               |                    |
       +-------+----+        +-----+------+
       |            |        |            |
  +----+----+  +----+----+  +----+----+  +----+----+
  | Leaf-1  |  | Leaf-2  |  | Leaf-1  |  | Leaf-2  |
  +-+----+--+  +-+----+--+  +----+----+  +-+----+--+
    |    |       |    |
    |    |       |    |
  host  host   host  host
  red   blue   red   blue
  VRF-R VRF-B  VRF-R VRF-B

  VRF-Red:   VLAN 2301 / VNI 130001 / GW 10.10.10.1/24
  VRF-Blue:  VLAN 2302 / VNI 130002 / GW 10.20.20.1/24

Scale by adding more spines, leafs, VRFs, or networks in config.yaml. The deployer generates all the wiring and NDFC data model files automatically.

Prerequisites

Requirement	Details
Proxmox VE 8.x	Single node with enough RAM (128 GB recommended)
Nexus Dashboard	OVA imported as a Proxmox VM, NDFC app installed
Cisco Modeling Labs	VM on same Proxmox host, licensed
N9Kv image	Uploaded to CML node definitions
Nested virtualization	KVM nested virt enabled on Proxmox (`nested=1`)
Python 3.11+	On the machine running `deploy.py`
SSH key auth	To Proxmox host (root)

Resource Estimates

VM	vCPU	RAM	Disk	Purpose
ND (100)	8	64 GB	500 GB	Nexus Dashboard + NDFC
CML (101)	16	48 GB	200 GB	Virtual switch host
Automation (102)	4	8 GB	50 GB	GitLab + Ansible
Per N9Kv switch	4	4-8 GB	-	Full image needs 8 GB

Minimum Proxmox host: 128 GB RAM, 16+ cores, 750 GB storage.

Quick Start

# 1. Clone the repo
git clone https://github.com/beye91/evpn-in-a-box.git
cd evpn-in-a-box

# 2. Install Python dependencies
pip install -r requirements.txt

# 3. Copy and edit the config file
cp config.example.yaml config.yaml
# Fill in your Proxmox IP, ND credentials, CML credentials, etc.
# Every CHANGE_ME must be replaced with your actual values.

# 4. Deploy everything
python3 deploy.py --all

Step-by-Step

If you prefer to run phases individually:

# Phase 0: Set up Proxmox networking + automation VM
python3 deploy.py --phase 0

# Phase 1: Deploy CML lab, bootstrap switches, collect serials
python3 deploy.py --phase 1

# Phase 2: Push fabric config to NDFC via Ansible
python3 deploy.py --phase 2

Configuration

All settings live in config.yaml. Copy config.example.yaml to get started. Nothing is hardcoded in the code, every IP, credential, VLAN, VNI, and fabric parameter is read from this single file.

Key Sections

proxmox:         # Proxmox host IP, bridges, SSH user
nexus_dashboard: # ND VM ID, IPs, credentials, NIC MAC addresses
cml:             # CML VM ID, IP, credentials, lab name
automation_host: # Ubuntu VM specs, GitLab config, Docker registry
fabric:          # Fabric name, BGP ASN, anycast MAC, NTP
underlay:        # OSPF area, loopback ranges, MTU, replication mode
switches:        # N9Kv image, credentials, node list with roles/IPs
hosts:           # Test hosts attached to leaf switches
overlay:         # VRFs (L3 VNI) and networks (L2 VNI, gateways)
orchestrator:    # Lifecycle management, timeouts, backup config

Scaling the Fabric

Add switches by appending to the switches.nodes list:

switches:
  nodes:
    - name: "Spine-1"
      role: "spine"
      mgmt_ip: "172.16.1.101"
    - name: "Spine-2"
      role: "spine"
      mgmt_ip: "172.16.1.102"
    - name: "Leaf-1"
      role: "leaf"
      mgmt_ip: "172.16.1.103"
    - name: "Leaf-2"
      role: "leaf"
      mgmt_ip: "172.16.1.104"
    # Add more leafs as needed:
    - name: "Leaf-3"
      role: "leaf"
      mgmt_ip: "172.16.1.105"

Add VRFs and networks in the overlay section. Add test hosts in hosts. Re-run the relevant phase and the deployer reconciles.

CLI Reference

# Full deployment
python3 deploy.py --all

# Individual phases
python3 deploy.py --phase 0              # Proxmox infra
python3 deploy.py --phase 1              # CML lab + serial collection
python3 deploy.py --phase 2              # Ansible NDFC deployment

# Phase 1 skip options (when lab already exists)
python3 deploy.py --phase 1 --skip-import   # Lab imported, just start + bootstrap
python3 deploy.py --phase 1 --skip-start    # Lab running, just bootstrap switches

# Remove resources from NDFC (declarative removal)
python3 deploy.py --remove networks      # Remove networks not in YAML
python3 deploy.py --remove vrfs          # Remove VRFs not in YAML
python3 deploy.py --remove all           # Remove all undeclared resources

# Cleanup
python3 deploy.py --cleanup              # Stop CML lab

# Lifecycle orchestrator (runs on automation VM)
python3 orchestrator.py startup          # Resource-aware sequential boot
python3 orchestrator.py shutdown         # Graceful reverse-order stop
python3 orchestrator.py status           # Current state
python3 orchestrator.py update           # Backup + apt upgrade

# Ansible direct (Phase 2 only, requires .env)
ansible-playbook -i inventory.yaml vxlan.yaml --tags role_validate
ansible-playbook -i inventory.yaml vxlan.yaml --tags role_create,role_deploy

# Debug logging
python3 deploy.py --all -v
python3 deploy.py --all -vvv

How It Works

Phase 0: Infrastructure

Creates vmbr1 (internal bridge) on Proxmox if it doesn't exist
Moves ND's fabric0 NIC from vmbr0 to vmbr1 so NDFC can reach switches on the isolated network
Adds a second NIC to CML on vmbr1 for switch management traffic
Creates an Ubuntu 24.04 automation VM (VM 102) with GitLab CE, Docker, Ansible, and the Cisco NAC collections pre-installed

Phase 1: CML Lab

Generates a CML topology YAML from your config.yaml (spines, leafs, hosts, links)
Imports and starts the lab in CML
Waits for all N9Kv switches to boot (5-10 min per switch)
SSHes into each switch and applies bootstrap config (hostname, mgmt IP, SSH, SNMP, NTP)
Collects serial numbers via show inventory (NDFC needs these to identify switches)
Generates NetAsCode YAML files under host_vars/evpn-lab/

Phase 2: NDFC Deployment

Sources .env for NDFC credentials
Runs ansible-playbook vxlan.yaml which invokes:
- cisco.nac_dc_vxlan.validate - schema + semantic validation against live NDFC
- cisco.nac_dc_vxlan.dtc.create - creates fabric state in NDFC
- cisco.nac_dc_vxlan.dtc.deploy - pushes configuration to all switches via NDFC

Generated Files

Phase 1 generates these files (all gitignored since they contain lab-specific data):

File	Content
`host_vars/evpn-lab/global.nac.yaml`	Fabric name, BGP ASN, NTP
`host_vars/evpn-lab/underlay.nac.yaml`	OSPF, multicast, IP ranges
`host_vars/evpn-lab/topology.nac.yaml`	Switch inventory with serial numbers
`host_vars/evpn-lab/vrfs.nac.yaml`	VRF definitions and attachments
`host_vars/evpn-lab/networks.nac.yaml`	L2 networks, VLANs, gateways
`.env`	NDFC credentials for Ansible

CI/CD Pipeline

The project includes a GitLab CI/CD pipeline (.gitlab-ci.yml) for Day 2 operations. After initial deployment, fabric changes follow a GitOps workflow:

Edit host_vars YAML -> git push -> validate (auto) -> deploy (manual approve)

A custom Docker runner image (docker/ansible-runner/Dockerfile) has all dependencies pre-installed, eliminating setup time:

# Build the runner image
docker build --no-cache -t ansible-runner:latest docker/ansible-runner/

Pipeline stages:

validate - Schema check + read-only diff against live NDFC (automatic)
deploy - Push config to switches via NDFC (manual gate, requires human review)
remove - Declarative removal of resources absent from YAML (manual)
build-image - Rebuild the runner Docker image (manual, on dependency changes)

Lifecycle Orchestrator

The orchestrator.py script manages VM startup/shutdown sequences for the entire lab. It runs on the automation VM as a systemd service and handles:

Resource-aware boot: Starts VMs sequentially, checking CPU/RAM thresholds between each N9Kv node start to avoid overloading the Proxmox host
Graceful shutdown: Stops nodes in reverse order, then shuts down VMs cleanly
Backup: Optional vzdump snapshots to Proxmox Backup Server before shutdown
Self-update: Scheduled apt upgrades with automatic reboot if needed

# Install as systemd service on the automation VM
sudo ./scripts/install-orchestrator.sh

Inter-VRF Routing (Optional)

The project supports inter-VRF routing via an Alpine Linux firewall service node. Traffic between VRF-Red and VRF-Blue is routed through a lightweight VM with iptables, using VRF-Lite static routes pushed through NAC overlay extensions.

Enable it by uncommenting the service_nodes section in config.yaml. See docs/inter-vrf-routing.md for the full design.

Troubleshooting

Common issues and fixes are documented per phase in docs/troubleshooting.md.

Quick diagnostics:

# Check CML lab and node status
python3 deploy.py --phase 1 --skip-import --skip-start   # re-bootstrap only

# Validate NDFC config without applying changes
ansible-playbook -i inventory.yaml vxlan.yaml --tags role_validate -v

# Check if NDFC API is responding (401 = up, needs auth)
curl -sk -o /dev/null -w "%{http_code}" \
  https://<ND_DATA_IP>/appcenter/cisco/ndfc/api/v1/lan-fabric/rest/control/fabrics

# SSH to a switch and verify EVPN state
ssh admin@172.16.1.101
show bgp l2vpn evpn summary
show nve peers

Project Structure

evpn-in-a-box/
|-- deploy.py               # Main CLI orchestrator (Phase 0, 1, 2)
|-- orchestrator.py          # VM lifecycle manager (startup/shutdown/backup)
|-- config.example.yaml      # Configuration template (copy to config.yaml)
|-- ansible.cfg              # Ansible runtime settings
|-- inventory.yaml           # Ansible inventory (reads from env vars)
|-- vxlan.yaml               # Ansible playbook (NetAsCode roles)
|-- requirements.txt         # Python dependencies
|-- requirements.yaml        # Ansible collection dependencies
|
|-- lib/                     # Python modules
|   |-- config_loader.py     # YAML config loader + validation
|   |-- proxmox_prep.py      # Phase 0a: Proxmox bridge + NIC setup
|   |-- cml_bridge.py        # Phase 0b: CML bridge1 via SSH
|   |-- automation_host.py   # Phase 0c: Ubuntu VM + GitLab + Ansible
|   |-- topology_builder.py  # CML topology YAML generator
|   |-- cml_manager.py       # Phase 1: CML API + serial collection
|   |-- nac_generator.py     # NetAsCode YAML file generator
|   |-- wait_utils.py        # Polling/retry helpers
|   |-- proxmox_monitor.py   # Proxmox resource monitoring
|
|-- group_vars/ndfc/         # Ansible NDFC connection (env var based)
|-- host_vars/evpn-lab/      # Generated NAC data model (gitignored)
|-- docker/ansible-runner/   # CI runner Docker image
|-- scripts/                 # Helper scripts (build, install, patch)
|-- docs/                    # Architecture, deployment guide, troubleshooting

Documentation

Architecture - detailed system design, network segmentation, deployment workflow
Deployment Guide - step-by-step walkthrough
Docker CI/CD - runner image and pipeline setup
Startup Orchestrator - lifecycle management
Inter-VRF Routing - firewall service node design
MCP Platform - LLM integration with NDFC and CML
Troubleshooting - common issues and fixes per phase

License

MIT. See LICENSE for full text.

Navigation Menu

Repository files navigation

EVPN-in-a-Box

What It Does

Architecture

Fabric Topology

Prerequisites

Resource Estimates

Quick Start

Step-by-Step

Configuration

Key Sections

Scaling the Fabric

CLI Reference

How It Works

Phase 0: Infrastructure

Phase 1: CML Lab

Phase 2: NDFC Deployment

Generated Files

CI/CD Pipeline

Lifecycle Orchestrator

Inter-VRF Routing (Optional)

Troubleshooting

Project Structure

Documentation

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
docker/ansible-runner		docker/ansible-runner
docs		docs
group_vars/ndfc		group_vars/ndfc
lib		lib
scripts		scripts
.env.example		.env.example
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
LICENSE		LICENSE
README.md		README.md
ansible.cfg		ansible.cfg
config.example.yaml		config.example.yaml
deploy.py		deploy.py
inventory.yaml		inventory.yaml
orchestrator.py		orchestrator.py
requirements.txt		requirements.txt
requirements.yaml		requirements.yaml
vxlan.yaml		vxlan.yaml

Folders and files

Latest commit

History

Repository files navigation

EVPN-in-a-Box

What It Does

Architecture

Fabric Topology

Prerequisites

Resource Estimates

Quick Start

Step-by-Step

Configuration

Key Sections

Scaling the Fabric

CLI Reference

How It Works

Phase 0: Infrastructure

Phase 1: CML Lab

Phase 2: NDFC Deployment

Generated Files

CI/CD Pipeline

Lifecycle Orchestrator

Inter-VRF Routing (Optional)

Troubleshooting

Project Structure

Documentation

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages