Fix parser bugs, optimize Layer 3 subnet analysis, improve scripting & EOF menu safety

Author: leifdavisson

.gitignore

@@ -6,6 +6,7 @@ __pycache__/
 # Local outputs & logs
 raw_logs/
 backups/
+deliverables/
 nmap_results.xml
 failed_hosts.json
 asset_inventory.csv
@@ -17,8 +18,10 @@ migration_cabling_matrix.csv
 cisco_to_target_translation.md
 migration_config_variables.json
 migration_verification_report.md
-*.json
 oui.txt
+*.log
+*.cfg
+*.json
 
 # OS specific files
 .DS_Store

README.md

@@ -72,6 +72,14 @@ This shell will guide you through checking requirements, installing dependencies
 
 ---
 
+## 📚 Documentation
+
+Detailed guides and manuals are available in the `docs/` directory:
+* **[Junior Network Engineer Quick-Start Guide](docs/junior_engineer_guide.md)**: Walks through basic menu operations, using the simulated demo scan mode, and reading generated topology diagrams and reports.
+* **[Advanced Scripting & Automation Guide](docs/scripting_guide.md)**: Outlines command-line flags, environment variable parameterization, and integration scripts to run the crawler programmatically in cron jobs or CI/CD pipelines.
+
+---
+
 ### Manual CLI Execution (Alternative)
 
 If you prefer to run the scripts manually, follow these steps:

cisco_crawler.py

@@ -0,0 +1,96 @@
+# Cisco Switch Docu-Crawler - Junior Network Engineer Quick-Start Guide
+
+Welcome to the **Cisco Switch Docu-Crawler**! This tool is designed to automate the discovery, auditing, configuration backup, and migration mapping of Cisco network switches and routers (supporting Cisco IOS, IOS-XE, NX-OS, and IOS-XR).
+
+This guide is designed to help you get up to speed with running the tool safely and interpreting the generated outputs.
+
+---
+
+## 🚀 Running the Operator Shell
+
+To make execution easy and consistent, the tool provides operator menus for both Unix/Linux and Windows platforms. These scripts check system requirements (like Python 3, pip, and Nmap) and install necessary packages automatically.
+
+### On Unix/Linux/macOS:
+Open your terminal and run the shell script bootstrapper:
+```bash
+./run.sh
+```
+
+### On Windows (PowerShell):
+Double-click `run.bat` or run it from a PowerShell window:
+```powershell
+PowerShell.exe -ExecutionPolicy Bypass -File .\run.ps1
+```
+
+---
+
+## 📋 Operations Menu: Step-by-Step
+
+When you launch the script, you are presented with the **Operations Menu**:
+
+```
+Operations Menu:
+  1) Initialize Environment (Install Python packages)
+  2) Run a New Discovery Scan
+  3) Run Simulated Discovery (Demo Mode)
+  4) List Current Backups
+  5) Advanced Options Menu
+  6) Exit
+```
+
+### Step 1: Initialize Environment (Option 1)
+Run this the first time you clone the repository or set up a new machine. It ensures that the required Python libraries (like `netmiko` and `netaddr`) are installed and up-to-date.
+
+### Step 2: Run Simulated Discovery (Option 3 - Highly Recommended First Step)
+Before scanning a live network, run a simulation. 
+* **Safe Dry-Run**: It does not make any real network connections.
+* It uses mock switches (spanning different Cisco models like WS-C3850, WS-C2960X, and ISR4431) to generate a full set of deliverables.
+* **Purpose**: Allows you to check the report formatting and understand what the tool collects without any risk of affecting production systems.
+
+### Step 3: Run a New Discovery Scan (Option 2)
+When you are ready to scan the live network:
+1. **Subnets Input**: The script will prompt you for target subnets (e.g. `192.168.1.0/24`). You can enter a single subnet or multiple subnets separated by commas.
+2. **Credentials Input**: Provide the SSH/Telnet username and password. 
+   * **AAA Lockout Prevention**: The crawler is built to gracefully halt retries if it hits a `NetmikoAuthenticationException` (authentication error). This prevents the crawler from locking out operator credentials in TACACS+/RADIUS AAA servers.
+3. **Scan Execution**: The tool will scan the subnets to locate active switches, validate login credentials, and crawl configuration details concurrently.
+
+---
+
+## 🛠️ Advanced Options Menu (Option 5)
+
+Advanced operations are grouped under the **Advanced Options Menu** to prevent accidental misconfigurations:
+
+* **Save/Compare Baselines**: Allows saving the running network state to a JSON file and comparing it later to detect changes (like new routes, modified neighbors, or interfaces going down).
+* **Retry/Resume**: Loads `failed_hosts.json` from a previous run to retry only the hosts that failed to connect, saving time.
+* **Crawler Customizations**:
+  * **Thread Count**: Defaults to `10`. Raising this too high (e.g. `50+`) on older network platforms can saturate switch CPU control planes. Keep it low for safety.
+  * **Connection Timeout**: The socket wait duration (default `10` seconds). Useful to raise only when scanning across high-latency WAN links.
+  * **Telnet Fallback**: Toggles Telnet fallback. Toggling this off enforces SSH-only connections.
+
+---
+
+## 📁 Understanding the Deliverables
+
+Every successful crawl creates a new folder under `deliverables/run_<timestamp>/`. The contents are organized as follows:
+
+### 1. `inventory/asset_inventory.csv`
+A spreadsheet listing every discovered switch and router. It contains the Hostname, IP, MAC address, Hardware Model, Firmware Version, Serial Number, and Management Method.
+
+### 2. `diagrams/L2_network_diagrams.md`
+Contains a **Mermaid.js** physical diagram showing how switches connect.
+* **Green Highlighted Nodes**: Represent Spanning Tree (STP) Root Bridges.
+* **Dashed Red Lines**: Represent active blocking ports (`BLK`) where STP is actively breaking topology loops.
+* **Solid Lines**: Forwarding connections (line thickness corresponds to link speed).
+
+### 3. `diagrams/L3_network_diagrams.md`
+Contains a logical diagram mapping IP networks, SVIs (Switched Virtual Interfaces), Loopbacks, and default routing boundaries.
+
+### 4. `analysis/network_analysis_report.md`
+An audit report highlighting issues:
+* Physical Layer: Speed/duplex mismatches, input/output packet errors, and CRC checksum issues.
+* Layer 2 Spanning Tree: STP Disabled switches (which present loop risks) and blocking ports list.
+* Layer 3 Subnet Conflicts: Flags **IP Address Conflicts** (same IP on two different switches) and **Subnet Overlaps** (overlapping routing ranges).
+* Services Audit: Flags missing NTP, DNS, or AAA security configurations.
+
+### 5. `migration/`
+Contains configuration mapping variables (`migration_config_variables.json`), physical patching details (`migration_cabling_matrix.csv`), and translation guidelines (`cisco_to_target_translation.md`) to assist in planning migrations.

docs/junior_engineer_guide.md

@@ -0,0 +1,167 @@
+# Cisco Switch Docu-Crawler - Advanced Scripting & Integration Guide
+
+The **Cisco Switch Docu-Crawler** is designed to be fully scriptable. This guide explains how to integrate the crawler into automated environments (such as cron jobs, Ansible playbooks, CI/CD pipelines, or custom Python orchestration scripts) by leveraging environment variables, CLI flags, and structured output files.
+
+---
+
+## 🔐 Credentials Automation
+
+To run the crawler non-interactively without prompting for input, you can pass credentials using environment variables:
+
+| Variable | Description |
+| --- | --- |
+| `CRAWLER_USER` or `CRAWLER_USERNAME` | SSH/Telnet Login Username |
+| `CRAWLER_PASSWORD` | SSH/Telnet Login Password |
+| `CRAWLER_SECRET` or `CRAWLER_ENABLE_SECRET` | Cisco Privileged EXEC Enable Secret |
+
+### Bash Example:
+```bash
+export CRAWLER_USER="admin"
+export CRAWLER_PASSWORD="SecureLogin123"
+export CRAWLER_SECRET="SecureEnable123"
+
+python3 cisco_crawler.py --subnets 10.0.1.0/24,10.0.2.0/24 --disable-telnet
+```
+
+---
+
+## ⚙️ CLI Reference Table
+
+The `cisco_crawler.py` command line supports the following flags:
+
+| Argument | Type | Default | Description |
+| --- | --- | --- | --- |
+| `--subnets` | string | *Local subnet* | Comma-separated target networks to scan (e.g. `10.0.0.0/24,192.168.1.0/24`) |
+| `--simulate` | flag | `False` | Run simulated scan & crawl (uses local mock switches without network traffic) |
+| `--disable-telnet` | flag | `False` | Disables Telnet connections and restricts connection protocol to SSH-only |
+| `--threads` | integer | `10` | Number of concurrent workers for switch scanning and crawls |
+| `--timeout` | integer | `10` | Connection and socket read timeout in seconds |
+| `--retry` | string | `None` | Path to `failed_hosts.json` to retry only previously failed targets |
+| `--baseline` | string | `None` | Filepath to save the collected network operational state (for baseline auditing) |
+| `--compare` | string | `None` | Filepath to a baseline JSON to compare the current network state against |
+| `--verbose` or `-v` | flag | `False` | Enable detailed debug logs printed to standard output |
+
+---
+
+## 🐍 Custom Python Integration Examples
+
+### Example 1: Nightly Automation Script (subprocess)
+This script runs the crawler, automatically passing subnets and credentials via the process environment, and exits with a status code.
+
+```python
+import subprocess
+import os
+import sys
+
+def run_nightly_backup():
+    print("[*] Starting scheduled backup run...")
+    
+    # Configure run environment
+    env = os.environ.copy()
+    env["CRAWLER_USER"] = "admin"
+    env["CRAWLER_PASSWORD"] = "SecurePassword123"
+    env["CRAWLER_SECRET"] = "SecureEnableSecret123"
+    
+    # Call crawler process
+    cmd = [
+        "python3", "cisco_crawler.py",
+        "--subnets", "192.168.1.0/24",
+        "--disable-telnet",
+        "--threads", "15",
+        "--timeout", "8"
+    ]
+    
+    result = subprocess.run(cmd, env=env, capture_output=True, text=True)
+    
+    if result.returncode == 0:
+        print("[+] Crawler completed successfully!")
+        print(result.stdout)
+    else:
+        print("[!] Crawler execution failed!", file=sys.stderr)
+        print(result.stderr, file=sys.stderr)
+        sys.exit(result.returncode)
+
+if __name__ == "__main__":
+    run_nightly_backup()
+```
+
+### Example 2: Parsing Config Variables JSON
+You can parse the structured JSON variables output file (`migration_config_variables.json`) inside your own automation scripts to build inventory assets or populate CMDBs.
+
+```python
+import json
+import glob
+import os
+
+def load_latest_device_variables():
+    # Locate the most recent run deliverables directory
+    runs = glob.glob("deliverables/run_*")
+    if not runs:
+        print("No crawler runs found.")
+        return
+        
+    latest_run = max(runs, key=os.path.getmtime)
+    vars_file = os.path.join(latest_run, "migration", "migration_config_variables.json")
+    
+    if not os.path.exists(vars_file):
+        print(f"Variables JSON not found in: {latest_run}")
+        return
+
+    with open(vars_file, "r") as f:
+        devices = json.load(f)
+        
+    print(f"--- Decoupled Configuration Parameters ({latest_run}) ---")
+    for hostname, config in devices.items():
+        print(f"\nDevice: {hostname}")
+        print(f"  Management IP: {config.get('management_ip')}")
+        print(f"  Model:         {config.get('model')}")
+        print(f"  DNS Servers:   {', '.join(config.get('dns_servers', []))}")
+        print(f"  NTP Servers:   {', '.join(config.get('ntp_servers', []))}")
+        print(f"  L3 Interfaces:")
+        for intf in config.get("l3_interfaces", []):
+            print(f"    - {intf.get('interface')}: {intf.get('ip_address')} (Subnet: {intf.get('subnet')})")
+
+if __name__ == "__main__":
+    load_latest_device_variables()
+```
+
+### Example 3: Running a Configuration State Baseline Audit
+This example automates baseline checks, saving the baseline config and raising alerts if the state changes in a future run.
+
+```python
+import subprocess
+import os
+import sys
+
+def check_network_baseline():
+    baseline_path = "backups/production_baseline.json"
+    env = os.environ.copy()
+    env["CRAWLER_USER"] = "admin"
+    env["CRAWLER_PASSWORD"] = "SecurePassword123"
+    env["CRAWLER_SECRET"] = "SecureEnableSecret123"
+    
+    # 1. Create baseline if it doesn't exist
+    if not os.path.exists(baseline_path):
+        print(f"[*] Baseline not found. Generating new baseline file at {baseline_path}...")
+        cmd = ["python3", "cisco_crawler.py", "--subnets", "192.168.1.0/24", "--baseline", baseline_path]
+        subprocess.run(cmd, env=env, check=True)
+        print("[+] Baseline successfully saved.")
+        return
+        
+    # 2. Compare current network state against the baseline
+    print("[*] Comparing current network state against baseline...")
+    cmd = ["python3", "cisco_crawler.py", "--subnets", "192.168.1.0/24", "--compare", baseline_path]
+    result = subprocess.run(cmd, env=env, capture_output=True, text=True)
+    
+    # Check output for configuration delta drift
+    if "State Comparison: CHANGES DETECTED" in result.stdout:
+        print("[!] WARNING: Network drift detected!")
+        # Print comparison report details
+        print(result.stdout)
+        # Here you could trigger email notifications or slack webhooks
+    else:
+        print("[+] Network state is consistent with the baseline.")
+
+if __name__ == "__main__":
+    check_network_baseline()
+```