🚀 ZPE Nodegrid OS Upgrade Automation

Bulk, hands-off OS upgrade automation for ZPE Systems Nodegrid devices over SSH. Feed it a list of device IPs — the script handles backup, image upload, checksum validation, upgrade, reboot monitoring, and post-upgrade verification for every device, sending email notifications at every stage.

🔄 How It Works

                    ┌─────────────────────────────────────────┐
                    │         devices.txt (IP list)           │
                    └───────────────────┬─────────────────────┘
                                        ▼
   ┌────────────────────────────────────────────────────────────────┐
   │ 1. CONNECT        SSH session established (paramiko)           │
   │ 2. BACKUP         save_settings → SFTP backup server           │
   │ 3. UPLOAD         ISO + .md5 pushed to /var/sw/ via SFTP       │
   │ 4. VALIDATE       md5sum computed on-device & compared         │
   │ 5. UPGRADE        software_upgrade triggered → device reboots  │
   │ 6. MONITOR        SSH polling until device goes DOWN and       │
   │                   comes back UP (no blind fixed sleeps)        │
   └────────────────────────────────────────────────────────────────┘
                                        ▼
              Post-upgrade validation: hostname, version, uptime,
              console device status compared before vs. after
                                        ▼
                     📧 Final summary email (SUCCESS/FAILED)

✨ Features

• Zero-touch bulk operation — one run upgrades every device in devices.txt, one by one
• Safety-first flow — aborts the device if any step fails (backup, upload, MD5 mismatch)
• Smart reboot detection — polls SSH instead of guessing with timers; knows the difference between "still upgrading", "rebooting", and "back online"
• Pre/post state comparison — captures console device statuses before and after to confirm nothing changed unexpectedly
• Email reporting at every stage — uses system mailx/mail, no SMTP libraries needed
• Per-stage failure isolation — a failed device doesn't stop the remaining ones
• Final summary report — success/failure count for the entire run

📂 Repository Contents

File	Description
invokeshell1.py	⭐ Main script (latest) — multi-step toolkit commands via invoke_shell, pre/post console status comparison
second.py	Earlier iteration — sends toolkit commands as heredoc blocks
firstpy.py, thirdpypypy.py	Development iterations
mail.py	Email helper experiments

Use invokeshell1.py — it's the most robust version.

⚙️ Configuration

Edit the settings block at the top of the script:

IP_FILE      = "devices.txt"          # one IP per line
SSH_USERNAME = "admin"
SSH_PASSWORD = "your_password_here"

EMAIL_TO   = "noc@yourcompany.com"
EMAIL_FROM = "nodegrid-upgrade@yourserver"

ISO_FILE     = "/opt/upgrades/Nodegrid_Platform_v6.0.0.iso"
MD5_FILE     = "/opt/upgrades/Nodegrid_Platform_v6.0.0.iso.md5"

BACKUP_SERVER_URL  = "sftp://192.168.10.50/backups"
BACKUP_SERVER_USER = "backupuser"
BACKUP_SERVER_PASS = "backuppass"

POLL_EVERY = 15      # seconds between SSH probes
MAX_WAIT   = 1800    # give up after 30 minutes

devices.txt example:

192.168.1.10
192.168.1.11
# comments are ignored
192.168.1.12

▶️ Usage

pip install paramiko
python3 invokeshell1.py

That's it. Watch the console or your inbox — every stage reports its status, and a final summary lands when all devices are done:

============================================================
  FINAL SUMMARY
============================================================
  [OK]  192.168.1.10       SUCCESS
  [OK]  192.168.1.11       SUCCESS
  [!!]  192.168.1.12       FAILED
============================================================
  2 succeeded,  1 failed  out of 3 total.

📋 Requirements

• Python 3.x with paramiko
• mailx or mail available on the runner host
• Network reachability to all target devices (SSH/22)
• An SFTP backup server for configuration backups
• Nodegrid ISO + matching .md5 file

⚠️ Notes

• Test on a lab device first — this performs real firmware upgrades and reboots.
• Devices are upgraded sequentially, never in parallel, by design.
• If a device doesn't return within MAX_WAIT, it's flagged FAILED and flagged for manual follow-up — the run continues with the next device.

---

About

Built during production network operations to turn a manual, multi-hour per-device upgrade procedure into a supervised one-command run. 🛠️