Operator runbooks: document production network patterns

[Reccetech]

Summary

No documentation exists for operating a real network: node failure recovery, rolling upgrades, state backup/restore, or scaling. An operator running production infrastructure has no runbook. The state-save-and-restore example exists in the Solo repo but is not linked from any operator-facing docs today.

Proposed Scope

A new "Operating Your Network" page or section covering at minimum:

• Handling a node failure
• Rolling upgrade procedure
• State backup and restore (linking the existing state-save-and-restore example)

Blocked

Requires input from someone who has operated a real network. Content cannot be written from spec alone. If you have production Solo experience, please comment.

[deshmukhpranali]

@Reccetech - this one needs SME input before I can write anything accurate. Some of this is already in our docs. I can structure and edit the runbooks, but I need someone from the Solo team who has operated a real network to provide the source material for the following:

1. Node failure recovery - we have solo consensus node restart and post-restart health checks documented. What I'm missing is the recovery procedure when a node has actually failed (crashed pod, corrupted state, won't come back with a restart). What's the correct sequence in that case?

2. Rolling upgrade - we have a solo consensus network upgrade (full network) and solo consensus dev-node-update (per-node) documented. What I'm missing is the production rolling sequence: which node first, what health check do you run before moving to the next, and what's the rollback path if an upgrade fails midway?

3. State backup/restore - nothing documented here yet. Is the state-save-and-restore example in the Solo repo the production procedure, or does production require additional steps?

[Reccetech]

I think we can defer this work for now - P2.