diff --git a/docs/postmortem-template.md b/docs/postmortem-template.md new file mode 100644 index 0000000..b8949e6 --- /dev/null +++ b/docs/postmortem-template.md @@ -0,0 +1,139 @@ +# Incident Post-Mortem Template + +Use this template for every production incident that causes user-visible impact or requires an emergency response. Complete all sections within 48 hours of resolution. + +> **How to use:** copy this file to `docs/postmortems/YYYY-MM-DD-.md`, fill in each section, and open a PR for team review. Blameless culture — focus on systems and processes, not individuals. + +--- + +## Incident Summary + +| Field | Value | +|-------|-------| +| **Incident ID** | INC-YYYY-NNN | +| **Date** | YYYY-MM-DD | +| **Severity** | P1 / P2 / P3 | +| **Status** | Resolved / Ongoing | +| **Duration** | X hours Y minutes | +| **Affected component(s)** | e.g. governance contract, token contract, frontend | +| **Incident commander** | @handle | +| **Authors** | @handle, @handle | + +### Description + + + +--- + +## Timeline + +All times in UTC. + +| Time (UTC) | Event | +|------------|-------| +| HH:MM | Incident begins / first alert fires | +| HH:MM | On-call engineer paged | +| HH:MM | Investigation started | +| HH:MM | Root cause identified | +| HH:MM | Mitigation applied (e.g. contract paused) | +| HH:MM | Service restored | +| HH:MM | Incident declared resolved | + +--- + +## Root Cause Analysis + +### What happened? + + + +### Why did it happen? + + + +**5-Whys:** +1. Why? — +2. Why? — +3. Why? — +4. Why? — +5. Why? — + +### Contributing factors + + + +- +- + +--- + +## Impact Assessment + +| Dimension | Details | +|-----------|---------| +| **Users affected** | e.g. all voters during the window, proposers only | +| **Proposals affected** | IDs of any proposals whose state was incorrect or inaccessible | +| **Votes lost / corrupted** | number and addresses if known | +| **Financial impact** | e.g. none / estimated gas costs / token loss | +| **Data integrity** | on-chain state consistent / inconsistent — explain | +| **Regulatory / compliance** | any audit trail gaps | + +--- + +## Action Items + +Each item must have an owner and a due date. + +| # | Action | Owner | Due date | Status | +|---|--------|-------|----------|--------| +| 1 | | @handle | YYYY-MM-DD | Open | +| 2 | | @handle | YYYY-MM-DD | Open | +| 3 | | @handle | YYYY-MM-DD | Open | + +Action item categories to consider: +- **Prevent recurrence** — code fix, config change, additional validation +- **Improve detection** — new alert, monitoring dashboard, event index +- **Improve response** — runbook update, on-call training, escalation path +- **Reduce blast radius** — pause automation, circuit breaker, rate limiting + +--- + +## Lessons Learned + +### What went well? + + + +- +- + +### What went poorly? + + + +- +- + +### Where did we get lucky? + + + +- +- + +--- + +## Appendix + +### Relevant links + +- Stellar Horizon event stream query: +- Contract transaction hash(es): +- GitHub issue / PR: +- Monitoring dashboard snapshot: + +### Raw logs / evidence + +``` +# paste relevant log lines, contract error codes, or event payloads here +``` diff --git a/docs/runbook.md b/docs/runbook.md index 87b2892..c46d265 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -157,3 +157,116 @@ Update `GOVERNANCE_CONTRACT_ID` and `TOKEN_CONTRACT_ID` in your `.env` file afte | `AlreadyInitialized` | 1 | `initialize` called on an already-initialized contract | Deploy a fresh contract instead | For the full error reference see [docs/errors.md](./errors.md). + +--- + +## Deployment Procedure + +Use this section when deploying CosmosVote contracts to testnet or mainnet for the first time, or when upgrading to a new version. + +### Pre-Flight Checks + +Complete every item before running any deployment script. + +- [ ] **Environment variables set** — all required vars are exported: + ```bash + echo "$GOVERNANCE_CONTRACT_ID" "$TOKEN_CONTRACT_ID" "$STELLAR_SECRET_KEY" \ + "$STELLAR_RPC_URL" "$NETWORK_PASSPHRASE" + ``` + None of the above should be blank. +- [ ] **Tests passing** — the full test suite is green: + ```bash + make test + ``` +- [ ] **Backup taken** — export the current contract state (proposal IDs, quorum settings, admin address) and save it off-chain before any upgrade: + ```bash + stellar contract invoke --id "$GOVERNANCE_CONTRACT_ID" \ + --rpc-url "$STELLAR_RPC_URL" \ + --network-passphrase "$NETWORK_PASSPHRASE" \ + -- get_config + ``` +- [ ] **Quorum config verified** — confirm `min_quorum_bps` and per-proposal quorums are correct for this network (testnet vs. mainnet values differ). +- [ ] **Old contract paused** (upgrades only) — pause the existing contract before deploying the replacement (see [Section 1](#1-pause-the-contract)). + +### Step-by-Step Deployment + +1. **Build WASM binaries:** + ```bash + make build + # Artifacts: target/wasm32-unknown-unknown/release/cosmosvote_governance.wasm + # target/wasm32-unknown-unknown/release/cosmosvote_token.wasm + ``` + +2. **Run the deployment script:** + - Testnet / local: + ```bash + bash scripts/deploy.sh + ``` + - Mainnet: + ```bash + bash scripts/deploy_mainnet.sh + ``` + Both scripts print the new `CONTRACT_ID` values on success. + +3. **Record the new contract IDs** returned by the script and update `.env`: + ```bash + GOVERNANCE_CONTRACT_ID= + TOKEN_CONTRACT_ID= + ``` + +4. **Initialize the governance contract** (fresh deploy only — skip for upgrades that reuse an existing instance): + ```bash + stellar contract invoke \ + --id "$GOVERNANCE_CONTRACT_ID" \ + --source "$STELLAR_SECRET_KEY" \ + --rpc-url "$STELLAR_RPC_URL" \ + --network-passphrase "$NETWORK_PASSPHRASE" \ + -- initialize \ + --admin "$(stellar keys address "$STELLAR_SECRET_KEY")" \ + --voting_token "$TOKEN_CONTRACT_ID" \ + --min_proposal_balance 0 \ + --proposal_cooldown 0 \ + --min_quorum_bps \ + --restrict_admin_vote false + ``` + +5. **Unpause** (upgrades only — if old contract was paused in pre-flight): + ```bash + # Unpause the NEW contract if it was deployed in paused state + # (fresh deploys are unpaused by default) + ``` + +6. **Verify WASM build integrity** using the test script: + ```bash + bash scripts/test_wasm.sh + ``` + +### Post-Flight Checks + +Perform these checks immediately after deployment completes. + +- [ ] **Health check** — confirm the contract responds to a read call: + ```bash + stellar contract invoke \ + --id "$GOVERNANCE_CONTRACT_ID" \ + --rpc-url "$STELLAR_RPC_URL" \ + --network-passphrase "$NETWORK_PASSPHRASE" \ + -- get_config + ``` + Expected: returns the config struct without error. +- [ ] **Event emission test** — create a test proposal and verify an `proposal_created` event appears in the Horizon event stream: + ```bash + stellar events --network testnet \ + --contract-id "$GOVERNANCE_CONTRACT_ID" \ + --start-ledger + ``` +- [ ] **Active proposals still accessible** (upgrades only) — query each known active proposal ID and confirm status is still `Active`: + ```bash + stellar contract invoke \ + --id "$GOVERNANCE_CONTRACT_ID" \ + --rpc-url "$STELLAR_RPC_URL" \ + --network-passphrase "$NETWORK_PASSPHRASE" \ + -- get_proposal --proposal_id + ``` +- [ ] **Frontend connected** — update `VITE_GOVERNANCE_CONTRACT_ID` and `VITE_TOKEN_CONTRACT_ID` in `frontend/.env`, restart the dev server, and confirm proposals load. +- [ ] **Notification service updated** — restart the notification service with the new contract ID so it resumes event polling from the correct ledger.