Dev/alpha

README.md

@@ -50,17 +50,20 @@ cd NetEngine
 # 2. Install dependencies
 poetry install
 
-# 3. Start local Postgres + pgmq (includes pgmq extension pre-installed)
-docker compose up -d db
+# 3. Verify host prerequisites
+poetry run netengine doctor
 
-# 4. Apply migrations
+# 4. Start local Postgres + pgmq (includes pgmq extension pre-installed)
+docker compose up -d postgres
+
+# 5. Apply migrations
 poetry run python -m netengine.utils.run_migrations
 
-# 5. Boot a minimal world
+# 6. Boot a minimal world
 poetry run netengine up examples/minimal.yaml
 ```
 
-Check status at any time:
+If you only want host/container checks before configuring Postgres, run `poetry run netengine doctor --skip-db`. Check status at any time:
 
 ```bash
 poetry run netengine status
@@ -84,6 +87,60 @@ Worlds are defined in YAML. See `examples/` for reference:
 | `examples/single-org.yaml` | One organisation with residential AND |
 | `examples/dev-sandbox.yaml` | Full dev setup with orgs, ANDs, mail, storage |
 
+### Alpha golden paths
+
+These are the official alpha operator paths. Run them after the Quickstart setup to validate the supported lifecycle.
+
+#### Path A — Minimal smoke world
+
+```bash
+poetry run netengine up examples/minimal.yaml
+poetry run netengine up examples/minimal.yaml
+poetry run netengine status
+poetry run netengine diagnose examples/minimal.yaml
+poetry run netengine down --dry-run
+poetry run netengine down --yes
+```
+
+The second `up` proves idempotency.
+
+#### Path B — Single-org world
+
+Uses `examples/single-org.yaml` to prove org identity, DNS delegation, AND profile basics, and registry records.
+
+```bash
+poetry run netengine up examples/single-org.yaml
+poetry run netengine up examples/single-org.yaml
+poetry run netengine status
+poetry run netengine diagnose examples/single-org.yaml
+poetry run netengine down --dry-run
+poetry run netengine down --yes
+```
+
+#### Path C — Dev sandbox
+
+Uses `examples/dev-sandbox.yaml` as the feature-rich alpha demo. It is more experimental than Paths A/B if some integrations are still stabilizing.
+
+```bash
+poetry run netengine up examples/dev-sandbox.yaml
+poetry run netengine up examples/dev-sandbox.yaml
+poetry run netengine status
+poetry run netengine diagnose examples/dev-sandbox.yaml
+poetry run netengine down --dry-run
+poetry run netengine down --yes
+```
+
+#### Acceptance checklist
+
+- Fresh install works
+- Boot completes
+- Re-running `up` is idempotent
+- `status` is accurate
+- `diagnose` explains failures
+- `reload` rejects immutable changes
+- `down --dry-run` lists resources
+- `down --yes` leaves no project-owned Docker resources behind
+
 ### Spec composition
 
 Large specs can be split across files:
@@ -108,6 +165,27 @@ poetry run netengine up spec.yaml --set metadata.name=my-world
 
 ---
 
+
+## Operator migration guidance
+
+Alpha migrations are forward-only unless a migration file includes explicit manual rollback notes. Inspect applied migrations with:
+
+```bash
+psql "$NETENGINE_DB_URL" -c "SELECT version, dirty FROM schema_migrations;"
+psql "$NETENGINE_DB_URL" -c "SELECT version FROM schema_migrations ORDER BY version DESC LIMIT 1;"
+```
+
+The second command identifies the last applied migration. If a migration fails, stop writers, capture the error and current `schema_migrations` state, inspect partially-created objects, and follow that migration's manual recovery notes before retrying. Without explicit rollback notes, restore from backup or rebuild only if the database is disposable. Wiping and reapplying migrations is acceptable for local/dev/CI alpha databases with no durable data; it is unsafe for shared, persistent, staging, production, or customer-like environments without an approved backup/restore plan.
+
+pgmq queue additions should be treated as forward schema changes and should create both the queue and matching `*_dlq` queue. Prefer migrations over manual changes. If a queue must be created manually during alpha recovery, first inspect existing queues and then create both queues explicitly; remove queues only in disposable environments or after confirming no pending/audit messages are needed:
+
+```bash
+psql "$NETENGINE_DB_URL" -c "SELECT queue_name FROM pgmq.list_queues() ORDER BY queue_name;"
+psql "$NETENGINE_DB_URL" -c "SELECT pgmq.create('new_queue'); SELECT pgmq.create('new_queue_dlq');"
+```
+
+See `docs/runbook.md` for the full rollback and recovery procedure.
+
 ## Architecture
 
 ```
@@ -137,9 +215,15 @@ poetry run netengine up spec.yaml --set metadata.name=my-world
 └─────────────────────────────────────────────────┘
 ```
 
-**Runtime state** is persisted to `netengines_state.json` after each phase so interrupted runs can resume where they left off.
+**Runtime state** is persisted to `netengines_state.json` after each phase so interrupted runs can resume where they left off. The state file carries `schema_version: netengine.runtime_state.v1`; pre-version alpha.1 state files are detected as v1-compatible and stamped on the next save, while unknown future/foreign versions fail closed with instructions to export using the older release or migrate through a compatible release. The file is written atomically with `0600` permissions because it can contain runtime secrets (bootstrap admin password, OIDC client secrets, generated in-world admin passwords, and other phase outputs). Do not commit it, attach it to issues, or copy it into shared logs; use a sanitized support bundle instead.
+
+**Spec compatibility** is tracked in `metadata.schema_version` (default `netengine.spec.v1`). The loader accepts missing schema versions for existing alpha specs, rejects unsupported versions before boot/import, and includes the spec schema in support bundles so alpha.2+ can decide whether an alpha.1 world is safe to restore.
+
+**Support bundles** are produced with `netengine export --out netengine-support-bundle.json` or `GET /api/v1/export`. Bundles include schema metadata, the world spec, phase completion, public CA material, and sanitized phase outputs with secret-looking fields/private PEMs removed. Restore with `netengine import <bundle-file>` or `POST /api/v1/import`; import validates the bundle schema, spec schema compatibility, spec parseability, known phases, phase prerequisites, and required outputs before replacing local runtime state.
+
+**Persistent teardown safety** requires typed confirmation. CLI teardown of a persistent world must pass `netengine down --confirm <world-name>`; the operator API requires both `confirm=true` and `confirmation` equal to the world name. `--yes` is intentionally not enough for persistent destructive operations.
 
-**Events** flow phase-to-phase via pgmq queues: `dns_updates`, `oidc_provisioning`, `and_provisioning`, `inworld_admissions`, `services_admissions`. Each queue has a dead-letter queue (`*_dlq`) for failed messages.
+**Events** flow phase-to-phase via pgmq queues defined by `netengine/events/queues.py::PRIMARY_QUEUES`. There are currently 11 primary queues (`dns_updates`, `oidc_provisioning`, `and_provisioning`, `inworld_admissions`, `services_admissions`, `and_admissions`, `pki_cert_rotation_events`, `drift_events`, `world_health`, `gateway_portal_events`, `phase_events`) plus 11 matching dead-letter queues (`*_dlq`) for failed messages.
 
 ---
 

compose/compose.test-integration.yml

@@ -6,7 +6,7 @@ version: '3.8'
 
 services:
   postgres-integration:
-    image: postgres:15
+    image: ghcr.io/pgmq/pg15-pgmq:latest
     container_name: netengine_postgres_integration
     environment:
       POSTGRES_USER: netengine
@@ -38,13 +38,10 @@ services:
       PGPASSWORD: integration_test_password
     command: |
       sh -c "
-        psql -h postgres-integration -U netengine -d netengine_integration -c 'CREATE EXTENSION IF NOT EXISTS pgmq' &&
-        psql -h postgres-integration -U netengine -d netengine_integration -c 'SELECT pgmq.create_queue(''dns_updates'');' &&
-        psql -h postgres-integration -U netengine -d netengine_integration -c 'SELECT pgmq.create_queue(''oidc_provisioning'');' &&
-        psql -h postgres-integration -U netengine -d netengine_integration -c 'SELECT pgmq.create_queue(''and_provisioning'');' &&
-        psql -h postgres-integration -U netengine -d netengine_integration -c 'SELECT pgmq.create_queue(''inworld_admissions'');' &&
-        psql -h postgres-integration -U netengine -d netengine_integration -c 'SELECT pgmq.create_queue(''services_admissions'');'
+        psql -h postgres-integration -U netengine -d netengine_integration -f /migrations/001_initial.sql
       "
+    volumes:
+      - ../migrations/001_initial.sql:/migrations/001_initial.sql:ro
     networks:
       - netengine-integration
     restart: on-failure