arangodb · bluepal-prasanthi-moparthi · Jul 24, 2026 · Jul 24, 2026 · Jul 24, 2026
@@ -0,0 +1,50 @@
+# Driver resiliency — implementer checklist
+
+One-page handoff for official ArangoDB driver maintainers implementing shared resiliency / Toxiproxy tests. Behavioral contract and steps live elsewhere; this page only points.
+
+## Sources of truth
+
+| Need | Read |
+| --- | --- |
+| **Behavioral requirements & acceptance** | OpenSpec capabilities under `openspec/specs/<capability>/spec.md` |
+| **Scenario steps, timing, error examples** | [`driver-resiliency-reference.md`](./driver-resiliency-reference.md) |
+| **Scenario → capability map** | [Scenario → Capability Mapping](./driver-resiliency-reference.md#scenario--capability-mapping) in the reference |
+| **Harness environment contract** | Reference [Harness env contract](./driver-resiliency-reference.md#harness-env-contract-all-drivers) and overview slides [`driver-k8s-shared-infra-demo.html`](./driver-k8s-shared-infra-demo.html) |
+| **Shared runners** | `deploy/kubernetes/run-driver-tests.sh` (cluster + ingress) and `test/toxiproxy.sh` (proxy) — integrate; do not re-specify kubectl recipes here |
+
+## OpenSpec capabilities (implement these)
+
+| Capability | Covers |
+| --- | --- |
+| `error-classification` | Categories A–F, phase acceptance, C vs D, universal rejects |
+| `load-balancing` | Part A #0 — observational coordinator distribution |
+| `ingress-resiliency` | Part A #1–#3 — failover, idle/active ingress restart |
+| `coordinator-resiliency` | Part A #4–#8 — coordinator restart/kill scenarios |
+| `toxiproxy-connection-interruption` | Part B #1–#3 |
+| `toxiproxy-latency` | Part B #4–#8 |
+| `toxiproxy-packet-loss` | Part B #9–#10 |
+| `toxiproxy-streaming-operations` | Part B #11–#12 |
+| `toxiproxy-write-operations` | Part B #13–#14 |
+
+## Recommended implementation order
+
+1. `error-classification` (shared by all fault scenarios)
+2. Harness env wiring (endpoints, auth, Host header, Toxiproxy admin/listen — per demo HTML / harness table)
+3. `load-balancing`
+4. `ingress-resiliency`
+5. `coordinator-resiliency`
+6. Toxiproxy subsets: connection → latency → packet loss → streaming → writes
+
+## Checklist
+
+- [ ] Honor the harness env contract (read or remap `TEST_*` / `TOXIPROXY_*` names; preserve semantics).
+- [ ] Classify failures by **category A–F**, not by one exact error string (HTTP/1 vs HTTP/2 text may differ).
+- [ ] For each scenario, follow reference steps and assert OpenSpec acceptance criteria for that capability.
+- [ ] Cover HTTP/1 and HTTP/2 when the driver and server support both (skip only where the reference allows).
+- [ ] Assert recovery on the required client after the fault is cleared.
+- [ ] Reject hangs, unexpected process termination, category F during fault windows, and “successful” cursor completion across an intentional kill.
+- [ ] Keep language-specific helpers local; do not copy another driver’s assertion APIs into the shared specs.
+
+## Out of scope for this page
+
+Kubernetes command dumps, Make targets, CI logs, and language-specific helper implementations — use the reference and shared scripts when those details are needed.
@@ -23,6 +23,8 @@
 | **If you want to…**                | **Read…**                                                                                                                     |
 | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
 | **See all scenarios at a glance**  | **[Scenario catalog](#scenario-catalog)**                                                                                     |
+| **Map scenarios to OpenSpec**      | **[Scenario → Capability Mapping](#scenario--capability-mapping)** — which OpenSpec capability owns each scenario             |
+| **Start implementing in another driver** | **[`driver-resiliency-implementer-checklist.md`](./driver-resiliency-implementer-checklist.md)** — one-page OpenSpec + harness pointers |
 | **Know which errors to accept**    | **[Error categories](#error-categories-language-agnostic)**                                                                   |
 | **See which scenario gets A–E**    | **[Which scenarios produce which categories?](#which-scenarios-produce-which-categories)**                                    |
 | **Implement one scenario**         | [Part A — Kubernetes scenarios](#part-a--kubernetes-scenarios) or [Part B — Toxiproxy network faults](#part-b--toxiproxy-network-faults) |
@@ -61,6 +63,26 @@
 
 ---
 
+## **Scenario → Capability Mapping**
+
+Behavioral requirements for these scenarios live in OpenSpec under `openspec/specs/<capability>/spec.md`. This table is a pointer only — it does **not** duplicate requirements, acceptance criteria, or step-level procedures (those remain in the scenario sections below and in the OpenSpec).
+
+| **Part** | **Scenario(s)** | **OpenSpec capability** |
+| -------- | --------------- | ----------------------- |
+| **A #0** | **LoadBalancerCoordinatorDistribution** | `load-balancing` |
+| **A #1–#3** | **IngressCoordinatorFailover**, **IngressRestartWhileIdle**, **IngressRestartDuringActiveWorkload** | `ingress-resiliency` |
+| **A #4–#8** | **CoordinatorRestartWhileIdle**, **CoordinatorRestartDuringActiveWorkload**, **CoordinatorKillDuringRead**, **CoordinatorKillDuringInsert**, **CoordinatorKillDuringCursorIteration** | `coordinator-resiliency` |
+| **B #1–#3** | **AbruptTCPConnectionClose**, **NetworkDisconnect**, **ConnectionResetByPeer** | `toxiproxy-connection-interruption` |
+| **B #4–#8** | **HighLatency**, **ExtremeLatency**, **LatencyRemoved**, **ContextTimeout**, **ServerTimeout** | `toxiproxy-latency` |
+| **B #9–#10** | **PartialPacketLoss**, **FullPacketLoss** | `toxiproxy-packet-loss` |
+| **B #11–#12** | **DisconnectDuringCursorIteration**, **DisconnectDuringQueryExecution** | `toxiproxy-streaming-operations` |
+| **B #13–#14** | **DisconnectDuringInsert**, **DisconnectDuringTransactionCommit** | `toxiproxy-write-operations` |
+| **Cross-cutting** | Error categories A–F, phase acceptance, universal rejects | `error-classification` |
+
+Each Part A/B scenario maps to **exactly one** scenario capability above. `error-classification` is shared by all fault scenarios and is not a substitute for those capabilities.
+
+---
+
 ## **Error categories (language-agnostic)**
 
 **Classify every failure into one of these categories. Your test helpers should accept the category, not a single exact string.**
@@ -143,7 +165,7 @@ Did the client authenticate wrong or send a bad request?
 | **Connection reset**   | `connection reset by peer`**,** `Connection reset`**,** `ECONNRESET`                    |
 | **Unexpected EOF**     | `unexpected EOF`**,** `EOF`**,** `end of stream`**,** `Premature end of Content-Length` |
 | **Broken pipe**        | `broken pipe`**,** `EPIPE`**,** `write: broken pipe`                                    |
-| **Closed connection**  | `use of closed network connection`**,** `Socket closed`                                 |
+| **Closed connection**  | `use of closed network connection`**,** `Socket closed`**,** `http: server closed idle connection` |
 
 
 **Full message example (HTTP/1):**
@@ -409,7 +431,7 @@ go-driver (Docker) → https://arangodb.local → nginx ingress → coordinator
 | **Step** | **Action**                                                                    |
 | -------- | ----------------------------------------------------------------------------- |
 | **1**    | **Record baseline coordinator via** `GET /_admin/status` **(fresh client)**   |
-| **2**    | **Delete 1 random coordinator pod**                                           |
+| **2**    | **Delete 1 random coordinator pod** that maps to a **live** pod (skip stale Health server IDs left after a prior recover) |
 | **3**    | **Retry** `GET /_admin/status` **with fresh clients until a response (90 s)** |
 | **4**    | **Log whether coordinator ID changed (either outcome is valid)**              |
 | **5**    | **Wait for cluster recovery (3 pods, ArangoDeployment ready, ingress OK)**    |
@@ -557,7 +579,7 @@ Other valid during-fault outcomes on different runs:
 | **Category** | **Example**                                                                      | **When**                                                                        |
 | ------------ | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
 | **A**        | `connection refused`                                                             | New connection before new ingress pod listens                                   |
-| **A**        | `connection reset by peer`, `EOF`, `http2: client conn could not be established` | Connection to pod being terminated / dial race while ingress is down            |
+| **A**        | `connection reset by peer`, `EOF`, `http2: client conn could not be established`, `http: server closed idle connection` | Connection to pod being terminated / dial race / keep-alive closed while ingress is down |
 | **B**        | `context deadline exceeded`                                                      | Request waits through 10 s timeout                                              |
 | **C**        | `Code 503`                                                                       | JSON gateway error from proxy/coordinator                                       |
 | **D**        | HTTP **502/503/504** + non-JSON `Content-Type` (e.g. `text/html`)                | Ingress returns non-JSON error page; Go v2 observable: `invalid character '<'…` |
@@ -702,7 +724,7 @@ FOR i IN 1..200 RETURN { i: i, burn: SLEEP(0.25) }
 
 | **Step** | **Action**                                                                                                                                               |
 | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| **1**    | Connect to cluster through ingress; create database and collection                                                                                       |
+| **1**    | Connect to cluster through ingress; create database and collection. After a prior HTTP protocol’s coordinator kill/recover, **CreateCollection** MAY briefly return **database not found** until agency propagation completes — drivers MUST retry that setup error (not treat it as scenario failure). |
 | **2**    | Start streaming AQL cursor: `FOR i IN 1..200 RETURN { i: i, burn: SLEEP(0.25) }` with `batchSize: 1`, `stream: true`                                     |
 | **3**    | Read **1** document successfully                                                                                                                         |
 | **4**    | Delete **all 3** coordinator pods                                                                                                                        |
@@ -2098,28 +2120,20 @@ K8S_INGRESS_ADDRESS=127.0.0.1 make run-k8s-v2-toxiproxy-e2e-tls
 
 # **Implementing in your driver — checklist**
 
+**Start here (cross-driver, one page):** [`driver-resiliency-implementer-checklist.md`](./driver-resiliency-implementer-checklist.md) — OpenSpec capabilities, reference steps, and harness/demo env contract. No language-specific helper dumps.
+
+Operational reminders (detail remains in the reference scenario sections):
+
 1. **Reuse shared infra** — `run-driver-tests.sh` for the cluster; reuse or copy `test/toxiproxy.sh` to start Toxiproxy. Write language-specific admin helpers + error classifiers only.
-2. **Connect through ingress / Toxiproxy** — same endpoints and auth as in [Running tests](#running-tests-go-reference).
+2. **Connect through ingress / Toxiproxy** — same endpoints and auth as in the [Harness env contract](#harness-env-contract-all-drivers).
 3. **Run each scenario** — match the steps in Part A/B exactly (same fault, same API calls, same timing).
-4. **Classify errors by category (A–E)** — do not hard-code one string.
+4. **Classify errors by category (A–F)** — do not hard-code one string.
 5. **Test HTTP/1 and HTTP/2 separately** if your driver supports both.
 6. **Log the full error in test output** — helps compare across drivers.
 7. **Assert recovery** — same client instance works after fault is removed.
 8. **Never accept:** hang past timeout, panic/crash, cursor completing after kill, data corruption.
 
-### **Suggested helper names (map to categories)**
-
-
-| **Category**          | **Suggested helper name**  | **Used in**                                       |
-| --------------------- | -------------------------- | ------------------------------------------------- |
-| **A + B + C + D**     | `isTransientOutageError()` | **Active workload tests (version loop, inserts)** |
-| **A + B + C + D + E** | `isCursorKilledError()`    | **Cursor interrupt + dead cursor + close**        |
-| **A**                 | `isConnectionError()`      | **Toxiproxy connection tests**                    |
-| **B**                 | `isTimeoutError()`         | **Toxiproxy timeout tests** (Go: `isDriverTimeoutError`) |
-| **A or B**            | `isIntermittentNetworkError()` | **Toxiproxy partial packet loss (#9)**        |
-
-
-**Go reference helpers:** `isResiliencyTransientError`, `isCoordinatorKillInterruptedError`, `isDeadCursorError` in `v2/tests/network_fault_error_util_test.go`.
+Language-specific predicate names are an implementation choice per driver. Map them to OpenSpec `error-classification` categories A–F; do not treat another driver’s helper names as normative.
 
 ---
 

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-07-22