diff --git a/deploy/kubernetes/documentation/driver-resiliency-implementer-checklist.md b/deploy/kubernetes/documentation/driver-resiliency-implementer-checklist.md new file mode 100644 index 00000000..4d06679e --- /dev/null +++ b/deploy/kubernetes/documentation/driver-resiliency-implementer-checklist.md @@ -0,0 +1,50 @@ +# Driver resiliency — implementer checklist + +One-page handoff for official ArangoDB driver maintainers implementing shared resiliency / Toxiproxy tests. Behavioral contract and steps live elsewhere; this page only points. + +## Sources of truth + +| Need | Read | +| --- | --- | +| **Behavioral requirements & acceptance** | OpenSpec capabilities under `openspec/specs//spec.md` | +| **Scenario steps, timing, error examples** | [`driver-resiliency-reference.md`](./driver-resiliency-reference.md) | +| **Scenario → capability map** | [Scenario → Capability Mapping](./driver-resiliency-reference.md#scenario--capability-mapping) in the reference | +| **Harness environment contract** | Reference [Harness env contract](./driver-resiliency-reference.md#harness-env-contract-all-drivers) and overview slides [`driver-k8s-shared-infra-demo.html`](./driver-k8s-shared-infra-demo.html) | +| **Shared runners** | `deploy/kubernetes/run-driver-tests.sh` (cluster + ingress) and `test/toxiproxy.sh` (proxy) — integrate; do not re-specify kubectl recipes here | + +## OpenSpec capabilities (implement these) + +| Capability | Covers | +| --- | --- | +| `error-classification` | Categories A–F, phase acceptance, C vs D, universal rejects | +| `load-balancing` | Part A #0 — observational coordinator distribution | +| `ingress-resiliency` | Part A #1–#3 — failover, idle/active ingress restart | +| `coordinator-resiliency` | Part A #4–#8 — coordinator restart/kill scenarios | +| `toxiproxy-connection-interruption` | Part B #1–#3 | +| `toxiproxy-latency` | Part B #4–#8 | +| `toxiproxy-packet-loss` | Part B #9–#10 | +| `toxiproxy-streaming-operations` | Part B #11–#12 | +| `toxiproxy-write-operations` | Part B #13–#14 | + +## Recommended implementation order + +1. `error-classification` (shared by all fault scenarios) +2. Harness env wiring (endpoints, auth, Host header, Toxiproxy admin/listen — per demo HTML / harness table) +3. `load-balancing` +4. `ingress-resiliency` +5. `coordinator-resiliency` +6. Toxiproxy subsets: connection → latency → packet loss → streaming → writes + +## Checklist + +- [ ] Honor the harness env contract (read or remap `TEST_*` / `TOXIPROXY_*` names; preserve semantics). +- [ ] Classify failures by **category A–F**, not by one exact error string (HTTP/1 vs HTTP/2 text may differ). +- [ ] For each scenario, follow reference steps and assert OpenSpec acceptance criteria for that capability. +- [ ] Cover HTTP/1 and HTTP/2 when the driver and server support both (skip only where the reference allows). +- [ ] Assert recovery on the required client after the fault is cleared. +- [ ] Reject hangs, unexpected process termination, category F during fault windows, and “successful” cursor completion across an intentional kill. +- [ ] Keep language-specific helpers local; do not copy another driver’s assertion APIs into the shared specs. + +## Out of scope for this page + +Kubernetes command dumps, Make targets, CI logs, and language-specific helper implementations — use the reference and shared scripts when those details are needed. diff --git a/deploy/kubernetes/documentation/driver-resiliency-reference.md b/deploy/kubernetes/documentation/driver-resiliency-reference.md index 47dd849b..9b5eb7c1 100644 --- a/deploy/kubernetes/documentation/driver-resiliency-reference.md +++ b/deploy/kubernetes/documentation/driver-resiliency-reference.md @@ -23,6 +23,8 @@ | **If you want to…** | **Read…** | | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | **See all scenarios at a glance** | **[Scenario catalog](#scenario-catalog)** | +| **Map scenarios to OpenSpec** | **[Scenario → Capability Mapping](#scenario--capability-mapping)** — which OpenSpec capability owns each scenario | +| **Start implementing in another driver** | **[`driver-resiliency-implementer-checklist.md`](./driver-resiliency-implementer-checklist.md)** — one-page OpenSpec + harness pointers | | **Know which errors to accept** | **[Error categories](#error-categories-language-agnostic)** | | **See which scenario gets A–E** | **[Which scenarios produce which categories?](#which-scenarios-produce-which-categories)** | | **Implement one scenario** | [Part A — Kubernetes scenarios](#part-a--kubernetes-scenarios) or [Part B — Toxiproxy network faults](#part-b--toxiproxy-network-faults) | @@ -61,6 +63,26 @@ --- +## **Scenario → Capability Mapping** + +Behavioral requirements for these scenarios live in OpenSpec under `openspec/specs//spec.md`. This table is a pointer only — it does **not** duplicate requirements, acceptance criteria, or step-level procedures (those remain in the scenario sections below and in the OpenSpec). + +| **Part** | **Scenario(s)** | **OpenSpec capability** | +| -------- | --------------- | ----------------------- | +| **A #0** | **LoadBalancerCoordinatorDistribution** | `load-balancing` | +| **A #1–#3** | **IngressCoordinatorFailover**, **IngressRestartWhileIdle**, **IngressRestartDuringActiveWorkload** | `ingress-resiliency` | +| **A #4–#8** | **CoordinatorRestartWhileIdle**, **CoordinatorRestartDuringActiveWorkload**, **CoordinatorKillDuringRead**, **CoordinatorKillDuringInsert**, **CoordinatorKillDuringCursorIteration** | `coordinator-resiliency` | +| **B #1–#3** | **AbruptTCPConnectionClose**, **NetworkDisconnect**, **ConnectionResetByPeer** | `toxiproxy-connection-interruption` | +| **B #4–#8** | **HighLatency**, **ExtremeLatency**, **LatencyRemoved**, **ContextTimeout**, **ServerTimeout** | `toxiproxy-latency` | +| **B #9–#10** | **PartialPacketLoss**, **FullPacketLoss** | `toxiproxy-packet-loss` | +| **B #11–#12** | **DisconnectDuringCursorIteration**, **DisconnectDuringQueryExecution** | `toxiproxy-streaming-operations` | +| **B #13–#14** | **DisconnectDuringInsert**, **DisconnectDuringTransactionCommit** | `toxiproxy-write-operations` | +| **Cross-cutting** | Error categories A–F, phase acceptance, universal rejects | `error-classification` | + +Each Part A/B scenario maps to **exactly one** scenario capability above. `error-classification` is shared by all fault scenarios and is not a substitute for those capabilities. + +--- + ## **Error categories (language-agnostic)** **Classify every failure into one of these categories. Your test helpers should accept the category, not a single exact string.** @@ -143,7 +165,7 @@ Did the client authenticate wrong or send a bad request? | **Connection reset** | `connection reset by peer`**,** `Connection reset`**,** `ECONNRESET` | | **Unexpected EOF** | `unexpected EOF`**,** `EOF`**,** `end of stream`**,** `Premature end of Content-Length` | | **Broken pipe** | `broken pipe`**,** `EPIPE`**,** `write: broken pipe` | -| **Closed connection** | `use of closed network connection`**,** `Socket closed` | +| **Closed connection** | `use of closed network connection`**,** `Socket closed`**,** `http: server closed idle connection` | **Full message example (HTTP/1):** @@ -409,7 +431,7 @@ go-driver (Docker) → https://arangodb.local → nginx ingress → coordinator | **Step** | **Action** | | -------- | ----------------------------------------------------------------------------- | | **1** | **Record baseline coordinator via** `GET /_admin/status` **(fresh client)** | -| **2** | **Delete 1 random coordinator pod** | +| **2** | **Delete 1 random coordinator pod** that maps to a **live** pod (skip stale Health server IDs left after a prior recover) | | **3** | **Retry** `GET /_admin/status` **with fresh clients until a response (90 s)** | | **4** | **Log whether coordinator ID changed (either outcome is valid)** | | **5** | **Wait for cluster recovery (3 pods, ArangoDeployment ready, ingress OK)** | @@ -557,7 +579,7 @@ Other valid during-fault outcomes on different runs: | **Category** | **Example** | **When** | | ------------ | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | **A** | `connection refused` | New connection before new ingress pod listens | -| **A** | `connection reset by peer`, `EOF`, `http2: client conn could not be established` | Connection to pod being terminated / dial race while ingress is down | +| **A** | `connection reset by peer`, `EOF`, `http2: client conn could not be established`, `http: server closed idle connection` | Connection to pod being terminated / dial race / keep-alive closed while ingress is down | | **B** | `context deadline exceeded` | Request waits through 10 s timeout | | **C** | `Code 503` | JSON gateway error from proxy/coordinator | | **D** | HTTP **502/503/504** + non-JSON `Content-Type` (e.g. `text/html`) | Ingress returns non-JSON error page; Go v2 observable: `invalid character '<'…` | @@ -702,7 +724,7 @@ FOR i IN 1..200 RETURN { i: i, burn: SLEEP(0.25) } | **Step** | **Action** | | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **1** | Connect to cluster through ingress; create database and collection | +| **1** | Connect to cluster through ingress; create database and collection. After a prior HTTP protocol’s coordinator kill/recover, **CreateCollection** MAY briefly return **database not found** until agency propagation completes — drivers MUST retry that setup error (not treat it as scenario failure). | | **2** | Start streaming AQL cursor: `FOR i IN 1..200 RETURN { i: i, burn: SLEEP(0.25) }` with `batchSize: 1`, `stream: true` | | **3** | Read **1** document successfully | | **4** | Delete **all 3** coordinator pods | @@ -2098,28 +2120,20 @@ K8S_INGRESS_ADDRESS=127.0.0.1 make run-k8s-v2-toxiproxy-e2e-tls # **Implementing in your driver — checklist** +**Start here (cross-driver, one page):** [`driver-resiliency-implementer-checklist.md`](./driver-resiliency-implementer-checklist.md) — OpenSpec capabilities, reference steps, and harness/demo env contract. No language-specific helper dumps. + +Operational reminders (detail remains in the reference scenario sections): + 1. **Reuse shared infra** — `run-driver-tests.sh` for the cluster; reuse or copy `test/toxiproxy.sh` to start Toxiproxy. Write language-specific admin helpers + error classifiers only. -2. **Connect through ingress / Toxiproxy** — same endpoints and auth as in [Running tests](#running-tests-go-reference). +2. **Connect through ingress / Toxiproxy** — same endpoints and auth as in the [Harness env contract](#harness-env-contract-all-drivers). 3. **Run each scenario** — match the steps in Part A/B exactly (same fault, same API calls, same timing). -4. **Classify errors by category (A–E)** — do not hard-code one string. +4. **Classify errors by category (A–F)** — do not hard-code one string. 5. **Test HTTP/1 and HTTP/2 separately** if your driver supports both. 6. **Log the full error in test output** — helps compare across drivers. 7. **Assert recovery** — same client instance works after fault is removed. 8. **Never accept:** hang past timeout, panic/crash, cursor completing after kill, data corruption. -### **Suggested helper names (map to categories)** - - -| **Category** | **Suggested helper name** | **Used in** | -| --------------------- | -------------------------- | ------------------------------------------------- | -| **A + B + C + D** | `isTransientOutageError()` | **Active workload tests (version loop, inserts)** | -| **A + B + C + D + E** | `isCursorKilledError()` | **Cursor interrupt + dead cursor + close** | -| **A** | `isConnectionError()` | **Toxiproxy connection tests** | -| **B** | `isTimeoutError()` | **Toxiproxy timeout tests** (Go: `isDriverTimeoutError`) | -| **A or B** | `isIntermittentNetworkError()` | **Toxiproxy partial packet loss (#9)** | - - -**Go reference helpers:** `isResiliencyTransientError`, `isCoordinatorKillInterruptedError`, `isDeadCursorError` in `v2/tests/network_fault_error_util_test.go`. +Language-specific predicate names are an implementation choice per driver. Map them to OpenSpec `error-classification` categories A–F; do not treat another driver’s helper names as normative. --- diff --git a/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/.openspec.yaml b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/.openspec.yaml new file mode 100644 index 00000000..7250f8fb --- /dev/null +++ b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-07-22 diff --git a/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/design.md b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/design.md new file mode 100644 index 00000000..9b1a93dd --- /dev/null +++ b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/design.md @@ -0,0 +1,102 @@ +## Context + +The Go Driver already implements a comprehensive resiliency test suite (Part A Kubernetes resiliency #0–#8 and Part B Toxiproxy #1–#14 under build tags `resiliency` and `toxiproxy`) using the shared Kubernetes and Toxiproxy test infrastructure — primarily `deploy/kubernetes/run-driver-tests.sh` and `test/toxiproxy.sh`, plus the documented environment contract. + +To enable consistent resiliency testing across all official ArangoDB drivers, a language-independent OpenSpec is needed so other drivers can implement the same behaviors and acceptance criteria without relying on Go-specific implementation details or duplicating Go-specific test logic. Other official drivers are expected to integrate with the same shared Kubernetes and Toxiproxy test infrastructure (including `run-driver-tests.sh`, `test/toxiproxy.sh`, and the documented environment contract) while implementing the behavioral requirements defined by this OpenSpec. + +This design codifies the existing Go Driver resiliency behavior into capability-oriented specifications that serve as the shared behavioral contract for all official drivers. It does not add driver product features. Step-level detail remains in `deploy/kubernetes/documentation/driver-resiliency-reference.md`; harness env naming is also summarized in `deploy/kubernetes/documentation/driver-k8s-shared-infra-demo.html`. + +Stakeholders: Go, Java, JavaScript, Python, and other official driver maintainers; shared k8s harness owners. + +## Goals / Non-Goals + +**Goals:** + +- Codify existing Go Driver resiliency behavior into language-independent requirements, scenarios, expected behavior, and acceptance criteria per capability. +- Provide a shared behavioral contract so other official drivers can implement the same suite against the shared Kubernetes/Toxiproxy harness. +- Split the suite into reviewable capabilities (ingress, LB, coordinators, Toxiproxy fault classes, error classification). +- Point implementers to the reference doc for fault injection steps, timing budgets, and example messages — not to Go helpers or runner internals. +- Require classification by error category (A–F), not exact strings; require HTTP/1 and HTTP/2 coverage where the driver supports both. +- Document the shared harness environment contract that drivers MUST honor when integrating with `run-driver-tests.sh` and `test/toxiproxy.sh`. + +**Non-Goals:** + +- Re-implement or modify the Go Driver resiliency and Toxiproxy test suites as part of this change (Go is the reference implementation; work is alignment verification only). +- Embed Kubernetes commands, Make targets, Docker recipes, Go helper names, or CI log dumps in specs. +- Mandate identical helper APIs or assertion libraries across languages. +- Require Ingress TLS / end-to-end TLS for Part B CI (HTTP path is the default contract; TLS modes remain optional). +- Change shared infrastructure scripts (`run-driver-tests.sh`, `toxiproxy.sh`) unless a gap is found during later apply work. +- Introduce new product functionality or driver APIs. + +## Decisions + +### 1. Capability split over a monolithic resiliency spec + +**Choice:** Nine capabilities as listed in the proposal. + +**Rationale:** Part A and Part B already group by fault domain. Splitting Toxiproxy into connection, latency, packet loss, streaming, and write mirrors how other drivers will implement and CI-gate subsets. Error classification is shared cross-cutting and must not be re-copied into every scenario spec. + +**Alternatives considered:** Single `driver-resiliency` spec (harder to review/partially implement); Part A vs Part B only (too coarse for Toxiproxy). + +### 2. Reference doc remains the step-level source of truth + +**Choice:** Specs state *what* must hold; `driver-resiliency-reference.md` remains authoritative for *how* to inject faults, wait budgets, and observed message patterns. + +**Rationale:** Avoid duplicating and drifting from the long reference. Specs cite scenario names (#0–#8, Toxiproxy #1–#14) that map 1:1 to the reference catalog. + +**Alternatives considered:** Copy all steps into OpenSpec (duplication/drift); replace the reference with OpenSpec only (loses operational detail other teams already use). + +### 3. Language-agnostic norms; Go as reference implementation + +**Choice:** Normative text uses driver-neutral terms (`Version()`, cursor read, document create, transaction commit, HTTP client). Go paths are examples only. + +**Rationale:** Other drivers map APIs differently; acceptance is behavioral (fail cleanly, recover, correct categories). + +### 4. Harness env contract is mandatory wiring, not a separate product capability + +**Choice:** Document env expectations in design + light requirements inside relevant specs / error-classification cross-cutting notes; do not create a tenth “harness” capability. + +**Rationale:** Env vars are integration plumbing shared by all scenarios. Source of truth for names/values: reference harness table + demo HTML. + + +| Concern | Default env (drivers MUST read or remap) | +| ------------------- | ---------------------------------------------------------------------------------------------------------------------- | +| Connection URL | `TEST_ENDPOINTS_OVERRIDE` (Part A: `http(s)://arangodb.local`; Part B: `http(s)://127.0.0.1:17001`) | +| Auth | `TEST_AUTHENTICATION_OVERRIDE` / `TEST_AUTHENTICATION` — `basic::` | +| Ingress Host header | `TEST_INGRESS_HOST` (required for Toxiproxy IP URLs → `arangodb.local`) | +| Docker net / mounts | `TEST_NET_OVERRIDE`, `K8S_TEST_DOCKER_EXTRA_ARGS` | +| Toxiproxy | `TOXIPROXY_LISTEN_PORT` / `TOXIPROXY_ADMIN_PORT`, `TOXIPROXY_UPSTREAM` / `TOXIPROXY_PROXY_NAME`, `TEST_TOXIPROXY_ADMIN`, `TEST_TOXIPROXY_PROXY` | + + +Drivers MAY map these environment variables through `K8S_TEST_*_ENV` or equivalent wrapper scripts, provided the semantics remain unchanged. + +### 5. Category-based acceptance; zero during-fault failures allowed where documented + +**Choice:** Accept any error in the allowed category set for the phase. For coordinator restart during active workload and coordinator kill during insert, `failuresDuring = 0` is a valid pass when recovery succeeds. + +**Rationale:** Matches reference and real CI timing variance. Exact string matching is forbidden as the sole acceptance rule. + +### 6. Universal reject conditions + +**Choice:** Across all capabilities, tests MUST fail on: process panic/crash, hang past the scenario timeout, category F during a fault window, failure to recover after the fault condition has been removed (where recovery is required), and cursor “success” completing as if no kill occurred when interrupt is required. + +## Risks / Trade-offs + +- **[Drift between OpenSpec and reference]** → Specs cite scenario IDs/names; update both in the same PR when behavior changes; prefer shortening specs over copying tables. +- **[Over-constraining other drivers]** → Keep APIs abstract; allow language-idiomatic timeouts/clients as long as categories and recovery hold. +- **[HTTP/1 vs HTTP/2 message differences]** → Classify by category; document that different text for the same fault is expected. +- **[Partial suite adoption]** → Capability split allows phased implementation; error-classification SHOULD land early because other capabilities depend on it. +- **[Go-only observability gaps (e.g. category D via decode error)]** → Specs prefer status + Content-Type when exposed; allow equivalent observable symptoms when the driver cannot surface headers. + +## Migration Plan + +1. Land this OpenSpec change in the Go Driver repository as the authoritative specification for cross-driver resiliency behavior. +2. Archive into `openspec/specs/` when approved (no code migration required for Go if already compliant — it remains the reference implementation on `run-driver-tests.sh` / `toxiproxy.sh`). +3. Other official driver repositories consume the archived specifications together with the reference documentation, integrate with the shared test infrastructure, and implement only language-specific wrappers where required. +4. Rollback: revert/archive-revert of OpenSpec artifacts only; no cluster or product rollback. + +## Open Questions + +- Whether other driver repos will vendor these specs, link to this repo, or mirror after archive (process, not behavior). +- Whether optional TLS Toxiproxy modes should later become a separate capability or remain “optional modes” of Part B (currently optional / non-blocking for the contract). + diff --git a/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/proposal.md b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/proposal.md new file mode 100644 index 00000000..f0120386 --- /dev/null +++ b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/proposal.md @@ -0,0 +1,47 @@ +## Why + +The Go Driver already implements a comprehensive resiliency test suite using the shared Kubernetes and Toxiproxy test infrastructure. + +To enable consistent resiliency testing across all official ArangoDB drivers, a language-independent OpenSpec is needed so other drivers can implement the same behaviors and acceptance criteria without relying on Go-specific implementation details or duplicating Go-specific test logic. + +This proposal codifies the existing Go Driver resiliency behavior into capability-oriented specifications that serve as the shared behavioral contract for all official ArangoDB drivers. + +## What Changes + +- Introduce OpenSpec capabilities that standardize resiliency and Toxiproxy network-fault **behaviors** across official ArangoDB drivers (Go, Java, JavaScript, Python, and others). +- Organize requirements by logical capability (ingress, load balancing, coordinators, Toxiproxy fault classes, error classification) rather than one monolithic spec. +- Treat `deploy/kubernetes/documentation/driver-resiliency-reference.md` as the authoritative step-level reference; specs state requirements, scenarios, expected behavior, and acceptance criteria only. +- Treat the shared harness environment contract (documented in +`deploy/kubernetes/documentation/driver-k8s-shared-infra-demo.html` and +`deploy/kubernetes/documentation/driver-resiliency-reference.md`) as the interface that other drivers integrate with. +- The OpenSpec defines behavioral requirements only and does not duplicate Kubernetes commands, runner scripts, or implementation details. +- Use the Go Driver v2 suite (`resiliency` / `toxiproxy` build tags) as the reference implementation while keeping specification language driver-agnostic. +- **No new product functionality** — this change documents and standardizes existing test behaviors; it does not add driver APIs or cluster features. + +## Capabilities + +### New Capabilities + +- `ingress-resiliency`: Behavior when ingress fails over or restarts (idle vs active workload), including recovery on the same or fresh clients. +- `load-balancing`: Observational expectations for coordinator distribution through ingress (no fault injection). +- `coordinator-resiliency`: Behavior when coordinators restart or are killed during idle, active version probes, cursor reads, inserts, and cursor iteration. +- `toxiproxy-connection-interruption`: Abrupt TCP close, proxy disable, and connection-reset faults with fail-then-recover expectations. +- `toxiproxy-latency`: High/extreme latency, latency removal, client context timeout, and server/header timeout behaviors. +- `toxiproxy-packet-loss`: Partial and full packet-loss faults and acceptable success/failure mixes. +- `toxiproxy-streaming-operations`: Disconnect during cursor iteration and during query startup. +- `toxiproxy-write-operations`: Disconnect during document insert and transaction commit (outcome unknown). +- `error-classification`: Language-agnostic categories A–F, phase-based acceptance, HTTP/1 vs HTTP/2 classification rules, and reject conditions (panic, hang, category F during faults). + +### Modified Capabilities + +- (none — no existing specs under `openspec/specs/`) + +## Impact + +- **Specs / process:** New capability specs under this change; after archive, main specs become the shared contract for all official drivers. +- **Go driver:** Reference implementation already exists; impact is alignment checks and any gaps vs the standardized acceptance rules (no intentional feature work). +- **Other drivers**: Implement the same scenarios against the shared Kubernetes/Toxiproxy test harness using the reference documentation for execution steps and the OpenSpec for behavioral requirements and acceptance criteria. +- **Shared infrastructure:** The existing shared Kubernetes and Toxiproxy test infrastructure (including `run-driver-tests.sh`, `test/toxiproxy.sh`, and the documented environment contract) serves as the common execution environment for all official drivers. The Go Driver already integrates with this infrastructure and acts as the reference implementation. Other drivers are expected to integrate with the same infrastructure while implementing the OpenSpec requirements. +- **Out of scope for this change:** Duplicating kubectl/Make recipes, Go helper code, CI log dumps, or language-specific assertion APIs inside the specs. +- This proposal standardizes existing behavior and serves as the basis for cross-driver implementation parity rather than introducing new functionality. + diff --git a/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/coordinator-resiliency/spec.md b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/coordinator-resiliency/spec.md new file mode 100644 index 00000000..40bd1beb --- /dev/null +++ b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/coordinator-resiliency/spec.md @@ -0,0 +1,81 @@ +## ADDED Requirements + +### Requirement: Coordinator restart while idle recovers on the same client + +The driver SHALL successfully perform a version operation after all coordinator pods are deleted and recreated, with no client requests issued during the outage. Steps MUST follow reference scenario **CoordinatorRestartWhileIdle** (Part A #4) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A version operation succeeds before the fault. No client requests are issued while coordinators are being recreated. Once all coordinators are ready, the same client successfully performs the version operation without requiring reinitialization. + +**Acceptance criteria:** The post-recovery version operation succeeds within the reference coordinator-ready budget using the same client. No hang or unexpected driver termination occurs. No client errors are expected during the outage because no requests are issued while coordinators are being recreated. + +#### Scenario: Idle client survives full coordinator recreation + +- **WHEN** all coordinator pods are deleted while the client is idle +- **THEN** version on the same client succeeds + +### Requirement: Coordinator restart during active workload recovers without requiring during-fault failures + +The driver SHALL keep a version loop alive while all coordinator pods are deleted and recreated. Recovery is mandatory; during-fault failures are optional. Steps MUST follow **CoordinatorRestartDuringActiveWorkload** (Part A #5). + +**Expected behavior:** Single client; `GET /_api/version` every ~100 ms with a bounded per-request timeout. Track successes/failures before, during, and after the fault. Recovery successes are counted only after coordinator readiness has been re-established. + +**Acceptance criteria:** `successesBefore ≥ 1`; `successesAfter ≥ 1`; baseline and recovery quality rules defined in the reference MUST hold; failuresDuring MAY be 0. If `failuresDuring > 0`, every observed error MUST belong to category A, B, C, or D. The workload MUST terminate cleanly after cancellation, with no hang or unexpected process termination, and category F MUST never occur during the fault window. + +#### Scenario: Recovery with zero during-fault failures + +- **WHEN** the version loop runs across full coordinator recreation and no request overlaps downtime +- **THEN** the scenario passes if recovery successes are recorded and other pass criteria hold + +#### Scenario: Recovery with transient during-fault failures + +- **WHEN** one or more version requests fail during the coordinator outage +- **THEN** each during-fault error is category A–D and recovery successes still occur + +### Requirement: Coordinator kill during cursor read interrupts and marks the cursor unusable + +The driver SHALL observe a clean interrupt when coordinators are killed during an active cursor read, then treat resume/close of that cursor according to dead-cursor rules. Steps MUST follow **CoordinatorKillDuringRead** (Part A #6). + +**Expected behavior:** The streaming cursor is interrupted after the coordinator kill. Interrupt-phase errors MUST belong to categories A–E as defined for that phase. After recovery, attempting to resume the same cursor accepts only categories A or E; gateway errors (502/503/504) are not valid dead-cursor resume outcomes. Closing the dead cursor primarily produces category E, although categories A–D are also accepted as defined in the reference. + +**Acceptance criteria:** The cursor MUST NOT complete successfully as though the coordinator interruption had not occurred. Interrupt and dead-cursor phases match allowed categories; cluster recovers for subsequent healthy operations; no hang or unexpected process termination. + +#### Scenario: Cursor interrupt on coordinator kill + +- **WHEN** all coordinators are deleted during an open cursor read +- **THEN** the next cursor operation fails with an allowed interrupt category (A–E) and does not pretend the query finished successfully + +#### Scenario: Dead cursor after recovery + +- **WHEN** the driver resumes or closes the same cursor after cluster recovery +- **THEN** errors match the dead-cursor acceptance set for that phase in the reference (see `error-classification`) + +### Requirement: Coordinator kill during insert recovers with optional during-fault failures + +The driver SHALL run an insert loop while one coordinator is deleted and SHALL recover afterward. Steps MUST follow **CoordinatorKillDuringInsert** (Part A #7). + +**Expected behavior:** An insert workload remains active throughout the fault window. `failuresDuring` MAY be `0`. If during-fault errors occur, they MUST belong to categories A–D. The outcome of an insert interrupted by the coordinator failure MAY be unknown; therefore the suite validates clean failure classification and post-recovery progress rather than the commit outcome of an individual interrupted write. + +**Acceptance criteria:** At least one successful insert (or equivalent post-recovery health check as defined in the reference) MUST occur after recovery. During-fault errors, if any, MUST belong only to categories A–D. The workload MUST terminate cleanly after cancellation with no hang or unexpected process termination, and category F MUST never occur during the fault window. + +#### Scenario: Insert loop survives coordinator kill + +- **WHEN** inserts run while one coordinator pod is deleted +- **THEN** the driver recovers after the fault, and any during-fault errors are only categories A–D + +### Requirement: Coordinator kill during cursor iteration interrupts mid-stream + +The driver SHALL interrupt cursor iteration after a defined number of documents when all coordinators are killed. Steps MUST follow **CoordinatorKillDuringCursorIteration** (Part A #8). + +**Expected behavior:** After reading the reference threshold of documents, all coordinator pods are deleted. The active cursor iteration is interrupted with an allowed interrupt category. After the cluster recovers, resuming or closing the same cursor follows the dead-cursor acceptance rules defined in `error-classification`. Differences in protocol-specific error messages or response bodies (for example between HTTP/1 and HTTP/2) are evaluated by error category rather than exact message text. + +**Acceptance criteria:** Cursor iteration MUST NOT complete successfully across the coordinator kill. Interrupt and dead-cursor phases MUST satisfy the allowed error categories defined in `error-classification`. The cluster MUST recover for subsequent healthy operations. The test MUST fail on hangs, unexpected process termination, or category F during the fault window. + +#### Scenario: Mid-iteration interrupt + +- **WHEN** coordinators are killed after partial cursor consumption +- **THEN** the next iteration step fails with an allowed category and the cursor is not treated as successfully exhausted + +#### Scenario: Dead cursor remains unusable after recovery + +- **WHEN** the cluster has recovered after the coordinator kill +- **THEN** resuming or closing the same cursor follows the dead-cursor acceptance rules defined in `error-classification` \ No newline at end of file diff --git a/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/error-classification/spec.md b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/error-classification/spec.md new file mode 100644 index 00000000..c0d79ff5 --- /dev/null +++ b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/error-classification/spec.md @@ -0,0 +1,93 @@ +## ADDED Requirements + +### Requirement: Failures are classified into language-agnostic categories A–F + +Every official driver resiliency/Toxiproxy suite SHALL classify observed failures into these categories (definitions are normative and correspond to the categories described in `driver-resiliency-reference.md`): + + +| Category | Meaning | +| -------- | ------------------------------------------------------------------------------------------- | +| **A** | Connection lost — TCP/HTTP transport broken before a valid HTTP response | +| **B** | Timeout — client or transport gave up waiting | +| **C** | Gateway HTTP error — HTTP 502/503/504 with JSON body | +| **D** | HTTP error status with a non-JSON body and/or non-JSON `Content-Type` (typical 502/503/504) | +| **E** | Cursor gone — HTTP 404/409/410 with JSON body on cursor APIs | +| **F** | Application error — auth, validation, or normal API not-found | + + +**Expected behavior:** Tests classify failures using categories (or equivalent language-specific predicates), rather than relying on a single exact error string. HTTP/1 and HTTP/2 MAY produce different message text for the same fault; category match is sufficient. + +**Acceptance criteria:** Tests assert allowed categories per scenario phase; exact-string-only matching is insufficient as the sole rule. + +#### Scenario: Same fault different protocol text + +- **WHEN** the same network fault is observed under HTTP/1 and HTTP/2 with different error strings +- **THEN** both are accepted if they map to the same allowed category for that phase + +#### Scenario: Category F never accepted during fault windows + +- **WHEN** an error during an injected fault window is category F +- **THEN** the test MUST fail + +### Requirement: Acceptance is phase- and scenario-specific + +Drivers SHALL apply the master scenario→category mapping from the reference (Kubernetes Part A and Toxiproxy Part B). Other capability specifications reference the allowed category sets defined here; this capability is the normative classification specification. + +**Summary of expected behavior by scenario:** + +- **Active ingress/coordinator outages:** Categories A–D are typically accepted during the fault window. +- **Cursor interruption:** Interrupt-phase errors accept categories A–E. +- **Dead-cursor resume:** Only categories A or E are accepted; gateway errors (502/503/504) are not valid dead-cursor resume outcomes. +- **Dead-cursor close:** Category E is the primary expected outcome, although categories A–D are also acceptable. +- **Idle restart scenarios:** No failures are expected during the fault window. +- **Coordinator restart and insert scenarios:** These scenarios MAY complete without any observed failures during the injected fault. When no failures occur, the scenario still passes provided all required recovery criteria are satisfied. If failures do occur, they MUST belong to the categories defined for that scenario. +- **Toxiproxy transport faults:** Connection interruption, streaming interruption, and write interruption primarily produce category A. Latency, timeout, and full packet-loss scenarios primarily produce category B. Partial packet loss MAY produce either category A or B. Categories C–E are not expected for pure transport faults. + +**Acceptance criteria:** Each automated scenario documents which categories are accepted at which checkpoint; violations fail the test. + +#### Scenario: Cursor interrupt vs dead cursor resume + +- **WHEN** coordinators are killed during cursor use +- **THEN** interrupt-phase errors accept A–E as listed for that scenario, and resume-phase errors follow the dead-cursor set (A or E), not unrestricted gateway success + +#### Scenario: Optional during-fault failures + +- **WHEN** no failures are observed during an injected fault in a scenario where this is permitted +- **THEN** the scenario passes if all required recovery criteria are satisfied + +### Requirement: Prefer status and Content-Type when distinguishing C vs D + +When the driver exposes HTTP status and `Content-Type`, category **C** MUST be used for 502/503/504 with JSON bodies, and category **D** for those statuses with non-JSON body and/or non-JSON `Content-Type`. If the driver cannot expose HTTP status or Content-Type and instead surfaces an equivalent decode or transport symptom, that observable MAY be classified as category D as described in the reference documentation. + +**Expected behavior:** Classification prefers protocol metadata over substring matching of HTML. + +**Acceptance criteria:** C and D are not conflated when headers/status are available; when status/`Content-Type` are not exposed, an equivalent decode/transport symptom of a non-JSON (e.g. HTML) body — such as a JSON parser rejecting a leading `'<'` — remains an allowed **D** observable. + +#### Scenario: JSON gateway error is category C + +- **WHEN** the fault response is HTTP 503 with `Content-Type: application/json` +- **THEN** the error is classified as category C + +#### Scenario: HTML gateway error is category D + +- **WHEN** the fault response is HTTP 502/503 with non-JSON `Content-Type` or non-JSON body +- **THEN** the error is classified as category D + +### Requirement: Universal reject conditions across resiliency suites + +All resiliency and Toxiproxy scenarios SHALL fail the test when any of the following occur: unexpected driver or test process termination; hang past the scenario/test timeout; missing required recovery after the fault is cleared; treating an interrupted cursor/query as a successful full completion; category F during a fault window. + +**Expected behavior:** Failures are clean, classified, and bounded in time; recovery uses the same client where the reference requires it. + +**Acceptance criteria:** CI marks the scenario failed on any universal reject condition. + +#### Scenario: Hang is a failure + +- **WHEN** a request or workload does not return within the scenario timeout under an injected fault +- **THEN** the test fails (hang), even if no wrong category was observed + +#### Scenario: Recovery required after clear + +- **WHEN** a scenario requires post-fault recovery and the fault has been cleared +- **THEN** a subsequent operation on the required client MUST succeed within the reference budget + diff --git a/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/ingress-resiliency/spec.md b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/ingress-resiliency/spec.md new file mode 100644 index 00000000..6f137d1b --- /dev/null +++ b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/ingress-resiliency/spec.md @@ -0,0 +1,51 @@ +## ADDED Requirements + +### Requirement: Ingress coordinator failover recovers through ingress + +The driver SHALL continue to reach the cluster through ingress after one coordinator pod is deleted. Implementation steps and wait budgets MUST follow reference scenario **IngressCoordinatorFailover** (Part A #1) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A baseline `GET /_admin/status` probe succeeds. After deleting one coordinator pod, the driver retries the probe using fresh clients until a coordinator responds successfully. After operator recovery, the original coordinator count is restored and a final probe succeeds. Whether the responding coordinator ID changes is observational only; either outcome is valid. + +**Acceptance criteria:** A probe succeeds within the reference failover budget after the coordinator kill. After operator recovery, the healthy coordinator count matches the pre-fault count and a final probe succeeds. The scenario MUST complete without hangs or unexpected process termination. Transient probe errors observed during the failover window, if any, MUST belong only to categories A–D (see `error-classification`). + +#### Scenario: Failover then full recovery + +- **WHEN** the driver repeatedly probes `GET /_admin/status` through ingress, one coordinator pod is deleted, and probing continues until success +- **THEN** a probe succeeds after the kill, the deployment recovers to the original coordinator count, and a post-recovery probe succeeds + +#### Scenario: HTTP/1 and HTTP/2 coverage + +- **WHEN** the driver supports both HTTP/1 and HTTP/2 +- **THEN** the failover scenario SHALL be executed for both protocols (skipping HTTP/2 only when unsupported by the server version, as defined in the reference). + +### Requirement: Ingress restart while idle recovers on the same client + +The driver SHALL successfully perform a version operation on the same client instance after an ingress-nginx restart, while no client requests are issued during the outage. Steps MUST follow reference scenario **IngressRestartWhileIdle** (Part A #2). + +**Expected behavior:** A version operation succeeds before the restart. No client requests are issued while ingress is unavailable. After ingress becomes ready again, the same client successfully performs the version operation without requiring reinitialization. + +**Acceptance criteria:** Version operations succeed before and after the ingress restart using the same client. The scenario MUST complete without hangs or unexpected process termination. Category F MUST NOT occur. No during-fault errors are expected because no client requests are issued while ingress is restarting. + +#### Scenario: Idle client survives ingress restart + +- **WHEN** version succeeds, ingress is restarted with no driver traffic during the outage, and ingress becomes ready +- **THEN** version on the same client succeeds after recovery + +### Requirement: Ingress restart during active workload tolerates transient errors and recovers + +The driver SHALL keep a version-request loop alive while ingress-nginx is restarted. Recovery is mandatory; transient failures during the fault window are permitted. Steps MUST follow **IngressRestartDuringActiveWorkload** (Part A #3). + +**Expected behavior:** A single client continuously performs `GET /_api/version` approximately every 100 ms with a bounded per-request timeout. Successes and failures are tracked before, during, and after the ingress restart. Recovery successes are counted only after ingress has become ready again. + +**Acceptance criteria:** `successesBefore ≥ 1`; `successesAfter ≥ 1`; baseline and recovery quality rules defined in the reference MUST hold. `failuresDuring` MAY be `0`. If `failuresDuring > 0`, every observed error MUST belong to category A, B, C, or D. The workload MUST terminate cleanly after cancellation, with no hang or unexpected process termination, and category F MUST never occur during the fault window. + +#### Scenario: Recovery with zero during-fault failures + +- **WHEN** the version loop runs across an ingress-nginx restart and no request overlaps the outage +- **THEN** the scenario passes if recovery successes are recorded and the other acceptance criteria hold + +#### Scenario: Recovery with transient during-fault failures + +- **WHEN** one or more version requests fail during the ingress-nginx restart +- **THEN** each during-fault error belongs to category A, B, C, or D and recovery successes still occur + diff --git a/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/load-balancing/spec.md b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/load-balancing/spec.md new file mode 100644 index 00000000..2ca7bc2c --- /dev/null +++ b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/load-balancing/spec.md @@ -0,0 +1,32 @@ +## ADDED Requirements + +### Requirement: Load balancer coordinator distribution is observable through ingress + +The driver SHALL probe coordinator identity through ingress with no fault injected, recording which coordinator answered each request. Steps MUST follow reference scenario **LoadBalancerCoordinatorDistribution** (Part A #0). + +**Expected behavior:** Repeated `GET /_admin/status` probes succeed and expose `serverInfo.serverId` (or equivalent). Requests may be served by a single coordinator or distributed across multiple coordinators. The distribution is observational and is not required to follow any specific balancing pattern. + +**Acceptance criteria:** Every probe succeeds. The responding coordinator ID for each successful probe is recorded. The scenario MUST NOT fail solely because only one coordinator is observed. No error-category validation applies because no fault is injected. + +#### Scenario: Probes succeed and record coordinator IDs + +- **WHEN** the driver issues status probes through ingress using the shared and fresh HTTP/1 and HTTP/2 client modes defined by the reference +- **THEN** every probe succeeds and the responding coordinator ID is recorded + +#### Scenario: Coordinator distribution is observational + +- **WHEN** probes are answered by one or more distinct coordinator IDs +- **THEN** the scenario still passes because ingress routing behavior is implementation-dependent + +### Requirement: Load-balancing baseline uses the shared resiliency harness endpoints + +Drivers SHALL connect using the Part A harness endpoint and auth contract (`TEST_ENDPOINTS_OVERRIDE`, `TEST_AUTHENTICATION_OVERRIDE` / `TEST_AUTHENTICATION`) as documented in the reference and `deploy/kubernetes/documentation/driver-k8s-shared-infra-demo.html`, remapping names only if semantics are preserved. + +**Expected behavior:** Traffic path is driver → ingress (`arangodb.local`) → coordinators. ClientIP session affinity on the coordinator Service does not apply on this out-of-cluster path. + +**Acceptance criteria:** Probes use the harness-exported URL and auth; Host header behavior matches the reference when required by the URL form. + +#### Scenario: Harness endpoint wiring + +- **WHEN** the load-balancing scenario runs under the shared Kubernetes driver-test runner +- **THEN** the driver uses the exported endpoints/auth (or an equivalent remap) to reach the cluster through ingress diff --git a/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/toxiproxy-connection-interruption/spec.md b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/toxiproxy-connection-interruption/spec.md new file mode 100644 index 00000000..7b818cff --- /dev/null +++ b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/toxiproxy-connection-interruption/spec.md @@ -0,0 +1,54 @@ +## ADDED Requirements + +### Requirement: Abrupt TCP close on a live connection fails then recovers + +The driver SHALL fail a version request with category A when Toxiproxy injects an upstream `reset_peer` toxic on an established connection, then SHALL succeed on the same client after the toxic is removed. Steps MUST follow Toxiproxy **AbruptTCPConnectionClose** (#1) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** Baseline version succeeds (keep-alive established); after `reset_peer` upstream, version fails quickly with a connection-loss error; after toxic removal, version succeeds again. + +**Acceptance criteria:** Fault error is category A; recovery succeeds on the same client; fail-fast (no hang until the safety deadline); no unexpected process termination; category F rejected. Run HTTP/1 and HTTP/2 when supported. + +#### Scenario: Reset peer then recover + +- **WHEN** `reset_peer` is applied upstream on the Toxiproxy proxy and version is called +- **THEN** the call fails with category A and a later version call succeeds after the toxic is removed + +### Requirement: Network disconnect via disabled proxy fails then recovers + +The driver SHALL fail a version request with **category A** when the Toxiproxy proxy is disabled, then SHALL successfully perform a version operation on the same client after the proxy is re-enabled. Steps MUST follow **NetworkDisconnect** (#2) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A baseline version operation succeeds. While the proxy is disabled, a version request fails with a transport-level connection error (category A). After the proxy is re-enabled, the same client successfully performs a version operation again. + +**Acceptance criteria:** The during-fault error MUST be category A. Recovery on the same client MUST succeed after the proxy is re-enabled. The operation MUST NOT hang or unexpectedly terminate. Category F MUST NOT occur. + +#### Scenario: Proxy disable then recover + +- **WHEN** the Toxiproxy proxy is disabled during a version call +- **THEN** the call fails with category A and succeeds after the proxy is re-enabled + +### Requirement: Downstream connection reset fails then recovers + +The driver SHALL fail a version request with category A when Toxiproxy applies a `reset_peer` toxic on the downstream path, then SHALL successfully perform a version operation on the same client after the toxic is removed. Steps MUST follow `ConnectionResetByPeer` (#3) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A baseline version operation succeeds. While a downstream `reset_peer` toxic is active, a version request fails with a category A transport error. HTTP/1 and HTTP/2 MAY surface different messages (for example `EOF` or `unexpected EOF`), but both classify as category A. After the toxic is removed, the same client successfully performs a version operation again. + +**Acceptance criteria:** The during-fault error MUST be category A. Recovery on the same client MUST succeed after the toxic is removed. The operation MUST NOT hang or unexpectedly terminate. Category F MUST NOT occur. + +#### Scenario: Downstream reset then recover + +- **WHEN** a downstream `reset_peer` toxic is active and a version request is issued +- **THEN** the request fails with category A, and a later version request succeeds after the toxic is removed + +### Requirement: Toxiproxy tests use the shared harness proxy contract + +Drivers SHALL send traffic to the Toxiproxy listen endpoint from the harness (`TEST_ENDPOINTS_OVERRIDE` → listen port) and MUST send `Host: TEST_INGRESS_HOST` (`arangodb.local`) on every request. Admin control uses `TEST_TOXIPROXY_ADMIN` / `TEST_TOXIPROXY_PROXY` (see demo HTML and reference harness table). + +**Expected behavior:** Driver → Toxiproxy listen → ingress → coordinator. Without the Host header, ingress will not route correctly. + +**Acceptance criteria:** Connection scenarios only pass when wired through the proxy with correct Host header; faults are injected via the admin API, not by changing the driver URL mid-test except as the reference describes. + +#### Scenario: Host header required through Toxiproxy + +- **WHEN** the driver connects via the Toxiproxy listen address for connection-interruption tests +- **THEN** every request includes the ingress Host header from `TEST_INGRESS_HOST` + diff --git a/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/toxiproxy-latency/spec.md b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/toxiproxy-latency/spec.md new file mode 100644 index 00000000..2b7eecab --- /dev/null +++ b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/toxiproxy-latency/spec.md @@ -0,0 +1,67 @@ +## ADDED Requirements + +### Requirement: High latency completes successfully but slower + +The driver SHALL successfully perform a version operation when a moderate Toxiproxy latency toxic (reference: 2000 ms) is applied, provided the client timeout exceeds the injected latency. Steps MUST follow `HighLatency` (#4) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A baseline version operation completes successfully. After applying a 2000 ms latency toxic, a version operation also succeeds using a client timeout greater than the injected latency. The observed elapsed time SHOULD reflect approximately the additional network latency. + +**Acceptance criteria:** The version operation succeeds under the injected latency. The observed duration is measurably greater than the baseline and consistent with the injected delay. The operation MUST NOT hang beyond the scenario timeout or unexpected process termination. No error-category validation applies because successful completion is expected. The scenario SHALL run for HTTP/1 and HTTP/2 when supported. + +#### Scenario: Version succeeds under high latency + +- **WHEN** a 2000 ms latency toxic is applied and a version request is issued with a sufficient client timeout +- **THEN** the request succeeds and completes more slowly than the baseline + +### Requirement: Extreme latency surfaces a client timeout + +The driver SHALL fail a version request with category B when injected latency exceeds the client timeout, then SHALL successfully perform a version operation on the same client after the latency toxic is removed. Steps MUST follow `ExtremeLatency` (#5) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A version operation fails when latency (reference: 30 s) exceeds the client timeout (reference: 10 s). The error is category B. After the latency toxic is removed, a subsequent version operation on the same client succeeds. + +**Acceptance criteria:** During-fault failure is category B; recovery succeeds after the toxic is removed; the operation MUST NOT hang or unexpectedly terminate; category F MUST NOT occur. + +#### Scenario: Extreme latency times out then recovers + +- **WHEN** extreme latency is applied with a shorter client timeout +- **THEN** version fails with category B and succeeds after the latency toxic is removed + +### Requirement: Removing latency restores faster responses + +The driver SHALL successfully perform a version operation after a Toxiproxy latency toxic is removed, with the observed response time improving relative to requests made while the toxic was active. Steps MUST follow `LatencyRemoved` (#6) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** Latency toxic applied then removed; subsequent version succeeds without the prior delay penalty. + +**Acceptance criteria:** Post-removal version succeeds; timing improves per reference checkpoints; no hang or unexpected process termination. + +#### Scenario: Latency removed + +- **WHEN** a latency toxic is removed after a slowed successful call +- **THEN** a subsequent version request succeeds without the injected latency effect + +### Requirement: Context timeout fails quickly under high latency + +The driver SHALL fail a version operation with category B when a client/context timeout is shorter than an injected Toxiproxy latency. Steps MUST follow **ContextTimeout** (#7) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A latency toxic greater than the client/context timeout is applied. The version operation fails because the client deadline expires rather than waiting for the full injected latency. + +**Acceptance criteria:** The failure is category B. The request fails within the reference fail-fast bound. The driver does not hang or unexpectedly terminate. + +#### Scenario: Short context deadline under latency + +- **WHEN** injected latency exceeds the client/context timeout +- **THEN** the version operation fails with category B before the injected latency fully elapses + +### Requirement: HTTP/1 response-header timeout fails under downstream latency + +Where the driver supports configuring an HTTP/1 response-header timeout, it SHALL fail a version operation with category B when downstream latency exceeds that timeout. Steps MUST follow **ServerTimeout** (#8) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A downstream latency toxic exceeds the configured response-header timeout, causing a category B timeout failure. + +**Acceptance criteria:** The failure is category B. The driver does not hang or unexpectedly terminate. Drivers that do not support configuring a response-header timeout SHALL document the limitation and skip this scenario. HTTP/2 is not applicable. + +#### Scenario: HTTP/1 response-header timeout + +- **WHEN** downstream latency exceeds the configured HTTP/1 response-header timeout +- **THEN** the version operation fails with category B + diff --git a/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/toxiproxy-packet-loss/spec.md b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/toxiproxy-packet-loss/spec.md new file mode 100644 index 00000000..9a6e2a61 --- /dev/null +++ b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/toxiproxy-packet-loss/spec.md @@ -0,0 +1,32 @@ +## ADDED Requirements + +### Requirement: Partial packet loss yields a mix of successes and connection or timeout failures + +The driver SHALL run a burst of version requests under intermittent `reset_peer` toxics (reference: 40% toxicity, 40 requests) and accept a mix of successes and failures. Steps MUST follow **PartialPacketLoss** (#9) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** Some requests succeed while others fail due to intermittent connection interruptions. Failed requests belong to category A or B. The scenario does not require either all requests to fail or all requests to succeed. + +**Acceptance criteria:** At least one request succeeds and at least one request fails. Every failure belongs to category A or B. After the toxic is removed, subsequent version requests succeed. The driver does not hang or unexpectedly terminate, and category F is rejected. This scenario is exercised on HTTP/1. Drivers that cannot meaningfully model per-request packet loss on HTTP/2 MAY skip the HTTP/2 variant. + +#### Scenario: Intermittent packet loss + +- **WHEN** intermittent `reset_peer` toxics are active during multiple version requests +- **THEN** the results include both successes and failures, and every failure belongs to category A or B + +#### Scenario: Recovery after packet loss + +- **WHEN** the intermittent packet-loss toxic is removed +- **THEN** subsequent version requests succeed + +### Requirement: Full packet loss fails then recovers + +The driver SHALL fail a version operation when Toxiproxy applies a timeout toxic that simulates complete packet loss. Steps MUST follow **FullPacketLoss** (#10) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** While the timeout toxic is active, the version operation does not complete successfully. The failure is either a timeout (category B) or an equivalent transport failure caused by the connection becoming unusable (category A). After the toxic is removed, the same client successfully performs the version operation again. + +**Acceptance criteria:** The fault error belongs to category A or B. Recovery succeeds on the same client after the toxic is removed. The request returns within the reference timeout budget (that is, it does not hang beyond the allowed safety timeout). The driver does not unexpectedly terminate, and category F is rejected. + +#### Scenario: Full packet loss then recover + +- **WHEN** a full packet-loss (timeout) toxic is applied and the version operation is performed +- **THEN** the operation fails with category A or B and succeeds after the toxic is removed diff --git a/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/toxiproxy-streaming-operations/spec.md b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/toxiproxy-streaming-operations/spec.md new file mode 100644 index 00000000..ab0cbaac --- /dev/null +++ b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/toxiproxy-streaming-operations/spec.md @@ -0,0 +1,28 @@ +## ADDED Requirements + +### Requirement: Disconnect during cursor iteration fails the next read + +The driver SHALL fail the next cursor read with category A when the Toxiproxy proxy is disabled after a partial number of documents have been read (reference: after 5 docs). Steps MUST follow **DisconnectDuringCursorIteration** (#11) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** Cursor streaming begins successfully; proxy disabled mid-iteration; subsequent read fails with connection-loss (category A); after proxy re-enable, the driver can perform new healthy operations (per reference recovery checks). Mid-stream cancel/connection errors that are still category A are acceptable. + +**Acceptance criteria:** The next cursor read fails with category A; the cursor does not continue successfully across the disconnect; after proxy re-enable the driver performs new operations successfully; no hang or unexpected process termination; category F rejected. Run HTTP/1 and HTTP/2 when supported. + +#### Scenario: Mid-cursor proxy disable + +- **WHEN** the proxy is disabled after partial cursor consumption +- **THEN** the next cursor read fails with category A + +### Requirement: Disconnect during query startup fails the query + +The driver SHALL fail query startup (`POST /_api/cursor` or driver `Query` equivalent) with category A when the proxy is disabled while the query is starting. Steps MUST follow **DisconnectDuringQueryExecution** (#12). + +**Expected behavior:** Fault overlaps query creation; the operation fails with connection-loss; HTTP/1 vs HTTP/2 message text may differ (e.g. EOF vs unexpected EOF) while category remains A. + +**Acceptance criteria:** Query fails with category A; no hang or unexpected process termination; recovery of new operations after proxy re-enable per reference; category F rejected. + +#### Scenario: Query start interrupted + +- **WHEN** the Toxiproxy proxy is disabled during query startup before a cursor is returned +- **THEN** the query fails with category A and no usable cursor is returned + diff --git a/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/toxiproxy-write-operations/spec.md b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/toxiproxy-write-operations/spec.md new file mode 100644 index 00000000..965f65aa --- /dev/null +++ b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/specs/toxiproxy-write-operations/spec.md @@ -0,0 +1,28 @@ +## ADDED Requirements + +### Requirement: Disconnect during document insert fails cleanly with unknown write outcome + +The driver SHALL fail document create with category A when the Toxiproxy proxy is disabled during the insert, and MUST treat the write outcome as unknown (neither assert committed nor assert absent unless the reference requires a specific follow-up check). Steps MUST follow **DisconnectDuringInsert** (#13) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** The insert request is in flight when the proxy is disabled. The create operation fails with a category A transport error. The final commit state of the interrupted write is intentionally unspecified. + +**Acceptance criteria:** Error is category A; no hang or unexpected process termination; category F rejected; recovery after clear per reference. HTTP/1 and HTTP/2 when supported. + +#### Scenario: Insert interrupted by proxy disable + +- **WHEN** the proxy is disabled during document create +- **THEN** the create fails with category A and the test does not assert a definitive write success or failure for that document’s durability + +### Requirement: Disconnect during transaction commit fails cleanly with unknown commit outcome + +The driver SHALL fail transaction commit with category A when the Toxiproxy proxy is disabled while the commit request is in flight, and MUST treat the transaction outcome as unknown (neither assert committed nor assert aborted unless the reference requires a specific follow-up check). Steps MUST follow **DisconnectDuringTransactionCommit** (#14) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A transaction containing writes is prepared successfully; the commit request is sent; the proxy is disabled while the commit is in flight; commit fails with a connection-loss error. After the proxy is re-enabled, new operations succeed. Tests MUST NOT require the interrupted transaction to have either committed or rolled back. + +**Acceptance criteria:** Commit fails with category A; the test MUST NOT assert whether the transaction committed or aborted; recovery succeeds after the proxy is restored; no hang or unexpected process termination; category F rejected. Run HTTP/1 and HTTP/2 when supported. + +#### Scenario: Transaction commit interrupted by disconnect + +- **WHEN** the proxy is disabled while a transaction commit request is in flight +- **THEN** the commit fails with category A, the transaction outcome is treated as unknown, and new operations succeed after recovery + diff --git a/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/tasks.md b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/tasks.md new file mode 100644 index 00000000..ffd03492 --- /dev/null +++ b/openspec/changes/archive/2026-07-24-standardize-driver-resiliency-tests/tasks.md @@ -0,0 +1,37 @@ +## 1. Spec coherence and reference linkage + +- [x] 1.1 Verify each Part A scenario (#0–#8) and Part B scenario (#1–#14) in `driver-resiliency-reference.md` maps to exactly one OpenSpec capability specification +- [x] 1.2 Add a short Scenario → Capability Mapping table in `driver-resiliency-reference.md` linking scenario groups to `openspec` capability names without duplicating requirements +- [x] 1.3 Confirm environment contract in specs/design matches the shared harness documentation +- [x] 1.4 Verify every requirement in each capability spec is traceable back to an existing scenario in driver-resiliency-reference.md + +## 2. Error classification (Categories A–F) + +- [x] 2.1 Document error classification requirements (Categories A–F) in `error-classification` — meanings, phase/scenario acceptance, C vs D, universal rejects +- [x] 2.2 Verify the Go Driver reference implementation conforms to the specification — helpers in `v2/tests/network_fault_error_util_test.go` map to A–F; HTTP/1 vs HTTP/2 dual coverage/skip rules match specs; scenarios that allow `failuresDuring = 0` are not over-asserted + +## 3. Capability gap check — Kubernetes resiliency + +- [x] 3.1 Gap-check Go Part A against `load-balancing` and `ingress-resiliency` acceptance criteria +- [x] 3.2 Gap-check Go Part A against `coordinator-resiliency` (idle, active restart, kill during read/insert/iteration) +- [x] 3.3 Record any deviations between the Go Driver reference implementation and the specification; fix only clear acceptance mismatches (no new product features) + - Deviations: none that require code changes. Non-blocking notes: Go remaps `TEST_ENDPOINTS_OVERRIDE`→`TEST_ENDPOINTS` via Makefile; coordinator idle baseline uses `prepareResiliencyClient`/`waitForClusterStable` rather than a single explicit pre-fault `Version()` call; internal `markIngressReady` naming is reused for coordinator recovery; kill-during-read threshold is 1 doc and kill-during-iteration is 30 docs (aligned with reference). + +## 4. Capability gap check — Toxiproxy + +- [x] 4.1 Gap-check Toxiproxy #1–#3 against `toxiproxy-connection-interruption` +- [x] 4.2 Gap-check Toxiproxy #4–#8 against `toxiproxy-latency` +- [x] 4.3 Gap-check Toxiproxy #9–#10 against `toxiproxy-packet-loss` +- [x] 4.4 Gap-check Toxiproxy #11–#12 against `toxiproxy-streaming-operations` +- [x] 4.5 Gap-check Toxiproxy #13–#14 against `toxiproxy-write-operations` (unknown write/commit outcome) + - Fixes applied: corrected ExtremeLatency expected behavior/scenario text in delta+main `toxiproxy-latency` (had LatencyRemoved content); removed duplicate ContextTimeout scenario; Go `TestToxiproxy_ExtremeLatency` now asserts post-toxic recovery. + +## 5. Cross-driver handoff package + +- [x] 5.1 Draft a one-page implementer checklist pointing to OpenSpec capabilities + reference doc + demo HTML env contract (no kubectl/Go helper dumps) + - Added `deploy/kubernetes/documentation/driver-resiliency-implementer-checklist.md`; linked from reference "How to use"; removed Go helper-name table from the reference checklist section. +- [x] 5.2 List recommended implementation order for other drivers: `error-classification` → harness env wiring → load-balancing → ingress → coordinator → Toxiproxy subsets + - Included in the implementer checklist "Recommended implementation order" section. +- [x] 5.3 Run `openspec validate --change standardize-driver-resiliency-tests` and resolve any schema issues before archive +- [x] 5.4 Verify all capability examples and acceptance criteria are language-independent (no Go-specific APIs, helper names, or implementation details). + - Removed Go-only `invalid character '<'` wording (kept language-neutral decode/HTML symptom for category D); replaced “panic” with “unexpected process termination” across capability specs. Re-validated change + main specs. diff --git a/openspec/config.yaml b/openspec/config.yaml new file mode 100644 index 00000000..392946c6 --- /dev/null +++ b/openspec/config.yaml @@ -0,0 +1,20 @@ +schema: spec-driven + +# Project context (optional) +# This is shown to AI when creating artifacts. +# Add your tech stack, conventions, style guides, domain knowledge, etc. +# Example: +# context: | +# Tech stack: TypeScript, React, Node.js +# We use conventional commits +# Domain: e-commerce platform + +# Per-artifact rules (optional) +# Add custom rules for specific artifacts. +# Example: +# rules: +# proposal: +# - Keep proposals under 500 words +# - Always include a "Non-goals" section +# tasks: +# - Break tasks into chunks of max 2 hours diff --git a/openspec/specs/coordinator-resiliency/spec.md b/openspec/specs/coordinator-resiliency/spec.md new file mode 100644 index 00000000..b62068cc --- /dev/null +++ b/openspec/specs/coordinator-resiliency/spec.md @@ -0,0 +1,85 @@ +## Purpose + +Define driver behavior when ArangoDB coordinators restart or are killed during idle periods, active version probes, cursor reads, inserts, and cursor iteration. + +## Requirements + +### Requirement: Coordinator restart while idle recovers on the same client + +The driver SHALL successfully perform a version operation after all coordinator pods are deleted and recreated, with no client requests issued during the outage. Steps MUST follow reference scenario **CoordinatorRestartWhileIdle** (Part A #4) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A version operation succeeds before the fault. No client requests are issued while coordinators are being recreated. Once all coordinators are ready, the same client successfully performs the version operation without requiring reinitialization. + +**Acceptance criteria:** The post-recovery version operation succeeds within the reference coordinator-ready budget using the same client. No hang or unexpected driver termination occurs. No client errors are expected during the outage because no requests are issued while coordinators are being recreated. + +#### Scenario: Idle client survives full coordinator recreation + +- **WHEN** all coordinator pods are deleted while the client is idle +- **THEN** version on the same client succeeds + +### Requirement: Coordinator restart during active workload recovers without requiring during-fault failures + +The driver SHALL keep a version loop alive while all coordinator pods are deleted and recreated. Recovery is mandatory; during-fault failures are optional. Steps MUST follow **CoordinatorRestartDuringActiveWorkload** (Part A #5). + +**Expected behavior:** Single client; `GET /_api/version` every ~100 ms with a bounded per-request timeout. Track successes/failures before, during, and after the fault. Recovery successes are counted only after coordinator readiness has been re-established. + +**Acceptance criteria:** `successesBefore ≥ 1`; `successesAfter ≥ 1`; baseline and recovery quality rules defined in the reference MUST hold; failuresDuring MAY be 0. If `failuresDuring > 0`, every observed error MUST belong to category A, B, C, or D. The workload MUST terminate cleanly after cancellation, with no hang or unexpected process termination, and category F MUST never occur during the fault window. + +#### Scenario: Recovery with zero during-fault failures + +- **WHEN** the version loop runs across full coordinator recreation and no request overlaps downtime +- **THEN** the scenario passes if recovery successes are recorded and other pass criteria hold + +#### Scenario: Recovery with transient during-fault failures + +- **WHEN** one or more version requests fail during the coordinator outage +- **THEN** each during-fault error is category A–D and recovery successes still occur + +### Requirement: Coordinator kill during cursor read interrupts and marks the cursor unusable + +The driver SHALL observe a clean interrupt when coordinators are killed during an active cursor read, then treat resume/close of that cursor according to dead-cursor rules. Steps MUST follow **CoordinatorKillDuringRead** (Part A #6). + +**Expected behavior:** The streaming cursor is interrupted after the coordinator kill. Interrupt-phase errors MUST belong to categories A–E as defined for that phase. After recovery, attempting to resume the same cursor accepts only categories A or E; gateway errors (502/503/504) are not valid dead-cursor resume outcomes. Closing the dead cursor primarily produces category E, although categories A–D are also accepted as defined in the reference. + +**Acceptance criteria:** The cursor MUST NOT complete successfully as though the coordinator interruption had not occurred. Interrupt and dead-cursor phases match allowed categories; cluster recovers for subsequent healthy operations; no hang or unexpected process termination. + +#### Scenario: Cursor interrupt on coordinator kill + +- **WHEN** all coordinators are deleted during an open cursor read +- **THEN** the next cursor operation fails with an allowed interrupt category (A–E) and does not pretend the query finished successfully + +#### Scenario: Dead cursor after recovery + +- **WHEN** the driver resumes or closes the same cursor after cluster recovery +- **THEN** errors match the dead-cursor acceptance set for that phase in the reference (see `error-classification`) + +### Requirement: Coordinator kill during insert recovers with optional during-fault failures + +The driver SHALL run an insert loop while one coordinator is deleted and SHALL recover afterward. Steps MUST follow **CoordinatorKillDuringInsert** (Part A #7). + +**Expected behavior:** An insert workload remains active throughout the fault window. `failuresDuring` MAY be `0`. If during-fault errors occur, they MUST belong to categories A–D. The outcome of an insert interrupted by the coordinator failure MAY be unknown; therefore the suite validates clean failure classification and post-recovery progress rather than the commit outcome of an individual interrupted write. + +**Acceptance criteria:** At least one successful insert (or equivalent post-recovery health check as defined in the reference) MUST occur after recovery. During-fault errors, if any, MUST belong only to categories A–D. The workload MUST terminate cleanly after cancellation with no hang or unexpected process termination, and category F MUST never occur during the fault window. + +#### Scenario: Insert loop survives coordinator kill + +- **WHEN** inserts run while one coordinator pod is deleted +- **THEN** the driver recovers after the fault, and any during-fault errors are only categories A–D + +### Requirement: Coordinator kill during cursor iteration interrupts mid-stream + +The driver SHALL interrupt cursor iteration after a defined number of documents when all coordinators are killed. Steps MUST follow **CoordinatorKillDuringCursorIteration** (Part A #8). + +**Expected behavior:** After reading the reference threshold of documents, all coordinator pods are deleted. The active cursor iteration is interrupted with an allowed interrupt category. After the cluster recovers, resuming or closing the same cursor follows the dead-cursor acceptance rules defined in `error-classification`. Differences in protocol-specific error messages or response bodies (for example between HTTP/1 and HTTP/2) are evaluated by error category rather than exact message text. + +**Acceptance criteria:** Cursor iteration MUST NOT complete successfully across the coordinator kill. Interrupt and dead-cursor phases MUST satisfy the allowed error categories defined in `error-classification`. The cluster MUST recover for subsequent healthy operations. The test MUST fail on hangs, unexpected process termination, or category F during the fault window. + +#### Scenario: Mid-iteration interrupt + +- **WHEN** coordinators are killed after partial cursor consumption +- **THEN** the next iteration step fails with an allowed category and the cursor is not treated as successfully exhausted + +#### Scenario: Dead cursor remains unusable after recovery + +- **WHEN** the cluster has recovered after the coordinator kill +- **THEN** resuming or closing the same cursor follows the dead-cursor acceptance rules defined in `error-classification` diff --git a/openspec/specs/error-classification/spec.md b/openspec/specs/error-classification/spec.md new file mode 100644 index 00000000..602a8a1d --- /dev/null +++ b/openspec/specs/error-classification/spec.md @@ -0,0 +1,96 @@ +## Purpose + +Define language-agnostic error categories A–F, phase- and scenario-specific acceptance rules, C vs D classification guidance, and universal reject conditions for resiliency and Toxiproxy suites. + +## Requirements + +### Requirement: Failures are classified into language-agnostic categories A–F + +Every official driver resiliency/Toxiproxy suite SHALL classify observed failures into these categories (definitions are normative and correspond to the categories described in `driver-resiliency-reference.md`): + + +| Category | Meaning | +| -------- | ------------------------------------------------------------------------------------------- | +| **A** | Connection lost — TCP/HTTP transport broken before a valid HTTP response | +| **B** | Timeout — client or transport gave up waiting | +| **C** | Gateway HTTP error — HTTP 502/503/504 with JSON body | +| **D** | HTTP error status with a non-JSON body and/or non-JSON `Content-Type` (typical 502/503/504) | +| **E** | Cursor gone — HTTP 404/409/410 with JSON body on cursor APIs | +| **F** | Application error — auth, validation, or normal API not-found | + + +**Expected behavior:** Tests classify failures using categories (or equivalent language-specific predicates), rather than relying on a single exact error string. HTTP/1 and HTTP/2 MAY produce different message text for the same fault; category match is sufficient. + +**Acceptance criteria:** Tests assert allowed categories per scenario phase; exact-string-only matching is insufficient as the sole rule. + +#### Scenario: Same fault different protocol text + +- **WHEN** the same network fault is observed under HTTP/1 and HTTP/2 with different error strings +- **THEN** both are accepted if they map to the same allowed category for that phase + +#### Scenario: Category F never accepted during fault windows + +- **WHEN** an error during an injected fault window is category F +- **THEN** the test MUST fail + +### Requirement: Acceptance is phase- and scenario-specific + +Drivers SHALL apply the master scenario→category mapping from the reference (Kubernetes Part A and Toxiproxy Part B). Other capability specifications reference the allowed category sets defined here; this capability is the normative classification specification. + +**Summary of expected behavior by scenario:** + +- **Active ingress/coordinator outages:** Categories A–D are typically accepted during the fault window. +- **Cursor interruption:** Interrupt-phase errors accept categories A–E. +- **Dead-cursor resume:** Only categories A or E are accepted; gateway errors (502/503/504) are not valid dead-cursor resume outcomes. +- **Dead-cursor close:** Category E is the primary expected outcome, although categories A–D are also acceptable. +- **Idle restart scenarios:** No failures are expected during the fault window. +- **Coordinator restart and insert scenarios:** These scenarios MAY complete without any observed failures during the injected fault. When no failures occur, the scenario still passes provided all required recovery criteria are satisfied. If failures do occur, they MUST belong to the categories defined for that scenario. +- **Toxiproxy transport faults:** Connection interruption, streaming interruption, and write interruption primarily produce category A. Latency, timeout, and full packet-loss scenarios primarily produce category B. Partial packet loss MAY produce either category A or B. Categories C–E are not expected for pure transport faults. + +**Acceptance criteria:** Each automated scenario documents which categories are accepted at which checkpoint; violations fail the test. + +#### Scenario: Cursor interrupt vs dead cursor resume + +- **WHEN** coordinators are killed during cursor use +- **THEN** interrupt-phase errors accept A–E as listed for that scenario, and resume-phase errors follow the dead-cursor set (A or E), not unrestricted gateway success + +#### Scenario: Optional during-fault failures + +- **WHEN** no failures are observed during an injected fault in a scenario where this is permitted +- **THEN** the scenario passes if all required recovery criteria are satisfied + +### Requirement: Prefer status and Content-Type when distinguishing C vs D + +When the driver exposes HTTP status and `Content-Type`, category **C** MUST be used for 502/503/504 with JSON bodies, and category **D** for those statuses with non-JSON body and/or non-JSON `Content-Type`. If the driver cannot expose HTTP status or Content-Type and instead surfaces an equivalent decode or transport symptom, that observable MAY be classified as category D as described in the reference documentation. + +**Expected behavior:** Classification prefers protocol metadata over substring matching of HTML. + +**Acceptance criteria:** C and D are not conflated when headers/status are available; when status/`Content-Type` are not exposed, an equivalent decode/transport symptom of a non-JSON (e.g. HTML) body — such as a JSON parser rejecting a leading `'<'` — remains an allowed **D** observable. + +#### Scenario: JSON gateway error is category C + +- **WHEN** the fault response is HTTP 503 with `Content-Type: application/json` +- **THEN** the error is classified as category C + +#### Scenario: HTML gateway error is category D + +- **WHEN** the fault response is HTTP 502/503 with non-JSON `Content-Type` or non-JSON body +- **THEN** the error is classified as category D + +### Requirement: Universal reject conditions across resiliency suites + +All resiliency and Toxiproxy scenarios SHALL fail the test when any of the following occur: unexpected driver or test process termination; hang past the scenario/test timeout; missing required recovery after the fault is cleared; treating an interrupted cursor/query as a successful full completion; category F during a fault window. + +**Expected behavior:** Failures are clean, classified, and bounded in time; recovery uses the same client where the reference requires it. + +**Acceptance criteria:** CI marks the scenario failed on any universal reject condition. + +#### Scenario: Hang is a failure + +- **WHEN** a request or workload does not return within the scenario timeout under an injected fault +- **THEN** the test fails (hang), even if no wrong category was observed + +#### Scenario: Recovery required after clear + +- **WHEN** a scenario requires post-fault recovery and the fault has been cleared +- **THEN** a subsequent operation on the required client MUST succeed within the reference budget diff --git a/openspec/specs/ingress-resiliency/spec.md b/openspec/specs/ingress-resiliency/spec.md new file mode 100644 index 00000000..b62d763b --- /dev/null +++ b/openspec/specs/ingress-resiliency/spec.md @@ -0,0 +1,54 @@ +## Purpose + +Define driver behavior when ingress fails over a coordinator or restarts (idle vs active workload), including recovery on the same or fresh clients. + +## Requirements + +### Requirement: Ingress coordinator failover recovers through ingress + +The driver SHALL continue to reach the cluster through ingress after one coordinator pod is deleted. Implementation steps and wait budgets MUST follow reference scenario **IngressCoordinatorFailover** (Part A #1) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A baseline `GET /_admin/status` probe succeeds. After deleting one coordinator pod, the driver retries the probe using fresh clients until a coordinator responds successfully. After operator recovery, the original coordinator count is restored and a final probe succeeds. Whether the responding coordinator ID changes is observational only; either outcome is valid. + +**Acceptance criteria:** A probe succeeds within the reference failover budget after the coordinator kill. After operator recovery, the healthy coordinator count matches the pre-fault count and a final probe succeeds. The scenario MUST complete without hangs or unexpected process termination. Transient probe errors observed during the failover window, if any, MUST belong only to categories A–D (see `error-classification`). + +#### Scenario: Failover then full recovery + +- **WHEN** the driver repeatedly probes `GET /_admin/status` through ingress, one coordinator pod is deleted, and probing continues until success +- **THEN** a probe succeeds after the kill, the deployment recovers to the original coordinator count, and a post-recovery probe succeeds + +#### Scenario: HTTP/1 and HTTP/2 coverage + +- **WHEN** the driver supports both HTTP/1 and HTTP/2 +- **THEN** the failover scenario SHALL be executed for both protocols (skipping HTTP/2 only when unsupported by the server version, as defined in the reference). + +### Requirement: Ingress restart while idle recovers on the same client + +The driver SHALL successfully perform a version operation on the same client instance after an ingress-nginx restart, while no client requests are issued during the outage. Steps MUST follow reference scenario **IngressRestartWhileIdle** (Part A #2). + +**Expected behavior:** A version operation succeeds before the restart. No client requests are issued while ingress is unavailable. After ingress becomes ready again, the same client successfully performs the version operation without requiring reinitialization. + +**Acceptance criteria:** Version operations succeed before and after the ingress restart using the same client. The scenario MUST complete without hangs or unexpected process termination. Category F MUST NOT occur. No during-fault errors are expected because no client requests are issued while ingress is restarting. + +#### Scenario: Idle client survives ingress restart + +- **WHEN** version succeeds, ingress is restarted with no driver traffic during the outage, and ingress becomes ready +- **THEN** version on the same client succeeds after recovery + +### Requirement: Ingress restart during active workload tolerates transient errors and recovers + +The driver SHALL keep a version-request loop alive while ingress-nginx is restarted. Recovery is mandatory; transient failures during the fault window are permitted. Steps MUST follow **IngressRestartDuringActiveWorkload** (Part A #3). + +**Expected behavior:** A single client continuously performs `GET /_api/version` approximately every 100 ms with a bounded per-request timeout. Successes and failures are tracked before, during, and after the ingress restart. Recovery successes are counted only after ingress has become ready again. + +**Acceptance criteria:** `successesBefore ≥ 1`; `successesAfter ≥ 1`; baseline and recovery quality rules defined in the reference MUST hold. `failuresDuring` MAY be `0`. If `failuresDuring > 0`, every observed error MUST belong to category A, B, C, or D. The workload MUST terminate cleanly after cancellation, with no hang or unexpected process termination, and category F MUST never occur during the fault window. + +#### Scenario: Recovery with zero during-fault failures + +- **WHEN** the version loop runs across an ingress-nginx restart and no request overlaps the outage +- **THEN** the scenario passes if recovery successes are recorded and the other acceptance criteria hold + +#### Scenario: Recovery with transient during-fault failures + +- **WHEN** one or more version requests fail during the ingress-nginx restart +- **THEN** each during-fault error belongs to category A, B, C, or D and recovery successes still occur diff --git a/openspec/specs/load-balancing/spec.md b/openspec/specs/load-balancing/spec.md new file mode 100644 index 00000000..d0c4dd9f --- /dev/null +++ b/openspec/specs/load-balancing/spec.md @@ -0,0 +1,36 @@ +## Purpose + +Define observational expectations for how ingress distributes driver traffic across ArangoDB coordinators, with no fault injected, using the shared resiliency harness endpoints. + +## Requirements + +### Requirement: Load balancer coordinator distribution is observable through ingress + +The driver SHALL probe coordinator identity through ingress with no fault injected, recording which coordinator answered each request. Steps MUST follow reference scenario **LoadBalancerCoordinatorDistribution** (Part A #0). + +**Expected behavior:** Repeated `GET /_admin/status` probes succeed and expose `serverInfo.serverId` (or equivalent). Requests may be served by a single coordinator or distributed across multiple coordinators. The distribution is observational and is not required to follow any specific balancing pattern. + +**Acceptance criteria:** Every probe succeeds. The responding coordinator ID for each successful probe is recorded. The scenario MUST NOT fail solely because only one coordinator is observed. No error-category validation applies because no fault is injected. + +#### Scenario: Probes succeed and record coordinator IDs + +- **WHEN** the driver issues status probes through ingress using the shared and fresh HTTP/1 and HTTP/2 client modes defined by the reference +- **THEN** every probe succeeds and the responding coordinator ID is recorded + +#### Scenario: Coordinator distribution is observational + +- **WHEN** probes are answered by one or more distinct coordinator IDs +- **THEN** the scenario still passes because ingress routing behavior is implementation-dependent + +### Requirement: Load-balancing baseline uses the shared resiliency harness endpoints + +Drivers SHALL connect using the Part A harness endpoint and auth contract (`TEST_ENDPOINTS_OVERRIDE`, `TEST_AUTHENTICATION_OVERRIDE` / `TEST_AUTHENTICATION`) as documented in the reference and `deploy/kubernetes/documentation/driver-k8s-shared-infra-demo.html`, remapping names only if semantics are preserved. + +**Expected behavior:** Traffic path is driver → ingress (`arangodb.local`) → coordinators. ClientIP session affinity on the coordinator Service does not apply on this out-of-cluster path. + +**Acceptance criteria:** Probes use the harness-exported URL and auth; Host header behavior matches the reference when required by the URL form. + +#### Scenario: Harness endpoint wiring + +- **WHEN** the load-balancing scenario runs under the shared Kubernetes driver-test runner +- **THEN** the driver uses the exported endpoints/auth (or an equivalent remap) to reach the cluster through ingress diff --git a/openspec/specs/toxiproxy-connection-interruption/spec.md b/openspec/specs/toxiproxy-connection-interruption/spec.md new file mode 100644 index 00000000..40bb3070 --- /dev/null +++ b/openspec/specs/toxiproxy-connection-interruption/spec.md @@ -0,0 +1,57 @@ +## Purpose + +Define driver behavior for Toxiproxy connection-interruption faults (abrupt TCP close, proxy disable, downstream reset) including fail-then-recover expectations and the shared proxy harness contract. + +## Requirements + +### Requirement: Abrupt TCP close on a live connection fails then recovers + +The driver SHALL fail a version request with category A when Toxiproxy injects an upstream `reset_peer` toxic on an established connection, then SHALL succeed on the same client after the toxic is removed. Steps MUST follow Toxiproxy **AbruptTCPConnectionClose** (#1) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** Baseline version succeeds (keep-alive established); after `reset_peer` upstream, version fails quickly with a connection-loss error; after toxic removal, version succeeds again. + +**Acceptance criteria:** Fault error is category A; recovery succeeds on the same client; fail-fast (no hang until the safety deadline); no unexpected process termination; category F rejected. Run HTTP/1 and HTTP/2 when supported. + +#### Scenario: Reset peer then recover + +- **WHEN** `reset_peer` is applied upstream on the Toxiproxy proxy and version is called +- **THEN** the call fails with category A and a later version call succeeds after the toxic is removed + +### Requirement: Network disconnect via disabled proxy fails then recovers + +The driver SHALL fail a version request with **category A** when the Toxiproxy proxy is disabled, then SHALL successfully perform a version operation on the same client after the proxy is re-enabled. Steps MUST follow **NetworkDisconnect** (#2) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A baseline version operation succeeds. While the proxy is disabled, a version request fails with a transport-level connection error (category A). After the proxy is re-enabled, the same client successfully performs a version operation again. + +**Acceptance criteria:** The during-fault error MUST be category A. Recovery on the same client MUST succeed after the proxy is re-enabled. The operation MUST NOT hang or unexpectedly terminate. Category F MUST NOT occur. + +#### Scenario: Proxy disable then recover + +- **WHEN** the Toxiproxy proxy is disabled during a version call +- **THEN** the call fails with category A and succeeds after the proxy is re-enabled + +### Requirement: Downstream connection reset fails then recovers + +The driver SHALL fail a version request with category A when Toxiproxy applies a `reset_peer` toxic on the downstream path, then SHALL successfully perform a version operation on the same client after the toxic is removed. Steps MUST follow `ConnectionResetByPeer` (#3) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A baseline version operation succeeds. While a downstream `reset_peer` toxic is active, a version request fails with a category A transport error. HTTP/1 and HTTP/2 MAY surface different messages (for example `EOF` or `unexpected EOF`), but both classify as category A. After the toxic is removed, the same client successfully performs a version operation again. + +**Acceptance criteria:** The during-fault error MUST be category A. Recovery on the same client MUST succeed after the toxic is removed. The operation MUST NOT hang or unexpectedly terminate. Category F MUST NOT occur. + +#### Scenario: Downstream reset then recover + +- **WHEN** a downstream `reset_peer` toxic is active and a version request is issued +- **THEN** the request fails with category A, and a later version request succeeds after the toxic is removed + +### Requirement: Toxiproxy tests use the shared harness proxy contract + +Drivers SHALL send traffic to the Toxiproxy listen endpoint from the harness (`TEST_ENDPOINTS_OVERRIDE` → listen port) and MUST send `Host: TEST_INGRESS_HOST` (`arangodb.local`) on every request. Admin control uses `TEST_TOXIPROXY_ADMIN` / `TEST_TOXIPROXY_PROXY` (see demo HTML and reference harness table). + +**Expected behavior:** Driver → Toxiproxy listen → ingress → coordinator. Without the Host header, ingress will not route correctly. + +**Acceptance criteria:** Connection scenarios only pass when wired through the proxy with correct Host header; faults are injected via the admin API, not by changing the driver URL mid-test except as the reference describes. + +#### Scenario: Host header required through Toxiproxy + +- **WHEN** the driver connects via the Toxiproxy listen address for connection-interruption tests +- **THEN** every request includes the ingress Host header from `TEST_INGRESS_HOST` diff --git a/openspec/specs/toxiproxy-latency/spec.md b/openspec/specs/toxiproxy-latency/spec.md new file mode 100644 index 00000000..71f7e390 --- /dev/null +++ b/openspec/specs/toxiproxy-latency/spec.md @@ -0,0 +1,70 @@ +## Purpose + +Define driver behavior for Toxiproxy latency faults: high latency success, extreme latency timeout, latency removal, client context timeout, and HTTP/1 response-header timeout. + +## Requirements + +### Requirement: High latency completes successfully but slower + +The driver SHALL successfully perform a version operation when a moderate Toxiproxy latency toxic (reference: 2000 ms) is applied, provided the client timeout exceeds the injected latency. Steps MUST follow `HighLatency` (#4) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A baseline version operation completes successfully. After applying a 2000 ms latency toxic, a version operation also succeeds using a client timeout greater than the injected latency. The observed elapsed time SHOULD reflect approximately the additional network latency. + +**Acceptance criteria:** The version operation succeeds under the injected latency. The observed duration is measurably greater than the baseline and consistent with the injected delay. The operation MUST NOT hang beyond the scenario timeout or unexpected process termination. No error-category validation applies because successful completion is expected. The scenario SHALL run for HTTP/1 and HTTP/2 when supported. + +#### Scenario: Version succeeds under high latency + +- **WHEN** a 2000 ms latency toxic is applied and a version request is issued with a sufficient client timeout +- **THEN** the request succeeds and completes more slowly than the baseline + +### Requirement: Extreme latency surfaces a client timeout + +The driver SHALL fail a version request with category B when injected latency exceeds the client timeout, then SHALL successfully perform a version operation on the same client after the latency toxic is removed. Steps MUST follow `ExtremeLatency` (#5) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A version operation fails when latency (reference: 30 s) exceeds the client timeout (reference: 10 s). The error is category B. After the latency toxic is removed, a subsequent version operation on the same client succeeds. + +**Acceptance criteria:** During-fault failure is category B; recovery succeeds after the toxic is removed; the operation MUST NOT hang or unexpectedly terminate; category F MUST NOT occur. + +#### Scenario: Extreme latency times out then recovers + +- **WHEN** extreme latency is applied with a shorter client timeout +- **THEN** version fails with category B and succeeds after the latency toxic is removed + +### Requirement: Removing latency restores faster responses + +The driver SHALL successfully perform a version operation after a Toxiproxy latency toxic is removed, with the observed response time improving relative to requests made while the toxic was active. Steps MUST follow `LatencyRemoved` (#6) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** Latency toxic applied then removed; subsequent version succeeds without the prior delay penalty. + +**Acceptance criteria:** Post-removal version succeeds; timing improves per reference checkpoints; no hang or unexpected process termination. + +#### Scenario: Latency removed + +- **WHEN** a latency toxic is removed after a slowed successful call +- **THEN** a subsequent version request succeeds without the injected latency effect + +### Requirement: Context timeout fails quickly under high latency + +The driver SHALL fail a version operation with category B when a client/context timeout is shorter than an injected Toxiproxy latency. Steps MUST follow **ContextTimeout** (#7) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A latency toxic greater than the client/context timeout is applied. The version operation fails because the client deadline expires rather than waiting for the full injected latency. + +**Acceptance criteria:** The failure is category B. The request fails within the reference fail-fast bound. The driver does not hang or unexpectedly terminate. + +#### Scenario: Short context deadline under latency + +- **WHEN** injected latency exceeds the client/context timeout +- **THEN** the version operation fails with category B before the injected latency fully elapses + +### Requirement: HTTP/1 response-header timeout fails under downstream latency + +Where the driver supports configuring an HTTP/1 response-header timeout, it SHALL fail a version operation with category B when downstream latency exceeds that timeout. Steps MUST follow **ServerTimeout** (#8) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A downstream latency toxic exceeds the configured response-header timeout, causing a category B timeout failure. + +**Acceptance criteria:** The failure is category B. The driver does not hang or unexpectedly terminate. Drivers that do not support configuring a response-header timeout SHALL document the limitation and skip this scenario. HTTP/2 is not applicable. + +#### Scenario: HTTP/1 response-header timeout + +- **WHEN** downstream latency exceeds the configured HTTP/1 response-header timeout +- **THEN** the version operation fails with category B diff --git a/openspec/specs/toxiproxy-packet-loss/spec.md b/openspec/specs/toxiproxy-packet-loss/spec.md new file mode 100644 index 00000000..d95c1393 --- /dev/null +++ b/openspec/specs/toxiproxy-packet-loss/spec.md @@ -0,0 +1,36 @@ +## Purpose + +Define driver behavior for Toxiproxy partial and full packet-loss faults, including mixed success/failure acceptance and recovery after the toxic is cleared. + +## Requirements + +### Requirement: Partial packet loss yields a mix of successes and connection or timeout failures + +The driver SHALL run a burst of version requests under intermittent `reset_peer` toxics (reference: 40% toxicity, 40 requests) and accept a mix of successes and failures. Steps MUST follow **PartialPacketLoss** (#9) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** Some requests succeed while others fail due to intermittent connection interruptions. Failed requests belong to category A or B. The scenario does not require either all requests to fail or all requests to succeed. + +**Acceptance criteria:** At least one request succeeds and at least one request fails. Every failure belongs to category A or B. After the toxic is removed, subsequent version requests succeed. The driver does not hang or unexpectedly terminate, and category F is rejected. This scenario is exercised on HTTP/1. Drivers that cannot meaningfully model per-request packet loss on HTTP/2 MAY skip the HTTP/2 variant. + +#### Scenario: Intermittent packet loss + +- **WHEN** intermittent `reset_peer` toxics are active during multiple version requests +- **THEN** the results include both successes and failures, and every failure belongs to category A or B + +#### Scenario: Recovery after packet loss + +- **WHEN** the intermittent packet-loss toxic is removed +- **THEN** subsequent version requests succeed + +### Requirement: Full packet loss fails then recovers + +The driver SHALL fail a version operation when Toxiproxy applies a timeout toxic that simulates complete packet loss. Steps MUST follow **FullPacketLoss** (#10) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** While the timeout toxic is active, the version operation does not complete successfully. The failure is either a timeout (category B) or an equivalent transport failure caused by the connection becoming unusable (category A). After the toxic is removed, the same client successfully performs the version operation again. + +**Acceptance criteria:** The fault error belongs to category A or B. Recovery succeeds on the same client after the toxic is removed. The request returns within the reference timeout budget (that is, it does not hang beyond the allowed safety timeout). The driver does not unexpectedly terminate, and category F is rejected. + +#### Scenario: Full packet loss then recover + +- **WHEN** a full packet-loss (timeout) toxic is applied and the version operation is performed +- **THEN** the operation fails with category A or B and succeeds after the toxic is removed diff --git a/openspec/specs/toxiproxy-streaming-operations/spec.md b/openspec/specs/toxiproxy-streaming-operations/spec.md new file mode 100644 index 00000000..32d09266 --- /dev/null +++ b/openspec/specs/toxiproxy-streaming-operations/spec.md @@ -0,0 +1,31 @@ +## Purpose + +Define driver behavior when Toxiproxy disconnects during cursor iteration or query startup, requiring category A failures and recovery of new operations after the proxy is restored. + +## Requirements + +### Requirement: Disconnect during cursor iteration fails the next read + +The driver SHALL fail the next cursor read with category A when the Toxiproxy proxy is disabled after a partial number of documents have been read (reference: after 5 docs). Steps MUST follow **DisconnectDuringCursorIteration** (#11) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** Cursor streaming begins successfully; proxy disabled mid-iteration; subsequent read fails with connection-loss (category A); after proxy re-enable, the driver can perform new healthy operations (per reference recovery checks). Mid-stream cancel/connection errors that are still category A are acceptable. + +**Acceptance criteria:** The next cursor read fails with category A; the cursor does not continue successfully across the disconnect; after proxy re-enable the driver performs new operations successfully; no hang or unexpected process termination; category F rejected. Run HTTP/1 and HTTP/2 when supported. + +#### Scenario: Mid-cursor proxy disable + +- **WHEN** the proxy is disabled after partial cursor consumption +- **THEN** the next cursor read fails with category A + +### Requirement: Disconnect during query startup fails the query + +The driver SHALL fail query startup (`POST /_api/cursor` or driver `Query` equivalent) with category A when the proxy is disabled while the query is starting. Steps MUST follow **DisconnectDuringQueryExecution** (#12). + +**Expected behavior:** Fault overlaps query creation; the operation fails with connection-loss; HTTP/1 vs HTTP/2 message text may differ (e.g. EOF vs unexpected EOF) while category remains A. + +**Acceptance criteria:** Query fails with category A; no hang or unexpected process termination; recovery of new operations after proxy re-enable per reference; category F rejected. + +#### Scenario: Query start interrupted + +- **WHEN** the Toxiproxy proxy is disabled during query startup before a cursor is returned +- **THEN** the query fails with category A and no usable cursor is returned diff --git a/openspec/specs/toxiproxy-write-operations/spec.md b/openspec/specs/toxiproxy-write-operations/spec.md new file mode 100644 index 00000000..dce2555d --- /dev/null +++ b/openspec/specs/toxiproxy-write-operations/spec.md @@ -0,0 +1,31 @@ +## Purpose + +Define driver behavior when Toxiproxy disconnects during document insert or transaction commit, requiring category A failures and treating write/commit outcomes as unknown. + +## Requirements + +### Requirement: Disconnect during document insert fails cleanly with unknown write outcome + +The driver SHALL fail document create with category A when the Toxiproxy proxy is disabled during the insert, and MUST treat the write outcome as unknown (neither assert committed nor assert absent unless the reference requires a specific follow-up check). Steps MUST follow **DisconnectDuringInsert** (#13) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** The insert request is in flight when the proxy is disabled. The create operation fails with a category A transport error. The final commit state of the interrupted write is intentionally unspecified. + +**Acceptance criteria:** Error is category A; no hang or unexpected process termination; category F rejected; recovery after clear per reference. HTTP/1 and HTTP/2 when supported. + +#### Scenario: Insert interrupted by proxy disable + +- **WHEN** the proxy is disabled during document create +- **THEN** the create fails with category A and the test does not assert a definitive write success or failure for that document’s durability + +### Requirement: Disconnect during transaction commit fails cleanly with unknown commit outcome + +The driver SHALL fail transaction commit with category A when the Toxiproxy proxy is disabled while the commit request is in flight, and MUST treat the transaction outcome as unknown (neither assert committed nor assert aborted unless the reference requires a specific follow-up check). Steps MUST follow **DisconnectDuringTransactionCommit** (#14) in `deploy/kubernetes/documentation/driver-resiliency-reference.md`. + +**Expected behavior:** A transaction containing writes is prepared successfully; the commit request is sent; the proxy is disabled while the commit is in flight; commit fails with a connection-loss error. After the proxy is re-enabled, new operations succeed. Tests MUST NOT require the interrupted transaction to have either committed or rolled back. + +**Acceptance criteria:** Commit fails with category A; the test MUST NOT assert whether the transaction committed or aborted; recovery succeeds after the proxy is restored; no hang or unexpected process termination; category F rejected. Run HTTP/1 and HTTP/2 when supported. + +#### Scenario: Transaction commit interrupted by disconnect + +- **WHEN** the proxy is disabled while a transaction commit request is in flight +- **THEN** the commit fails with category A, the transaction outcome is treated as unknown, and new operations succeed after recovery diff --git a/v2/tests/helper_test.go b/v2/tests/helper_test.go index d3dc8775..98e3545c 100644 --- a/v2/tests/helper_test.go +++ b/v2/tests/helper_test.go @@ -24,6 +24,7 @@ import ( "context" "fmt" "os" + "strings" "sync" "testing" "time" @@ -112,6 +113,16 @@ func WithCollectionV2(t testing.TB, db arangodb.Database, props *arangodb.Create if arangoErr.Code == 503 || arangoErr.Code == 408 || arangoErr.Code == 500 { return nil // Retry } + // After coordinator recreation, CreateDatabase can succeed on one + // coordinator before agency propagation reaches the coordinator that + // handles CreateCollection (common via ingress Maglev/round-robin). + if arangoErr.Code == 404 || arangoErr.ErrorNum == shared.ErrArangoDatabaseNotFound { + return nil // Retry + } + } + if shared.IsNotFound(createErr) || + strings.Contains(strings.ToLower(createErr.Error()), "database not found") { + return nil // Retry } // For other errors (like duplicate name), return them immediately diff --git a/v2/tests/network_fault_error_util_test.go b/v2/tests/network_fault_error_util_test.go index 0432507a..36cdb285 100644 --- a/v2/tests/network_fault_error_util_test.go +++ b/v2/tests/network_fault_error_util_test.go @@ -93,6 +93,8 @@ func isConnectionError(err error) bool { strings.Contains(msg, "broken pipe") || strings.Contains(msg, "connection refused") || strings.Contains(msg, "use of closed network connection") || + // net/http keep-alive: ingress/server closes an idle pooled connection during outage. + strings.Contains(msg, "server closed idle connection") || strings.Contains(msg, "client connection lost") || strings.Contains(msg, "transport connection broken") || // golang.org/x/net/http2 during ingress restart / dial races (no net.OpError wrapper). @@ -334,6 +336,18 @@ func TestIsConnectionError_detectsHTTP2ClientConnNotEstablished(t *testing.T) { require.True(t, isResiliencyTransientError(err)) } +func TestIsConnectionError_detectsServerClosedIdleConnection(t *testing.T) { + // HTTP/1 keep-alive during ingress restart: net/http surfaces this instead of + // (or in addition to) connection reset by peer when the peer closes an idle conn. + err := pkgerrors.WithStack(&url.Error{ + Op: "Get", + URL: "http://arangodb.local/_api/version", + Err: errors.New("http: server closed idle connection"), + }) + require.True(t, isConnectionError(err)) + require.True(t, isResiliencyTransientError(err)) +} + func TestIsResetOrEOFError_detectsDocumentedOutcomes(t *testing.T) { require.True(t, isResetOrEOFError(pkgerrors.WithStack(&url.Error{ Op: "Get", diff --git a/v2/tests/resiliency_coordinator_k8s_helper_test.go b/v2/tests/resiliency_coordinator_k8s_helper_test.go index e8169edd..12e4d73c 100644 --- a/v2/tests/resiliency_coordinator_k8s_helper_test.go +++ b/v2/tests/resiliency_coordinator_k8s_helper_test.go @@ -53,30 +53,55 @@ type CoordinatorTarget struct { } // killRandomCoordinator deletes one coordinator pod selected at random from cluster health. +// Only Health entries that map to a live pod are eligible: after a prior kill/recover, +// Health can briefly still list replaced server IDs with no matching pod (CircleCI flake). func killRandomCoordinator(t testing.TB, client arangodb.Client) CoordinatorTarget { t.Helper() ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() - health, err := client.Health(ctx) - require.NoError(t, err) + var target CoordinatorTarget + err := NewTimeout(func() error { + health, err := client.Health(ctx) + if err != nil { + t.Logf("killRandomCoordinator: Health() not ready: %v", err) + return nil + } - var targets []CoordinatorTarget - for id, server := range health.Health { - if server.Role != arangodb.ServerRoleCoordinator { - continue + pods, err := tryListCoordinatorPods() + if err != nil { + t.Logf("killRandomCoordinator: listing pods: %v", err) + return nil } - serverID := string(id) - targets = append(targets, CoordinatorTarget{ - ServerID: serverID, - Endpoint: server.Endpoint, - ResourceName: coordinatorPodForServerID(t, serverID), - }) - } - require.NotEmpty(t, targets, "no coordinators found in cluster health") + var targets []CoordinatorTarget + for id, server := range health.Health { + if server.Role != arangodb.ServerRoleCoordinator { + continue + } + serverID := string(id) + pod, ok := matchCoordinatorPod(pods, serverID) + if !ok { + t.Logf("killRandomCoordinator: skipping %s (no live pod among %v)", serverID, pods) + continue + } + targets = append(targets, CoordinatorTarget{ + ServerID: serverID, + Endpoint: server.Endpoint, + ResourceName: pod, + }) + } + if len(targets) == 0 { + t.Logf("killRandomCoordinator: no pod-backed coordinators yet (health entries may be stale)") + return nil + } + + target = targets[rand.Intn(len(targets))] + return Interrupt{} + }).Timeout(coordinatorRecoveryTimeout, time.Second) + require.NoError(t, err, "no pod-backed coordinator available to kill") + require.NotEmpty(t, target.ResourceName) - target := targets[rand.Intn(len(targets))] t.Logf("Killing coordinator pod %s (server %s)", target.ResourceName, target.ServerID) deleteCoordinatorPod(t, target.ResourceName) return target @@ -85,19 +110,27 @@ func killRandomCoordinator(t testing.TB, client arangodb.Client) CoordinatorTarg func coordinatorPodForServerID(t testing.TB, serverID string) string { t.Helper() - podToken := coordinatorPodToken(serverID) pods := listCoordinatorPods(t) + pod, ok := matchCoordinatorPod(pods, serverID) + require.True(t, ok, + "expected exactly one coordinator pod for server ID %q (token %q) among %v", + serverID, coordinatorPodToken(serverID), pods) + return pod +} + +// matchCoordinatorPod returns the unique live pod name for serverID, if any. +func matchCoordinatorPod(pods []string, serverID string) (string, bool) { + podToken := coordinatorPodToken(serverID) var matches []string for _, pod := range pods { if strings.Contains(strings.ToLower(pod), podToken) { matches = append(matches, pod) } } - - require.Len(t, matches, 1, - "expected exactly one coordinator pod for server ID %q (token %q), got %v among %v", - serverID, podToken, matches, pods) - return matches[0] + if len(matches) != 1 { + return "", false + } + return matches[0], true } // requireResiliencyK8sCoordinatorMode skips unless the test runs in k8s cluster mode with kubectl and @@ -145,17 +178,23 @@ func listCoordinatorPods(t testing.TB) []string { t.Helper() requireKubectl(t) + pods, err := tryListCoordinatorPods() + require.NoError(t, err) + require.NotEmpty(t, pods, "no coordinator pods found with selector %q", coordinatorLabelSelector()) + return pods +} + +func tryListCoordinatorPods() ([]string, error) { cmd := exec.Command( "kubectl", "-n", k8sNamespace(), "get", "pods", "-l", coordinatorLabelSelector(), "-o", "jsonpath={range .items[*]}{.metadata.name}{\"\\n\"}{end}", ) output, err := cmd.CombinedOutput() - require.NoError(t, err, "kubectl get coordinator pods failed: %s", string(output)) - - pods := strings.Fields(string(output)) - require.NotEmpty(t, pods, "no coordinator pods found with selector %q", coordinatorLabelSelector()) - return pods + if err != nil { + return nil, fmt.Errorf("kubectl get coordinator pods failed: %w: %s", err, strings.TrimSpace(string(output))) + } + return strings.Fields(string(output)), nil } func deleteCoordinatorPod(t testing.TB, pod string) { @@ -448,9 +487,9 @@ func countHealthyCoordinators(ctx context.Context, client arangodb.Client) int { return coordinators } -// waitForHealthyCoordinatorCount retries client.Health until it reports at least expected -// coordinators. A single Health() call can fail or return an incomplete map right after -// pod recovery even when Version() already succeeds (CircleCI flake: expected 3, got 0). +// waitForHealthyCoordinatorCount retries until cluster health reports at least expected +// coordinators that each map to a live pod. A bare Health() count can include replaced +// server IDs after kill/recover (no pod yet / ghost entry), which then breaks the next kill. func waitForHealthyCoordinatorCount(t testing.TB, client arangodb.Client, expected int, timeout time.Duration) { t.Helper() @@ -462,22 +501,32 @@ func waitForHealthyCoordinatorCount(t testing.TB, client arangodb.Client, expect return nil } - coordinators := 0 - for _, server := range health.Health { - if server.Role == arangodb.ServerRoleCoordinator { - coordinators++ + pods, err := tryListCoordinatorPods() + if err != nil { + t.Logf("Waiting for cluster health coordinator count: list pods: %v", err) + return nil + } + + matched := 0 + for id, server := range health.Health { + if server.Role != arangodb.ServerRoleCoordinator { + continue + } + if _, ok := matchCoordinatorPod(pods, string(id)); ok { + matched++ } } - if coordinators >= expected { - t.Logf("Cluster health reports %d coordinator(s) (want >= %d)", coordinators, expected) + if matched >= expected && len(pods) >= expected { + t.Logf("Cluster health reports %d pod-backed coordinator(s) (want >= %d; pods=%d)", + matched, expected, len(pods)) return Interrupt{} } - t.Logf("Waiting for coordinator count in cluster health: got %d, want >= %d", coordinators, expected) + t.Logf("Waiting for pod-backed coordinators: matched=%d pods=%d want>=%d", matched, len(pods), expected) return nil }) }).Timeout(timeout, time.Second) - require.NoError(t, err, "cluster health did not report at least %d coordinators within %s", expected, timeout) + require.NoError(t, err, "cluster health did not report at least %d pod-backed coordinators within %s", expected, timeout) } func readyCoordinatorPodCount(namespace, selector string) (int, error) { diff --git a/v2/tests/resiliency_failover_test.go b/v2/tests/resiliency_failover_test.go index cd25a214..7216308e 100644 --- a/v2/tests/resiliency_failover_test.go +++ b/v2/tests/resiliency_failover_test.go @@ -58,11 +58,20 @@ func runIngressCoordinatorFailover( ) { t.Helper() + // Ensure pods from a prior protocol's kill/recover are ready before Health()-based kill. + if isK8S() { + waitForCoordinatorsReady(t, expectedCoordinatorCount()) + } + client := newResiliencyClient(t, newClientConn(t)) requireMinimumCoordinators(t, client, 2) - expectedCoordinators := countHealthyCoordinators(ctx, client) - t.Logf("Cluster has %d coordinators", expectedCoordinators) + expectedCoordinators := expectedCoordinatorCount() + if expectedCoordinators < 2 { + expectedCoordinators = countHealthyCoordinators(ctx, client) + } + t.Logf("Cluster has %d coordinators (health reports %d)", + expectedCoordinators, countHealthyCoordinators(ctx, client)) baselineCoordinatorID := waitForCoordinatorResponse(t, resiliencyFailoverResponseTimeout, freshProbeConn) t.Logf("Baseline coordinator before kill: %s", baselineCoordinatorID) diff --git a/v2/tests/resiliency_util_test.go b/v2/tests/resiliency_util_test.go index adb1fe3b..799bcc74 100644 --- a/v2/tests/resiliency_util_test.go +++ b/v2/tests/resiliency_util_test.go @@ -149,7 +149,9 @@ func connectionJsonHttpFresh(t testing.TB) connection.Connection { return c } -// waitForClusterWritable retries a create/delete database probe until ingress serves writes. +// waitForClusterWritable retries create database + collection until ingress serves +// cluster writes end-to-end. CreateDatabase alone can succeed while agency propagation +// still lags for CreateCollection (seen on HTTP/2 after coordinator kill/recover). func waitForClusterWritable(t testing.TB, client arangodb.Client, timeout time.Duration) { t.Helper() @@ -163,6 +165,15 @@ func waitForClusterWritable(t testing.TB, client arangodb.Client, timeout time.D return nil } + colName := GenerateUUID("resiliency-probe-col") + if _, err := db.CreateCollectionV2(ctx, colName, nil); err != nil { + t.Logf("cluster writable probe: create collection failed: %v", err) + removeCtx, removeCancel := context.WithTimeout(context.Background(), 30*time.Second) + _ = db.Remove(removeCtx) + removeCancel() + return nil + } + removeCtx, removeCancel := context.WithTimeout(context.Background(), 30*time.Second) defer removeCancel() if err := db.Remove(removeCtx); err != nil { diff --git a/v2/tests/toxiproxy_latency_test.go b/v2/tests/toxiproxy_latency_test.go index c49307c0..9bda9cfb 100644 --- a/v2/tests/toxiproxy_latency_test.go +++ b/v2/tests/toxiproxy_latency_test.go @@ -101,6 +101,9 @@ func testExtremeLatency(t *testing.T, connFactory toxiproxyConnectionFactory) { require.True(t, isContextDeadlineExceeded(err), "expected context deadline exceeded with 30s latency and 10s timeout, got: %v", err) }) + + require.NoError(t, proxy.RemoveToxic("latency_up")) + waitForSuccessfulVersion(t, client, 1*time.Minute) } // TestToxiproxy_LatencyRemoved validates that request duration returns to normal