Skip to content

Build a Network-Partition Chaos Engineering Framework With Automated Recovery Verification Across Contract, Backend, and Gateway #1629

Description

@Emmyt24

Description

chaosEngine.ts and the campaign-chaos.yml CI workflow exist, but there's no framework specifically for injecting network partitions between the contract RPC layer, the backend, and the gateway, and automatically verifying the system recovers to a consistent state once the partition heals — the exact failure mode a real cloud network blip causes.

Requirements and Context

  • Build a partition-injection harness that can sever connectivity between any two of {contract RPC, backend, gateway} for a configured duration while the rest of the system keeps running
  • Automatically verify, once the partition heals, that no events were permanently lost and all projections reconverge to the correct state within a bounded recovery window
  • Integrate with the existing chaosEngine.ts and campaign-chaos.yml infrastructure rather than duplicating it
  • Add three partition scenarios (contract↔backend, backend↔gateway, contract↔gateway) as automated chaos test cases
  • Proof of work (required): this is a hard/complex issue — the PR description must include (1) a screenshot of the full relevant test suite run showing all tests passing, and (2) a screenshot of a successful project/module build or compile (npm run build) with no errors. PRs missing either screenshot will not be reviewed.

Suggested Execution

Branch: feat/integration-network-partition-chaos-framework

Implement Changes

  • Extend chaosEngine.ts with a partition-injection primitive (e.g. a proxy layer that can be told to drop traffic between two named endpoints)
  • Add a post-partition recovery verification pass reusing campaignConsistencyChecker.ts-style drift detection across all affected projections
  • Add the three partition scenarios as automated chaos test cases
  • Wire the new scenarios into campaign-chaos.yml

Test and Commit

Run npx vitest run src/__tests__/chaos/networkPartition.chaos.test.ts — all three partition scenarios heal with zero permanently-lost events and full projection reconvergence. Then run npm run build and confirm it completes with no errors. Attach a screenshot of both the passing test run and the successful build to the PR description as proof of work.

Example Commit Message

feat(integration): network-partition chaos framework with automated recovery verification

Closes #<issue>

Guidelines

  • Branch from main, open a PR back to main
  • All new code must have corresponding tests
  • Run npm run lint and npm run test before pushing
  • Follow existing naming conventions and file structure
  • PR description must reference this issue number (e.g., Closes #<issue>)
  • Keep commits atomic and use conventional commit format
  • This is a hard/complex issue: PR must include a screenshot of the test suite passing and a screenshot of a successful build/compile — PRs without both screenshots will be marked incomplete and not merged

Metadata

Metadata

Assignees

No one assigned

    Labels

    hardHard/complex, cross-cutting contributionintegrationIntegration / external services

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions