Implement Leader-Election-Based High-Availability Event Listener With Split-Brain Prevention and Automatic Failover

[Emmyt24]

Description

stellarEventListener.ts currently runs as a single instance; if it crashes or its host dies, event ingestion stops until manual intervention. Running multiple instances naively causes duplicate processing and split-brain cursor writes.

Requirements and Context

• Design and implement Redis-based distributed leader election (ioredis is already a dependency) using a renewable lease/lock with a monotonically increasing fencing token
• Only the elected leader instance may advance the event cursor; standby instances stay hot but idle
• On leader failure (lease expiry), a standby must take over within a bounded, documented and tested failover window
• A demoted former leader must be provably prevented from writing a stale cursor advance after a new leader is elected — enforce a fencing-token check on every cursor write
• Proof of work (required): this is a hard/complex issue — the PR description must include (1) a screenshot of the full relevant test suite run showing all tests passing, and (2) a screenshot of a successful project/module build or compile (npm run build) with no errors. PRs missing either screenshot will not be reviewed.

Suggested Execution

Branch: feat/backend-event-listener-leader-election

Implement Changes

• Add a LeaderElection module wrapping ioredis with a TTL-based lock, lease-renewal heartbeat, and a fencing token issued on each election
• Wire stellarEventListener.ts to run its ingestion loop only while holding leadership, and to validate the fencing token before every eventCursorStore write
• Add a kill-switch integration test that kills the current leader process mid-run and asserts a standby takes over and resumes from the correct cursor within the target failover window
• Add structured logs/metrics for election events (became-leader, lost-leadership, fencing-token-rejected)

Test and Commit

Run npx vitest run src/__tests__/leaderElection.test.ts src/__tests__/stellarEventListener.failover.test.ts — leader election, failover timing, and fencing-token rejection all pass. Then run npm run build and confirm it completes with no errors. Attach a screenshot of both the passing test run and the successful build to the PR description as proof of work.

Example Commit Message

feat(backend): leader-election-based HA event listener with split-brain prevention

Closes #<issue>

Guidelines

• Branch from main, open a PR back to main
• All new code must have corresponding tests
• Run npm run lint and npm run test before pushing
• Follow existing naming conventions and file structure
• PR description must reference this issue number (e.g., Closes #<issue>)
• Keep commits atomic and use conventional commit format
• This is a hard/complex issue: PR must include a screenshot of the test suite passing and a screenshot of a successful build/compile — PRs without both screenshots will be marked incomplete and not merged