fix: Calc Engine Concurrent Upsert Race and Orchestrator OOM Deadlocks#148
Merged
Merged
Conversation
…sede When two calc jobs process the same product+period+calc_type concurrently and no prior active row exists, both transactions see no row to supersede and both attempt to INSERT — the second hits the uk_cpc_active unique constraint. Wrap the supersede+insert in a retry loop (max 3 attempts) that catches the 23505 violation, rolls back, and re-runs in a fresh transaction where the winner's row is now visible and can be properly superseded. Add finance_cost_upsert_retry_total Prometheus counter for observability. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… crashes When a worker pod crashes (OOM) mid-chunk, it never publishes a ChunkCompletedEvent. The orchestrator's advanceAfterChunk sees completed < total and waits forever, stalling the entire job. Add a sweeper goroutine (every 2m, configurable) that detects DISPATCHED chunks older than 10m, marks them FAILED, and publishes a synthetic ChunkCompletedEvent so the orchestrator can advance past stuck waves. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ilramdhan
added
bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces critical resilience and fault-tolerance mechanisms to the distributed Calculation Engine.
First, it resolves a database transaction race condition (
uk_cpc_activeunique constraint violation) that occurs when two parallel calculation jobs attempt to upsert the same product cost record simultaneously. Second, it implements a self-healing "sweeper" mechanism in the Job Orchestrator to prevent the entire calculation job from stalling indefinitely if a worker pod silently crashes (e.g., due to an Out-Of-Memory kill) mid-chunk.Type of Change
Service(s) Affected
Changes Made
🛡️ Database Transaction Resilience
UpsertWithSupersede. If two concurrent transactions race to insert a new cost record and one hits theuk_cpc_activeunique constraint violation (SQLSTATE 23505), the loser catches the error, rolls back, and retries in a fresh transaction where it can properly see and supersede the winner's newly inserted row.finance_cost_upsert_retry_totalto monitor how often this race condition is encountered and successfully mitigated in production.⚙️ Orchestrator Fault Tolerance
DISPATCHEDchunks that are older than 10 minutes (indicating the assigned worker likely crashed via OOM and failed to emit aChunkCompletedEvent). It automatically marks these stuck chunks asFAILEDand publishes a syntheticChunkCompletedEvent, allowing the orchestrator to successfully advance past the stuck wave instead of hanging forever.Related Issues
Fixes #
Related to #
API Changes (if applicable)
Proto Changes
Breaking Changes
None.
Testing Performed
Unit Tests
Integration Tests
Manual Testing
Lint & Build
golangci-lint run ./...passesgo build ./...succeedsgo test -race ./...passesDatabase (if applicable)
Documentation
Rollback Plan
Revert the
finance-serviceandfinance-workerdeployments to the previous stable tag. No database schema rollback is required.Screenshots/Logs (if applicable)
Pre-merge Checklist
Reviewer Notes
finance_cost_upsert_retry_totalif the rate spikes unusually high, as this could indicate severe DB contention requiring a reduction in worker concurrency.