SYM-7915: Prevent initial loads deadlocks due to a JVM restart mid-extract

[gp510]

Fixes SYM-7915.

The problem

An initial load whose extract is interrupted by a JVM restart comes back permanently stuck. sym_extract_request reads OK with extracted_rows=0 while its batches are still RQ. Every extraction selector filters on NE so nothing re-extracts it, extractBatch refuses to deliver an RQ batch, nothing reconciles the two, and nothing surfaces it.

The only symptom in the product is a pair of INFO log lines. The console shows an ACTIVE pipeline making no progress, indistinguishable from a slow load. At the reporting site it ran three days, was never recovered, and the load had to be discarded.

Root cause

extractOutgoingBatch marked the entire request — all of its batches — OK from a single batch's extract_row_count and extract_millis, with no check that anything was extracted and no check that the rest of the range had finished. A batch that skipped extraction because it was already extracted contributes a zero row count and a near-zero duration, which is exactly the row observed.

Three layers

• Guard — isExtractRequestComplete gates the status write. When incomplete, statistics accumulate through a new incremental statement instead, so progress stays visible without claiming completion.
• Reconciler — recoverStuckExtractRequests returns such requests to NE, called from queueWork, which InitialLoadExtractorJob already drives. That covers startup for free. A startup-only check would not have helped the reporting site, whose node ran three days in this state.
• restartExtractRequest now also zeroes extracted_rows / extracted_millis, which it previously left behind.

Why the detection has no false positives

MultiBatchStagingWriter.close() advances every remaining batch in the range, so a finished extract leaves none at RQ. A request at OK with an RQ batch in its range is an impossible state, not a healthy in-flight one. That is what makes both the guard and the reconciler safe against a running extract.

The constraint that shapes the recovery

restartExtractRequest flips the whole range back to RQ through updateOutgoingBatchStatusSql, which carries no status predicate — so it re-sends batches that already landed. Requests whose range contains delivered batches are therefore not restarted automatically; they are reported at ERROR every pass with the recovery choices and require force. The product cannot know whether the target tolerates a re-apply, and the field evidence (500,000 rows committed with no sym_incoming_batch row) shows the target's own record can be untrustworthy.

Review notes

• No schema change. Everything needed is derivable from sym_outgoing_batch.
• DataExtractorService is the only implementer of the interface and FileSyncExtractorService inherits it; there is no PRO implementer, so the interface addition breaks nothing.
• FileSyncExtractorService returns 0 from the recovery, matching how updateExtractRequestsForThreading is already handled there — otherwise it would scan and restart ordinary data extract requests and resolve their staging through the file-sync override.
• The requested/delivered lookups deliberately select one row rather than count(*): the answers are booleans, and a count visits the whole batch range, so across a load split into N batches the cost would grow with N².
• The recovery scan is throttled to its own staleness threshold. queueWork runs every 10s but a request cannot qualify for 5 minutes, so 29 of every 30 scans could not have a new answer.

781 tests pass in :symmetric-core, 0 failures.

Not covered here

The end-to-end kill-and-recover path needs a real database — it is on the ticket as manual QA.

Found while reading and deliberately not folded in: sym_extract_request rows are inserted as LS and flipped LS→NE by DataService, and nothing recovers an LS request either, so a kill during load setup strands a load the same way. The reconciler is the natural home, but it is a separate defect and should be its own ticket.

🤖 Generated with Claude Code

[sonarqubecloud]

Quality Gate failed

Failed conditions
4.9% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

[Pavel_JM (pavel-jm)]

Unfortunately this condition is hard to read... Making it hard to support.
There are plenty of variables inplay:

• Clustered servers overtaking each other's load requests (without prior crash).
• Own multi-threaded processing streams racing to re-do potentially crashed reqiuest.

I wonder if a more productive way to move forward is a demo with this issue discussed in a medium-sized gathering?..

Also, voluminous comments are a "code smell". It alerts reviewer that logic is so complex/convoluted/unrefined that code alone is not no longer self-documenting...

[Pavel_JM (pavel-jm)]

Let's schedule a code review/debate meeting to settle on the best idea to solve this?

[Pavel_JM (pavel-jm)]

Not sure why this is new parameter is needed?
I was hoping the issue is fixed without user needing to analyse it and set parameters?