Harden RunSimulateDataPoint worker against permanent lock deadlock and stale-job re-dispatch

[anchapin]

Summary

Production incident (2026-07-07/08): 50+ analyses with corrupted seed zips triggered a permanent deadlock in the worker-side lock/receipt coordination in DjJobs::RunSimulateDataPoint, wedging ~9,988 worker pods (99.6% of the worker tier) into an infinite wait. Root-caused and fixed here; not related to (but complementary to) the seed-zip validation fix in #841/#844, which covers a different, once-per-analysis web-node code path (Analysis#run_initialization). This fix targets the per-datapoint, per-worker-pod download+extract path (initialize_worker/write_lock).

Root cause

write_lock released its flock on failure but never deleted the lock file, and never wrote a receipt. The waiter loop in initialize_worker only polled for the receipt file appearing — it never rechecked whether the lock file itself had disappeared (holder crashed) — so every subsequent worker for the same analysis blocked for the full initialize_worker_timeout (8h default), and then, due to a second bug, fell through to return true anyway: reporting fake success and letting perform proceed to simulate against a directory that was never populated.

Changes

• write_lock: rescue any exception in the protected block, delete the lock file before re-raising (previously only released the flock — leaving a stat-able "still locked" file behind). Also closes the file handle in ensure (minor fd leak).
• initialize_worker's receipt-wait loop: detect lock_holder_gone (lock file disappears without a receipt appearing) and return false immediately instead of looping to timeout. On genuine Timeout::Error, explicitly delete the stale lock file and return false instead of the old fall-through return true.
• Zip extraction retry: add rescue ::Zip::Error, ::Zlib::Error ahead of the generic retry rescue — fail fast on deterministic corruption (no futile 3x retry), remove the partial analysis_dir. Worker-side counterpart to Enhance InitializeAnalysis resilience for corrupt seed.zip files #841/Fail fast on corrupt seed.zip: validate at upload, no futile retries, terminal 'failed' state (#841) #844's fail-fast, applied to the different per-datapoint download path.
• @@data_point → @data_point typo fix (5 occurrences: 3 in dj_jobs/run_simulate_data_point.rb, 2 in dj_jobs/urban_opt.rb, which is included into the same class): the invalid-timeout fallback branches referenced an uninitialized class variable instead of the instance variable, since ebc2361b (2019). Confirmed also present on origin/develop. Previously any invalid timeout value raised NameError instead of applying the intended 28800s default.
• resque_jobs/run_simulate_data_point.rb#perform: added a narrow analysis.run_flag == false guard (checked ahead of the existing completed-status guard) to skip stale queue payloads for analyses explicitly stopped/quarantined via stop_analysis/soft_stop_analysis. Deliberately not a blanket status == 'completed' check — that would break the admin manual-retry workflow, since requeue/requeue_started don't reset status before re-enqueueing.

Testing

Added server/spec/models/dj_run_simulate_data_point_hardening_spec.rb (7 examples) and server/spec/models/resque_run_simulate_data_point_hardening_spec.rb (7 examples). All 14 executed against a real local Mongo/Redis and passed. Each behavior also has a companion check that reconstructs the pre-fix code path in isolation to confirm the new assertions would have failed against the old bug (discriminating power), e.g.:

• old fall-through logic returns true after a real Timeout::Error with the lock file still present — spec expecting false + lock gone would fail against it.
• old write_lock (flock-release only) leaves the lock file on disk after the block raises — spec expecting it gone would fail against it.
• old 3-retry rescue triggers a 2nd extract_archive call — expect(...).to receive(:extract_archive).once raises MockExpectationError against it.
• old @@data_point reference raises NameError (uninitialized class variable, never assigned anywhere in the codebase) rather than silently applying an invalid timeout — differs from the incident hypothesis but confirms the bug is real and the fix's 28800 readback behavior is correct.

No project-wide test suite/linter run (per repo convention, left to CI).

Not in scope here

• Seed-zip re-upload validation — already covered by Enhance InitializeAnalysis resilience for corrupt seed.zip files #841/Fail fast on corrupt seed.zip: validate at upload, no futile retries, terminal 'failed' state (#841) #844 (different code path).
• The 85 corrupted seed-zip batches from this incident are being resubmitted separately once this fix (and Enhance InitializeAnalysis resilience for corrupt seed.zip files #841/Fail fast on corrupt seed.zip: validate at upload, no futile retries, terminal 'failed' state (#841) #844, confirmed not yet deployed to the production image as of this incident) are live.