Auto-reprocess criterion misjudges completed nights in both directions

[dvida]

Follow-up to #418 / #942. Fixing the standalone reprocess cleanup in #942 removes one source of the problem; the criterion that turns leftovers into a reprocess decision is still wrong, and it is wrong in two directions.

The criterion

processIncompleteCaptures() (RMS/StartCapture.py:921, called from :1347) decides whether a captured night needs reprocessing from two facts inside the captured directory (:957-978):

pickle_files = glob.glob("{:s}/rms_queue_bkup_*.pickle".format(captured_dir_path))
...
FTPdetectinfo_files = glob.glob('{:s}/FTPdetectinfo_*.txt'.format(captured_dir_path))
...
run_reprocess = False
if any_pickle_files:
    run_reprocess = True
else:
    if not any_ftpdetectinfo_files:
        run_reprocess = True

It never consults ArchivedFiles. The only config.archived_dir uses in StartCapture.py are retention deletion and locating the latest archive for the slideshow and similar.

A. False negative: archive missing, nothing happens

This is the first half of @peschman's original #418 report, and it was never addressed. He moved ArchivedFiles/<night> and its tar aside and restarted capture, expecting RMS to notice and reprocess:

I then restarted capture, expecting that the system would detect the missing ArchivedFiles directory and reprocess the data. It did not detect the missing ArchivedFiles directory and re-create it, so I ran a manual reprocessing job

Of course it didn't — the captured dir still had its FTPdetectinfo and no pickles, so run_reprocess = False and the folder was logged as ... fully processed!. The hand-run he then did is what triggered B.

B. False positive: any leftover pickle forces a reprocess

if any_pickle_files is tested before the FTPdetectinfo check, so a single stale backup outweighs a complete set of outputs. #942 fixes the standalone-reprocess-with-uploading-off source. It is not the only one: a crash, a kill -9, a full disk, or any future caller that skips deleteBackupFiles() puts the night straight back into the same state. The criterion should not be this brittle.

It is also self-concealing — the duplicate run goes through StartCapture, which does clean up, so the evidence deletes itself and the night never triggers twice. That is why #418 sat for two years looking irreproducible.

C. A deterministic failure retries on every start

except Exception as e:
    log.error("An error occurred when trying to reprocess partially processed data!")

StartCapture.py:1014 logs and moves on, leaving the pickles in place. A night that fails reliably is therefore retried on every single start, burning the pre-capture window each time; :1021 only breaks the loop once capture is already due. This is roughly the ground #756 ("Fix infinite auto-reprocessing loop and QueuedPool memory leak") was aiming at before it was closed with "Closing. Will remove the time check instead."

Why A and B are the same defect

"Is this night finished?" is being inferred from partial artifacts in the input directory, when the real evidence is elsewhere. Pickles are a resume aid for detection, not a progress record.

There is already a completion marker in the captured directory, maintained by code nobody needs to touch. processNight calls finalizeObservationSummary (RMS/Formats/ObservationSummary.py:1759, from RMS/Reprocess.py:651), which writes <night>_observation_summary.json and deletes <night>_observation_summary_working.json that startObservationSummaryReport created at the start (StartCapture.py:617, Reprocess.py:846). Final JSON present means processing reached the end of the summary step, it lives with the captured dir, and it survives ArchivedFiles purging.

Two honest caveats: the summary step runs before archiveDetections (Reprocess.py:651 vs :672), so the marker means "detection and summary done", not "archived"; and finalizeObservationSummary sits in a try/except that logs and continues (Reprocess.py:646-658), so a failure there leaves the night looking unprocessed — the safe direction.

Proposed scope

B and C fixed unconditionally. A night whose final summary is present is processed: delete any stale pickles instead of reprocessing it, which closes B for every leftover source rather than one at a time. Nights predating the change (no summary of either kind, but an FTPdetectinfo present) are treated as processed, so upgrading does not reprocess every existing captured dir on the first start. For C, a per-night attempt count with a configurable cap, in the same spirit as the reboot loop guard in Scripts/MultiCamLinux/GRMSUpdater.sh.

A behind a config flag, defaulting to off. This is the part worth arguing with, so stating the reasoning plainly: a missing archive directory does not reliably mean "not processed", because retention deletes archives independently of captured dirs. arch_dirs_to_keep (20) and capt_dirs_to_keep (8) are separate settings (RMS/ConfigReader.py:290, :296), deleteOldDirs purges each on its own count (RMS/DeleteOldObservations.py:1045-1078), and deleteByQuota applies arch_dir_quota and bz2_files_quota separately from the captured allowance (:998-1028). The defaults keep archives longer than captured dirs, so the common case is fine — but on a station where quota management purges ArchivedFiles ahead of CapturedFiles, a default-on rule would drag old nights into a reprocess on every start. That is a worse failure than the one being fixed, so it should be opt-in with the caveat documented next to the option.

Happy to be argued out of the default if someone has a better completion signal for the archive side.

Refs #418, #942, #756.