Orphaned `resources_restart` marker blocks recovery

[robjarawan]

The housekeeping resources callback can leave a configuration permanently stuck in resources_restart if the restarting process dies after it creates the marker and before the new process removes it.

Reproduction

I used the real Resources.restart() path in a child process. The first child reaches the patched exec boundary and exits with status 17, which simulates the process dying after the marker is written. The marker is present afterward. A second child then calls the same restart() path.

On the current development branch, the second child is still waiting after the two-second fixture boundary:

owner exit code: 17
resources_restart exists: true
recovery process alive after 2 seconds: true
recovery reached exec: false

The no-failure control keeps the first owner alive at the exec boundary. A contender waits while that owner is alive, then continues after the owner completes. That control passes.

The repository regression result on the current baseline is:

2 failed, 2 passed

The two failures cover a crash-generated orphan and an aged legacy timestamp-only marker. The two passing cases prove that a live identified owner and a recent legacy marker still serialize a concurrent restart.

Expected behavior

A live restart owner should keep the marker and make other instances wait. If the owner is gone, a later restart should be able to identify the orphan, replace it, and continue recovery.

Actual behavior

The marker contains only nowstr(). The loop only checks os.path.exists() and has no owner identity or recovery condition:

while file_not_ready:
    if not os.path.exists(self.state_file):
        ...
    else:
        time.sleep(self.randomSleep)

Once orphaned, the marker also leaves the configuration in the resources_restart state, so normal automatic recovery does not make progress.

Root cause

The marker records a timestamp instead of the owning process, and acquisition uses a separate existence check followed by a normal write. There is no way to distinguish a live restart from a dead one.

I have a minimal fix and regression ready. It creates the marker exclusively, stores the owner PID and process creation time, waits while that exact owner is alive, and removes a marker only after its owner is gone. It gives timestamp-only markers from current releases a 60-second compatibility window before treating them as legacy orphans.

Related history: MetPX#1439 and merged PR MetPX#1553 introduced the resources_restart coordination path. I did not find an issue that covers orphan-marker recovery itself.

[robjarawan]

I finished the fork proof and fix in draft PR #166.

The baseline test now generates the orphan through the real Resources.restart() path. The owner creates resources_restart, then exits with status 17 at the exec boundary. The marker remains and the next restart is still waiting after the two-second fixture boundary. A timestamp-only marker from the current release has the same result. Baseline: 2 failed, 2 passed.

With the fix, both orphan cases continue, a live owner still blocks takeover, and a recent legacy marker gets a compatibility window: 4 passed. The original audit's stale/absent-marker controls also pass 2/2. The broader local suite is 358 passed and 1 skipped, and the isolated maintenance flow passes with all five messages downloaded and cleanup completed.