Repository navigation
Retry of syncs seems to be broken #18
Description
I ran a large backfill sync that I had to cancel midway, but when I retried the sync, it was not apparent to me if it would pick up where it stopped.
Before cancelling:
Mode: Backfill
Range: 2014-12-01 -> 2014-12-01
Queued tasks: activities 1219, gpx 975, health 4228, performance 604
Planning complete.
Act 4/1223 | GPX 4/993 | Health 6/4228 | Perf 6/604 | elapsed 2m 41s
After restarting:
Mode: Backfill
Range: 2025-06-28 -> 2025-06-28
Queued tasks: activities 1, gpx 13, health 4221, performance 598
Planning complete.
Act 2/3 | GPX 3/46 | Health 2/4221 | Perf 2/598 | elapsed 6s
The resumed counts did not line up with the queued work, which made it hard to tell whether the sync was truly resuming from the interrupted state.
Questions:
- Are in-progress tasks recovered after interruption?
- Should progress totals be based on queued/discovered task counts when resuming?
- Can resumed backfills show a range/count summary that is easier to trust?
Current status after PRs #19, #20, and #23
Part of this report is fixed. On restart, activity and GPX totals now include outstanding tasks from the saved queue. Task claiming is also safe between concurrent workers, and a worker holding an old claim cannot complete a task or add follow-up work after another worker has reclaimed it.
Immediate restart is still ambiguous. An in-progress task is only returned to the pending queue after its claim is 30 minutes old. This avoids taking work away from another process that may still be alive, but the queue does not record which sync run owns the claim or whether that process was killed.
A representative scenario is:
10:00:00 P0 claims GPX task #4839
10:00:05 the user cancels the sync
10:00:10 the user starts the same sync again
The new run counts #4839 as outstanding work
The claim is only 10 seconds old, so it is not returned to pending
No worker in the new run can claim #4839 yet
The command can therefore show remaining work without being able to continue it
There are two other presentation gaps:
- If saved work already exists, the command skips planning and then builds the displayed date range from the new command's defaults rather than from the saved tasks. The range can therefore look unrelated to the work being resumed.
- Progress starts again at zero and uses the outstanding tasks as the new total. That describes work remaining in this invocation, but it does not explain how much of the original backfill was already completed.
A trustworthy resumed run would make the outcome explicit in ordinary language:
Resuming backfill originally covering 2014-12-01 through 2026-06-28
Already completed: 914 tasks
Ready to continue: 5,102 tasks
Waiting for old claims to expire: 3 tasks
Next recovery in: 29m 50s
The exact display does not need to match this example. The important result is that the user can tell what was preserved, what can run now, what is temporarily blocked, and which date range the saved work actually covers.
Activity
hxu commentedon Jun 28, 2026
Just encountered this again and I have the outputs. Before cancelling, it showed:
It hung at this status for a long time. Maybe due to throttling?
Then when restarting after killing the process, I got this:
hxu commentedon Jun 28, 2026
Deferred branch for the remaining resume-count fix:
hxu/fix-sync-resume-countsThis is not opened as a PR yet because it is stacked on the progress display branch.
Prerequisite:
hxu/sync-progress-loggingtracked in issue Show more progress in sync mode #17Once the progress/logging PR can be opened and merged, this branch can be opened as a follow-up PR.