Skip to content

Retry of syncs seems to be broken #18

Open

Description

@hxu

I ran a large backfill sync that I had to cancel midway, but when I retried the sync, it was not apparent to me if it would pick up where it stopped.

Before cancelling:

Mode: Backfill
Range: 2014-12-01 -> 2014-12-01
Queued tasks: activities 1219, gpx 975, health 4228, performance 604
Planning complete.
Act 4/1223 | GPX 4/993 | Health 6/4228 | Perf 6/604 | elapsed 2m 41s

After restarting:

Mode: Backfill
Range: 2025-06-28 -> 2025-06-28
Queued tasks: activities 1, gpx 13, health 4221, performance 598
Planning complete.
Act 2/3 | GPX 3/46 | Health 2/4221 | Perf 2/598 | elapsed 6s

The resumed counts did not line up with the queued work, which made it hard to tell whether the sync was truly resuming from the interrupted state.

Questions:

  • Are in-progress tasks recovered after interruption?
  • Should progress totals be based on queued/discovered task counts when resuming?
  • Can resumed backfills show a range/count summary that is easier to trust?

Current status after PRs #19, #20, and #23

Part of this report is fixed. On restart, activity and GPX totals now include outstanding tasks from the saved queue. Task claiming is also safe between concurrent workers, and a worker holding an old claim cannot complete a task or add follow-up work after another worker has reclaimed it.

Immediate restart is still ambiguous. An in-progress task is only returned to the pending queue after its claim is 30 minutes old. This avoids taking work away from another process that may still be alive, but the queue does not record which sync run owns the claim or whether that process was killed.

A representative scenario is:

10:00:00 P0 claims GPX task #4839
10:00:05 the user cancels the sync
10:00:10 the user starts the same sync again

The new run counts #4839 as outstanding work
The claim is only 10 seconds old, so it is not returned to pending
No worker in the new run can claim #4839 yet
The command can therefore show remaining work without being able to continue it

There are two other presentation gaps:

  • If saved work already exists, the command skips planning and then builds the displayed date range from the new command's defaults rather than from the saved tasks. The range can therefore look unrelated to the work being resumed.
  • Progress starts again at zero and uses the outstanding tasks as the new total. That describes work remaining in this invocation, but it does not explain how much of the original backfill was already completed.

A trustworthy resumed run would make the outcome explicit in ordinary language:

Resuming backfill originally covering 2014-12-01 through 2026-06-28
Already completed: 914 tasks
Ready to continue: 5,102 tasks
Waiting for old claims to expire: 3 tasks
Next recovery in: 29m 50s

The exact display does not need to match this example. The important result is that the user can tell what was preserved, what can run now, what is temporarily blocked, and which date range the saved work actually covers.

Activity

  1. hxu commented on Jun 28, 2026

    @hxu
    Author

    Just encountered this again and I have the outputs. Before cancelling, it showed:

    Mode: Backfill
    Range: 2014-12-01 -> 2014-12-01
    Queued tasks: activities 1219, gpx 975, health 4228, performance 604
    Planning complete.
    Act: 50/1267 | GPX: 872/1847 | Health: 0/4228 | Perf: 0/604 | 58m 44s
    

    It hung at this status for a long time. Maybe due to throttling?

    Then when restarting after killing the process, I got this:

    $ ./target/release/garmin sync run --backfill
    Using storage: /home/exedev/.local/share/garmin
    Planning sync...
    Profile: hgcrpd
    Mode: Backfill
    Range: 2025-06-28 -> 2025-06-28
    Queued tasks: activities 1, gpx 0, health 4228, performance 604
    Planning complete.
    Act: 0/1 | GPX: 0/0 | Health: 0/4228 | Perf: 0/604 | 2s ^C
    
  2. hxu commented on Jun 28, 2026

    @hxu
    Author

    Deferred branch for the remaining resume-count fix:

    • hxu/fix-sync-resume-counts

    This is not opened as a PR yet because it is stacked on the progress display branch.

    Prerequisite:

    Once the progress/logging PR can be opened and merged, this branch can be opened as a follow-up PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions