When doing a large backfill sync, the process can take quite a long time and I have had cases where it might get apparently stuck without any feedback.
Example from a verbose backfill run:
Act 4/1223 | GPX 4/993 | Health 6/4228 | Perf 6/604 | elapsed 2m 41s
P0 Activities #4855 Activities 200-250 waiting for rate limit 0s
P1 Performance #4236 2015-01-12 waiting for rate limit 1s
P2 GPX #4839 2026-03-17 Singapore Running waiting for rate limit 1s
P3 Health #8 2014-12-07 waiting for rate limit 0s
I would like to:
- Show what each worker is doing and how long it has been on the task.
- Add detailed logging when workers encounter issues.
- Add per-task timeouts so one stuck request does not block the whole sync and can be retried.
Current status after PRs #20, #21, and #23
The queue is safer now: workers claim tasks without racing, old workers cannot finish a newer worker's claim, and individual HTTP requests stop after 30 seconds. The client also has request-logging support.
The original visibility problem remains. A running sync prints totals, but it does not keep a stable line for each worker. Workers processing the same kind of data share one "current item", so one worker can replace or clear the item shown by another. Request logging exists in the client but is not currently connected to sync run. The 30-second limit applies to each HTTP request, not to the whole task, which may include waiting, several requests, queueing, and writing data.
A representative scenario is:
P0 starts activity page 200-250
P1 starts health data for 2015-01-12
P2 starts a GPX download
P1 updates the shared Health current item
P0 is still waiting or working, but the display only shows aggregate totals
No totals move for a while
The user cannot tell which worker is active, waiting, retrying, or writing
A useful outcome would let a person answer these questions while the command is running:
- What is each worker doing now?
- How long has that worker been on this task?
- Is it waiting for Garmin, waiting because of rate limiting, writing locally, or retrying?
- If it failed, where can I find the task and request details?
- Will this task retry during this run, or only after starting another sync?
When doing a large backfill sync, the process can take quite a long time and I have had cases where it might get apparently stuck without any feedback.
Example from a verbose backfill run:
I would like to:
Current status after PRs #20, #21, and #23
The queue is safer now: workers claim tasks without racing, old workers cannot finish a newer worker's claim, and individual HTTP requests stop after 30 seconds. The client also has request-logging support.
The original visibility problem remains. A running sync prints totals, but it does not keep a stable line for each worker. Workers processing the same kind of data share one "current item", so one worker can replace or clear the item shown by another. Request logging exists in the client but is not currently connected to
sync run. The 30-second limit applies to each HTTP request, not to the whole task, which may include waiting, several requests, queueing, and writing data.A representative scenario is:
A useful outcome would let a person answer these questions while the command is running: