Add resumable CLI scraping - #313
Open
gosom wants to merge 1 commit into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a resumable mode for CLI file-based scrapes so long-running jobs can be restarted without overwriting existing output or re-emitting already-written places. It does so by loading existing result identities, appending new rows/objects, and tracking fully-completed input IDs in a sidecar state file to skip finished inputs on subsequent runs.
Changes:
- Add
-resumeCLI flag and enforce compatibility constraints (file output only; no stdout/custom writer/LeadsDB). - Introduce
runner/resumehelpers to load existing result identities (CSV/JSONL), append without duplication, and persist completed inputs via a sidecar file. - Add deterministic seed IDs + per-input completion tracking wired through gmaps seed/place/email jobs to know when an input query is “done”.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| runner/runner.go | Adds Resume config field and -resume CLI flag. |
| runner/resume/writer.go | Adds CSV/JSONL append writers that skip already-seen place identities. |
| runner/resume/writer_test.go | Tests header behavior, JSONL line output, and identity skipping. |
| runner/resume/state.go | Implements persistent sidecar state for completed input IDs. |
| runner/resume/state_test.go | Tests state load/missing file and persistence behavior. |
| runner/resume/results.go | Loads existing CSV/JSONL results into an identity set for dedupe/resume. |
| runner/resume/results_test.go | Tests identity extraction from CSV and JSONL result files. |
| runner/resume/progress.go | Tracks per-input discovery vs. completion to decide when to mark inputs done. |
| runner/resume/progress_test.go | Tests completion marking logic for zero/multi-place inputs. |
| runner/jobs.go | Adds seed job options (skip completed, deterministic IDs, completion tracker hookup). |
| runner/jobs_test.go | Tests completed-input skipping and completion tracker attachment. |
| runner/filerunner/filerunner.go | Wires resume mode into file runner: append output, load identities/state, seed options. |
| runner/filerunner/filerunner_test.go | Tests resume compatibility validation and append vs truncate behavior. |
| README.md | Documents -resume usage and constraints. |
| gmaps/place.go | Adds completion tracker support to place job and forwards tracker to email jobs. |
| gmaps/job.go | Adds completion tracker support to seed (GmapJob) and notifies discovery counts. |
| gmaps/emailjob.go | Adds completion tracker support for email extraction completion. |
| gmaps/completion.go | Introduces CompletionTracker interface used for per-input progress tracking. |
| gmaps/completion_test.go | Tests tracker notifications from seed/place/email job processing. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+130
to
+131
| scanner := bufio.NewScanner(r) | ||
| lineNumber := 0 |
Comment on lines
+48
to
+53
| w.headerOnce.Do(func() { | ||
| if w.writeHeader { | ||
| _ = w.w.Write(entries[0].CsvHeaders()) | ||
| } | ||
| }) | ||
|
|
Comment on lines
+79
to
+81
| func (j *PlaceJob) Process(_ context.Context, resp *scrapemate.Response) (any, []scrapemate.IJob, error) { | ||
| completed := true | ||
|
|
Comment on lines
156
to
157
| return nil, []scrapemate.IJob{emailJob}, nil | ||
| } else if j.ExitMonitor != nil && !j.WriterManagedCompletion { |
Allow interrupted CLI file scrapes to continue without starting over or duplicating already-written places. Large scrape jobs can fail or be stopped after producing useful partial output. Previously, rerunning the same command would recreate the run from scratch and risk duplicating results or overwriting output. The new resume mode lets users keep existing CSV/JSONL output, append missing places and skip inputs that have already completed. This keeps the default behavior unchanged while giving long-running CLI jobs a practical recovery path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allow interrupted CLI file scrapes to continue without starting over or duplicating already-written places.
Large scrape jobs can fail or be stopped after producing useful partial output. Previously, rerunning the same command would recreate the run from scratch and risk duplicating results or overwriting output.
The new resume mode lets users keep existing CSV/JSONL output, append missing places and skip inputs that have already completed.
This keeps the default behavior unchanged while giving long-running CLI jobs a practical recovery path.