Resolve some annoying issues in the periodic scraper#27
Open
salcock wants to merge 4 commits into
Open
Conversation
salcock
commented
Jul 13, 2026
salcock
commented
Collaborator
- fix issue where the periodic scraper would always do a full historical scrape on start up
- add 48 hour grace period to avoid missing files if they were uploaded out of order
- fixed outdated arguments in the main.go query CLI
This method scraped every RIB and UPDATE for every known collector on startup, which is a) unnecessary, b) takes a long time and therefore prevents scraping newer files in the meantime. The regular scraping routines should suffice and also respect the last_completed_crawl times in the database so won't waste time crawling files that we already know about.
This resolves issues when updates appear on the collector site in non-chronological order. When this happens, the last completed crawl time is set to a timestamp after the missing updates and therefore they are never subsequently scraped once they arrive. Now, the scraper will look back up to 48 hours from the last completed crawl time to ensure that it sees those missing files on a subsequent crawl. Note: currently our scraper attempts to upsert each file that it scrapes so we are now going to be doing a lot of redundant updates in the general case. Might be worth re-thinking how we want to handle conflicts in db.go?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.