We currently don't have a nice way of simply periodically checking for new collectors and adding them to the collectors table. This means that whenever RV or RIS add new collectors, we aren't able to include them without manual intervention.
scrape_collectors.go looks like it might do it, but it also tries to fetch all of the dumps which is something I'd rather leave up to the main scraper.
It doesn't need to run often (once per day is probably enough), but it needs to be smart enough to not upsert to existing collectors (as that might mess with their existing time columns).
Note that often the bgpfinder-service is actually often able to recognize that new collectors exist, because the getCollectors() methods in the RV finder actually does a quick scrape of the page rather than going on what is in the database. I'm not convinced this is the right approach, as that presumably means that the service could be making a lot of needless HTTP requests. Once we have reliable and timely collector information in the database, then we should revisit this approach and consider just falling back to the DB wherever possible.
We currently don't have a nice way of simply periodically checking for new collectors and adding them to the
collectorstable. This means that whenever RV or RIS add new collectors, we aren't able to include them without manual intervention.scrape_collectors.golooks like it might do it, but it also tries to fetch all of the dumps which is something I'd rather leave up to the main scraper.It doesn't need to run often (once per day is probably enough), but it needs to be smart enough to not upsert to existing collectors (as that might mess with their existing time columns).
Note that often the bgpfinder-service is actually often able to recognize that new collectors exist, because the getCollectors() methods in the RV finder actually does a quick scrape of the page rather than going on what is in the database. I'm not convinced this is the right approach, as that presumably means that the service could be making a lot of needless HTTP requests. Once we have reliable and timely collector information in the database, then we should revisit this approach and consider just falling back to the DB wherever possible.